Actual RPM readout

[cncdrive]

I will have to check what is the time constant we used, but the 5000RPM is a so fast speed that I don't think it will be the issue, because it would cause problems on low speed when the spindle is rotating slower than the time constant of the measurements and this is sure not the case with the high 5000 RPM.
I think it is more likely that the problem is the opposite of that, that the index signal is so short that the controller can't read it.
A quick calculation: 5000 RPM is about 83.33 rotations per second and that would be about 0.012 seconds per rotation.
Assuming that you have an encoder with e.g. 1000 counts per second and that the index pulse is 1 pulse long only then one pulse is only 0.012/1000 seconds long which is only 12 microseconds.
The controller needs at least about 20usec to safely read the signal, because there is a small low pass filtering implemented mechanically on the inputs to avoid noise spikes to make false triggers.

[Derek]

It's an AMT 103 if that helps.

[cncdrive]

With AMT encoders the index signal length is always as long as one pulse.
So, for example if you set it to 250 pulses per per rotation will read as 1000 signal edges (standard 4x encoder decoding mode) and then the signal is too short on 5000 RPM as I described above.
Because the AMT has changable resolution and so the index signal width also changing with the resolution I advice to calculate with minimum 20usec required signal length and adjust the resolution to that to make the index signal longer than that on the highest possible spindle RPM.
Also take into account if any breakout boards are used to connect this signal, because optical isolators may make the 20usec requirement even much higher depending on the optocoupler and the circuit used on the BOB, because optocouplers always delay the signal and so they will cut bandwidth lower. But if the signal is directly connected to the -5LPT motherboard inputs then you can calculate with 20usec which the controller can still read. It is not guaranteed that it will read shorter signals properly.