"In fact, it seems that no action, which normally MUST stop the spindle does actually stop it. E-Stop, hard limits or any other stop command just ignores M5 and only reset the spindle relay if such relay is defined in the configuration, so as it is now, it is pretty dangerous to use it with Modbus in my opinion. I think you should put out a warning about this because it is pretty nasty to see the spindle continuing to spin when you expect it to be stopped. As it is now, only spindles controlled by the spindle relay are stopped.
Stop and Reset and hardlimit etc. actions all stop the spindle also accidental disconnection of the motion controller stops the spindle.
The stop is done with the spindle relay off and the PWM set to it's defined inactive state or the step signal to zero frequency depending on how the spindle is configured.
The spindle signals are special kind of signals in the software (we call them hardware signals, because they are controlled directly by the motion controller with highest priority), the motion controller handles these in a special way where if a stop is requested or anything problematic happens then the configured inactive states of these signals are output by the motion controller to make sure the spindle will stop.
With modbus the software has no idea which register and what value of that register will stop your spindle, there is no configuration for that, you could freely code that many different ways for different hardwares.
Also I'm not sure how safe is that to leave the spindle stop on a modbus or any kind of serial communication, I would probably not do that and instead would use the relay signals,
because however they still depends on software, but depends only on the closest level of software piece to the device, because that part of the software runs on the motion controller microcontroller.
As far as I remember the spindle to comply the CE rules must directly disabled it by the external e-stop button with the e-stop button must have at least 2 normally closed contacts and the signal must go through on both in series so if any of the contactors will fail the circuit will still disconnect in case of an e-stop button activation. Or at least this was the rule about a year ago when I read them.
I don't know how would you comply these safety rules without using at least one physical input on the VFD instead of the modbus or a power relay discharger.
My description started getting a bit long, so I cut it short now so the most important is that I don't think leaving the spindle enable signal to depend from modbus register values is a safe way of controlling a spindle.
And I think that a solution for your suggestion can be is if you running a macro loop you could add a poll to the stop button press and if it is pressed then you could change your modbus variable which stops your spindle. Also you could check the reset button or LED state and do the same. But again I don't feel this a safe solution...