I don't think your conclusion is be correct.
I mean I don't know about any possible causes which could cause high CPU usage when a serial port is just opened and in idle.
Isn't it possible that you doing other things in your loop or thread and those things requires CPU time and those things happen more often when you not emitting anything,
because while the writting is happening you blocking the thread, but when you not writting anything then the thread loops more frequent, because the loop finishes faster.
Another idea is that the device you talking to maybe sending requests to communicate when you not transmitting data?
I mean if you've attached a Receive event to the serialport object then it may be loading the CPU too much if the external device sending data very frequently and if you have CPU time consuming code in that event handler which code then works with that data too frequently, for example converting the data and/or writting the data to the GUI somewhere making too frequent updates of screen items.
If you can show your code I would be happy to check to see if I can see any possible problem causes.