I had the same problem, and one of my displays is insanely picky when it comes to switching sources. It seems like it will only attempt a handshake once, and wh

I had the same problem, and one of my displays is insanely picky when it comes to switching sources. It seems like it will only attempt a handshake once, and when the source is not ready for it, it will switch back. I bought HDMI/DisplayPort switches which are capable of 4k60 and are very cheap. Only these combined KVMs have outrageous prices, but if you're fine with using multiple switches it can be achieved quite cheaply.