Even outside of impedence issues, you can't just use relays because then every switch is basically an unplug/replug, meaning that: a) the display will flip out

Even outside of impedence issues, you can't just use relays because then every switch is basically an unplug/replug, meaning that: a) the display will flip out and spend multiple seconds re-syncing itself, and b) the sources may take actions such as resizing the desktop, switching to the internal display, pausing the game, whatever. Part of the point of a hardware KVM is that it negotiates its own connection to all of the devices involved, and only changes which signals are passed through at any given moment, resulting in instantaneous switchovers. Now, a multi-input monitor can also do this, but then you have to deal with whatever compat issues are involved in commanding it to switch over DDC (see for example https://github.com/haimgel/display-switch https://github.com/haimgel/display-switch), and it typically also won't be as fast as a hardware solution since input switching speed isn't a metric most people care about when monitor shopping.