The number of technology layers one must understand and control to make GPU draw what VM wants is frankly insane. Hypervisor channels, GPU driver code, graphics

The number of technology layers one must understand and control to make GPU draw what VM wants is frankly insane. Hypervisor channels, GPU driver code, graphics server API, graphics toolkit libraries - all these have several variants. It seems close to impossible to get anything done in this space. I just want to send drawing commands from a node over network to another node with a GPU. Like "draw a black rectangle 20,20,200,200 on main GPU on VM at 192.168.1.102". How would I do that in the simplest possible way? Is there some network graphics command protocol? Like X11, but simpler and faster, just the raw drawing commands.