Maybe it would make sense to have a "framebuffer chip", that handles things at a fairly abstract level-- commands like "set screen dimensions to 640x480x8bpp",

Maybe it would make sense to have a "framebuffer chip", that handles things at a fairly abstract level-- commands like "set screen dimensions to 640x480x8bpp", "write buffer #12, pixel (203, 122) as colour 0xF3", "toggle buffer #5 to active display". Let the "framebuffer chip" handle the timing sensitive HDMI/DVI dance, and then you can use the general-purpose controller for things like "map a more well-supported graphics protocol to this language". If the controller fails to meet a timing requirement, the framebuffer chip doesn't cause the monitor to lose synch, you'd just get a torn image because it wasn't fully populated in time for the frame to render. What I've been waiting for is some sort of "modern MCU as replacement for expensive, clunky, slow old video cards." It seems like if you want to use something like a Pi Pico, it would use almost all of the resources just to maintain synch and timing, so it might not be amenable for the other half of the job-- monitoring the external bus and waiting for inquiries on the framebuffer/firmware/support register address ranges.