Others have pointed out DirectFB and SVGALib. Both very interesting. SVGALib was supported by a lot of commercial Linux game ports in the earlier days, whereas

Others have pointed out DirectFB and SVGALib. Both very interesting. SVGALib was supported by a lot of commercial Linux game ports in the earlier days, whereas DirectFB was possibly one of the most serious alternatives to X11, although I think like most such projects it mainly found use in embedded devices (Coincidentally, Wayland seems to have finally unified some of the "embedded" and desktop use cases.) These days SVGALib doesn't work and fbdev is deprecated. However, the "next-generation" Linux graphics stack can achieve a fairly similar goal using the DRM/KMS APIs, albeit it's definitely a lot more work if you just want a framebuffer. On the other hand, though, it's relatively straight-forward to get an EGL context and start doing hardware accelerated things. Maybe that takes some of the fun out of it, but I think it's pretty cool.