The X server is mostly irrelevant, these days you'd use something like SDL for a simple 2D game, which abstracts it away entirely and works just the same on the

The X server is mostly irrelevant, these days you'd use something like SDL for a simple 2D game, which abstracts it away entirely and works just the same on the framebuffer and under X. So no reason not to use X/Wayland since it's just more comfortable to develop for.