The problem with the X server code is most of the protocol code is very complex and it's hand rolled code predating C89. It (still!) does all the parsing and by

The problem with the X server code is most of the protocol code is very complex and it's hand rolled code predating C89. It (still!) does all the parsing and byte swaps manually. In a modern project you would have the compiler generate the parsers and the dispatch table based on an IDL but X is way too old to have done this. There are just two implementations in the code for everything. No one wants to touch it due to fear of breaking everything even worse. So instead they just slowly try to stop the bleeding by disabling things that cause the worst security issues. It's a bad situation overall, and the time has already passed for a full rewrite of the X server to be viable. The SSH protocol is relatively simple by comparison, and most importantly, there actually are multiple independent modernized implementations of it.