Generally the limiting factor for approaching the asymptote you observe is power. There is a figure of merit with units of Joules per bit per second that gets w

Generally the limiting factor for approaching the asymptote you observe is power. There is a figure of merit with units of Joules per bit per second that gets worse - much worse - as you start to get fancier with your modulation. Most of the fancier schemes take advantage of the fact that traditional binary signalling has excess noise margin, ie, they were throwing away energy to start with, and they are encoding extra bits in the energy budget. But to maintain noise margins as you cram more bits in, you have to up the amount of energy per bit. The other half is that the physical layer implementation that does the encoding and decoding consumes more energy because it's doing more computation. This also figures into the energy per bit metric if you're being honest about your comparisons (and because it is not always clear if this is included in a metric you find papers where people cherry pick numbers to make their case). This number can become quite big because the baseline of a binary tx/rx is so low compared to doing effectively a DAC/ADC and phase recovery system. What you find is that QAM or more schemes are certainly possible, but they can consume more power than the CPU just to keep the link idle and trained. The real art is picking the implementation and developing new circuit tricks that we hadn't thought of before to wring a little more bandwidth without killing the power budget.