Intereting question. The answer to the second part: we have much faster switching transistors (GaAs, SiGe, InP, now GaN) already but they cannot be miniaturized

Intereting question. The answer to the second part: we have much faster switching transistors (GaAs, SiGe, InP, now GaN) already but they cannot be miniaturized easily and the production technology isn't as simple as CMOS. One can build computers with them, but due to physical size and large distances it wouldn't be performing good compared to a CMOS chip. So the answer is: size matters. Large devices cannot be used for building complex fast computers.