The problem though is that you lose any external trust. Now external trust is not very helpful for some things. It won't protect you from state-sponsored MITM

The problem though is that you lose any external trust. Now external trust is not very helpful for some things. It won't protect you from state-sponsored MITM attacks. Also non-EV certs may not inspire a whole lot of trust. But you have the fundamental question which X.509 tries to answer: "Before I give you my credit card number, how do I know you are who you say you are?" You need some form of external trust there to answer that question. External trust is not foolproof. See Thompson's important paper on the limitations of it. However, it is very good at addressing certain classes of threats (and very bad at others). In the end we need both models and there is no real way around that.