You often have to be a subject expert to be able to distinguish genuine content from genuine-sounding guff, especially the more technical the subject becomes.

You often have to be a subject expert to be able to distinguish genuine content from genuine-sounding guff, especially the more technical the subject becomes. That’s why a lot (though not all!) of the over-the-top LLM hype you see online is coming from people with very little experience and no serious expertise in a technical domain. If it walks like a duck, and quacks like a duck… …possibly it’s just an LLM trained on the output of real ducks, and you’re not a duck so you can’t tell the difference. I think LLMs are simply a less general technology than we (myself included) might have predicted at first interaction. They’re incredibly good at what they do — fluidly manipulating and interpreting natural language. But humans are prone to believing that anything that can speak their language to a high degree of fluency (in the case of GPT-3+, beyond almost all native speakers) must also be hugely intelligent and therefore capable of general reasoning. And in LLMs, we finally have the perfect counterexample.