Humans write. LLMs pretend to write. Really what they are doing is a statistical process called "continuation". By training an LLM on human-written content, we

Humans write. LLMs pretend to write. Really what they are doing is a statistical process called "continuation". By training an LLM on human-written content, we are essentially restricting the problem domain from "any possible next character" to "something similar to what the humans wrote". What the LLM does at the end of the day can be as simple and intelligent as blindly stumbling around whatever data it was given, because the domain it is stumbling around is made exclusively out of human-written content. An LLM won't generate nonsense, because nonsense lies outside its domain: the LLM was never trained on nonsense, so there is no nonsense for it to stumble into. An LLM can stumble into lies, offensive language, threats, incorrect arithmetic, logical fallacies, etc. These are generally presented as "limitations". The limitation is that we can't keep them out of the domain. It would be more clear to say those are limitations of the LLM creator. To the LLM itself, these limitations are a feature; though not the kind we are happy about. Humans are able to think about our writing before we write it. That's a feature called objectivity, and LLMs don't have it. There's no obvious place to add that feature, either.