Well, duh ... of course there are statistical regularities in the data, which is what LLMs learn. However, the LLM has no first hand experience of the world des

Well, duh ... of course there are statistical regularities in the data, which is what LLMs learn. However, the LLM has no first hand experience of the world described by those words, so the words (to an LLM) are not grounded. So, the LLM is doomed to only be able to predict patterns in the dataset, with no knowledge or care as to whether what it is outputting is objectively true or not. Calling it bullshitting would be an anthromorphism since it implies intention, but it's effectively what an LLM does - just spits out words without any care (or knowledge) as to the truth of them. Of course if there is more truth than misinformation and lies in the dataset, then statistically that is more likely to be output by an LLM, but to the LLM it's all the same - just word stats, just the same as when it "hallucinates" and "makes something up", since there is always a non-zero possibility that the next word could be ... anything.