This really shouldn't be surprising, given it's such a known bullshitter. As a language model it will present answers to almost any question, because it knows h

This really shouldn't be surprising, given it's such a known bullshitter. As a language model it will present answers to almost any question, because it knows how to generate text of the same form as answers to questions. There's no guarantee those answers will be correct though. A key point is that it doesn't encode facts directly, only probabilistic word sequences that may or may not encode facts, and which it may or may not reproduce accurately. It only has a weighted random chance of generating any given word at any given time, in any given answer, so there's a lot of semantic noise layered on to the answers. The grammar and syntax are generally very good because the word sequence weightings strongly select for that, but facts simply don't exist as a feature in it's architecture. They just probabilistically emerge from some word sequences. Even if a word sequence containing a relevant true fact is highly likely to be generated to a given prompt, there's still a reasonable chance it won't be. You'll get some other similarly weighted but semantically subtly different words swapped in instead that make sense as english but just don't mean what they should mean if we know the right answer.