To some extent it does contain facts, but those facts are indistinguishable from non-facts, are just a small proportion of the dataset compared to everything el

To some extent it does contain facts, but those facts are indistinguishable from non-facts, are just a small proportion of the dataset compared to everything else are indistinguishable from non factual information. E.g. using text-davinci-003 (this is GPT3, not ChatGPT), "The moon is made of" completes to: Cheese 48.74%, rock: 31.66%, green 4.09% (98.75% followed by cheese), rocks 3.86%, and several other lower percentage tokens. I wonder if there eventually will be a type of model that incorporates the ability to simultaneously do text completion while adhering to facts at the model level (rather than having to bolt it on top via context).