It's also not used to learn facts about stuff: it's used to learn how language works and is used to describe the world. The sum of all variants of human commun

It's also not used to learn facts about stuff: it's used to learn how language works and is used to describe the world. The sum of all variants of human communication in text is pretty good for this and examples of wrong or different also matter: there was that article a few months ago about how a chemical prediction model performed better when it was also trained on invalid SMILES representations compared to totally sanitized datasets.