Anyone that has paid any attention to ChatGPT, or LLMs in general, knows that this is absolutely par for the course. There’s no reliability that you get correct

Anyone that has paid any attention to ChatGPT, or LLMs in general, knows that this is absolutely par for the course. There’s no reliability that you get correct answers with current training methods. They are optimized to create plausible (from a word-statistics perspective), human sounding sentences. They’re also overfit on an incredible amount of data, which is why they are so large—-they memorize web scale data. But we can’t tell the difference between a correctly memorized fact and an incorrectly conjured one. Until someone figures out, and implements, a good way to wire these up to an external fact data store, I don’t see a way around it. But I also don’t think that’s a very hard problem to solve and will likely result in far smaller models. I suspect companies launch these products knowing these deficiencies because right now it’s about market share and UX, and the technical ML problems will be confidently solved. Especially if you already own a search engine.