3 ms·
When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string
by simonw 2y ago
When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string together grammatically correct sentences. As a result, many people assumed that even GPT-3.5 was wildly more "intelligent" than it actually was.
I think Deep Research (and tools like it) offer an even stronger illustration of that same effect. Anything that can produce a well-formatted multiple page report with headings and citations surely must be of PhD-level intelligence, right?
(Clearly not.)
- SubiculumCode 2y agoIn some ways, it's a good tool to teach yourself to sus out the real clues to reliability, not format and authoritative tone.
- j_maffe 2y agoBut that's the thing. The only way to truly find out if it's reliable (>90%) is to check the data yourself.
- SubiculumCode 2y agoThis is why metrics and leaderboards like these are so important (but under reported on): https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard https://www.kaggle.com/facts-leaderboard https://www.kaggle.com/facts-leaderboard Google Gemni models seem to lead...hopefully the metrics aren't being gamed.
- immibis 2y agoThat's been every LLM since GPT-2.
- isaiahwp 2y agoTo be fair, OpenAI's the one marketing it as such.
- jcgrillo 2y agocomputer use big big words ergo computer rull rull smrt