3 ms·

It doesn’t “make everything up”. On a hallucination benchmark [1] Llama 3 8B has a hallucination rate of 5.4%, compared to GPT4 at 3%, and GPT-4 Turbo at 2.5%.

• ein0p 2y ago
  Benchmarks don’t tell the whole story. Obscure questions are unlikely to be in benchmarks. Large models simply encode more knowledge, which means that they are _way_ more likely to not hallucinate an answer to practical questions that are off the beaten path and that the model hasn’t seen before. Try asking your Llama 8B about some less commonly known details in some topic you know well, and then repeat the same with GPT4. Even Claude Opus hallucinates way more than GPT4, despite the claims that it matches it according to the various leaderboards and whatever.
  • anon373839 2y ago
    I agree that larger models have more memorization capacity, but that isn't what find exciting about LLMs, personally. You can ask GPT-4 some esoteric question and get back an answer that's more likely to be truthful than Llama 3's, but either way, it's the same result: you have to look it up for yourself. What I think is much more interesting about LLMs is their ability to transform natural language inputs. Applications like open-book question answering, summarization, data extraction, and so forth. For these things, small models are at the point where they are commercially useful now. And if you do fine-tune them on your own domain, they will work better than larger general-purpose models.