4 ms·
Random anecdote: I tried a few trivia questions for 90s video games and llama 3 8b and 70b are hallucinating quite a bit. Command R+ and Mistral 8x22B were bet
by tosh 3y ago
Random anecdote: I tried a few trivia questions for 90s video games and llama 3 8b and 70b are hallucinating quite a bit.
Command R+ and Mistral 8x22B were better.
Llama 3 still feels like a huge jump forward in capability. I wonder why the 70b is not better at recall (for lack of a better word?).
- swatcoder 3y agoI wouldn't be surprised if the field's increased emphasis on reasoning, summarization, tool use, etc and optimizing for benchmarks on those factors might come at the expense of less emphasized/measured features like explicit recall. And that might not really be a bad thing. It's generally more useful to have something that can access (say) a specific dataset about video games and summarize its content to answer questions than it is to have one that needs to encode and recall every answer to every long tail question itself.
- tosh 3y agoAgreed, I prefer a more capable model that I can connect to Wikipedia than a less capable model.
- acchow 3y agoThose models are 50% and 150% larger. I imagine they just contain more data.