3 ms·
It's important to note that while the LLM Leaderboard (and the MTEB leaderboard) are good guides, being the top scoring model doesn't always mean much. For exa
by dmezzetti 3y ago
It's important to note that while the LLM Leaderboard (and the MTEB leaderboard) are good guides, being the top scoring model doesn't always mean much.
For example, in my projects, I've consistently found that the OpenOrca Mistral fine-tune (https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca) works better. Once again, that's just my experience and probably isn't universal. But it's important to try a couple models out to see if one that scores lower perhaps is better for your use case.
- hskalin 3y agoThat is what my experience has been. I have tried OpenHermes2.5 (very hyped 7B model) vs basic Mistral 7B and tbh I prefer Mistral's answer for most things.
- dmezzetti 3y agoExactly, benchmarks are just a guide. Same thing with vector models. Likely just a matter of how much your data overlaps with the fine-tuning dataset for a particular model.