4 ms·
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as
by simonw 2y ago
I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front.
The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models.
I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
- jerpint 2y agoCompletely agree, and it’s been great to see to some extent that the gap between open source and 3rd party models is really starting to close
- FergusArgyll 2y agoSomeone made a framework to make and test llms on your own benchmarks you might find it useful https://github.com/carlini/yet-another-applied-llm-benchmark https://github.com/carlini/yet-another-applied-llm-benchmark