4 ms·
"[...] we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not." What other benchmarks do you thi
by simonw 2y ago
"[...] we talk about LMSys's because they agree with our feelings, and we don't talk about other results because they do not."
What other benchmarks do you think are being overlooked?
- doctorpangloss 2y agoNo idea. Better to ask the people training models what they use internally, not me. A good benchmark itself could be worth millions. LMSys has the same energy as Amazon reviews of brand name products. It's measuring something, and that something might be valuable. I don't know if it is valuable for answering subjective, spicy questions like, "What LLM is likely to replace most or all of your job right now, giving you some time to arbitrage 1h of chatbot with a full time employment?" which is what everyone actually cares about. That is what this is all about. I can't tell you what benchmark is answering that question or others. I don't know!
- simonw 2y agoI've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front. The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models. I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.
- jerpint 2y agoCompletely agree, and it’s been great to see to some extent that the gap between open source and 3rd party models is really starting to close
- FergusArgyll 2y agoSomeone made a framework to make and test llms on your own benchmarks you might find it useful https://github.com/carlini/yet-another-applied-llm-benchmark https://github.com/carlini/yet-another-applied-llm-benchmark
- bravura 2y agoPredicting the future. I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861 https://arxiv.org/abs//2402.00861 "Evaluating Large Language Models for Generalization and Robustness via Data Compression" They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments. This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.
- mejutoco 2y agoWouldn t this get you survivorship bias but with models instead of investors?