4 ms·
Regular reminder that most LLM benchmarks are pretty terrible. I've looked inside the datasets and it's not stuff that I'd want to dictate what determines which
by tikkun 3y ago
Regular reminder that most LLM benchmarks are pretty terrible. I've looked inside the datasets and it's not stuff that I'd want to dictate what determines which models are best!
- cosmojg 3y agoIn my experience, HellaSwag seems to correlate highly enough with model performance for my use case (i.e., complex autocompletion of prose and code rather than rather than glorified chatbotting). MMLU and ARC aren't too bad, but TruthfulQA can sometimes be a straight-up countersignal.
- brucethemoose2 3y ago> 36% of HellaSwag benchmark contains errors https://old.reddit.com/r/LanguageTechnology/comments/zkia03/36_of_hellaswag_benchmark_contains_errors/ https://old.reddit.com/r/LanguageTechnology/comments/zkia03/...