4 ms·
In my experience, HellaSwag seems to correlate highly enough with model performance for my use case (i.e., complex autocompletion of prose and code rather than
by cosmojg 3y ago
In my experience, HellaSwag seems to correlate highly enough with model performance for my use case (i.e., complex autocompletion of prose and code rather than rather than glorified chatbotting). MMLU and ARC aren't too bad, but TruthfulQA can sometimes be a straight-up countersignal.
- brucethemoose2 3y ago> 36% of HellaSwag benchmark contains errors https://old.reddit.com/r/LanguageTechnology/comments/zkia03/36_of_hellaswag_benchmark_contains_errors/ https://old.reddit.com/r/LanguageTechnology/comments/zkia03/...