3 ms·
Once benchmarks exist for a while, they become meaningless - even if it's not specifically training on the test set, actions (what used to be called "graduate s
by andy99 3y ago
Once benchmarks exist for a while, they become meaningless - even if it's not specifically training on the test set, actions (what used to be called "graduate student descent") end up optimizing new models towards overfitting on benchmark tasks.
- acchow 3y agoAlso, the technological leader focuses less on the benchmarks
- bilbo0s 3y ago"graduate student descent" Ahhh that takes me back!
- karmasimida 3y agoEven random seed could cause bad big shift in human eval performance if you know you know. It is perfectly illegal to choose one ckpt that looks best on those benchmarks and move along HumanEval is meaningless regardless, those 164 problems have been overfit to the tea. Hook this up to LLM arena we will get a better picture regarding how powerful they really are