4 ms·
> I analyzed 1,232 models from the Open LLM Leaderboard and 88 models from the General Language Understanding Evaluation (GLUE) Leaderboard. This analysis seem
by verteu 3y ago
> I analyzed 1,232 models from the Open LLM Leaderboard and 88 models from the General Language Understanding Evaluation (GLUE) Leaderboard.
This analysis seems highly sensitive to the population of "models." What if I submit the same model 1000x to the Open LLM Leaderboard (perhaps with small tweaks)? I'd expect the models' performance would be highly correlated, increasing your measure of `g`.
Given this, does the value of `g` truly say something meaningful about LLMs?
- famouswaffles 3y agoTo be fair, he did this twice. With the LLM leaderboard and again with Glue to the same results. I don't know how gamed the model for Glue could be in this way ? Should be easy enough to check i guess
- verteu 3y agoGood point!