4 ms·
> The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb https://hugg
by behnamoh 3y ago
> The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there.
I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
- minimaxir 3y agoIt's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/ https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination. If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
- visarga 3y agoI evaluated many Mistrals for an information extraction task and the merged models were much better than direct fine-tunes. About 5% better.