4 ms·
LLM Leaderboard
- ilaksh 3y agoWhere is the new giant Falcon model?
- rahidz 3y agoGPT-4, and I'm surprised, but even Turbo, win out over every open-source LLM on the leaderboard, though Anthropic's Claude beats Turbo which is true in my experience. How likely is it that this will change anytime soon? Or are LLMs at the level of GPT-4 just out of reach for anyone other than a huge company like OpenAI?
- htrp 3y agoGPT4 uses that MoE approach so its effectively a fine-tuned model for about a dozen different domain spaces.
- famouswaffles 3y agoNot really. That's not what/how MoE models work. It's not an ensemble.
- yieldcrv 3y agoIts only 6 months in for GPT4 I think open source things will match gpt 4 in a year or a little after. OpenAI will need additional models or competitive advantages at the same clip they’ve been doing. This is going to be very expensive for them. What I really predict will happen is that late 2024 hardware refreshes (being worked on since mid 2023) will prioritize everything related to accelerating LLM training and output More RAM to different chips, different busses, different coprocessor architecture that we currently call GPUs and this will lower the barrier of entry enough
- flangola7 3y agoGPT-4-base was completed in early 2022, its existence only went public much later. Whatever OpenAI has right now is a year and a half more advanced than current models. Since open source models can't even touch GPT-3.5 yet, they are very far behind. OpenAI has been cutting deals left and right with newspapers, publishers, and other organizations to train on vast high quality private datasets that simply are not available to open source development. Looking at how datasets are at least as important as volume of compute, we can expect the gap between OAI and open source models to continue to widen. None of this addresses the gigantic (possibly 1:1000) advantage services with constant workload have when it comes to memory bottleneck efficacy.
- yieldcrv 3y agoI think there will be diminishing utility of “smarter” models People won’t need them They’ll need more efficient models: smaller parameter count, faster output, thats just as smart as some current baseline benchmark
- oezi 3y agoI also think this will happen. Graphics cards with humongous RAM, etc. But I don't see how that changes the advantages that companies such as OpenAI have. Consumers will have access to GPT4-level LLMs, OpenAI will have a next gen offering.
- yieldcrv 3y ago
- araes 3y agoOf these, I think I'm most surprised by Vicuna-33B. For having MIT Bench and MMLU scores significantly worse than #4 and #5, it fairs quite well (47% and 42% choice rate 1v1). Its only vs the #3+ things get bad (although notably also does better vs GPT-4 than many) Really, the choice rate's not bad when you really look at it. Even vs the top its 1/3, 2/3. Nowhere near as bad as NBA/NFL vs high-schoolers or similar "Professional Grade" situation. Also the first Non-Commercial top option.
- deleted 3y ago[deleted]
- adt 3y agoMore comprehensive list: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/