4 ms·

GPT-4, and I'm surprised, but even Turbo, win out over every open-source LLM on the leaderboard, though Anthropic's Claude beats Turbo which is true in my exper

by rahidz 3y ago

GPT-4, and I'm surprised, but even Turbo, win out over every open-source LLM on the leaderboard, though Anthropic's Claude beats Turbo which is true in my experience. How likely is it that this will change anytime soon? Or are LLMs at the level of GPT-4 just out of reach for anyone other than a huge company like OpenAI?

htrp 3y ago
GPT4 uses that MoE approach so its effectively a fine-tuned model for about a dozen different domain spaces.
- famouswaffles 3y ago
  Not really. That's not what/how MoE models work. It's not an ensemble.
yieldcrv 3y ago
Its only 6 months in for GPT4 I think open source things will match gpt 4 in a year or a little after. OpenAI will need additional models or competitive advantages at the same clip they’ve been doing. This is going to be very expensive for them. What I really predict will happen is that late 2024 hardware refreshes (being worked on since mid 2023) will prioritize everything related to accelerating LLM training and output More RAM to different chips, different busses, different coprocessor architecture that we currently call GPUs and this will lower the barrier of entry enough
- flangola7 3y ago
  GPT-4-base was completed in early 2022, its existence only went public much later. Whatever OpenAI has right now is a year and a half more advanced than current models. Since open source models can't even touch GPT-3.5 yet, they are very far behind. OpenAI has been cutting deals left and right with newspapers, publishers, and other organizations to train on vast high quality private datasets that simply are not available to open source development. Looking at how datasets are at least as important as volume of compute, we can expect the gap between OAI and open source models to continue to widen. None of this addresses the gigantic (possibly 1:1000) advantage services with constant workload have when it comes to memory bottleneck efficacy.
  - yieldcrv 3y ago
    I think there will be diminishing utility of “smarter” models People won’t need them They’ll need more efficient models: smaller parameter count, faster output, thats just as smart as some current baseline benchmark
- oezi 3y ago
  I also think this will happen. Graphics cards with humongous RAM, etc. But I don't see how that changes the advantages that companies such as OpenAI have. Consumers will have access to GPT4-level LLMs, OpenAI will have a next gen offering.
  - yieldcrv 3y ago
    A more efficient and faster GPT4-level LLM coming from a smaller model running in consumer hardware is going to be good enough. It will be competitive if it is easier to train on current events and keep updated, by anyone for nearly free. OpenAI's next gen offerings need to be something entirely different than smarter, and good for them if they do. All the multimodal capabilities have to be out, readily available, and stuff I haven't thought of yet. I think everyone's right, or "accurate" that OpenAI will have far more resources, I don't think its important.
    - mycall 3y ago
      At the end of last year, OpenAI was talking about GPT-5 being a dead-end and something besides transformers needs to be made to do much better. We all wait for this.
    - Nevermark 3y ago
      > OpenAI will have far more resources, I don't think its important. Who knows which direction they go, but I expect improvements like the following would obsolete GPT-4 and equivalents for most people: Multi-modal? I.e. text, audio, image, video, physics in video and interactive simulation, etc. the result will be major new features and a much deeper understanding of many topics. Audio and video input output? Engage with the model using voice, tone, gestures expressions and any artifacts you have: whiteboards, objects, whatever. Bring the model into conversations and meetings with other people to add its input. Long term memory, summarizing previous interactions, providing long term continuity? “Your” instance keeps up with you without a context limit. For days. Or months. Or longer. Have multiple instances for work and home. Share an instance with family, as a joint assistant. Live audio and long term memory would create the ultimate voice memo machine. Never forget anything again. An accurate confidence in its own accuracy? Clear distinction between answers backed by real information vs. confabulation. Built in ability to use the web and other tools, such as programming and executing calculations on the fly? Much wider access to up to date information and ability to come up with complex calculated answers in real time. Ability to manage your resources? Make sure bills are paid, proactively suggest ways to save, services to cancel. Manage your health by proactively suggesting & tracking changes, making appointments for troubling symptoms, etc. Remind you of tasks with all information you need proactively collected. Schedule one off services to solve problems, and follow up to be sure they were solved. Not only create a trip itinerary, but manage the whole trip, including intervening to solve inevitable problems. On and on here.
araes 3y ago
Of these, I think I'm most surprised by Vicuna-33B. For having MIT Bench and MMLU scores significantly worse than #4 and #5, it fairs quite well (47% and 42% choice rate 1v1). Its only vs the #3+ things get bad (although notably also does better vs GPT-4 than many) Really, the choice rate's not bad when you really look at it. Even vs the top its 1/3, 2/3. Nowhere near as bad as NBA/NFL vs high-schoolers or similar "Professional Grade" situation. Also the first Non-Commercial top option.

by rahidz 3y ago

htrp 3y ago

GPT4 uses that MoE approach so its effectively a fine-tuned model for about a dozen different domain spaces.

famouswaffles 3y ago
Not really. That's not what/how MoE models work. It's not an ensemble.

yieldcrv 3y ago

Its only 6 months in for GPT4 I think open source things will match gpt 4 in a year or a little after. OpenAI will need additional models or competitive advantages at the same clip they’ve been doing. This is going to be very expensive for them. What I really predict will happen is that late 2024 hardware refreshes (being worked on since mid 2023) will prioritize everything related to accelerating LLM training and output More RAM to different chips, different busses, different coprocessor architecture that we currently call GPUs and this will lower the barrier of entry enough

flangola7 3y ago
GPT-4-base was completed in early 2022, its existence only went public much later. Whatever OpenAI has right now is a year and a half more advanced than current models. Since open source models can't even touch GPT-3.5 yet, they are very far behind. OpenAI has been cutting deals left and right with newspapers, publishers, and other organizations to train on vast high quality private datasets that simply are not available to open source development. Looking at how datasets are at least as important as volume of compute, we can expect the gap between OAI and open source models to continue to widen. None of this addresses the gigantic (possibly 1:1000) advantage services with constant workload have when it comes to memory bottleneck efficacy.
- yieldcrv 3y ago
  I think there will be diminishing utility of “smarter” models People won’t need them They’ll need more efficient models: smaller parameter count, faster output, thats just as smart as some current baseline benchmark
oezi 3y ago
I also think this will happen. Graphics cards with humongous RAM, etc. But I don't see how that changes the advantages that companies such as OpenAI have. Consumers will have access to GPT4-level LLMs, OpenAI will have a next gen offering.
- yieldcrv 3y ago
  A more efficient and faster GPT4-level LLM coming from a smaller model running in consumer hardware is going to be good enough. It will be competitive if it is easier to train on current events and keep updated, by anyone for nearly free. OpenAI's next gen offerings need to be something entirely different than smarter, and good for them if they do. All the multimodal capabilities have to be out, readily available, and stuff I haven't thought of yet. I think everyone's right, or "accurate" that OpenAI will have far more resources, I don't think its important.
  - mycall 3y ago
    At the end of last year, OpenAI was talking about GPT-5 being a dead-end and something besides transformers needs to be made to do much better. We all wait for this.
  - Nevermark 3y ago
    > OpenAI will have far more resources, I don't think its important. Who knows which direction they go, but I expect improvements like the following would obsolete GPT-4 and equivalents for most people: Multi-modal? I.e. text, audio, image, video, physics in video and interactive simulation, etc. the result will be major new features and a much deeper understanding of many topics. Audio and video input output? Engage with the model using voice, tone, gestures expressions and any artifacts you have: whiteboards, objects, whatever. Bring the model into conversations and meetings with other people to add its input. Long term memory, summarizing previous interactions, providing long term continuity? “Your” instance keeps up with you without a context limit. For days. Or months. Or longer. Have multiple instances for work and home. Share an instance with family, as a joint assistant. Live audio and long term memory would create the ultimate voice memo machine. Never forget anything again. An accurate confidence in its own accuracy? Clear distinction between answers backed by real information vs. confabulation. Built in ability to use the web and other tools, such as programming and executing calculations on the fly? Much wider access to up to date information and ability to come up with complex calculated answers in real time. Ability to manage your resources? Make sure bills are paid, proactively suggest ways to save, services to cancel. Manage your health by proactively suggesting & tracking changes, making appointments for troubling symptoms, etc. Remind you of tasks with all information you need proactively collected. Schedule one off services to solve problems, and follow up to be sure they were solved. Not only create a trip itinerary, but manage the whole trip, including intervening to solve inevitable problems. On and on here.

araes 3y ago

Of these, I think I'm most surprised by Vicuna-33B. For having MIT Bench and MMLU scores significantly worse than #4 and #5, it fairs quite well (47% and 42% choice rate 1v1). Its only vs the #3+ things get bad (although notably also does better vs GPT-4 than many) Really, the choice rate's not bad when you really look at it. Even vs the top its 1/3, 2/3. Nowhere near as bad as NBA/NFL vs high-schoolers or similar "Professional Grade" situation. Also the first Non-Commercial top option.