4 ms·

> Meta released Llama-3 only three days ago, and it already feels like the inflection point when open source models finally closed the gap with proprietary mode

> Meta released Llama-3 only three days ago, and it already feels like the inflection point when open source models finally closed the gap with proprietary models. The benchmarks show that Llama-3 70B matches GPT-4 and Claude Opus in most tasks, and the even more powerful Llama-3 400B+ model is still training. I'm all for open models, but where do the benchmarks show that? The official page https://llama.meta.com/llama3/ https://llama.meta.com/llama3/ does not show any comparisons with GPT-4 or Claude Opus Looking at https://arena.lmsys.org/ https://arena.lmsys.org/, Llama-3-70b-Instruct is ranked #5 while current GPT-4 models and Claude Opus are still tied at #1. Meanwhile, Llama-3-8b-Instruct is ranked #14 Would love to be corrected, but either way an article should include sources for these types of claims.

afro88 3y ago
Came here to say this. Thought I missed something! 400B does look set to meet GPT-4, which will be exciting, but it's not finished yet.
- faeriechangling 3y ago
  OpenAI is also a moving target and Llama3 is probably due to get a few months of parity before GPT5 wallops it. To me, it looks like OpenAI is ahead of the competition by 12-18 months, which isn't nothing but the competition is certainly nipping at their heels.
  - boppo1 3y ago
    I thought OpenAI said they weren't training gpt5 and were doing more smaller multimodal models
    - woleium 3y ago
      what they say and what they do are different things
    - pskkk 3y ago
      I wonder if that was a case of being literal as in, technically speaking gpt-5 isn’t being _trained_, while omitting that they are working on collecting and preparing the data.
    - faeriechangling 3y ago
      They have recently confirmed that they were working on GPT5, without confirming or denying that they were training it. I guess it isn't unthinkable that GPT5 is more than a year out given the gap between GPT3 and GPT4 was 3 years. At the same time, the rumourmongers at businessinsider said their sources are expecting a Summer release, which seems plausible. Nobody really knows when it will release besides Sam Altman though.
      - airstrike 3y ago
        > Nobody really knows when it will release besides Sam Altman though. The day after some model truly beats GPT-4, plus or minus 1-2 weeks, is my guess.
  - nabakin 3y ago
    Not ahead of Anthropic. Anthropic is trading blows with the best GPT-4 models these days. For the open weight/source community I'd put it at 3-6 months behind and catching up. Llama 3 405B might catch the community up unless OpenAI has another LLM up their sleeve that makes a significant jump forward, but we haven't seen evidence of that yet.
    - faeriechangling 3y ago
      Anthropic's Claude 3 came out 12 months after GPT4.
      - nabakin 3y ago
        GPT-4 has been updated multiple times since then. Anthropic has been trading blows with those updates to GPT-4.
  - anon373839 3y ago
    At the risk of this comment aging poorly, I am not convinced there is a magic GPT-5 around the corner, waiting to wallop anybody. I have a suspicion that we’re into diminishing returns with model scaling. All of the recent flagship releases (Gemini Ultra, Claude Opus, GPT-4 updates) have only advanced the frontier a little. Could OpenAI train something another 1-2 orders of magnitude bigger? Perhaps, but then it wouldn’t be economical to deploy it.
    - tomohelix 3y ago
      Sora is a thing. It literally walloped everything else in the field to the point the second closest video generation model looked like a broken nonfunctional mess compared to it. GPT5 might not be the thing we are all assuming it would be. It might not just be a chatbot but an actual AI that can "see" and "talk". By that point, I don't care if performance wise it isn't a big improvement over GPT4 (though it should as we know these models improve significantly when trained with another modal of data, e.g. text+image was much better than text only)
willsmith72 3y ago
Yeah I came to post the same thing. It's nothing like gpt4, just great for the size.
natrys 3y ago
It's above Opus in second position in the "English" only category. It probably suffers in the overall score due to poor multilingual ability (afaik 95% of its training data was English only). Though usual caveat about small sample size applies, as of now the CI is fairly big. It's also not at the level of those two in "Code" category, I hope Meta will give the CodeLlama variant an update again.
rb2k_ 3y ago
Some Llama3 400B numbers for the April 15th checkpoint are listed at https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ Direct image link: https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/439015366_1603174683862748_5008894608826037916_n.png?_nc_cat=105&ccb=1-7&_nc_sid=e280be&_nc_ohc=LMkkIsSw4qQAb4n9Jka&_nc_ht=scontent-atl3-2.xx&oh=00_AfBVj_TjflKu3gB1K201kFHoWArwzxdZSs11wN9upDw3gw&oe=663FD7AA https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/439015366_...)
gtt 3y ago
How significant is the difference in Elo scores for the end user? Should I use Claude Opus if I already have GPT4, despite its similar score? Additionally, how much better is Claude for coding?
- floydnoel 3y ago
  I use Poe.com, which gives me access to every model available for $20. When a new model comes out it is quickly added to the available list. So, I've been doing some comparison against Claude & GPT4. I still think GPT4 is better at following instructions and giving me the result I'm after.

afro88 3y ago

Came here to say this. Thought I missed something! 400B does look set to meet GPT-4, which will be exciting, but it's not finished yet.

faeriechangling 3y ago
OpenAI is also a moving target and Llama3 is probably due to get a few months of parity before GPT5 wallops it. To me, it looks like OpenAI is ahead of the competition by 12-18 months, which isn't nothing but the competition is certainly nipping at their heels.
- boppo1 3y ago
  I thought OpenAI said they weren't training gpt5 and were doing more smaller multimodal models
  - woleium 3y ago
    what they say and what they do are different things
  - pskkk 3y ago
    I wonder if that was a case of being literal as in, technically speaking gpt-5 isn’t being _trained_, while omitting that they are working on collecting and preparing the data.
  - faeriechangling 3y ago
    They have recently confirmed that they were working on GPT5, without confirming or denying that they were training it. I guess it isn't unthinkable that GPT5 is more than a year out given the gap between GPT3 and GPT4 was 3 years. At the same time, the rumourmongers at businessinsider said their sources are expecting a Summer release, which seems plausible. Nobody really knows when it will release besides Sam Altman though.
    - airstrike 3y ago
      > Nobody really knows when it will release besides Sam Altman though. The day after some model truly beats GPT-4, plus or minus 1-2 weeks, is my guess.
- nabakin 3y ago
  Not ahead of Anthropic. Anthropic is trading blows with the best GPT-4 models these days. For the open weight/source community I'd put it at 3-6 months behind and catching up. Llama 3 405B might catch the community up unless OpenAI has another LLM up their sleeve that makes a significant jump forward, but we haven't seen evidence of that yet.
  - faeriechangling 3y ago
    Anthropic's Claude 3 came out 12 months after GPT4.
    - nabakin 3y ago
      GPT-4 has been updated multiple times since then. Anthropic has been trading blows with those updates to GPT-4.
- anon373839 3y ago
  At the risk of this comment aging poorly, I am not convinced there is a magic GPT-5 around the corner, waiting to wallop anybody. I have a suspicion that we’re into diminishing returns with model scaling. All of the recent flagship releases (Gemini Ultra, Claude Opus, GPT-4 updates) have only advanced the frontier a little. Could OpenAI train something another 1-2 orders of magnitude bigger? Perhaps, but then it wouldn’t be economical to deploy it.
  - tomohelix 3y ago
    Sora is a thing. It literally walloped everything else in the field to the point the second closest video generation model looked like a broken nonfunctional mess compared to it. GPT5 might not be the thing we are all assuming it would be. It might not just be a chatbot but an actual AI that can "see" and "talk". By that point, I don't care if performance wise it isn't a big improvement over GPT4 (though it should as we know these models improve significantly when trained with another modal of data, e.g. text+image was much better than text only)

willsmith72 3y ago

Yeah I came to post the same thing. It's nothing like gpt4, just great for the size.

natrys 3y ago

It's above Opus in second position in the "English" only category. It probably suffers in the overall score due to poor multilingual ability (afaik 95% of its training data was English only). Though usual caveat about small sample size applies, as of now the CI is fairly big. It's also not at the level of those two in "Code" category, I hope Meta will give the CodeLlama variant an update again.

rb2k_ 3y ago

Some Llama3 400B numbers for the April 15th checkpoint are listed at https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ Direct image link: https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/439015366_1603174683862748_5008894608826037916_n.png?_nc_cat=105&ccb=1-7&_nc_sid=e280be&_nc_ohc=LMkkIsSw4qQAb4n9Jka&_nc_ht=scontent-atl3-2.xx&oh=00_AfBVj_TjflKu3gB1K201kFHoWArwzxdZSs11wN9upDw3gw&oe=663FD7AA https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/439015366_...)

gtt 3y ago

How significant is the difference in Elo scores for the end user? Should I use Claude Opus if I already have GPT4, despite its similar score? Additionally, how much better is Claude for coding?

floydnoel 3y ago
I use Poe.com, which gives me access to every model available for $20. When a new model comes out it is quickly added to the available list. So, I've been doing some comparison against Claude & GPT4. I still think GPT4 is better at following instructions and giving me the result I'm after.