4 ms·

Have we got any details on the benchmarks that show LLaMa's 13B architecture outperforming GPT-3? Because that seems kindsof fantastical. Is it just a product o

Have we got any details on the benchmarks that show LLaMa's 13B architecture outperforming GPT-3? Because that seems kindsof fantastical. Is it just a product of a very specific benchmark or does it reflect real world performance?

rnosov 4y ago
They list 7B, 13B, 33B, 65B architectures. Presumably, they compare 65b one to GPT-3 175B. Chinchilla model which is about 70B outperformed a much larger GPT-3 model. So not that fantastical. EDIT: I stand corrected. They do compare 13B model with a large GPT-3 model which is hard to believe without a bit more concrete evidence
- rileyphone 4y ago
  You're incorrect. > For example, LLaMA's 13B architecture outperforms GPT-3 despite being 10 times smaller. Of course this is all on benchmarks but it's a big improvement if true.
rafaelero 4y ago
The GPT-3 they are comparing to is the one that was released on 2020. Since then OpenAI made a lot of improvements and nowadays I believe GPT-3.5 is competitive to Palm (540b). Still, LLaMa is in the same tier, with much less parameters.
- famouswaffles 4y ago
  GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75. https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416
  - tikkun 4y ago
    Flan-Palm is 75 with five-shot. text-davinci-003 is it 56.9 with five-shot or zero-shot?
    - famouswaffles 4y ago
      You're right. Equivalent scenarios, the gap is smaller - about a difference of 10. you can check the end of the flan paper for some equivalent comparisons.
- famouswaffles 4y ago
  Just a heads up with my comparison. Under equivalent scenarios, the gap is smaller. davinci-003 gets about 10 more points using five shot (which is what the palm comp does)

rnosov 4y ago

They list 7B, 13B, 33B, 65B architectures. Presumably, they compare 65b one to GPT-3 175B. Chinchilla model which is about 70B outperformed a much larger GPT-3 model. So not that fantastical. EDIT: I stand corrected. They do compare 13B model with a large GPT-3 model which is hard to believe without a bit more concrete evidence

rileyphone 4y ago
You're incorrect. > For example, LLaMA's 13B architecture outperforms GPT-3 despite being 10 times smaller. Of course this is all on benchmarks but it's a big improvement if true.

rafaelero 4y ago

The GPT-3 they are comparing to is the one that was released on 2020. Since then OpenAI made a lot of improvements and nowadays I believe GPT-3.5 is competitive to Palm (540b). Still, LLaMa is in the same tier, with much less parameters.

famouswaffles 4y ago
GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75. https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416
- tikkun 4y ago
  Flan-Palm is 75 with five-shot. text-davinci-003 is it 56.9 with five-shot or zero-shot?
  - famouswaffles 4y ago
    You're right. Equivalent scenarios, the gap is smaller - about a difference of 10. you can check the end of the flan paper for some equivalent comparisons.
famouswaffles 4y ago
Just a heads up with my comparison. Under equivalent scenarios, the gap is smaller. davinci-003 gets about 10 more points using five shot (which is what the palm comp does)