4 ms·
The GPT-3 they are comparing to is the one that was released on 2020. Since then OpenAI made a lot of improvements and nowadays I believe GPT-3.5 is competitive
by rafaelero 4y ago
The GPT-3 they are comparing to is the one that was released on 2020. Since then OpenAI made a lot of improvements and nowadays I believe GPT-3.5 is competitive to Palm (540b). Still, LLaMa is in the same tier, with much less parameters.
- famouswaffles 4y agoGPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75. https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416
- tikkun 4y agoFlan-Palm is 75 with five-shot. text-davinci-003 is it 56.9 with five-shot or zero-shot?
- famouswaffles 4y agoYou're right. Equivalent scenarios, the gap is smaller - about a difference of 10. you can check the end of the flan paper for some equivalent comparisons.
- famouswaffles 4y agoJust a heads up with my comparison. Under equivalent scenarios, the gap is smaller. davinci-003 gets about 10 more points using five shot (which is what the palm comp does)