2 ms·
GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanfor
by famouswaffles 4y ago
GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios
For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75.
https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416
- tikkun 4y agoFlan-Palm is 75 with five-shot. text-davinci-003 is it 56.9 with five-shot or zero-shot?
- famouswaffles 4y agoYou're right. Equivalent scenarios, the gap is smaller - about a difference of 10. you can check the end of the flan paper for some equivalent comparisons.