3 ms·
I thought LLaMA outscored GPT-3
by flangola7 4y ago
I thought LLaMA outscored GPT-3
- simonw 4y agoGPT-3 is a very different model from GPT-3.5. My understanding is that they were comparing LLaMA's performance to benchmark scores published for the original GPT-3, which came out in 2020 and had not yet had instruction tuning, so was significantly harder to use.
- flangola7 4y agoI know, that is why I said GPT-3 (Davinci) not GPT-3.5|ChatGPT.
- simonw 4y agoDa Vinci 002 and 003 are actually classified as GPT 3.5 by OpenAI: https://platform.openai.com/docs/models/gpt-3-5 https://platform.openai.com/docs/models/gpt-3-5 ChatGPT is GPT-3.5 Turbo.
- eshack94 4y agoWould you mind summarizing the difference between GPT 3.5 and GPT-3.5 Turbo? I'm not clear about that.
- simonw 4y agoGPT 3.5 is the instruction tuned modern GPT models, such as Da Vinci 002 and 003. 3.5 Turbo is the ChatGPT model: it's cheaper (1/10th the price), faster and has a bunch of extra RLHF training to make it work well as a safe and usable chatbot. https://openai.com/blog/introducing-chatgpt-and-whisper-apis https://openai.com/blog/introducing-chatgpt-and-whisper-apis introduced the turbo model.
- vletal 4y agoHard to measure these days. The training sets are so large they might contain leaks of test sets. Take these numbers with a grain of salt.
- code51 4y agoOr... it could be that Chinchilla study has deficiencies in measuring capabilities of models maybe? Either that or your explanation. Frankly I don't think 13B is better than GPT-3 (text-davinci-001 which I think is not RLHF - but maybe better than base)
- simonw 4y agotext-davinci-001 is currently classed as "GPT 3.5" by OpenAI, and it did indeed have RLHF in the form of instruction tuning: https://openai.com/research/instruction-following https://openai.com/research/instruction-following MY MISTAKE: 002 and 003 are 3.5, but 001 looks to have pre-dated the InstructGPT work.