3 ms·

GPT3.5 was 175B parameters and GPT4 is larger (nobody knows how larger) and it blows GPT3.5 out of the water. So, GPT4 is, well, way more than 100B+. And yes,

GPT3.5 was 175B parameters and GPT4 is larger (nobody knows how larger) and it blows GPT3.5 out of the water. So, GPT4 is, well, way more than 100B+. And yes, I share your opinion

jml78 3y ago
I used chatgpt4 daily. I tried using some of these 13b models and was completely underwhelmed when compared gpt4.
- tmalsburg2 3y ago
  I think some of these 13B models may only be pretrained and may not have benefited from RLHF. So it’s not clear how much of the difference is due to parameter count.
emadm 3y ago
The ability to run it on your own private data and train/distill it from the larger models is very useful (see Orca for example: https://arxiv.org/abs/2306.02707 https://arxiv.org/abs/2306.02707). Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.
- MuffinFlavored 3y ago
  > is very useful How? Is there not extreme correlation between "13B parameters (low/not a lot) most likely means lots of hallucinations"
  - emadm 3y ago
    What’s better a 130bn model or 10 13bn models working together.
    - jmiskovic 3y ago
      The GPT4 cognitive leaps and breadth of knowledge seem impossible to replicate with any number of 13B models. 13B x10 might be the only feasible business model for your company, but that alone doesn't make it a superior option. Can you share more details on how the models work together?
brianjking 3y ago
There is no information about the size of GPT-4; it could very well be smaller than GPT-3.5. Sam has said multiple times "the age of large models is over" https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of... Either way, the LLaMa models are quite powerful and now that we have an open, commercially usable version in 3, 7, and 13b sizes is huge. The 65b model I'm assuming is coming soon as well.

GPT3.5 was 175B parameters and GPT4 is larger (nobody knows how larger) and it blows GPT3.5 out of the water. So, GPT4 is, well, way more than 100B+. And yes, I share your opinion

jml78 3y ago

I used chatgpt4 daily. I tried using some of these 13b models and was completely underwhelmed when compared gpt4.

tmalsburg2 3y ago
I think some of these 13B models may only be pretrained and may not have benefited from RLHF. So it’s not clear how much of the difference is due to parameter count.

emadm 3y ago

The ability to run it on your own private data and train/distill it from the larger models is very useful (see Orca for example: https://arxiv.org/abs/2306.02707 https://arxiv.org/abs/2306.02707). Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.

MuffinFlavored 3y ago
> is very useful How? Is there not extreme correlation between "13B parameters (low/not a lot) most likely means lots of hallucinations"
- emadm 3y ago
  What’s better a 130bn model or 10 13bn models working together.
  - jmiskovic 3y ago
    The GPT4 cognitive leaps and breadth of knowledge seem impossible to replicate with any number of 13B models. 13B x10 might be the only feasible business model for your company, but that alone doesn't make it a superior option. Can you share more details on how the models work together?

brianjking 3y ago

There is no information about the size of GPT-4; it could very well be smaller than GPT-3.5. Sam has said multiple times "the age of large models is over" https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of... Either way, the LLaMa models are quite powerful and now that we have an open, commercially usable version in 3, 7, and 13b sizes is huge. The 65b model I'm assuming is coming soon as well.