8 ms·

https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... It looks like at least one other

https://twitter.com/soumithchintala/status/1671267150101721090 https://twitter.com/soumithchintala/status/16712671501017210... It looks like at least one other person has also heard the same information.

throwaway9274 3y ago
2x0=0
startupsfail 3y ago
It’s funny that this post is trending on HN right next to the post about a paper showing how to build a model 1000x smaller than 1.7T that can code better than LLMs 10x larger.
- TeMPOraL 3y ago
  I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is now.
  - candiodari 3y ago
    If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required. That would negate, at least partially, the "we have 20 datacenters" advantage.
    - ChatGTP 3y ago
      Is it a large language model anymore ? Something doesn’t quite add up?
      - TeMPOraL 3y ago
        It's still large. But it might no longer need to be "only few entities on the planet can afford to make one, and not at the same time, since NVIDIA can pump out GPUs only so fast" large.
    - startupsfail 3y ago
      I think it is. I also think this is what OpenAI did. They’ve carefully crafted the data. I don’t think they have an ensemble of 8 models. First, this is not elegant. Second, I don’t see how this could be compatible with the streaming output. I’d guess that GPT4 is around 200B parameters, and it’s trained on a dataset made with love, that goes from Lorem Ipsum to a doctorate degree. Love is all you need ;)
  - mrtranscendence 3y ago
    This doesn't seem likely in the near term. An iPhone 13 Pro has 6gb of memory, which might be enough for a single 1.3b parameter model if you purged everything else out of RAM. Combining 8 (or even 4) of them on a single phone won't happen anytime soon at the rate phones improve. Plus, none of the smaller models are really appreciably close to GPT-4 on most metrics. It's not clear to me you could get there at all with 1.3b models. Maybe somebody gets there someday with 65b models, but then you're far out of the reach of phones.
    - Dylan16807 3y ago
      Are you assuming 32 bit parameters? You definitely don't need to keep that much. People have been cramming llama into 3-4 bits. You wouldn't want 8x1.3b but it should fit. Also Apple has never gone for high RAM in mobile devices. I could go get 12GB in a brand new phone for $400, and some high end phones have had 16GB since 2020. So combined, you could do normal app stuff with 2.5GB and 20-25b 3-bit inference with the other 9.5GB.
espadrine 3y ago
To be fair, this was already a common whisper at the time, so it could be a Chinese whisper effect. Even I thought this on release day: https://news.ycombinator.com/item?id=35165874 https://news.ycombinator.com/item?id=35165874 What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-source model on the leaderboard beats it 30% of the time. [0]: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard [1]: https://news.ycombinator.com/item?id=35893357 https://news.ycombinator.com/item?id=35893357
- machdiamonds 3y ago
  From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more complex tasks, such as code generation, GPT4 unequivocally outshines all open source models. That said, the advancements in models like Orca and the "Textbooks are all you need" paper are noteworthy (https://arxiv.org/pdf/2306.02707.pdf https://arxiv.org/pdf/2306.02707.pdf, https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644). I'm optimistic about what future smaller models could achieve.
  - espadrine 3y ago
    I agree. Also, ELO rankings can artificially deflate strong performers when having too few binary comparisons. The Guanaco paper points that out, giving instead a 326-point lead to GPT-4 over Guanaco-65B across 10k orderings, based on GPT-4’s opinion, which corresponds to Guanaco winning about 13% of the time. But even that only relies on the Vicuna benchmark set.
  - mark_l_watson 3y ago
    There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements). I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I could do with T5-flan-XXL, and it is useful, but not as effective tool as the closed OpenAI models. I will do the same with Orca this week.
manojlds 3y ago
And from Bing - https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304

throwaway9274 3y ago

2x0=0

startupsfail 3y ago

It’s funny that this post is trending on HN right next to the post about a paper showing how to build a model 1000x smaller than 1.7T that can code better than LLMs 10x larger.

TeMPOraL 3y ago
I don't find it funny, I find it scary and mind-blowing: the impact of these headlines is additive - this one confirms the effectiveness of combining models, and the other one suggests you could cut the model size a couple orders of magnitude if you train on clean enough data. Together, this points at a way to achieve both GPT-4 that fits on your phone, and a much more powerful model that's not larger than GPT-4 is now.
- candiodari 3y ago
  If it truly is the training data that's making models smart, then that would explain that there is both a minimum and maximum "useful" size to LLMs. The recent stream of papers seems to indicate that the cleaner the input data, the less size is required. That would negate, at least partially, the "we have 20 datacenters" advantage.
  - ChatGTP 3y ago
    Is it a large language model anymore ? Something doesn’t quite add up?
    - TeMPOraL 3y ago
      It's still large. But it might no longer need to be "only few entities on the planet can afford to make one, and not at the same time, since NVIDIA can pump out GPUs only so fast" large.
  - startupsfail 3y ago
    I think it is. I also think this is what OpenAI did. They’ve carefully crafted the data. I don’t think they have an ensemble of 8 models. First, this is not elegant. Second, I don’t see how this could be compatible with the streaming output. I’d guess that GPT4 is around 200B parameters, and it’s trained on a dataset made with love, that goes from Lorem Ipsum to a doctorate degree. Love is all you need ;)
- mrtranscendence 3y ago
  This doesn't seem likely in the near term. An iPhone 13 Pro has 6gb of memory, which might be enough for a single 1.3b parameter model if you purged everything else out of RAM. Combining 8 (or even 4) of them on a single phone won't happen anytime soon at the rate phones improve. Plus, none of the smaller models are really appreciably close to GPT-4 on most metrics. It's not clear to me you could get there at all with 1.3b models. Maybe somebody gets there someday with 65b models, but then you're far out of the reach of phones.
  - Dylan16807 3y ago
    Are you assuming 32 bit parameters? You definitely don't need to keep that much. People have been cramming llama into 3-4 bits. You wouldn't want 8x1.3b but it should fit. Also Apple has never gone for high RAM in mobile devices. I could go get 12GB in a brand new phone for $400, and some high end phones have had 16GB since 2020. So combined, you could do normal app stuff with 2.5GB and 20-25b 3-bit inference with the other 9.5GB.

espadrine 3y ago

To be fair, this was already a common whisper at the time, so it could be a Chinese whisper effect. Even I thought this on release day: https://news.ycombinator.com/item?id=35165874 https://news.ycombinator.com/item?id=35165874 What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-source model on the leaderboard beats it 30% of the time. [0]: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard [1]: https://news.ycombinator.com/item?id=35893357 https://news.ycombinator.com/item?id=35893357

machdiamonds 3y ago
From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more complex tasks, such as code generation, GPT4 unequivocally outshines all open source models. That said, the advancements in models like Orca and the "Textbooks are all you need" paper are noteworthy (https://arxiv.org/pdf/2306.02707.pdf https://arxiv.org/pdf/2306.02707.pdf, https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644). I'm optimistic about what future smaller models could achieve.
- espadrine 3y ago
  I agree. Also, ELO rankings can artificially deflate strong performers when having too few binary comparisons. The Guanaco paper points that out, giving instead a 326-point lead to GPT-4 over Guanaco-65B across 10k orderings, based on GPT-4’s opinion, which corresponds to Guanaco winning about 13% of the time. But even that only relies on the Vicuna benchmark set.
- mark_l_watson 3y ago
  There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements). I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I could do with T5-flan-XXL, and it is useful, but not as effective tool as the closed OpenAI models. I will do the same with Orca this week.

manojlds 3y ago

And from Bing - https://twitter.com/MParakhin/status/1670666605427298304 https://twitter.com/MParakhin/status/1670666605427298304