4 ms·

Do you realize I'm not disagreeing with you about the difference between 3 and 4? Reread what I wrote. I contrasted 3 and 4 with 2 and 3, which you seem to be

Do you realize I'm not disagreeing with you about the difference between 3 and 4? Reread what I wrote. I contrasted 3 and 4 with 2 and 3, which you seem to be entirely ignoring. 3 and 4 could be worlds apart, but wouldn't matter if 2 and 3 were two worlds apart, for example. And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. You would need a matching exponential growth in processing capability which is most certainly not occurring. So compute will not fall at the rate you would need it to for LLMs to actually compete in any meaningful way with human software engineers. We are not guaranteed to continue to progress in anything just because we have in the past.

famouswaffles 3y ago
>And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. This is a lot of unfounded assumptions. You don't need Moore's Law. GPU's are not really made with ML training in mind. You don't need exponential growth for anything. The money Open ai spent on GPT-4 a year ago could train a model twice as large today. and that amount is a drop in the bucket for the R&D of large corporations. Microsoft gave open ai 10B. amazon gave anthropic 4B >So compute will not fall at the rate you would need it to for LLMs to actually compete in any meaningful way with human software engineers. I don't think the compute reuired is anywhere near as much as you think it is. https://arxiv.org/abs/2309.12499 https://arxiv.org/abs/2309.12499 >We are not guaranteed to continue to progress in anything just because we have in the past. Nothing is guaranteed. But the scaling plots show no indication of a slow down so it's up to you to provide a concrete reason this object in motion is going to stop immediately and conveniently right now. If all you have is "well it just can't keep getting better right" then visit the 2 and 3 threads to see how meaningless such unfounded assertions are.
- WendyTheWillow 3y ago
  I think you fundamentally don't understand the nature of exponential growth, and the power of diminishing returns. Even if you double the GPU capacity over the next year, you won't even remotely begin to come close enough to producing a step-level growth of capability such as what we experienced between 2 to 3, or even 3 to 4. The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You generally can't just push an innovation infinitely, it will have a drop-off point somewhere. the "Large" part of LLMs is probably done. We've gotten as far as we can with those style of models, and the next innovation will be in smaller, more targeted models. > As costs have skyrocketed while benefits have leveled off, the economics of scale have turned against ever-larger models. Progress will instead come from improving model architectures, enhancing data efficiency, and advancing algorithmic techniques beyond copy-paste scale. The era of unlimited data, computing and model size that remade AI over the past decade is finally drawing to a close. [0] > Altman, who was interviewed over Zoom at the Imagination in Action event at MIT yesterday, believes we are approaching the limits of LLM size for size’s sake. “I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said. [1] [0] https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai-models-is-ending-a-gpu-crisis-could-be-one-reason-why/ https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai... [1] https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/?guccounter=1 https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...
  - famouswaffles 3y ago
    >The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You don't know that. This is literally just an assertion. An unfounded one at that. If you couldn't predict how far in 2017 the LLM concept would take us today, then you definitely have no idea how far it could actually go. >believes we are approaching the limits of LLM size for size’s sake Nothing to do with thinking they wouldn't improve from scale. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop... An interview from Altman later clarifying. "6. The scaling laws still hold Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant. OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude" Yes there are economic compute walls. But that's the kind of problem you want, not "innovation".
Karrot_Kream 3y ago
> And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. I think the stronger argument here won't necessarily be Moore's Law related but a change in architecture. Things like Apple's Neural Engine, Google's TPMs, or Geohot's Tinybox. In Intel's Tick-Tock model, this is the Tock for the previous Tick of larger datasets so to speak. (Note: I don't necessarily agree, just trying to make a stronger argument than just invoking Moore's Law.)

famouswaffles 3y ago

>And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. This is a lot of unfounded assumptions. You don't need Moore's Law. GPU's are not really made with ML training in mind. You don't need exponential growth for anything. The money Open ai spent on GPT-4 a year ago could train a model twice as large today. and that amount is a drop in the bucket for the R&D of large corporations. Microsoft gave open ai 10B. amazon gave anthropic 4B >So compute will not fall at the rate you would need it to for LLMs to actually compete in any meaningful way with human software engineers. I don't think the compute reuired is anywhere near as much as you think it is. https://arxiv.org/abs/2309.12499 https://arxiv.org/abs/2309.12499 >We are not guaranteed to continue to progress in anything just because we have in the past. Nothing is guaranteed. But the scaling plots show no indication of a slow down so it's up to you to provide a concrete reason this object in motion is going to stop immediately and conveniently right now. If all you have is "well it just can't keep getting better right" then visit the 2 and 3 threads to see how meaningless such unfounded assertions are.

WendyTheWillow 3y ago
I think you fundamentally don't understand the nature of exponential growth, and the power of diminishing returns. Even if you double the GPU capacity over the next year, you won't even remotely begin to come close enough to producing a step-level growth of capability such as what we experienced between 2 to 3, or even 3 to 4. The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You generally can't just push an innovation infinitely, it will have a drop-off point somewhere. the "Large" part of LLMs is probably done. We've gotten as far as we can with those style of models, and the next innovation will be in smaller, more targeted models. > As costs have skyrocketed while benefits have leveled off, the economics of scale have turned against ever-larger models. Progress will instead come from improving model architectures, enhancing data efficiency, and advancing algorithmic techniques beyond copy-paste scale. The era of unlimited data, computing and model size that remade AI over the past decade is finally drawing to a close. [0] > Altman, who was interviewed over Zoom at the Imagination in Action event at MIT yesterday, believes we are approaching the limits of LLM size for size’s sake. “I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said. [1] [0] https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai-models-is-ending-a-gpu-crisis-could-be-one-reason-why/ https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai... [1] https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/?guccounter=1 https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...
- famouswaffles 3y ago
  >The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You don't know that. This is literally just an assertion. An unfounded one at that. If you couldn't predict how far in 2017 the LLM concept would take us today, then you definitely have no idea how far it could actually go. >believes we are approaching the limits of LLM size for size’s sake Nothing to do with thinking they wouldn't improve from scale. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop... An interview from Altman later clarifying. "6. The scaling laws still hold Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant. OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude" Yes there are economic compute walls. But that's the kind of problem you want, not "innovation".

Karrot_Kream 3y ago

> And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. I think the stronger argument here won't necessarily be Moore's Law related but a change in architecture. Things like Apple's Neural Engine, Google's TPMs, or Geohot's Tinybox. In Intel's Tick-Tock model, this is the Tock for the previous Tick of larger datasets so to speak. (Note: I don't necessarily agree, just trying to make a stronger argument than just invoking Moore's Law.)