4 ms·

I think you fundamentally don't understand the nature of exponential growth, and the power of diminishing returns. Even if you double the GPU capacity over the

I think you fundamentally don't understand the nature of exponential growth, and the power of diminishing returns. Even if you double the GPU capacity over the next year, you won't even remotely begin to come close enough to producing a step-level growth of capability such as what we experienced between 2 to 3, or even 3 to 4. The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You generally can't just push an innovation infinitely, it will have a drop-off point somewhere. the "Large" part of LLMs is probably done. We've gotten as far as we can with those style of models, and the next innovation will be in smaller, more targeted models. > As costs have skyrocketed while benefits have leveled off, the economics of scale have turned against ever-larger models. Progress will instead come from improving model architectures, enhancing data efficiency, and advancing algorithmic techniques beyond copy-paste scale. The era of unlimited data, computing and model size that remade AI over the past decade is finally drawing to a close. [0] > Altman, who was interviewed over Zoom at the Imagination in Action event at MIT yesterday, believes we are approaching the limits of LLM size for size’s sake. “I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said. [1] [0] https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai-models-is-ending-a-gpu-crisis-could-be-one-reason-why/ https://venturebeat.com/ai/openai-chief-says-age-of-giant-ai... [1] https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/?guccounter=1 https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...

famouswaffles 3y ago
>The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You don't know that. This is literally just an assertion. An unfounded one at that. If you couldn't predict how far in 2017 the LLM concept would take us today, then you definitely have no idea how far it could actually go. >believes we are approaching the limits of LLM size for size’s sake Nothing to do with thinking they wouldn't improve from scale. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop... An interview from Altman later clarifying. "6. The scaling laws still hold Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant. OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude" Yes there are economic compute walls. But that's the kind of problem you want, not "innovation".
- WendyTheWillow 3y ago
  Er, that's not how arguments work. What we can't know is that those trends will continue, so it's on you to demonstrate that they will, despite evidence suggesting they won't. As for as what you linked, Altman is saying the same thing I'm saying: > That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude. This is exactly my point; doubling or tripling of the size will be possible, but it won't result in a doubling of performance. We won't see a GPT 5 that's twice as good as GPT 4, for example. The jump from 2 to 3 was exponential. The jump from 3 to 4 was also exponential, though not as much. The jump from 4 to 5 will follow that curve, according to Altman, which means exactly what he said in my quote; the value will continue to decrease. For a 2 to 3 type jump, GPU technology would have to completely transform in capability, which there are no indications that we've found that innovation.
  - famouswaffles 3y ago
    My argument was that improvement from scale would continue. There is absolutely evidence suggesting this. Gpt-4 can perform nearly all tasks you throw at it with well above average human performance. There literally isn't any testable definition of intelligence it fails that a big chunks of humans wouldn't also fail. You seem to keep missing the fact that We do not need an exponential improvement from 4.
    - WendyTheWillow 3y ago
      > Gpt-4 can perform nearly all tasks you throw at it with well above average human performance. It can't even generate flashcards from a textbook chapter, because it can't load the entire chapter into memory. Heck, it doesn't even know what textbook I'm talking about; I have to provide the content! It fails constantly at real world coding problems, and often does so silently. If you tried to replace a software developer with GPT 4, you would be left with a gaping productivity hole where that developer you replaced once existed. The improvement GPT 5 would have to provide is multiple orders of magnitude in order for this to be a realistic proposition. I use it daily and know better than to trust its output.
famouswaffles 3y ago
Moreover you keep saying we can't scale infinitely. Sure...but nobody is saying we have to. 4 is not as scaled from 3 as 3 was from 2. Doesn't matter, still massive gap.
- WendyTheWillow 3y ago
  As I said already, the gap from 3 to 4 was substantially smaller than the gap between 2 to 3, and all indications are that the gap from 4 to 5 will also be further smaller than that.
  - deleted 3y ago
    [deleted]

famouswaffles 3y ago

>The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You don't know that. This is literally just an assertion. An unfounded one at that. If you couldn't predict how far in 2017 the LLM concept would take us today, then you definitely have no idea how far it could actually go. >believes we are approaching the limits of LLM size for size’s sake Nothing to do with thinking they wouldn't improve from scale. https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans https://web.archive.org/web/20230531203946/https://humanloop... An interview from Altman later clarifying. "6. The scaling laws still hold Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant. OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude" Yes there are economic compute walls. But that's the kind of problem you want, not "innovation".

WendyTheWillow 3y ago
Er, that's not how arguments work. What we can't know is that those trends will continue, so it's on you to demonstrate that they will, despite evidence suggesting they won't. As for as what you linked, Altman is saying the same thing I'm saying: > That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude. This is exactly my point; doubling or tripling of the size will be possible, but it won't result in a doubling of performance. We won't see a GPT 5 that's twice as good as GPT 4, for example. The jump from 2 to 3 was exponential. The jump from 3 to 4 was also exponential, though not as much. The jump from 4 to 5 will follow that curve, according to Altman, which means exactly what he said in my quote; the value will continue to decrease. For a 2 to 3 type jump, GPU technology would have to completely transform in capability, which there are no indications that we've found that innovation.
- famouswaffles 3y ago
  My argument was that improvement from scale would continue. There is absolutely evidence suggesting this. Gpt-4 can perform nearly all tasks you throw at it with well above average human performance. There literally isn't any testable definition of intelligence it fails that a big chunks of humans wouldn't also fail. You seem to keep missing the fact that We do not need an exponential improvement from 4.
  - WendyTheWillow 3y ago
    > Gpt-4 can perform nearly all tasks you throw at it with well above average human performance. It can't even generate flashcards from a textbook chapter, because it can't load the entire chapter into memory. Heck, it doesn't even know what textbook I'm talking about; I have to provide the content! It fails constantly at real world coding problems, and often does so silently. If you tried to replace a software developer with GPT 4, you would be left with a gaping productivity hole where that developer you replaced once existed. The improvement GPT 5 would have to provide is multiple orders of magnitude in order for this to be a realistic proposition. I use it daily and know better than to trust its output.

famouswaffles 3y ago

Moreover you keep saying we can't scale infinitely. Sure...but nobody is saying we have to. 4 is not as scaled from 3 as 3 was from 2. Doesn't matter, still massive gap.

WendyTheWillow 3y ago
As I said already, the gap from 3 to 4 was substantially smaller than the gap between 2 to 3, and all indications are that the gap from 4 to 5 will also be further smaller than that.
- deleted 3y ago
  [deleted]