3 ms·

Law of diminishing returns. The low-hanging fruit has been picked. > But the company’s CEO, Sam Altman, says further progress will not come from making models

Law of diminishing returns. The low-hanging fruit has been picked. > But the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...

moffkalast 3y ago
Given how they've struggled to even make even GPT 4 economical it's highly unlikely that larger models would be in any way cost effective for wide use. And there's certainly more to be found in training longer on better datasets and adjusting the architecture.
- int_19h 3y ago
  It's important to remember that "not cost-effective" is not the same as "not useful", though.
  - moffkalast 3y ago
    Depends on how you look at it I guess. If it's really too expensive and overly slow to run for end users but is able to say, generate perfect synthetic datasets over time which can then be used to train smaller and faster models which pay back for that original cost then it is cost effective, just in a different application.
    - int_19h 3y ago
      What I mean, rather, is that "cost-effective" implies that everything can be reduced down to some costing. So e.g. having humans do the mind-numbing work is often more "cost-effective" than robots doing the same thing, because humans can be hired for less. But from a broader perspective, a robot replacing a human in such a role is socially beneficial even if the robot is more expensive - and the ideal state of affairs is when all things that people would rather not do are fully automated away. LLMs are one piece of that puzzle, but we can only put it together if we stop thinking about everything in terms of money saved.
      - moffkalast 3y ago
        In capitalism everything can and is reduced down to some costing indeed. We don't have public schools and art classes because it's nice, it's because an educated society as a whole has a much higher economic output and you need someone to design good looking products which will sell better. I wouldn't say I agree with the process, but you can't ignore that it's the underlying reason for almost everything our society does. On topic of LLMs and diffusion models I doubt they're really having that much of a positive social impact, I would bet that in the long run they'll automate more of the jobs humans would like to do compared to the ones we don't, and erode trust on a global level. But the economic gain is undeniable so we're doing it anyway. Automating all work would be a good thing in an ideal world, but in practice it's the one thing that still gives the lower classes some leverage over the capitalists and a world where the average person is expendable and unemployable isn't good for society.
        int_19h 3y ago
        I feel like this is a self-defeating approach. We don't need an ideal world for that - you rightly identified capitalism as the problem, and it's not utopian to think of other options. Nor do many of them require some sort of ideal man to work, as their opponents sometimes present. Change will happen - necessarily so, in fact, as more automation is brought online, because the crowd of people excluded from the economy as "redundant" will not be happy about that state of affairs, and it will only grow in size. At some point, they will understand that the real problem is not that they don't have work, but that they don't have a fair share of the automation pie. And things like, say, property rights on means of production / automation stop to matter if most people don't believe in them anymore.
stavros 3y ago
I don't think that's what that means. I think he means "we don't have to keep making models larger, because we've found ways to make very good models that are also small".

moffkalast 3y ago

Given how they've struggled to even make even GPT 4 economical it's highly unlikely that larger models would be in any way cost effective for wide use. And there's certainly more to be found in training longer on better datasets and adjusting the architecture.

int_19h 3y ago
It's important to remember that "not cost-effective" is not the same as "not useful", though.
- moffkalast 3y ago
  Depends on how you look at it I guess. If it's really too expensive and overly slow to run for end users but is able to say, generate perfect synthetic datasets over time which can then be used to train smaller and faster models which pay back for that original cost then it is cost effective, just in a different application.
  - int_19h 3y ago
    What I mean, rather, is that "cost-effective" implies that everything can be reduced down to some costing. So e.g. having humans do the mind-numbing work is often more "cost-effective" than robots doing the same thing, because humans can be hired for less. But from a broader perspective, a robot replacing a human in such a role is socially beneficial even if the robot is more expensive - and the ideal state of affairs is when all things that people would rather not do are fully automated away. LLMs are one piece of that puzzle, but we can only put it together if we stop thinking about everything in terms of money saved.
    - moffkalast 3y ago
      In capitalism everything can and is reduced down to some costing indeed. We don't have public schools and art classes because it's nice, it's because an educated society as a whole has a much higher economic output and you need someone to design good looking products which will sell better. I wouldn't say I agree with the process, but you can't ignore that it's the underlying reason for almost everything our society does. On topic of LLMs and diffusion models I doubt they're really having that much of a positive social impact, I would bet that in the long run they'll automate more of the jobs humans would like to do compared to the ones we don't, and erode trust on a global level. But the economic gain is undeniable so we're doing it anyway. Automating all work would be a good thing in an ideal world, but in practice it's the one thing that still gives the lower classes some leverage over the capitalists and a world where the average person is expendable and unemployable isn't good for society.
      - int_19h 3y ago
        I feel like this is a self-defeating approach. We don't need an ideal world for that - you rightly identified capitalism as the problem, and it's not utopian to think of other options. Nor do many of them require some sort of ideal man to work, as their opponents sometimes present. Change will happen - necessarily so, in fact, as more automation is brought online, because the crowd of people excluded from the economy as "redundant" will not be happy about that state of affairs, and it will only grow in size. At some point, they will understand that the real problem is not that they don't have work, but that they don't have a fair share of the automation pie. And things like, say, property rights on means of production / automation stop to matter if most people don't believe in them anymore.

stavros 3y ago

I don't think that's what that means. I think he means "we don't have to keep making models larger, because we've found ways to make very good models that are also small".