6 ms·

AI models that cost $1B to train are underway, $100B models coming

34 points by pulse7 2y ago

DrMiaow 2y ago
elmo-arms-up-world-burning.gif
- dialup_sounds 2y ago
  For the meme-impaired: https://knowyourmeme.com/memes/elmo-rise https://knowyourmeme.com/memes/elmo-rise
- bitwize 2y ago
  Somewhere, a Tamarian is posting shaka-when-the-walls-fell.jpg
  - hi_dang_ 2y ago
    It took an entire thread of nostradamus-tier bullshit before we finally got the first serious response. Bravo.
blowski 2y ago
What will the benefit be of more expensive models? More facts, because it's consumed more information? More ability to, say, adjust writing style? Or is this all necessary just to filter out the garbage recycled AI content it's now consuming?
- jiggawatts 2y ago
  Right around the time GPT-4 was first announced, OpenAI published a paper that basically said that training can "just keep going" with no obvious end in sight. Recently, Meta tried to train a model 75x as long as is naively optimal, and it just kept getting better. Better in this case means some combination of "less errors for the same size" and/or "bigger and smarter". Fundamentally, they're still the same thing, just more and better. Unfortunately, the scaling is (roughly) logarithmic. So for every 10x increase in scale you get a +1 better model. Scaling up 1,000x gets you just a +3 improvement, and so on.
  - iamleppert 2y ago
    And what, exactly is the ROI on "better"? Who cares if the model is better, is it 100B better? Who are going to buy these services, what consumer will pay for it?
    - jiggawatts 2y ago
      Meta has something like half a million modern GPUs that they’ve purchased outright. They can afford to keep training models “forever”. This is useful to eke out every last drop of quality per gigabyte of model file size. It also keeps the models up to date with current events. Obviously this scaling becomes too inefficient at infinite scale not just because of training costs that’ll never be recouped but also increasing inference cost with larger models. Some fundamentally new architectures will need to be developed to take much better advantage of increased computer power. I suspect the major players are investing in hardware now in the hope that some revolutionary new algorithm is invented soon and they’ll be ready for it. It’s… a bit of a gamble!
- lm28469 2y ago
  > What will the benefit be of more expensive models? Bleed investors dry before the next fad pops up
  - blitzar 2y ago
    > What will the benefit be of more expensive models? A G650 to fly to your 85m yacht in the med doesnt come cheap.
fuzzfactor 2y ago
If you had an extra $100 Billion, some people could think of something better to spend it on, some not.
- user90131313 2y ago
  Metaverse! oh wait that is too old and forgotten already.
  - xinayder 2y ago
    web3 metaverse powered by genAI and NFTs?
    - bratwurst3000 2y ago
      And we put that in the cloud
danpalmer 2y ago
No company can afford to spend $100B on something that will be obsolete a year later, you just can't recover the investment from sales that quickly. $100m is manageable, if you've got 100m paying subscribers or companies using your API for a year you can recoup the costs, but there aren't many companies with 100m users to monetise for it. $1B feels like it's pushing it, only a few companies in the world can monetise, and realistically it's about lasting through the next round to be able to continue competing, not about making the money back. $100B though, that's a whole different game again. That's like asking for the biggest private investment ever made, for capex that depreciates at $50B a year. You'd have to be stupid to do it. The public markets wouldn't take it. Investing that much in hardware that depreciates over 5+ years and is theoretically still usable at the end, maybe, but even then the biggest companies in the world are still spending an order of magnitude less per year, so the numbers end up working out very differently. Plus that's companies with 1B users ready to monetise.
- ca_tech 2y ago
  I agree and do not think any company would make that investment directly. Nvidia selling to Microsoft renting to OpenAI, I'm sure you could make that add up to $100B on paper. In the long run the economics are likely much more complicated and consist of "agreements worth $x".
- bcherny 2y ago
  That’s true for AI, but it is not the right way to think about AGI. For AGI, the bet is that someone will build an AI capable enough to automate AI development. Once we get there it will pay for itself. The question is what the cost-speed tradeoff to get there looks like.
  - awakeasleep 2y ago
    for 100B they would probably want a realistic description of how they get to AGI. Thats a bit too much money for the handwavy answers we have right now for the path between LLMs and AGI (which doesn't even have a great definition)
  - 2y ago
htrp 2y ago
X to Doubt. This is the Anthropic CEO talking up his company's capital needs to the Norwegian Sovereign Wealth Fund ( Norges Bank Investment Management ) and trying to justify some absurd 100bn valuation.
- belter 2y ago
  Yes. The release of GPT-5 will make or break the AI movement. If the capabilities are not another quantum leap, it will become clear the scaling laws are not all. These investments will be unsustainable on the basis of any economic metrics you use.
  - MuffinFlavored 2y ago
    > If the capabilities are not another quantum leap While I don't disagree 100%, my question to you is: who/what says this is the case/why? GPT-3.5 was released/made popular "to the masses" not too long ago. Where do you feel the pressure for a quantum leap "quickly" is coming from?
    - wormlord 2y ago
      To steal from another comment in the thread: > That’s true for AI, but it is not the right way to think about AGI. For AGI, the bet is that someone will build an AI capable enough to automate AI development. Once we get there it will pay for itself. The question is what the cost-speed tradeoff to get there looks like. I don't think people are treating AI as a typical investment. They are valuing it as a potential replacement for like 95% of human workers. Once the plateau becomes obvious to even the biggest fanatics, people are going to realize all this money has been used to create really good chatbots that just make shit up 25% of the time. The whole sales pitch for the last 1-2 years has been that AGI is "just around the corner" and we can get there via the magic of exponential growth.
      - pulse7 2y ago
        "AGI is just around the corner" but we haven't be able to build "fully automatic car driving" for a decade... So first I would like to see car drivers replaced by non-general AI, then I will start believing in AGI...
demondemidi 2y ago
Wow, this CEO entitlement and wealth pissing contests are laughable.
- ai4ever 2y ago
  altman, and amodei are speaking their book, but in doing so seem like shady snake-oil salesmen. they would be better off not bullshitting their investors.
  - blitzar 2y ago
    the people not bullshitting their investors have no investors investors with huge piles of cash should buy themselves a brain and stop funding bullshitters
seydor 2y ago
BigTech wants all your sovereign money
mensetmanusman 2y ago
If only this had came before crypto. We could have had a system that underwrites international finance and pays for training on the cheap. I wonder which timelines had this scenario…
- seydor 2y ago
  That sounds like a great idea for our next bubble
Temporary_31337 2y ago
All this burn and recruiters and bots still match on keywords in CV.
hurrdurr57 2y ago
Well, I guess the question I have is, what exactly does he mean by the "cost to train"? As in, just the cost of the electricity used to train that one model? That seems really excessive. Or is it the total overall cost of buying TPUs / GPUs, developing infrastructure, constructing data centers, putting together quality data sets, doing R&D, paying salaries, etc. as well as training the model itself? I could see that overall investment into AI scaling into the tens of billions over the next few years.
speedylight 2y ago
I could see the US subsidizing most of that $100B, just because they can, and more importantly, it would be the kind of tactical advantage that’s needed to make sure US tech companies stay relevant in a world where there’s a growing desire to breakaway from them in-favor of homegrown solutions.

34 points by pulse7 2y ago

DrMiaow 2y ago

elmo-arms-up-world-burning.gif

dialup_sounds 2y ago
For the meme-impaired: https://knowyourmeme.com/memes/elmo-rise https://knowyourmeme.com/memes/elmo-rise
bitwize 2y ago
Somewhere, a Tamarian is posting shaka-when-the-walls-fell.jpg
- hi_dang_ 2y ago
  It took an entire thread of nostradamus-tier bullshit before we finally got the first serious response. Bravo.

blowski 2y ago

What will the benefit be of more expensive models? More facts, because it's consumed more information? More ability to, say, adjust writing style? Or is this all necessary just to filter out the garbage recycled AI content it's now consuming?

jiggawatts 2y ago
Right around the time GPT-4 was first announced, OpenAI published a paper that basically said that training can "just keep going" with no obvious end in sight. Recently, Meta tried to train a model 75x as long as is naively optimal, and it just kept getting better. Better in this case means some combination of "less errors for the same size" and/or "bigger and smarter". Fundamentally, they're still the same thing, just more and better. Unfortunately, the scaling is (roughly) logarithmic. So for every 10x increase in scale you get a +1 better model. Scaling up 1,000x gets you just a +3 improvement, and so on.
- iamleppert 2y ago
  And what, exactly is the ROI on "better"? Who cares if the model is better, is it 100B better? Who are going to buy these services, what consumer will pay for it?
  - jiggawatts 2y ago
    Meta has something like half a million modern GPUs that they’ve purchased outright. They can afford to keep training models “forever”. This is useful to eke out every last drop of quality per gigabyte of model file size. It also keeps the models up to date with current events. Obviously this scaling becomes too inefficient at infinite scale not just because of training costs that’ll never be recouped but also increasing inference cost with larger models. Some fundamentally new architectures will need to be developed to take much better advantage of increased computer power. I suspect the major players are investing in hardware now in the hope that some revolutionary new algorithm is invented soon and they’ll be ready for it. It’s… a bit of a gamble!
lm28469 2y ago
> What will the benefit be of more expensive models? Bleed investors dry before the next fad pops up
- blitzar 2y ago
  > What will the benefit be of more expensive models? A G650 to fly to your 85m yacht in the med doesnt come cheap.

fuzzfactor 2y ago

If you had an extra $100 Billion, some people could think of something better to spend it on, some not.

user90131313 2y ago
Metaverse! oh wait that is too old and forgotten already.
- xinayder 2y ago
  web3 metaverse powered by genAI and NFTs?
  - bratwurst3000 2y ago
    And we put that in the cloud

danpalmer 2y ago

No company can afford to spend $100B on something that will be obsolete a year later, you just can't recover the investment from sales that quickly. $100m is manageable, if you've got 100m paying subscribers or companies using your API for a year you can recoup the costs, but there aren't many companies with 100m users to monetise for it. $1B feels like it's pushing it, only a few companies in the world can monetise, and realistically it's about lasting through the next round to be able to continue competing, not about making the money back. $100B though, that's a whole different game again. That's like asking for the biggest private investment ever made, for capex that depreciates at $50B a year. You'd have to be stupid to do it. The public markets wouldn't take it. Investing that much in hardware that depreciates over 5+ years and is theoretically still usable at the end, maybe, but even then the biggest companies in the world are still spending an order of magnitude less per year, so the numbers end up working out very differently. Plus that's companies with 1B users ready to monetise.

ca_tech 2y ago
I agree and do not think any company would make that investment directly. Nvidia selling to Microsoft renting to OpenAI, I'm sure you could make that add up to $100B on paper. In the long run the economics are likely much more complicated and consist of "agreements worth $x".
bcherny 2y ago
That’s true for AI, but it is not the right way to think about AGI. For AGI, the bet is that someone will build an AI capable enough to automate AI development. Once we get there it will pay for itself. The question is what the cost-speed tradeoff to get there looks like.
- awakeasleep 2y ago
  for 100B they would probably want a realistic description of how they get to AGI. Thats a bit too much money for the handwavy answers we have right now for the path between LLMs and AGI (which doesn't even have a great definition)
- 2y ago

htrp 2y ago

X to Doubt. This is the Anthropic CEO talking up his company's capital needs to the Norwegian Sovereign Wealth Fund ( Norges Bank Investment Management ) and trying to justify some absurd 100bn valuation.

belter 2y ago
Yes. The release of GPT-5 will make or break the AI movement. If the capabilities are not another quantum leap, it will become clear the scaling laws are not all. These investments will be unsustainable on the basis of any economic metrics you use.
- MuffinFlavored 2y ago
  > If the capabilities are not another quantum leap While I don't disagree 100%, my question to you is: who/what says this is the case/why? GPT-3.5 was released/made popular "to the masses" not too long ago. Where do you feel the pressure for a quantum leap "quickly" is coming from?
  - wormlord 2y ago
    To steal from another comment in the thread: > That’s true for AI, but it is not the right way to think about AGI. For AGI, the bet is that someone will build an AI capable enough to automate AI development. Once we get there it will pay for itself. The question is what the cost-speed tradeoff to get there looks like. I don't think people are treating AI as a typical investment. They are valuing it as a potential replacement for like 95% of human workers. Once the plateau becomes obvious to even the biggest fanatics, people are going to realize all this money has been used to create really good chatbots that just make shit up 25% of the time. The whole sales pitch for the last 1-2 years has been that AGI is "just around the corner" and we can get there via the magic of exponential growth.
    - pulse7 2y ago
      "AGI is just around the corner" but we haven't be able to build "fully automatic car driving" for a decade... So first I would like to see car drivers replaced by non-general AI, then I will start believing in AGI...

demondemidi 2y ago

Wow, this CEO entitlement and wealth pissing contests are laughable.

ai4ever 2y ago
altman, and amodei are speaking their book, but in doing so seem like shady snake-oil salesmen. they would be better off not bullshitting their investors.
- blitzar 2y ago
  the people not bullshitting their investors have no investors investors with huge piles of cash should buy themselves a brain and stop funding bullshitters

seydor 2y ago

BigTech wants all your sovereign money

mensetmanusman 2y ago

If only this had came before crypto. We could have had a system that underwrites international finance and pays for training on the cheap. I wonder which timelines had this scenario…

seydor 2y ago
That sounds like a great idea for our next bubble

Temporary_31337 2y ago

All this burn and recruiters and bots still match on keywords in CV.

hurrdurr57 2y ago

Well, I guess the question I have is, what exactly does he mean by the "cost to train"? As in, just the cost of the electricity used to train that one model? That seems really excessive. Or is it the total overall cost of buying TPUs / GPUs, developing infrastructure, constructing data centers, putting together quality data sets, doing R&D, paying salaries, etc. as well as training the model itself? I could see that overall investment into AI scaling into the tens of billions over the next few years.

speedylight 2y ago

I could see the US subsidizing most of that $100B, just because they can, and more importantly, it would be the kind of tactical advantage that’s needed to make sure US tech companies stay relevant in a world where there’s a growing desire to breakaway from them in-favor of homegrown solutions.