5 ms·

$730B pre-money for a company where each model is roughly 2x profitable on its own, but each next model costs 10x the last. The whole thing only works if scalin

$730B pre-money for a company where each model is roughly 2x profitable on its own, but each next model costs 10x the last. The whole thing only works if scaling keeps delivering. Research (Sara Hooker et al.) is not encouraging on that front, compact models already outperform massive predecessors on downstream tasks while scaling laws only predict pre-training loss reliably. Wrote about both the per-model math and the scaling question: (1) https://philippdubach.com/posts/ai-models-as-standalone-pls/ https://philippdubach.com/posts/ai-models-as-standalone-pls/ (2) https://philippdubach.com/posts/the-most-expensive-assumption-in-ai/ https://philippdubach.com/posts/the-most-expensive-assumptio... EDIT: Removed the dot after et; bc apparently it's an entire word (the more you know..)

onlyrealcuzzo 8mo ago
> each model is roughly 2x profitable on its own, but each next model costs 10x the last. The whole thing only works if scaling keeps delivering. This is a decent argument, but it's not the death knell you think. Models are getting 99% more efficient every 3 years - to get the same amount of output, combined with hardware and (mostly) software upgrades - you can use 99% less power. The number of applications where AI is already "good enough" keeps growing every day. If the cost goes down 99% every three years, it doesn't take long until you can make a ton of money on those applications. If AI stopped progressing today, it would take probably a decade or longer for us to take full advantage of it. So there is tons of forward looking revenue that isn't counted yet. For the foreseeable future, there are MANY MANY uses of models where a company would not want to host its own models and would be GLAD to pay an 4-5x cost for someone else to host the model and hardware for them. I'm as bullish on OpenAI being "worth" $730B as I was on Snap being worth what it IPO'd for - which it's still down about 80% (AFTER inflation, or about ~95% adjusting for gold inflation). But guess what - these are MINIMUM valuations based on 50-80% margins - i.e. they're really getting about ~$30B - the rest is market value of hardware and hosting. OpenAI could be worth 80% less, and they could still make a metric fuck-ton of money selling at IPO with a $1T+ market cap to speculative morons easily... Realistically, very rich people with high risk tolerance are saying that they think OpenAI has a MINIMUM value of ~$100B. That seems very reasonable given the risk tolerance and wealth.
- kortilla 8mo ago
  > Models are getting 99% more efficient every 3 years The LLM industry has only be around for like 4 years. Extrapolating trends from that is pretty naive.
- dfp33 8mo ago
  "If AI stopped progressing today, it would take probably a decade or longer for us to take full advantage of it." AI stopped progressing, or LLMs? I really dislike people throwing the term AI around.
  - grosswait 8mo ago
    For the purposes of their argument, I don’t think the distinction matters.
- blmarket 8mo ago
  > 99% more efficient every 3 years It's 2x efficiency. Then I'd take 50% less power instead of ridiculous 99% less power.
  - sigmoid10 8mo ago
    GPT-4 came out 3 years ago and you can run comparable models for 1% of the cost nowadays. That is not 2x efficiency. That's two orders of magnitude in end-to-end compute efficiency.
    - swingboy 8mo ago
      How do we know how much it costs? Or is this just based off the token pricing?
      - Xunjin 8mo ago
        That's the bingo of the question... The entire argument is token pricing, which can be subsidized.
    - 8mo ago
ohyoutravel 8mo ago
Et is an entire word and doesn’t need a period at the end.
nazgul17 8mo ago
What makes you think this trend will continue? In a situation with finite resources (eg the number of parameters), the default is to assume things will plateau.
rishabhaiover 8mo ago
I was reading a paper on dark silicon and how it broke the beautiful scaling laws of the past (Moore's law/Dennard Scaling). We hit a wall, innovated and at the moment, the hardware industry is thriving. To me, that means scaling the industry and riding that momentum wasn't wrong. In fact, it allowed us to be where we are today. Why are we so against, in principle, to the current pre-training scaling laws? Perhaps, we'll require new innovations at some point, but the momentum allows us to reach to newer heights that we've never climbed before.
credit_guy 8mo ago
> each next model costs 10x the last Yes, but there's a chance that actually training is done more or less for free by companies like OpenAI. The reason being that they do a gigantic amount of inference for end users (for which they get paid), but their servers can't be constantly utilized at 100% by inference. So, if they know how to schedule things correctly (and they probably do), they can do the training of their new model on the unutilized compute capacity. If you or I were to pay for that training, it would be billions of dollars, but for them it is just using compute that otherwise would be idle.
simonebrunozzi 8mo ago
> EDIT: Removed the dot after et; bc apparently it's an entire word (the more you know..) From latin "et alia", abbreviated as "et al." - it's not a single word but an expression.

onlyrealcuzzo 8mo ago

> each model is roughly 2x profitable on its own, but each next model costs 10x the last. The whole thing only works if scaling keeps delivering. This is a decent argument, but it's not the death knell you think. Models are getting 99% more efficient every 3 years - to get the same amount of output, combined with hardware and (mostly) software upgrades - you can use 99% less power. The number of applications where AI is already "good enough" keeps growing every day. If the cost goes down 99% every three years, it doesn't take long until you can make a ton of money on those applications. If AI stopped progressing today, it would take probably a decade or longer for us to take full advantage of it. So there is tons of forward looking revenue that isn't counted yet. For the foreseeable future, there are MANY MANY uses of models where a company would not want to host its own models and would be GLAD to pay an 4-5x cost for someone else to host the model and hardware for them. I'm as bullish on OpenAI being "worth" $730B as I was on Snap being worth what it IPO'd for - which it's still down about 80% (AFTER inflation, or about ~95% adjusting for gold inflation). But guess what - these are MINIMUM valuations based on 50-80% margins - i.e. they're really getting about ~$30B - the rest is market value of hardware and hosting. OpenAI could be worth 80% less, and they could still make a metric fuck-ton of money selling at IPO with a $1T+ market cap to speculative morons easily... Realistically, very rich people with high risk tolerance are saying that they think OpenAI has a MINIMUM value of ~$100B. That seems very reasonable given the risk tolerance and wealth.

kortilla 8mo ago
> Models are getting 99% more efficient every 3 years The LLM industry has only be around for like 4 years. Extrapolating trends from that is pretty naive.
dfp33 8mo ago
"If AI stopped progressing today, it would take probably a decade or longer for us to take full advantage of it." AI stopped progressing, or LLMs? I really dislike people throwing the term AI around.
- grosswait 8mo ago
  For the purposes of their argument, I don’t think the distinction matters.
blmarket 8mo ago
> 99% more efficient every 3 years It's 2x efficiency. Then I'd take 50% less power instead of ridiculous 99% less power.
- sigmoid10 8mo ago
  GPT-4 came out 3 years ago and you can run comparable models for 1% of the cost nowadays. That is not 2x efficiency. That's two orders of magnitude in end-to-end compute efficiency.
  - swingboy 8mo ago
    How do we know how much it costs? Or is this just based off the token pricing?
    - Xunjin 8mo ago
      That's the bingo of the question... The entire argument is token pricing, which can be subsidized.
  - 8mo ago

ohyoutravel 8mo ago

Et is an entire word and doesn’t need a period at the end.

nazgul17 8mo ago

What makes you think this trend will continue? In a situation with finite resources (eg the number of parameters), the default is to assume things will plateau.

rishabhaiover 8mo ago

I was reading a paper on dark silicon and how it broke the beautiful scaling laws of the past (Moore's law/Dennard Scaling). We hit a wall, innovated and at the moment, the hardware industry is thriving. To me, that means scaling the industry and riding that momentum wasn't wrong. In fact, it allowed us to be where we are today. Why are we so against, in principle, to the current pre-training scaling laws? Perhaps, we'll require new innovations at some point, but the momentum allows us to reach to newer heights that we've never climbed before.

credit_guy 8mo ago

> each next model costs 10x the last Yes, but there's a chance that actually training is done more or less for free by companies like OpenAI. The reason being that they do a gigantic amount of inference for end users (for which they get paid), but their servers can't be constantly utilized at 100% by inference. So, if they know how to schedule things correctly (and they probably do), they can do the training of their new model on the unutilized compute capacity. If you or I were to pay for that training, it would be billions of dollars, but for them it is just using compute that otherwise would be idle.

simonebrunozzi 8mo ago

> EDIT: Removed the dot after et; bc apparently it's an entire word (the more you know..) From latin "et alia", abbreviated as "et al." - it's not a single word but an expression.