4 ms·

Altman says: > The intelligence of an AI model roughly equals the log of the resources used to train and run it. Pretty sure how log curves work is you add ex

Altman says: > The intelligence of an AI model roughly equals the log of the resources used to train and run it. Pretty sure how log curves work is you add exponentially more inputs to get linear increases in outputs. That would mean it's going to get wildly more difficult and expensive to get each additional marginal gain, and that's what we're seeing.

AnotherGoodName 2y ago
Tie it with the second and third points though. Essentially the claim is that it gets exponentially cheaper at the same rate the logarithmic resource requirements go up. Which leads to point 3 - linear growth in intelligence is expected over the next few years and AGI is a certainty. I feel that's not happening at all. This costs 15x more to run compared to 4o and is marginally better. Perhaps the failing of this prediction is on the cost side but regardless it's a failing of the prediction for linear growth in intelligence. Would another 15x resources lead to another marginal gain? Repeat? At that rate of gain we could turn all the resources in the world to this and not hit anywhere near AGI.
- mullingitover 2y ago
  My take is that the second point is the models created by this crazy expensive training are just software, and software gets cheaper to operate by virtue of Moore's Law in an exponential fashion. Third point, I think, is that the intelligent work that the models can do will increase exponentially because they get cheaper and cheaper to operate as the get more and more capable. So I think GPT4.5 is in the short term more expensive (they have to pay their training bill) but eventually this being the new floor is just ratcheting toward the nerd singularity or whatever.
  - sifar 2y ago
    Moore's Law is a self-fulfilling prophecy derived from the value of computation. If a given computation does not provide enough value, it might not be invested with enough resources to benefit from Moore's Law. When scaling saturates, less computationally expensive models would benefit more.
- darkerside 2y ago
  Doesn't make sense. Requiring exponentially more resources is not surprising. Resources getting exponentially cheaper? Like, falling by more than half each cycle? That doesn't happen.
- danenania 2y ago
  > This costs 15x more to run compared to 4o and is marginally better. 4o has been very heavily optimized though. I'd say a more apples-to-apples comparison would be to original gpt-4, which was $30/$60 to 4.5's $75/$150. Still a significant difference, but not quite so stark.
- deepsquirrelnet 2y ago
  I felt like the point of llama 3 was to prove this out. 2T -> 15T tokens, 70B -> 405B. We now have llama 3.3 70B, which by most metrics outperforms the 405B model without further scaling, so it’s been my assumption that scaling is dead. Other innovations in training are taking the lead. Higher volumes of low quality data aren’t moving the needle.
nialv7 2y ago
This statement makes no sense because there is just no good measurement of intelligence for him to make a quantitative statement like that. What's he measuring intelligence with? MMLU? IQ? His gut feeling?
- dash2 2y ago
  There are very many tests, leaderboards etc. for AI models, and while none of them are perfect, in the aggregate they might not be a bad measure of intelligence (perhaps locally, in an interval around current models' performance level, since people develop new tests as old ones get saturated).
  - nialv7 2y ago
    Yes, using those methods you can have a blurry idea of maybe this model is "smarter" than the other one, etc. But none of them may be used to anchor the claim that "intelligence is proportional to the log of compute", which is the claim sama was making.
    - dash2 2y ago
      Suppose you take the first principal component of performance and fit it to log(compute). Seems like a fairly reasonable way to get an answer.
tomrod 2y ago
It's hard to tell the difference of a log and a sigmoid with an upper asymptote
- kenjackson 2y ago
  It's not hard to tell when you reach the upper asymptote though.
panarky 2y ago
Exponential increases in inputs yield linear increases in intelligence. Linear increases in intelligence yield exponential increases in economic gains. Therefore, exponential increases in inputs yield exponential increases in economic gains. Left as an exercise for the reader is to determine who will capture most of those economic gains.
- jcgrillo 2y ago
  So where are the economic gains? So far it's just been losses.

AnotherGoodName 2y ago

Tie it with the second and third points though. Essentially the claim is that it gets exponentially cheaper at the same rate the logarithmic resource requirements go up. Which leads to point 3 - linear growth in intelligence is expected over the next few years and AGI is a certainty. I feel that's not happening at all. This costs 15x more to run compared to 4o and is marginally better. Perhaps the failing of this prediction is on the cost side but regardless it's a failing of the prediction for linear growth in intelligence. Would another 15x resources lead to another marginal gain? Repeat? At that rate of gain we could turn all the resources in the world to this and not hit anywhere near AGI.

mullingitover 2y ago
My take is that the second point is the models created by this crazy expensive training are just software, and software gets cheaper to operate by virtue of Moore's Law in an exponential fashion. Third point, I think, is that the intelligent work that the models can do will increase exponentially because they get cheaper and cheaper to operate as the get more and more capable. So I think GPT4.5 is in the short term more expensive (they have to pay their training bill) but eventually this being the new floor is just ratcheting toward the nerd singularity or whatever.
- sifar 2y ago
  Moore's Law is a self-fulfilling prophecy derived from the value of computation. If a given computation does not provide enough value, it might not be invested with enough resources to benefit from Moore's Law. When scaling saturates, less computationally expensive models would benefit more.
darkerside 2y ago
Doesn't make sense. Requiring exponentially more resources is not surprising. Resources getting exponentially cheaper? Like, falling by more than half each cycle? That doesn't happen.
danenania 2y ago
> This costs 15x more to run compared to 4o and is marginally better. 4o has been very heavily optimized though. I'd say a more apples-to-apples comparison would be to original gpt-4, which was $30/$60 to 4.5's $75/$150. Still a significant difference, but not quite so stark.
deepsquirrelnet 2y ago
I felt like the point of llama 3 was to prove this out. 2T -> 15T tokens, 70B -> 405B. We now have llama 3.3 70B, which by most metrics outperforms the 405B model without further scaling, so it’s been my assumption that scaling is dead. Other innovations in training are taking the lead. Higher volumes of low quality data aren’t moving the needle.

nialv7 2y ago

This statement makes no sense because there is just no good measurement of intelligence for him to make a quantitative statement like that. What's he measuring intelligence with? MMLU? IQ? His gut feeling?

dash2 2y ago
There are very many tests, leaderboards etc. for AI models, and while none of them are perfect, in the aggregate they might not be a bad measure of intelligence (perhaps locally, in an interval around current models' performance level, since people develop new tests as old ones get saturated).
- nialv7 2y ago
  Yes, using those methods you can have a blurry idea of maybe this model is "smarter" than the other one, etc. But none of them may be used to anchor the claim that "intelligence is proportional to the log of compute", which is the claim sama was making.
  - dash2 2y ago
    Suppose you take the first principal component of performance and fit it to log(compute). Seems like a fairly reasonable way to get an answer.

tomrod 2y ago

It's hard to tell the difference of a log and a sigmoid with an upper asymptote

kenjackson 2y ago
It's not hard to tell when you reach the upper asymptote though.

panarky 2y ago

Exponential increases in inputs yield linear increases in intelligence. Linear increases in intelligence yield exponential increases in economic gains. Therefore, exponential increases in inputs yield exponential increases in economic gains. Left as an exercise for the reader is to determine who will capture most of those economic gains.

jcgrillo 2y ago
So where are the economic gains? So far it's just been losses.