4 ms·

Tie it with the second and third points though. Essentially the claim is that it gets exponentially cheaper at the same rate the logarithmic resource requireme

Tie it with the second and third points though. Essentially the claim is that it gets exponentially cheaper at the same rate the logarithmic resource requirements go up. Which leads to point 3 - linear growth in intelligence is expected over the next few years and AGI is a certainty. I feel that's not happening at all. This costs 15x more to run compared to 4o and is marginally better. Perhaps the failing of this prediction is on the cost side but regardless it's a failing of the prediction for linear growth in intelligence. Would another 15x resources lead to another marginal gain? Repeat? At that rate of gain we could turn all the resources in the world to this and not hit anywhere near AGI.

mullingitover 2y ago
My take is that the second point is the models created by this crazy expensive training are just software, and software gets cheaper to operate by virtue of Moore's Law in an exponential fashion. Third point, I think, is that the intelligent work that the models can do will increase exponentially because they get cheaper and cheaper to operate as the get more and more capable. So I think GPT4.5 is in the short term more expensive (they have to pay their training bill) but eventually this being the new floor is just ratcheting toward the nerd singularity or whatever.
- sifar 2y ago
  Moore's Law is a self-fulfilling prophecy derived from the value of computation. If a given computation does not provide enough value, it might not be invested with enough resources to benefit from Moore's Law. When scaling saturates, less computationally expensive models would benefit more.
darkerside 2y ago
Doesn't make sense. Requiring exponentially more resources is not surprising. Resources getting exponentially cheaper? Like, falling by more than half each cycle? That doesn't happen.
danenania 2y ago
> This costs 15x more to run compared to 4o and is marginally better. 4o has been very heavily optimized though. I'd say a more apples-to-apples comparison would be to original gpt-4, which was $30/$60 to 4.5's $75/$150. Still a significant difference, but not quite so stark.
deepsquirrelnet 2y ago
I felt like the point of llama 3 was to prove this out. 2T -> 15T tokens, 70B -> 405B. We now have llama 3.3 70B, which by most metrics outperforms the 405B model without further scaling, so it’s been my assumption that scaling is dead. Other innovations in training are taking the lead. Higher volumes of low quality data aren’t moving the needle.

mullingitover 2y ago

My take is that the second point is the models created by this crazy expensive training are just software, and software gets cheaper to operate by virtue of Moore's Law in an exponential fashion. Third point, I think, is that the intelligent work that the models can do will increase exponentially because they get cheaper and cheaper to operate as the get more and more capable. So I think GPT4.5 is in the short term more expensive (they have to pay their training bill) but eventually this being the new floor is just ratcheting toward the nerd singularity or whatever.

sifar 2y ago
Moore's Law is a self-fulfilling prophecy derived from the value of computation. If a given computation does not provide enough value, it might not be invested with enough resources to benefit from Moore's Law. When scaling saturates, less computationally expensive models would benefit more.

darkerside 2y ago

Doesn't make sense. Requiring exponentially more resources is not surprising. Resources getting exponentially cheaper? Like, falling by more than half each cycle? That doesn't happen.

danenania 2y ago

> This costs 15x more to run compared to 4o and is marginally better. 4o has been very heavily optimized though. I'd say a more apples-to-apples comparison would be to original gpt-4, which was $30/$60 to 4.5's $75/$150. Still a significant difference, but not quite so stark.

deepsquirrelnet 2y ago

I felt like the point of llama 3 was to prove this out. 2T -> 15T tokens, 70B -> 405B. We now have llama 3.3 70B, which by most metrics outperforms the 405B model without further scaling, so it’s been my assumption that scaling is dead. Other innovations in training are taking the lead. Higher volumes of low quality data aren’t moving the needle.