31 ms·

If the spend is X billion [inflation-adjusted] dollars per year on AI, we still get exponential returns on compute. That is not a stall in compute, obvs. So t

If the spend is X billion [inflation-adjusted] dollars per year on AI, we still get exponential returns on compute. That is not a stall in compute, obvs. So the question is, will that exponential gain in compute will yield exponential gains in AI capability? I suppose it depends on how you measure capability. I don't know how to measure that. But even if it only yields linear gains in AI capability, that is still not a stall, by any definition.

HarHarVeryFunny 3y ago
Right now it seems model-size/training-tokens are roughly going up 10x per year, resulting in training costs going up 10x per year. As long as investors are willing to increase spend 10x year over year then models can continue to grow 10x year over year. The "stall" scenario is where model performance from one year to the next, despite 10x increase in size, stops getting significantly better (i.e. better to extent that investors project increased revenue and worthwhile ROI). Let's say best case scenario here is that investors are willing to keep putting money in, but only at a flat year-over-year level. Without that 10x YoY spend increase, the model designers have lost a 10x factor in ability to increase model size. Perhaps compute prices halve YoY, so they can still get a 2x model size increase, but what good will this do if prior year's 10x increase just saw performance leveling off? This seems like the best case scenario in this "performance levelling off to point of minimal RoI" situation. Why would an investor "throw good money after bad" and spend another $10B after having judged that last year's $10B wasn't really worth it? So, under this logic, it seems that either: a) scaling/tweaks are all you need, and YoY performance gains continue to be impressive and support 10x model-size increases/spend b) scaling is not all you need, but some AGI-critical breakthroughs are made BEFORE investors give up on current trajectory c) scaling is not all you need, the AGI breakthroughs are not made in next few iterations, and the industry enters "stall" mode (i.e. no further progress towards AGI, other than ongoing research to find a new direction)
- TheDudeMan 3y ago
  Maybe the slowdown would actually result in us surviving!
  - HarHarVeryFunny 3y ago
    Maybe - definitely safer if it's slowly introduced!

HarHarVeryFunny 3y ago

Right now it seems model-size/training-tokens are roughly going up 10x per year, resulting in training costs going up 10x per year. As long as investors are willing to increase spend 10x year over year then models can continue to grow 10x year over year. The "stall" scenario is where model performance from one year to the next, despite 10x increase in size, stops getting significantly better (i.e. better to extent that investors project increased revenue and worthwhile ROI). Let's say best case scenario here is that investors are willing to keep putting money in, but only at a flat year-over-year level. Without that 10x YoY spend increase, the model designers have lost a 10x factor in ability to increase model size. Perhaps compute prices halve YoY, so they can still get a 2x model size increase, but what good will this do if prior year's 10x increase just saw performance leveling off? This seems like the best case scenario in this "performance levelling off to point of minimal RoI" situation. Why would an investor "throw good money after bad" and spend another $10B after having judged that last year's $10B wasn't really worth it? So, under this logic, it seems that either: a) scaling/tweaks are all you need, and YoY performance gains continue to be impressive and support 10x model-size increases/spend b) scaling is not all you need, but some AGI-critical breakthroughs are made BEFORE investors give up on current trajectory c) scaling is not all you need, the AGI breakthroughs are not made in next few iterations, and the industry enters "stall" mode (i.e. no further progress towards AGI, other than ongoing research to find a new direction)

TheDudeMan 3y ago
Maybe the slowdown would actually result in us surviving!
- HarHarVeryFunny 3y ago
  Maybe - definitely safer if it's slowly introduced!