3 ms·

I can't help but think this fixation on ever-bigger data centers with ever-more compute is a very American approach. It's the same line of thought that gave us

I can't help but think this fixation on ever-bigger data centers with ever-more compute is a very American approach. It's the same line of thought that gave us jumbo refrigerators, muscle cars and supersize meals. In the American mind, bigger = better. That that's not necessarily the best way to better results has been demonstrated by the Deepseek shock. I do wonder if more foundational research would be a better investment.

courseofaction 1y ago
But those damnable universities are just left wing brainwashing facilities, very little room for profit there. Cut their funding, educated people don't vote right /s
Closi 1y ago
It’s not an either/or - it can be both! If you have the best foundational research, but then find yourself blocked by compute, that’s a bad place to be in. At the moment compute is a constraint too (at least for the smaller shops, maybe not OpenAI!) We don’t necessarily know that the next AI breakthrough will need less compute either! A breakthrough that gives 5x performance for 20x training compute and 2x inference cost would still be a giant breakthrough and need a lot of hardware! Although this sort of breakthrough is unlikely, and it’s likely to be smaller steps forward - but I think it might be the case that models continue to get more computationally expensive even if there are breakthroughs that make models more efficient because we will want higher capability.
- saubeidl 1y ago
  I think it's often been shown that constraints are what brings out true breakthroughs, so I'm skeptical of scaling up throw brute force - to me it seems like a lazy way out that will prevent you from challenging your base assumptions and figuring out better ways of doing things.
  - Closi 1y ago
    I think there is a world where you have both - a portion of research sitting under constraints, and also massive computing capability to help build the next big model. Put another way - I doubt OpenAI would have built GPT-3 faster if they had less compute.
klipt 1y ago
> more foundational research So you're saying a bigger amount of research is better?
- saubeidl 1y ago
  I do. I'm saying more thinking, less throwing raw material at a problem.
  - deleted 1y ago
    [deleted]

courseofaction 1y ago

But those damnable universities are just left wing brainwashing facilities, very little room for profit there. Cut their funding, educated people don't vote right /s

Closi 1y ago

It’s not an either/or - it can be both! If you have the best foundational research, but then find yourself blocked by compute, that’s a bad place to be in. At the moment compute is a constraint too (at least for the smaller shops, maybe not OpenAI!) We don’t necessarily know that the next AI breakthrough will need less compute either! A breakthrough that gives 5x performance for 20x training compute and 2x inference cost would still be a giant breakthrough and need a lot of hardware! Although this sort of breakthrough is unlikely, and it’s likely to be smaller steps forward - but I think it might be the case that models continue to get more computationally expensive even if there are breakthroughs that make models more efficient because we will want higher capability.

saubeidl 1y ago
I think it's often been shown that constraints are what brings out true breakthroughs, so I'm skeptical of scaling up throw brute force - to me it seems like a lazy way out that will prevent you from challenging your base assumptions and figuring out better ways of doing things.
- Closi 1y ago
  I think there is a world where you have both - a portion of research sitting under constraints, and also massive computing capability to help build the next big model. Put another way - I doubt OpenAI would have built GPT-3 faster if they had less compute.

klipt 1y ago

> more foundational research So you're saying a bigger amount of research is better?

saubeidl 1y ago
I do. I'm saying more thinking, less throwing raw material at a problem.
- deleted 1y ago
  [deleted]