3 ms·
The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's
by FanaHOVA 3y ago
The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.
- p1esk 3y agoOr you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.
- FanaHOVA 3y ago> do something other than dumb scaling up This is exactly what people told OpenAI 8 years ago and look where we are now.
- p1esk 3y ago8 years ago the dumb scaling up was exactly what we needed. 8 years we’ve been riding that train. Don’t you think it’s time to try something new?
- voz_ 3y agoI love this comment. Very HN. You’re absolutely right, everyone should just try to make paradigm shifts in the field.
- p1esk 3y agoThe key word here is “try”. And we are not talking about “everyone“, just those who complain they don’t have access to $5k/hr GPU clusters.
- PoignardAzur 3y ago"Or better, you can do the same thing that three people managed to do in the entire industry in the last 36 years." I mean, don't get me wrong, I'm all for improvements in AI efficiency, but maybe there isn't that much low-hanging fruit to pick? Tons of papers get published on transformers optimization techniques and barely any of them seem to stick.