9 ms·

BERT is much smaller than GPT-3 (500x fewer parameters), not sure why the article doesn't point that out more explicitly. The mentioned paper itself notes a 30

BERT is much smaller than GPT-3 (500x fewer parameters), not sure why the article doesn't point that out more explicitly. The mentioned paper itself notes a 300,000x increase in compute used for training language models over the last 6 years. I think the point is, if nothing changes, the costs will be very significant very soon.

qeternity 6y ago
We’re already at soon. Another order of magnitude increase (which at our pace is all but guaranteed in the near term) will make large scale training a significant R&D line item even for the FAANGs. It will also push efforts to be commercially viable, as you can longer do proof-of-concept nets at $500m
PragmaticPulp 6y ago
> I think the point is, if nothing changes, the costs will be very significant very soon. The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.
- _jal 6y ago
  I think it is reasonable to assume that any advances in efficiency will be subsumed by increased use, much in the same way increased energy efficiency doesn't decrease absolute overall use.
- joe_the_user 6y ago
  ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down. ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given. [1] https://openai.com/blog/ai-and-compute/ https://openai.com/blog/ai-and-compute/ [2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
amelius 6y ago
So much for the "democratization of AI", I suppose.

qeternity 6y ago

We’re already at soon. Another order of magnitude increase (which at our pace is all but guaranteed in the near term) will make large scale training a significant R&D line item even for the FAANGs. It will also push efforts to be commercially viable, as you can longer do proof-of-concept nets at $500m

PragmaticPulp 6y ago

> I think the point is, if nothing changes, the costs will be very significant very soon. The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.

_jal 6y ago
I think it is reasonable to assume that any advances in efficiency will be subsumed by increased use, much in the same way increased energy efficiency doesn't decrease absolute overall use.
joe_the_user 6y ago
ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down. ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given. [1] https://openai.com/blog/ai-and-compute/ https://openai.com/blog/ai-and-compute/ [2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

amelius 6y ago

So much for the "democratization of AI", I suppose.