6 ms·

The article points out that the energy to train BERT is comparable to a flight across America. There are hundreds of thousands of flights every day - why should

The article points out that the energy to train BERT is comparable to a flight across America. There are hundreds of thousands of flights every day - why should we be concerned about the equivalent of one extra? Especially given that BERT improves ~10% of English language Google search results, it seems like we're getting a lot in return for relatively small energy use. On top of that, Google buys 100% of their energy usage from Green sources. I think it's great to talk about other methods of training or architectures that don't require so many parameters. The point about how BERT consumes vastly more text than humans do when they learn to read is interesting. But trying to phrase this like an environmental issue just seems disingenuous and misleading.

alisonkisk 6y ago
Google buying "green" isn't a justification because it fails the categorical imperative. It's impossible for everyone to buy "green" energy.
- toomuchtodo 6y ago
  As always, there’s nuance. You can schedule workloads, machine learning/training included, where the power is cleanest (low carbon). Google relies on electricitymap.org/Tomorrow to do this. https://blog.google/inside-google/infrastructure/data-centers-work-harder-sun-shines-wind-blows https://blog.google/inside-google/infrastructure/data-center... The more folks who elect to pick where to compute based on low electrical carbon intensity, the faster the grid turns over to clean generation. You must vote with your fiat. I encourage technologists to include this consideration in their workload scheduling requirements. Renewables are almost always cheaper than fossil generation as well.
- curiousllama 6y ago
  The categorical imperative doesn't require everyone to be practically able to do the thing. Is it not moral to feed a starving man because people in China aren't able to feed him?
- rictic 6y ago
  There's more than enough solar and wind, it's just a matter of how much we as a society want it. The green energy revolution has barely begun, and we're still climbing the learning curve. More demand means a faster climb means exponentially more green energy sooner.
- himinlomax 6y ago
  Buying more green energy creates demand for more green energy, which in turn creates more supply (wind, solar farms ...) Meanwhile, the aforementioned transatlantic flight cannot get greener, whether Bert is trained or not.
adamsea 6y ago
Not sure why folks are defensive? The article is mainly informational. TFA: "What does this mean for the future of AI research? Things may not be as bleak as they look. The cost of training might come down as more efficient training methods are invented. Similarly, while data center energy use was predicted to explode in recent years, this has not happened due to improvements in data center efficiency, more efficient hardware and cooling." Also. One flight in and of itself isn't an environemntal problem; thousands of flights are. Training this one instance of a model isn't an environmental problem; I would be curious to see some educated guesses about the number of models being trained over the next ten years. Not an expert but I know that use of ML is exploding - lots of new use-cases and thus lots of new models. So it makes sense to me think about this stuff. To
- ALittleLight 6y ago
  The article suggests that the energy usage of language models is a problem. I don't think energy usage is a problem. I'm not sure how you interpret this as being defensive. There are hundreds of thousands of flights per day. Adding an additional flight, or even an additional thousand flights, to substantially improve Google doesn't seem like a big cost. Consider: Would your life be worse if one random trans-American flight was cancelled today, or if Google searches became 10% worse? Another way of thinking about this same point, is, why is the author writing about language models if they are so concerned about the environment. Surely the airline industry is a better subject as, again, they fly hundreds of thousands of flights each day. It's hard to take someone seriously when they are focusing on an infinitesimal part of a huge problem. It's also misleading because there is a difference between energy used by an airline flight, where the energy comes from burning jet fuel, and energy used in a data center. In Google's case (the people training BERT) the energy used was 100% renewable - Google reached that goal in 2017. Perhaps Open AI didn't use renewable energy to train GPT-3, but I wager they didn't power their machines by burning jet fuel either. Maybe electricity used to power to train language models will become a meaningful issue at some point in the future. I don't think that future is close at hand though.
  - adamsea 6y ago
spiznnx 6y ago
BERT is much smaller than GPT-3 (500x fewer parameters), not sure why the article doesn't point that out more explicitly. The mentioned paper itself notes a 300,000x increase in compute used for training language models over the last 6 years. I think the point is, if nothing changes, the costs will be very significant very soon.
- qeternity 6y ago
  We’re already at soon. Another order of magnitude increase (which at our pace is all but guaranteed in the near term) will make large scale training a significant R&D line item even for the FAANGs. It will also push efforts to be commercially viable, as you can longer do proof-of-concept nets at $500m
- PragmaticPulp 6y ago
  > I think the point is, if nothing changes, the costs will be very significant very soon. The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.
  - _jal 6y ago
    I think it is reasonable to assume that any advances in efficiency will be subsumed by increased use, much in the same way increased energy efficiency doesn't decrease absolute overall use.
  - joe_the_user 6y ago
    ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down. ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given. [1] https://openai.com/blog/ai-and-compute/ https://openai.com/blog/ai-and-compute/ [2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
acchow 6y ago
> There are hundreds of thousands of flights every day 2019 was peak flight with 38.9 million flights over the year. That averages to 106k flights per day.
acdc4life 6y ago
My brain does nlp better than any system out there. I’m also able to ride bicycles and do motor control better than Boston Dynamics. I can also construct and prove math, do physics and code all of this in Matlab and C. My brain can handle all these wide range of tasks almost seamlessly with just 15 watts of power, that Silicon Valley’s super computers can barely do 1% of.
- omgwtfbyobbq 6y ago
  To be fair, it takes years of energy to train our brains to the point where it can do all those things, and our brains aren't extensible in the same way hardware/software is. I guess there's also a lot more variation in yields. ;)
  - acdc4life 6y ago
    Well, that’s not entirely true. Alpha go self played 29 million games of Go, which is in feasible for humans. Assuming a game of Go is 5mins, it would take a human 286 years to achieve the same, assuming this person doesn’t sleep, eat or do anything else other than just play Go. CPU time is an order of magnitude faster than real time, especially on GPU clusters.
- bumby 6y ago
  To be a fair comparison wouldn’t you have to include all the energy used to train “your” brain through generations of evolutionary training? Your latest model is like taking an already trained BERT model and adding a few tweaks
  - acdc4life 6y ago
    I see BERT as non sensical. You need to be scientific and have a mathematical theory on how humans learn language, which is a multi disciplinary task requiring physicists, mathematicians, neuroscientists, cognitive psychologists and linguists. Benchmarks are useless, theories, models, experiments and testable predictions is how science progresses. You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked. The psychological, linguistic, evolutionary biology and neuroscience evidence doesn’t seem to corroborate. The evidence points stronger to humans having general learning and problem solving abilities. For instance, there was no evolutionary pressure for humans to be good at math or programming. I was not born knowing english or calculus or probability theory, these were learned abilities. Evolution favoured brain mechanisms that lead to behaviour for success in a rapidly changing world. Had I been born in ancient Rome as a farmer, I would learn to speak Latin, and learn how to be a successful farmer, instead of the physics, math, probability, computer, driving, reading skills that I learned in my life time.
sgt101 6y ago
Also, I have downloaded and used Bert in several other applications; and I think that 10,000's of other folks have done the same.
jkochis 6y ago
The comparison between training BERT and a 5 year old made me wonder what the carbon footprint is to raise someone to the age of 5.
throwaway2245 6y ago
> There are hundreds of thousands of flights every day - why should we be concerned about the equivalent of one extra See also: Sorites fallacy. There is a limited desire for flights, which benefit people as far as it allows them to get things where they need to be, and no more than that; there is presumably an unlimited desire for machine learning.

alisonkisk 6y ago

Google buying "green" isn't a justification because it fails the categorical imperative. It's impossible for everyone to buy "green" energy.

toomuchtodo 6y ago
As always, there’s nuance. You can schedule workloads, machine learning/training included, where the power is cleanest (low carbon). Google relies on electricitymap.org/Tomorrow to do this. https://blog.google/inside-google/infrastructure/data-centers-work-harder-sun-shines-wind-blows https://blog.google/inside-google/infrastructure/data-center... The more folks who elect to pick where to compute based on low electrical carbon intensity, the faster the grid turns over to clean generation. You must vote with your fiat. I encourage technologists to include this consideration in their workload scheduling requirements. Renewables are almost always cheaper than fossil generation as well.
curiousllama 6y ago
The categorical imperative doesn't require everyone to be practically able to do the thing. Is it not moral to feed a starving man because people in China aren't able to feed him?
rictic 6y ago
There's more than enough solar and wind, it's just a matter of how much we as a society want it. The green energy revolution has barely begun, and we're still climbing the learning curve. More demand means a faster climb means exponentially more green energy sooner.
himinlomax 6y ago
Buying more green energy creates demand for more green energy, which in turn creates more supply (wind, solar farms ...) Meanwhile, the aforementioned transatlantic flight cannot get greener, whether Bert is trained or not.

adamsea 6y ago

Not sure why folks are defensive? The article is mainly informational. TFA: "What does this mean for the future of AI research? Things may not be as bleak as they look. The cost of training might come down as more efficient training methods are invented. Similarly, while data center energy use was predicted to explode in recent years, this has not happened due to improvements in data center efficiency, more efficient hardware and cooling." Also. One flight in and of itself isn't an environemntal problem; thousands of flights are. Training this one instance of a model isn't an environmental problem; I would be curious to see some educated guesses about the number of models being trained over the next ten years. Not an expert but I know that use of ML is exploding - lots of new use-cases and thus lots of new models. So it makes sense to me think about this stuff. To

ALittleLight 6y ago
The article suggests that the energy usage of language models is a problem. I don't think energy usage is a problem. I'm not sure how you interpret this as being defensive. There are hundreds of thousands of flights per day. Adding an additional flight, or even an additional thousand flights, to substantially improve Google doesn't seem like a big cost. Consider: Would your life be worse if one random trans-American flight was cancelled today, or if Google searches became 10% worse? Another way of thinking about this same point, is, why is the author writing about language models if they are so concerned about the environment. Surely the airline industry is a better subject as, again, they fly hundreds of thousands of flights each day. It's hard to take someone seriously when they are focusing on an infinitesimal part of a huge problem. It's also misleading because there is a difference between energy used by an airline flight, where the energy comes from burning jet fuel, and energy used in a data center. In Google's case (the people training BERT) the energy used was 100% renewable - Google reached that goal in 2017. Perhaps Open AI didn't use renewable energy to train GPT-3, but I wager they didn't power their machines by burning jet fuel either. Maybe electricity used to power to train language models will become a meaningful issue at some point in the future. I don't think that future is close at hand though.
- adamsea 6y ago

spiznnx 6y ago

BERT is much smaller than GPT-3 (500x fewer parameters), not sure why the article doesn't point that out more explicitly. The mentioned paper itself notes a 300,000x increase in compute used for training language models over the last 6 years. I think the point is, if nothing changes, the costs will be very significant very soon.

qeternity 6y ago
We’re already at soon. Another order of magnitude increase (which at our pace is all but guaranteed in the near term) will make large scale training a significant R&D line item even for the FAANGs. It will also push efforts to be commercially viable, as you can longer do proof-of-concept nets at $500m
PragmaticPulp 6y ago
> I think the point is, if nothing changes, the costs will be very significant very soon. The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.
- _jal 6y ago
  I think it is reasonable to assume that any advances in efficiency will be subsumed by increased use, much in the same way increased energy efficiency doesn't decrease absolute overall use.
- joe_the_user 6y ago
  ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down. ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given. [1] https://openai.com/blog/ai-and-compute/ https://openai.com/blog/ai-and-compute/ [2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

acchow 6y ago

> There are hundreds of thousands of flights every day 2019 was peak flight with 38.9 million flights over the year. That averages to 106k flights per day.

acdc4life 6y ago

My brain does nlp better than any system out there. I’m also able to ride bicycles and do motor control better than Boston Dynamics. I can also construct and prove math, do physics and code all of this in Matlab and C. My brain can handle all these wide range of tasks almost seamlessly with just 15 watts of power, that Silicon Valley’s super computers can barely do 1% of.

omgwtfbyobbq 6y ago
To be fair, it takes years of energy to train our brains to the point where it can do all those things, and our brains aren't extensible in the same way hardware/software is. I guess there's also a lot more variation in yields. ;)
- acdc4life 6y ago
  Well, that’s not entirely true. Alpha go self played 29 million games of Go, which is in feasible for humans. Assuming a game of Go is 5mins, it would take a human 286 years to achieve the same, assuming this person doesn’t sleep, eat or do anything else other than just play Go. CPU time is an order of magnitude faster than real time, especially on GPU clusters.
bumby 6y ago
To be a fair comparison wouldn’t you have to include all the energy used to train “your” brain through generations of evolutionary training? Your latest model is like taking an already trained BERT model and adding a few tweaks
- acdc4life 6y ago
  I see BERT as non sensical. You need to be scientific and have a mathematical theory on how humans learn language, which is a multi disciplinary task requiring physicists, mathematicians, neuroscientists, cognitive psychologists and linguists. Benchmarks are useless, theories, models, experiments and testable predictions is how science progresses. You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked. The psychological, linguistic, evolutionary biology and neuroscience evidence doesn’t seem to corroborate. The evidence points stronger to humans having general learning and problem solving abilities. For instance, there was no evolutionary pressure for humans to be good at math or programming. I was not born knowing english or calculus or probability theory, these were learned abilities. Evolution favoured brain mechanisms that lead to behaviour for success in a rapidly changing world. Had I been born in ancient Rome as a farmer, I would learn to speak Latin, and learn how to be a successful farmer, instead of the physics, math, probability, computer, driving, reading skills that I learned in my life time.

sgt101 6y ago

Also, I have downloaded and used Bert in several other applications; and I think that 10,000's of other folks have done the same.

jkochis 6y ago

The comparison between training BERT and a 5 year old made me wonder what the carbon footprint is to raise someone to the age of 5.

throwaway2245 6y ago

> There are hundreds of thousands of flights every day - why should we be concerned about the equivalent of one extra See also: Sorites fallacy. There is a limited desire for flights, which benefit people as far as it allows them to get things where they need to be, and no more than that; there is presumably an unlimited desire for machine learning.