4 ms·

Not sure why folks are defensive? The article is mainly informational. TFA: "What does this mean for the future of AI research? Things may not be as bleak as

Not sure why folks are defensive? The article is mainly informational. TFA: "What does this mean for the future of AI research? Things may not be as bleak as they look. The cost of training might come down as more efficient training methods are invented. Similarly, while data center energy use was predicted to explode in recent years, this has not happened due to improvements in data center efficiency, more efficient hardware and cooling." Also. One flight in and of itself isn't an environemntal problem; thousands of flights are. Training this one instance of a model isn't an environmental problem; I would be curious to see some educated guesses about the number of models being trained over the next ten years. Not an expert but I know that use of ML is exploding - lots of new use-cases and thus lots of new models. So it makes sense to me think about this stuff. To

ALittleLight 6y ago
The article suggests that the energy usage of language models is a problem. I don't think energy usage is a problem. I'm not sure how you interpret this as being defensive. There are hundreds of thousands of flights per day. Adding an additional flight, or even an additional thousand flights, to substantially improve Google doesn't seem like a big cost. Consider: Would your life be worse if one random trans-American flight was cancelled today, or if Google searches became 10% worse? Another way of thinking about this same point, is, why is the author writing about language models if they are so concerned about the environment. Surely the airline industry is a better subject as, again, they fly hundreds of thousands of flights each day. It's hard to take someone seriously when they are focusing on an infinitesimal part of a huge problem. It's also misleading because there is a difference between energy used by an airline flight, where the energy comes from burning jet fuel, and energy used in a data center. In Google's case (the people training BERT) the energy used was 100% renewable - Google reached that goal in 2017. Perhaps Open AI didn't use renewable energy to train GPT-3, but I wager they didn't power their machines by burning jet fuel either. Maybe electricity used to power to train language models will become a meaningful issue at some point in the future. I don't think that future is close at hand though.
- adamsea 6y ago
  tl;dr I think the tech industry doesn't like to be criticized. That's a good point about renewable energy sources. I'd still argue it's reasonable to assume that ML will consume more power as use-cases for it grow (i.e. more and more models are trained), and, therefore, I don't think it's unreasonable to consider the energy usage. Honestly - I think what you may have heard is "ML is bad/evil." And yeah some folks probably feel/vibe that. That's different from "Like any other technology the full context of ML deserves to be considered, and since it's so hyped by billion-dollar companies right now, maybe a bit of pushback on all the hype isn't a terrible idea." And yeah we should definitely look at the airline industry; pretty sure Mother Jones does, and there's no reason we can't look at the energy impact of multiple industries.
  - ALittleLight 6y ago
    If you were against lung cancer, and writing about the issues caused by scented candles emitting smoke and that smoke corrupting lungs - it would be hard to take you seriously or accept your criticism as valid. It's probably true that scented candles are infinitesimal contributors to lung cancer, but it's just silly to think of them that way. Likewise, yes, language models use energy, but objectively, it's not that much energy and the sources for the energy are sometimes green and always efficient (i.e. Coming from the grid and not by burning jet fuel). You can't really solve the problem of lung cancer by addressing scented candles. Reducing energy used by language models likewise won't have an effect.

ALittleLight 6y ago

The article suggests that the energy usage of language models is a problem. I don't think energy usage is a problem. I'm not sure how you interpret this as being defensive. There are hundreds of thousands of flights per day. Adding an additional flight, or even an additional thousand flights, to substantially improve Google doesn't seem like a big cost. Consider: Would your life be worse if one random trans-American flight was cancelled today, or if Google searches became 10% worse? Another way of thinking about this same point, is, why is the author writing about language models if they are so concerned about the environment. Surely the airline industry is a better subject as, again, they fly hundreds of thousands of flights each day. It's hard to take someone seriously when they are focusing on an infinitesimal part of a huge problem. It's also misleading because there is a difference between energy used by an airline flight, where the energy comes from burning jet fuel, and energy used in a data center. In Google's case (the people training BERT) the energy used was 100% renewable - Google reached that goal in 2017. Perhaps Open AI didn't use renewable energy to train GPT-3, but I wager they didn't power their machines by burning jet fuel either. Maybe electricity used to power to train language models will become a meaningful issue at some point in the future. I don't think that future is close at hand though.

adamsea 6y ago
tl;dr I think the tech industry doesn't like to be criticized. That's a good point about renewable energy sources. I'd still argue it's reasonable to assume that ML will consume more power as use-cases for it grow (i.e. more and more models are trained), and, therefore, I don't think it's unreasonable to consider the energy usage. Honestly - I think what you may have heard is "ML is bad/evil." And yeah some folks probably feel/vibe that. That's different from "Like any other technology the full context of ML deserves to be considered, and since it's so hyped by billion-dollar companies right now, maybe a bit of pushback on all the hype isn't a terrible idea." And yeah we should definitely look at the airline industry; pretty sure Mother Jones does, and there's no reason we can't look at the energy impact of multiple industries.
- ALittleLight 6y ago
  If you were against lung cancer, and writing about the issues caused by scented candles emitting smoke and that smoke corrupting lungs - it would be hard to take you seriously or accept your criticism as valid. It's probably true that scented candles are infinitesimal contributors to lung cancer, but it's just silly to think of them that way. Likewise, yes, language models use energy, but objectively, it's not that much energy and the sources for the energy are sometimes green and always efficient (i.e. Coming from the grid and not by burning jet fuel). You can't really solve the problem of lung cancer by addressing scented candles. Reducing energy used by language models likewise won't have an effect.