5 ms·

One of the criticisms in that paper was that training AI used a lot of electricity. That's technically true, but what's the proposal there? The power efficiency

One of the criticisms in that paper was that training AI used a lot of electricity. That's technically true, but what's the proposal there? The power efficiency of hardware is already being increased.

clankyclanker 6y ago
> what's the proposal there? That's what Google was afraid of: the alternative is to decide that the cost of the model isn't worth the effort (externalities < benefit) and to abandon it. Which might be the correct decision, but one that loses Google a lot of potential money.
- nindalf 6y ago
  The paper didn't attempt to examine if the benefits of the model outweighed the costs. They calculated the total energy used and called it a day.
  - joshuamorton 6y ago
    Correct. It left it up to the model creators to potentially evaluate those tradeoffs for future models.
  - deleted 6y ago
    [deleted]
- Nasrudith 6y ago
  I find the conclusions ludicrous. Should we also abandon particle accelerators because they use too much electrity for ill-defined benefits? Really the claim is toxic academia style knife fighting for tenure at best.
klipt 6y ago
I bet all the big tech companies spend way more electricity on things other than training big models. "You use too much electricity" is tantamount to "tech is a flawed business model". Arguably solvable with increased green energy sources but that's more a job for society in general than just tech companies.
jeffbee 6y ago
Part of the issue in these events is that the paper that was written by Google's own ethics researchers simply repeats estimates of AI energy consumption from outside academic papers, instead of examining them. The reality is that Google's ML infrastructure is dramatically more energy efficient than the rest of the industry. A person with access to these facts who chooses to ignore them isn't really doing their best as a researcher, they're just grinding an ax. There are other examples against the meme of AI using "a lot of electricity", for example a recent paper from Google where they show that a learned computational fluid dynamics technique uses only 2% the energy of standard techniques. https://arxiv.org/pdf/2102.01010.pdf https://arxiv.org/pdf/2102.01010.pdf
- dekhn 6y ago
  Not only are ML techniques finding ways to do physics with less energy, the amount of waste by non-ML computing absolutely swamps ML computing. I don't care if training a model uses the amount of CO2 that a translatlanic flight uses, because those flights are flying constantly, some of them not filled to the max. The whole criticism about energy in the paper seemed to miss major details that I'd expect top researchers to get right. BTW, I ran a large scale computing platform at Google for several years. It made absolutely enormous, stunningly huge amounts of CPU available to scientists and we used it to make several important breakthroughs. However, after a few years, I finally was shown what the power cost implications were, and I shut down the project because the costs were not justified by the scientific benefits. Compared to what I was doing, ML is relatively skimpy on CPU.
  - jeffbee 6y ago
    I'm not sure that your scientific facilities cumulatively spent as much CPU time as DeepMind spent playing StarCraft. If we want to talk about energy and ethics in machine learning I guess that's where I would start.
    - dekhn 6y ago
      Yes, my calculations used far more time than DM, with the caveat that my jobs ran on CPUs while most of their training is on accelerators. I believe we reported that Exacycle provided 700K high speed Xeon cores for over a year, and the actual number is far higher (I can't share it because you'd be able to make a reasonable estimate on how many computers Google has).
paganel 6y ago
Do less AI training, so less AI could be a valid alternative.
8note 6y ago
How often is the electricity use considered a constraint in algorithm design for AI models? Moore's law isn't the only way to gain efficiency, and even if the available compute goes up, doesn't mean the total power usage will go down
sudosysgen 6y ago
The proposal is not to continue the trend of simply adding titanic amounts of parameters to models and develop more efficient ones.

clankyclanker 6y ago

> what's the proposal there? That's what Google was afraid of: the alternative is to decide that the cost of the model isn't worth the effort (externalities < benefit) and to abandon it. Which might be the correct decision, but one that loses Google a lot of potential money.

nindalf 6y ago
The paper didn't attempt to examine if the benefits of the model outweighed the costs. They calculated the total energy used and called it a day.
- joshuamorton 6y ago
  Correct. It left it up to the model creators to potentially evaluate those tradeoffs for future models.
- deleted 6y ago
  [deleted]
Nasrudith 6y ago
I find the conclusions ludicrous. Should we also abandon particle accelerators because they use too much electrity for ill-defined benefits? Really the claim is toxic academia style knife fighting for tenure at best.

klipt 6y ago

I bet all the big tech companies spend way more electricity on things other than training big models. "You use too much electricity" is tantamount to "tech is a flawed business model". Arguably solvable with increased green energy sources but that's more a job for society in general than just tech companies.

jeffbee 6y ago

Part of the issue in these events is that the paper that was written by Google's own ethics researchers simply repeats estimates of AI energy consumption from outside academic papers, instead of examining them. The reality is that Google's ML infrastructure is dramatically more energy efficient than the rest of the industry. A person with access to these facts who chooses to ignore them isn't really doing their best as a researcher, they're just grinding an ax. There are other examples against the meme of AI using "a lot of electricity", for example a recent paper from Google where they show that a learned computational fluid dynamics technique uses only 2% the energy of standard techniques. https://arxiv.org/pdf/2102.01010.pdf https://arxiv.org/pdf/2102.01010.pdf

dekhn 6y ago
Not only are ML techniques finding ways to do physics with less energy, the amount of waste by non-ML computing absolutely swamps ML computing. I don't care if training a model uses the amount of CO2 that a translatlanic flight uses, because those flights are flying constantly, some of them not filled to the max. The whole criticism about energy in the paper seemed to miss major details that I'd expect top researchers to get right. BTW, I ran a large scale computing platform at Google for several years. It made absolutely enormous, stunningly huge amounts of CPU available to scientists and we used it to make several important breakthroughs. However, after a few years, I finally was shown what the power cost implications were, and I shut down the project because the costs were not justified by the scientific benefits. Compared to what I was doing, ML is relatively skimpy on CPU.
- jeffbee 6y ago
  I'm not sure that your scientific facilities cumulatively spent as much CPU time as DeepMind spent playing StarCraft. If we want to talk about energy and ethics in machine learning I guess that's where I would start.
  - dekhn 6y ago
    Yes, my calculations used far more time than DM, with the caveat that my jobs ran on CPUs while most of their training is on accelerators. I believe we reported that Exacycle provided 700K high speed Xeon cores for over a year, and the actual number is far higher (I can't share it because you'd be able to make a reasonable estimate on how many computers Google has).

paganel 6y ago

Do less AI training, so less AI could be a valid alternative.

8note 6y ago

How often is the electricity use considered a constraint in algorithm design for AI models? Moore's law isn't the only way to gain efficiency, and even if the available compute goes up, doesn't mean the total power usage will go down

sudosysgen 6y ago

The proposal is not to continue the trend of simply adding titanic amounts of parameters to models and develop more efficient ones.