8 ms·

Green AI

104 points by montalbano 7y ago

malux85 7y ago
This reminds me of that Simpsons quote "We can't fix your heart, but we can tell you exactly how damaged it is"
- swalsh 7y ago
  I'm not sure about that. I think you get what you measure, and if you start measuring effeciency, you're going to start seeing a major incentive to make it more effecient.
  - Nasrudith 7y ago
    I agree that it is important but think it is more understanding is a prerequisite to do so deliberately (careful of Goodhart's law) and from there the incentives become evident. Like Lanchester's laws. It is obvious how outnumbering the foe can help but knowing the scaling really hammers home how important it is to concentrate force against as weak a foe groups as possible. https://en.m.wikipedia.org/wiki/Lanchester%27s_laws https://en.m.wikipedia.org/wiki/Lanchester%27s_laws
  - malux85 7y ago
    Yes the first step to improvement is quantifying your objective, but it would have been nice to see some actual improvements and concrete implementations. The quantifying part is easy.
ausbah 7y ago
Like the abstract mentioned, I think this is also a good criteria for helping "level the playing field" for groups with lower budgets so the phenomenon of simply throwing more computational resources at a problem with an existing resource becomes less of a novelty.
- remon 7y ago
  Hm, I struggle to see an upside with levelling the playing field that way. Groups that have the budget to throw huge amounts of resources at problems are still providing important insights. That can happen in parallel to optimising wattage/computational unit. In fact, that can be an almost completely parallel track in AI research.
  - currymj 7y ago
    right now such a parallel track doesn't really exist, at least not on an even footing. as the article shows, there's still a heavy focus on beating accuracy benchmarks, at huge computational cost, in terms of what actually gets accepted at major venues. it can be quite difficult to make the case for a method that has worse performance, but is cheaper. hard to judge without real data but I think such papers are much less likely to be accepted. there are also sometimes demands to replicate very expensive techniques as baselines, which can be onerous for groups with limited resources.
  - Nasrudith 7y ago
    The upside is probably opportunity cost related essentially like mainframes of yore. Technically they could handle higher throughput than the workstations at the start and even today but once developed they became a dead end compared to more parallel approaches. That sort of quantity vs quality thing has happened a lot throughout history. The "VHS beats Betamax" moments that goes way further back but with more obscure examples.
taneq 7y ago
> The computations required for deep learning research have been doubling every few months I feel this is misleading. Computation is cheap, so the computation thrown at deep learning research has been doubling every few months, but that doesn't mean it's required to do research (unless your research is "throw huge datasets at a neural net architecture and see if it sticks".)
- atoav 7y ago
  I understood it as “computation required to replicate stuff outlined in a paper has doubled” not as “we do more research so we do more computation”.
- mattkrause 7y ago
  That...does seem to be a lot of the current research though.
mark_l_watson 7y ago
I like this idea! In last week’s Lex Firdman AI interview, Gary Marcus touches on the low twenty watt energy requirements of the human brain compared to deep learning energy requirements. Basically adding energy requirements to the loss function of automated neural model architecture search seems like a good idea also. (I am thinking of frameworks like AdaNet, etc.) I retired this year but I still spend a lot of time reviewing deep learning and also conventional AI literature (and I do tiny consulting projects to help people get started or build prototypes). Since I now mostly pay for my own computer resources I try to mostly limit myself to what I can do on my System 76 laptop with a single 1070 GPU. The availability of pretrained models makes this not so bad, at all. I really appreciate efforts by huggingface (and other organizations) of offering reduced size models that still provide good results.
- AndrewKemendo 7y ago
  The math is just wrong though. To train a brain to be competent in a classification task it takes 20W rms for years on end for the individual + all of the wattage from the parents, grandparents, teachers etc... that are training the individual for those years. Very hard to determine the power allocation used for a specific human being trained on a narrow task for eg. object classification, but that doesn't mean it's not the comparable measure. Comparing training of a single model to the "instant power" draw of the brain is not just over simplistic, the scale and time periods are wrong.
  - mirrorsedge 7y ago
    I agree. Using watts, which are energy (joules) per unit time (second), to measure the energy consumption of training a model is the wrong unit to begin with. One cannot measure the total energy consumption of something without knowing both the wattage and the time spent running at that voltage. While training a neural network is often done from scratch or a pre-trained model with a known training time, the brain of a human being does not start developing from zero their birth. Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. It would be very hard but also very interesting to approximate the energy spent on this, but it may not be relevant to machine learning as the processing power scales and time periods are completely different than those involved in the development of the human brain.
CoffeePython 7y ago
I find it strange that the part of the conversation where AI has the potential to save thousands of hours of human labor doesn’t show up more often in these types of threads. That being said, we need to definitely be thinking about lowering the cost (environmental and monetary) of training these models. I’m glad research is being done within this domain. I’d love to see a study on what the human labor cost potential vs. training environmental costs would be for certain large models.
- rahkiin 7y ago
  I find it interesting that you see 'save thousands of hours of human labor' as something that's surely positive. What should those people start doing instead? They also need an income to get bread on the table.
  - Nasrudith 7y ago
    Apply it backwards - would a welfare system of having people complete large N Tower of Hanoi locks that pay people money be a good idea? On one hand technically it is better than nothing immediately for those in need and is fraud resistant in that it consumes time and any better paying job without major negative externalities would be a better use of their time. The negative is that it does nothing productive and those who are dependent on it aren't able to use their time productively on anything else. While the problem of lack of income and lack of good uses for their labor certainly should be addressed just because it provides income doesn't make it a good thing overall. It is one of those things which puts the "dismal" in the "dismal science" term for economics - regardless of any misgivings over calling it a science. It is essentially a subset of human organizational and incentive problems which the economy "is designed" (technically more emergant even if it is literally a planned economy) to solve.
    - jkp56 7y ago
      What do you mean by "productive"?
  - deleted 7y ago
    [deleted]
css 7y ago
The citation for the "surprisingly large carbon footprint" [0] is crazy. The paper alleges that a car, including fuel, has a lifetime footprint of 126,000 lbs of CO₂e, and training a big NN transformer consumes 626,155 lbs of CO₂e, almost a 5-fold increase. [0] https://arxiv.org/pdf/1906.02243.pdf https://arxiv.org/pdf/1906.02243.pdf
- tsbinz 7y ago
  Training it _with neural architecture search_. Just training it with a given architecture they cite as 192lbs ...
  - dharma1 7y ago
    I thought Google etc datacentres run on mostly renewable energy?
sriku 7y ago
Research that demonstrates, say, 1000x reduction in power for training a known problem (ex: mnist) won't be considered irrelevant by the community. So is there a specific need to bias against large works apart from the carbon footprint argument? There remain questions to be answered in that space too - such as are current "neural" architectures adequate to cover the capabilities of the brain when upping only the scale? It was certainly worth knowing that scaling up was all that was required to compete with humans in DOTA. But will we hit wall as we near human level complexity? After all, the money spent on making movie which is "just" for entertainment trump's multi million $ deep learning experiments in costs which I guess have some correlation with the carbon footprint .. or the gases emitted in rocket launches. Do we really know whether this well intentioned call for green AI (which I right as hell want) will do too little for the greenness while biasing people against possible discoveries that can lead to a greener future a tad too early? Edit: pardon typos due to mobile device.
keithyjohnson 7y ago
Efficiency metrics would be really useful is evaluating DNNs for embedded solutions as well.
gwern 7y ago
This is ultimately a silly and misguided proposal, which focuses on the costs and not benefits. There is nothing special about training an AI; it is simply another thing to spend resources on, no intrinsically worse or better than any other way. A biology experiment has 'carbon footprint'. The ITER has a 'carbon footprint' (and probably one orders of magnitude larger than all AI research this year). HN burns CO2. Everything uses energy or it uses things which require energy, like human labor, and they involve other costs like opportunity cost which are just as real and important. They stand or fall on their net merits, not how much electricity they use. There is no need to go around criticizing people for 'green AI' by myopically focusing solely on an abstract electrical cost of training. And if there is, then that applies to everything which uses electricity, and is better handled by putting a carbon tax on energy sources, and letting the market find the most unprofitable uses of energy (which will probably not be AI research, I'll tell you that...) and stop it and substitute in more 'green' power sources for everything else. More importantly, if you are concerned about the costs of training AI, you should be concerned about the total costs as compared to the total benefits, not slicing out a completely arbitrary subset of costs and ranting about how many 'cars' it is equivalent to (which is not even strictly true in the first place considering that many data centers are located near cheap and renewable power like hydropower or nuclear power plants!) and shrugging away the issue that people consider these performance gains important and well-worth paying for. There are costs to a model which is worse than it could be. There are costs to models which run slower at deployment time even if they are faster to train. There are costs to models which cannot be used for transfer learning (as the criticized language models excel at, incidentally). And so on. What matters are the total costs, and corporations and researchers already pay considerable attention to that already. (Not a single one of their metrics - 'carbon emission', 'electricity usage', 'elapsed real time', 'number of parameters', 'FPO' - is an actual total cost!)
- 6gvONxR4sf7o 7y ago
  >This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. [emphasis mine] They suggest reporting cost metrics next to benefit metrics so readers can judge the tradeoff for themselves. There's no proposal to ignore the benefits.
esotericn 7y ago
This quote amused me: > deep learning was inspired by the human brain, which is remarkably energy efficient Yeah, sure, if you purely look at its' energy consumption in isolation and ignore all of the flights of fancy we engage in (such as, for example, this NN model) in order to maintain its' coherence.

Green AI

104 points by montalbano 7y ago

malux85 7y ago

This reminds me of that Simpsons quote "We can't fix your heart, but we can tell you exactly how damaged it is"

swalsh 7y ago
I'm not sure about that. I think you get what you measure, and if you start measuring effeciency, you're going to start seeing a major incentive to make it more effecient.
- Nasrudith 7y ago
  I agree that it is important but think it is more understanding is a prerequisite to do so deliberately (careful of Goodhart's law) and from there the incentives become evident. Like Lanchester's laws. It is obvious how outnumbering the foe can help but knowing the scaling really hammers home how important it is to concentrate force against as weak a foe groups as possible. https://en.m.wikipedia.org/wiki/Lanchester%27s_laws https://en.m.wikipedia.org/wiki/Lanchester%27s_laws
- malux85 7y ago
  Yes the first step to improvement is quantifying your objective, but it would have been nice to see some actual improvements and concrete implementations. The quantifying part is easy.

ausbah 7y ago

Like the abstract mentioned, I think this is also a good criteria for helping "level the playing field" for groups with lower budgets so the phenomenon of simply throwing more computational resources at a problem with an existing resource becomes less of a novelty.

remon 7y ago
Hm, I struggle to see an upside with levelling the playing field that way. Groups that have the budget to throw huge amounts of resources at problems are still providing important insights. That can happen in parallel to optimising wattage/computational unit. In fact, that can be an almost completely parallel track in AI research.
- currymj 7y ago
  right now such a parallel track doesn't really exist, at least not on an even footing. as the article shows, there's still a heavy focus on beating accuracy benchmarks, at huge computational cost, in terms of what actually gets accepted at major venues. it can be quite difficult to make the case for a method that has worse performance, but is cheaper. hard to judge without real data but I think such papers are much less likely to be accepted. there are also sometimes demands to replicate very expensive techniques as baselines, which can be onerous for groups with limited resources.
- Nasrudith 7y ago
  The upside is probably opportunity cost related essentially like mainframes of yore. Technically they could handle higher throughput than the workstations at the start and even today but once developed they became a dead end compared to more parallel approaches. That sort of quantity vs quality thing has happened a lot throughout history. The "VHS beats Betamax" moments that goes way further back but with more obscure examples.

taneq 7y ago

> The computations required for deep learning research have been doubling every few months I feel this is misleading. Computation is cheap, so the computation thrown at deep learning research has been doubling every few months, but that doesn't mean it's required to do research (unless your research is "throw huge datasets at a neural net architecture and see if it sticks".)

atoav 7y ago
I understood it as “computation required to replicate stuff outlined in a paper has doubled” not as “we do more research so we do more computation”.
mattkrause 7y ago
That...does seem to be a lot of the current research though.

mark_l_watson 7y ago

I like this idea! In last week’s Lex Firdman AI interview, Gary Marcus touches on the low twenty watt energy requirements of the human brain compared to deep learning energy requirements. Basically adding energy requirements to the loss function of automated neural model architecture search seems like a good idea also. (I am thinking of frameworks like AdaNet, etc.) I retired this year but I still spend a lot of time reviewing deep learning and also conventional AI literature (and I do tiny consulting projects to help people get started or build prototypes). Since I now mostly pay for my own computer resources I try to mostly limit myself to what I can do on my System 76 laptop with a single 1070 GPU. The availability of pretrained models makes this not so bad, at all. I really appreciate efforts by huggingface (and other organizations) of offering reduced size models that still provide good results.

AndrewKemendo 7y ago
The math is just wrong though. To train a brain to be competent in a classification task it takes 20W rms for years on end for the individual + all of the wattage from the parents, grandparents, teachers etc... that are training the individual for those years. Very hard to determine the power allocation used for a specific human being trained on a narrow task for eg. object classification, but that doesn't mean it's not the comparable measure. Comparing training of a single model to the "instant power" draw of the brain is not just over simplistic, the scale and time periods are wrong.
- mirrorsedge 7y ago
  I agree. Using watts, which are energy (joules) per unit time (second), to measure the energy consumption of training a model is the wrong unit to begin with. One cannot measure the total energy consumption of something without knowing both the wattage and the time spent running at that voltage. While training a neural network is often done from scratch or a pre-trained model with a known training time, the brain of a human being does not start developing from zero their birth. Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. It would be very hard but also very interesting to approximate the energy spent on this, but it may not be relevant to machine learning as the processing power scales and time periods are completely different than those involved in the development of the human brain.

CoffeePython 7y ago

I find it strange that the part of the conversation where AI has the potential to save thousands of hours of human labor doesn’t show up more often in these types of threads. That being said, we need to definitely be thinking about lowering the cost (environmental and monetary) of training these models. I’m glad research is being done within this domain. I’d love to see a study on what the human labor cost potential vs. training environmental costs would be for certain large models.

rahkiin 7y ago
I find it interesting that you see 'save thousands of hours of human labor' as something that's surely positive. What should those people start doing instead? They also need an income to get bread on the table.
- Nasrudith 7y ago
  Apply it backwards - would a welfare system of having people complete large N Tower of Hanoi locks that pay people money be a good idea? On one hand technically it is better than nothing immediately for those in need and is fraud resistant in that it consumes time and any better paying job without major negative externalities would be a better use of their time. The negative is that it does nothing productive and those who are dependent on it aren't able to use their time productively on anything else. While the problem of lack of income and lack of good uses for their labor certainly should be addressed just because it provides income doesn't make it a good thing overall. It is one of those things which puts the "dismal" in the "dismal science" term for economics - regardless of any misgivings over calling it a science. It is essentially a subset of human organizational and incentive problems which the economy "is designed" (technically more emergant even if it is literally a planned economy) to solve.
  - jkp56 7y ago
    What do you mean by "productive"?
- deleted 7y ago
  [deleted]

css 7y ago

The citation for the "surprisingly large carbon footprint" [0] is crazy. The paper alleges that a car, including fuel, has a lifetime footprint of 126,000 lbs of CO₂e, and training a big NN transformer consumes 626,155 lbs of CO₂e, almost a 5-fold increase. [0] https://arxiv.org/pdf/1906.02243.pdf https://arxiv.org/pdf/1906.02243.pdf

tsbinz 7y ago
Training it _with neural architecture search_. Just training it with a given architecture they cite as 192lbs ...
- dharma1 7y ago
  I thought Google etc datacentres run on mostly renewable energy?

sriku 7y ago

Research that demonstrates, say, 1000x reduction in power for training a known problem (ex: mnist) won't be considered irrelevant by the community. So is there a specific need to bias against large works apart from the carbon footprint argument? There remain questions to be answered in that space too - such as are current "neural" architectures adequate to cover the capabilities of the brain when upping only the scale? It was certainly worth knowing that scaling up was all that was required to compete with humans in DOTA. But will we hit wall as we near human level complexity? After all, the money spent on making movie which is "just" for entertainment trump's multi million $ deep learning experiments in costs which I guess have some correlation with the carbon footprint .. or the gases emitted in rocket launches. Do we really know whether this well intentioned call for green AI (which I right as hell want) will do too little for the greenness while biasing people against possible discoveries that can lead to a greener future a tad too early? Edit: pardon typos due to mobile device.

keithyjohnson 7y ago

Efficiency metrics would be really useful is evaluating DNNs for embedded solutions as well.

gwern 7y ago

This is ultimately a silly and misguided proposal, which focuses on the costs and not benefits. There is nothing special about training an AI; it is simply another thing to spend resources on, no intrinsically worse or better than any other way. A biology experiment has 'carbon footprint'. The ITER has a 'carbon footprint' (and probably one orders of magnitude larger than all AI research this year). HN burns CO2. Everything uses energy or it uses things which require energy, like human labor, and they involve other costs like opportunity cost which are just as real and important. They stand or fall on their net merits, not how much electricity they use. There is no need to go around criticizing people for 'green AI' by myopically focusing solely on an abstract electrical cost of training. And if there is, then that applies to everything which uses electricity, and is better handled by putting a carbon tax on energy sources, and letting the market find the most unprofitable uses of energy (which will probably not be AI research, I'll tell you that...) and stop it and substitute in more 'green' power sources for everything else. More importantly, if you are concerned about the costs of training AI, you should be concerned about the total costs as compared to the total benefits, not slicing out a completely arbitrary subset of costs and ranting about how many 'cars' it is equivalent to (which is not even strictly true in the first place considering that many data centers are located near cheap and renewable power like hydropower or nuclear power plants!) and shrugging away the issue that people consider these performance gains important and well-worth paying for. There are costs to a model which is worse than it could be. There are costs to models which run slower at deployment time even if they are faster to train. There are costs to models which cannot be used for transfer learning (as the criticized language models excel at, incidentally). And so on. What matters are the total costs, and corporations and researchers already pay considerable attention to that already. (Not a single one of their metrics - 'carbon emission', 'electricity usage', 'elapsed real time', 'number of parameters', 'FPO' - is an actual total cost!)

6gvONxR4sf7o 7y ago
>This position paper advocates a practical solution by making efficiency an evaluation criterion for research alongside accuracy and related measures. In addition, we propose reporting the financial cost or "price tag" of developing, training, and running models to provide baselines for the investigation of increasingly efficient methods. [emphasis mine] They suggest reporting cost metrics next to benefit metrics so readers can judge the tradeoff for themselves. There's no proposal to ignore the benefits.

esotericn 7y ago

This quote amused me: > deep learning was inspired by the human brain, which is remarkably energy efficient Yeah, sure, if you purely look at its' energy consumption in isolation and ignore all of the flights of fancy we engage in (such as, for example, this NN model) in order to maintain its' coherence.