6 ms·

I like this idea! In last week’s Lex Firdman AI interview, Gary Marcus touches on the low twenty watt energy requirements of the human brain compared to deep le

I like this idea! In last week’s Lex Firdman AI interview, Gary Marcus touches on the low twenty watt energy requirements of the human brain compared to deep learning energy requirements. Basically adding energy requirements to the loss function of automated neural model architecture search seems like a good idea also. (I am thinking of frameworks like AdaNet, etc.) I retired this year but I still spend a lot of time reviewing deep learning and also conventional AI literature (and I do tiny consulting projects to help people get started or build prototypes). Since I now mostly pay for my own computer resources I try to mostly limit myself to what I can do on my System 76 laptop with a single 1070 GPU. The availability of pretrained models makes this not so bad, at all. I really appreciate efforts by huggingface (and other organizations) of offering reduced size models that still provide good results.

AndrewKemendo 7y ago
The math is just wrong though. To train a brain to be competent in a classification task it takes 20W rms for years on end for the individual + all of the wattage from the parents, grandparents, teachers etc... that are training the individual for those years. Very hard to determine the power allocation used for a specific human being trained on a narrow task for eg. object classification, but that doesn't mean it's not the comparable measure. Comparing training of a single model to the "instant power" draw of the brain is not just over simplistic, the scale and time periods are wrong.
- mirrorsedge 7y ago
  I agree. Using watts, which are energy (joules) per unit time (second), to measure the energy consumption of training a model is the wrong unit to begin with. One cannot measure the total energy consumption of something without knowing both the wattage and the time spent running at that voltage. While training a neural network is often done from scratch or a pre-trained model with a known training time, the brain of a human being does not start developing from zero their birth. Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. It would be very hard but also very interesting to approximate the energy spent on this, but it may not be relevant to machine learning as the processing power scales and time periods are completely different than those involved in the development of the human brain.
  - Invictus0 7y ago
    > Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. That's completely ridiculous. It's like saying that to measure the energy used by a computer on a task, you have to start by measuring the entire energy usage of the human race since the invention of the transistor.
- Invictus0 7y ago
  8 year olds can accomplish pretty much any classification task. 20W * 3.154E7 seconds in a year * 8 years / 3600 seconds per hour gives about 70 kilowatt-hours, which would cost about $14 from the grid at $0.20/kwh. Our AI is nowhere near the intelligence of an 8 year old, who can presumably also do more things than just decide if a photo depicts a bus or an avocado.
  - Tade0 7y ago
    I think your math is off. 8 years at 20W is: 0.02kW * 24h * 365.25 * 8 = 1 402.56kWh
    - Invictus0 7y ago
      Yup, you're correct, forgot to multiply by the 20W. Still only about 1.5X average American monthly electricity usage.
  - AndrewKemendo 7y ago
    Our AI is nowhere near the intelligence of an 8 year old, who can presumably also do more things than just decide if a photo depicts a bus or an avocado. Indeed, but at least you describe a slightly more accurate possible representation.
- Tade0 7y ago
  Also it's important to note that the energy consumed by the brain is really really dirty, considering it's taken from sugars, which require farming to produce.
  - romaniv 7y ago
    I'm willing to bet this will be used as one of the arguments in a corporate push for global automation in ~10 years.
dharma1 7y ago
Don't forget biological brain "neural architecture search" has also had a significant time and energy cost - billions of years of evolution, powered by a pretty large fusion reactor for those billions of years. That's the reason brains are so energy efficient today. Though you are right, energy constraints are built in with biological evolutionary search
mlthoughts2018 7y ago
Back when I used to frequently visit Less Wrong, I made a post about this related to Watson from IBM. I remember feeling really discouraged by the level of discourse and downvoting in the comments at that time. https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son
- gwern 7y ago
  That's too bad, because your comments there are really bad. You argue solely by assertion and personal feeling, and then wonder why no one agrees with you. Why does it matter in the least that you are not 'impressed' or feel using more than 1 server 'waters it down'? Why does it matter how much IBM Watson uses? AlphaGo uses more electricity than any human; does it not win at Go? If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter? Is there some law of physics I am unaware of which stipulates that no computer is allowed to use more than 150 watts, or that we are unable to build systems which have more than 1 computer involved? Is there some theorem in computer science which proves that AI isn't real AI unless it solves problems in exactly the same way as a human brain at exactly the same efficiency, never mind how well it solves problems or how much superior to humans it may be otherwise? ('Jet planes, you see, do not actually provide "artificial flight" because they burn more joules than a bird would.') Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity? Have deep learning techniques not continued to scale over the 8 years since you posted that? IBM Watson failed; did it have anything to do with using 15TB RAM; would Watson now be a success and closer to 'real AI' if they had hand-optimized its routines for memory efficiency to do the same thing but only needing 1.5TB RAM?
  - mlthoughts2018 7y ago
    > “AlphaGo uses more electricity than any human; does it not win at Go?” I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. If you present me a Go-based Turing test, and I’m pretty fooled by AlphaGo, except I get blasted by the hot air venting out from a big cooling system, then it probably reveals to me that it’s failed the test (at least until some far future where there are life forms that regularly vent computer-levels of hot air from cooling systems). This effect is much more pronounced with a larger system like Watson, hence the article. > “If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter?” It matters because pure computing achievements are not isolated from their resource constraints. If a NN can generate flawless natural language responses, but it takes 1000 years per word, we would not be interested and certainly would not see it as valuable progress towards artificial intelligence (likely not even strong AI, but surely not tool AI, there just aren’t a lot of ways to demonstrate a conceptual breakthrough so wasteful of resources). Intelligence is (partly) generating language within human-level time constraints. Why not also human level RAM, physical volume, material density, etc. Solving it within these constraints is much more valuable than solving it without them. Likewise with power. Maybe using 100x power is an OK tradeoff for pure tool usage (heck, I work professionally in computer vision and NLP and I make this concession to use machine learning models running on computers to solve my work daily). But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. > “Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity?” Huh? Where did I ever say anything like this? I mean, humans misuse intelligence and cause problems on less power. It sounds like you’re trying to propagate your feelings about the “green” aspect of the post into what I’ve said, but if you’ll notice, I did not say anything about that, and don’t particularly care about this OP article’s take on “green” AI power consumption, it’s not related at all to what I’m talking about. (Plus they are more focused on power demands at training time, while I’m interested in power constraints, like runtime or space constraints, at inference time).
m0zg 7y ago
If you do consulting, you should consider using pre-tax money to buy whatever hardware you need for work. Trump made it much easier to amortize business expenses up to a certain limit, and this could save you a lot of money if you're in the higher tax brackets. I spent ~16K on deep learning hardware for my business this year so far (consumer GPUs, since I'm not running them in the "datacenter") because I'm not made of money and think current cloud GPU pricing is a rip-off.

AndrewKemendo 7y ago

The math is just wrong though. To train a brain to be competent in a classification task it takes 20W rms for years on end for the individual + all of the wattage from the parents, grandparents, teachers etc... that are training the individual for those years. Very hard to determine the power allocation used for a specific human being trained on a narrow task for eg. object classification, but that doesn't mean it's not the comparable measure. Comparing training of a single model to the "instant power" draw of the brain is not just over simplistic, the scale and time periods are wrong.

mirrorsedge 7y ago
I agree. Using watts, which are energy (joules) per unit time (second), to measure the energy consumption of training a model is the wrong unit to begin with. One cannot measure the total energy consumption of something without knowing both the wattage and the time spent running at that voltage. While training a neural network is often done from scratch or a pre-trained model with a known training time, the brain of a human being does not start developing from zero their birth. Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. It would be very hard but also very interesting to approximate the energy spent on this, but it may not be relevant to machine learning as the processing power scales and time periods are completely different than those involved in the development of the human brain.
- Invictus0 7y ago
  > Measuring the amount of energy spent on a specific human being trained on a task would also have to account for the billions of years of evolution that lead up to the present-day structure of the human brain. That's completely ridiculous. It's like saying that to measure the energy used by a computer on a task, you have to start by measuring the entire energy usage of the human race since the invention of the transistor.
Invictus0 7y ago
8 year olds can accomplish pretty much any classification task. 20W * 3.154E7 seconds in a year * 8 years / 3600 seconds per hour gives about 70 kilowatt-hours, which would cost about $14 from the grid at $0.20/kwh. Our AI is nowhere near the intelligence of an 8 year old, who can presumably also do more things than just decide if a photo depicts a bus or an avocado.
- Tade0 7y ago
  I think your math is off. 8 years at 20W is: 0.02kW * 24h * 365.25 * 8 = 1 402.56kWh
  - Invictus0 7y ago
    Yup, you're correct, forgot to multiply by the 20W. Still only about 1.5X average American monthly electricity usage.
- AndrewKemendo 7y ago
  Our AI is nowhere near the intelligence of an 8 year old, who can presumably also do more things than just decide if a photo depicts a bus or an avocado. Indeed, but at least you describe a slightly more accurate possible representation.
Tade0 7y ago
Also it's important to note that the energy consumed by the brain is really really dirty, considering it's taken from sugars, which require farming to produce.
- romaniv 7y ago
  I'm willing to bet this will be used as one of the arguments in a corporate push for global automation in ~10 years.

dharma1 7y ago

Don't forget biological brain "neural architecture search" has also had a significant time and energy cost - billions of years of evolution, powered by a pretty large fusion reactor for those billions of years. That's the reason brains are so energy efficient today. Though you are right, energy constraints are built in with biological evolutionary search

mlthoughts2018 7y ago

Back when I used to frequently visit Less Wrong, I made a post about this related to Watson from IBM. I remember feeling really discouraged by the level of discourse and downvoting in the comments at that time. https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son

gwern 7y ago
That's too bad, because your comments there are really bad. You argue solely by assertion and personal feeling, and then wonder why no one agrees with you. Why does it matter in the least that you are not 'impressed' or feel using more than 1 server 'waters it down'? Why does it matter how much IBM Watson uses? AlphaGo uses more electricity than any human; does it not win at Go? If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter? Is there some law of physics I am unaware of which stipulates that no computer is allowed to use more than 150 watts, or that we are unable to build systems which have more than 1 computer involved? Is there some theorem in computer science which proves that AI isn't real AI unless it solves problems in exactly the same way as a human brain at exactly the same efficiency, never mind how well it solves problems or how much superior to humans it may be otherwise? ('Jet planes, you see, do not actually provide "artificial flight" because they burn more joules than a bird would.') Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity? Have deep learning techniques not continued to scale over the 8 years since you posted that? IBM Watson failed; did it have anything to do with using 15TB RAM; would Watson now be a success and closer to 'real AI' if they had hand-optimized its routines for memory efficiency to do the same thing but only needing 1.5TB RAM?
- mlthoughts2018 7y ago
  > “AlphaGo uses more electricity than any human; does it not win at Go?” I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. If you present me a Go-based Turing test, and I’m pretty fooled by AlphaGo, except I get blasted by the hot air venting out from a big cooling system, then it probably reveals to me that it’s failed the test (at least until some far future where there are life forms that regularly vent computer-levels of hot air from cooling systems). This effect is much more pronounced with a larger system like Watson, hence the article. > “If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter?” It matters because pure computing achievements are not isolated from their resource constraints. If a NN can generate flawless natural language responses, but it takes 1000 years per word, we would not be interested and certainly would not see it as valuable progress towards artificial intelligence (likely not even strong AI, but surely not tool AI, there just aren’t a lot of ways to demonstrate a conceptual breakthrough so wasteful of resources). Intelligence is (partly) generating language within human-level time constraints. Why not also human level RAM, physical volume, material density, etc. Solving it within these constraints is much more valuable than solving it without them. Likewise with power. Maybe using 100x power is an OK tradeoff for pure tool usage (heck, I work professionally in computer vision and NLP and I make this concession to use machine learning models running on computers to solve my work daily). But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. > “Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity?” Huh? Where did I ever say anything like this? I mean, humans misuse intelligence and cause problems on less power. It sounds like you’re trying to propagate your feelings about the “green” aspect of the post into what I’ve said, but if you’ll notice, I did not say anything about that, and don’t particularly care about this OP article’s take on “green” AI power consumption, it’s not related at all to what I’m talking about. (Plus they are more focused on power demands at training time, while I’m interested in power constraints, like runtime or space constraints, at inference time).

m0zg 7y ago

If you do consulting, you should consider using pre-tax money to buy whatever hardware you need for work. Trump made it much easier to amortize business expenses up to a certain limit, and this could save you a lot of money if you're in the higher tax brackets. I spent ~16K on deep learning hardware for my business this year so far (consumer GPUs, since I'm not running them in the "datacenter") because I'm not made of money and think current cloud GPU pricing is a rip-off.