4 ms·

One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor whi

One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a huge power budget to get anywhere close to a brain; this estimate [2] puts training GPT-3 at about a TWh, which is about six million years' of power for a single brain. [1] https://www.pnas.org/doi/10.1073/pnas.2107022118 https://www.pnas.org/doi/10.1073/pnas.2107022118 [2] https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-planet/#:~:text=One%20popular%20model%2C%20GPT%2D3,GPT%2D3%20consumed%20936%20MWh https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-pl....

dsign 4y ago
Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or playing Go. Have you ever talked to a System's Biologist? Ask them about how anything works, and they'll start with 'oh, it's so complicated. You have X, and then Y, and then Z, and nobody knows about W. And then how they work together? Madness!"
- deleted 4y ago
  [deleted]
- TeMPOraL 4y ago
  > Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally). I also imagine it wouldn't be larger than six thousand people specifically raised and taught in coordinated fashion to fulfill this role. On the other other hand, a human brain does a lot more than learning language, ideas and conversion between one and the other. It also, simultaneously, learns a lot of video and audio processing, not to mention smell, proprioception, touch (including pain and temperature), and... a bunch of other stuff (I was actually surprised by the size of the "human" part of the Wikipedia infobox here: https://en.wikipedia.org/wiki/Template:Sensation_and_perception https://en.wikipedia.org/wiki/Template:Sensation_and_percept...). So it's probably hard to compare to specialized NN models until we learn to better classify and isolate how biological brains learn and encode all the various things they do.
  - IIsi50MHz 4y ago
    > On the other hand, > … > On the other other hand, See: "on the gripping hand" (-:
swyx 4y ago
> a TWh, which is about six million years' of power i'm not a physics guy but wanted to check this - a Watt is a per second measure, and a Terawatt is 1 trillion watts, so 1 TWh is 50 billion seconds of 20 Watts, which is 1585 years of power for a single brain, not 6 million. i'm sure i got this wrong as i'm not a physics guy but where did i go wrong here? a more neutral article (that doesn't have a clear "AI is harming our planet" agenda) estimates closer to 1404MWh to train GPT3: https://blog.scaleway.com/doing-ai-without-breaking-the-bank-yours-or-the-planets/ https://blog.scaleway.com/doing-ai-without-breaking-the-bank... i dont know either way but i'd like as best an estimate as possible since this seems an important number.
- belval 4y ago
  A watt is not a per-second measure, Wh are the energy measurement to W being the power. TWh = 10^12Wh which means a trillion-watt for 1 hour. 10^12 / 20 (power of brain) / 24 (hours in a day) / 365 (days in a year) = 5 707 762 years.
  - pfdietz 4y ago
    One watt is one joule per second. What exactly do you mean by "a per-second measure"?
    - belval 4y ago
      The seconds are not a denominator, OP was doing TWh / W as if W = 1 / 3600 * Wh which is not the case.
- primis 4y ago
  Watts are actually a time independent measurement, note that the TWh has "hour" affixed to the end. This is 1 Tera Watt over the course of one hour, not one second. Your numbers are off by a factor of 3600. 1TWh / 20 Watt brain = 50,000,000,000 (50 Billion) Hours. 50 Billion Hours / (24h * 365.25) = 5,703,855.8 Years
  - 4y ago
gnfargbl 4y ago
I can't edit my post any more, but I'm a moron: the article says about a GWh, not a TWh. So the calculation is out by a factor of 1000.
- deleted 4y ago
  [deleted]

dsign 4y ago

Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or playing Go. Have you ever talked to a System's Biologist? Ask them about how anything works, and they'll start with 'oh, it's so complicated. You have X, and then Y, and then Z, and nobody knows about W. And then how they work together? Madness!"

deleted 4y ago
[deleted]
TeMPOraL 4y ago
> Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally). I also imagine it wouldn't be larger than six thousand people specifically raised and taught in coordinated fashion to fulfill this role. On the other other hand, a human brain does a lot more than learning language, ideas and conversion between one and the other. It also, simultaneously, learns a lot of video and audio processing, not to mention smell, proprioception, touch (including pain and temperature), and... a bunch of other stuff (I was actually surprised by the size of the "human" part of the Wikipedia infobox here: https://en.wikipedia.org/wiki/Template:Sensation_and_perception https://en.wikipedia.org/wiki/Template:Sensation_and_percept...). So it's probably hard to compare to specialized NN models until we learn to better classify and isolate how biological brains learn and encode all the various things they do.
- IIsi50MHz 4y ago
  > On the other hand, > … > On the other other hand, See: "on the gripping hand" (-:

swyx 4y ago

> a TWh, which is about six million years' of power i'm not a physics guy but wanted to check this - a Watt is a per second measure, and a Terawatt is 1 trillion watts, so 1 TWh is 50 billion seconds of 20 Watts, which is 1585 years of power for a single brain, not 6 million. i'm sure i got this wrong as i'm not a physics guy but where did i go wrong here? a more neutral article (that doesn't have a clear "AI is harming our planet" agenda) estimates closer to 1404MWh to train GPT3: https://blog.scaleway.com/doing-ai-without-breaking-the-bank-yours-or-the-planets/ https://blog.scaleway.com/doing-ai-without-breaking-the-bank... i dont know either way but i'd like as best an estimate as possible since this seems an important number.

belval 4y ago
A watt is not a per-second measure, Wh are the energy measurement to W being the power. TWh = 10^12Wh which means a trillion-watt for 1 hour. 10^12 / 20 (power of brain) / 24 (hours in a day) / 365 (days in a year) = 5 707 762 years.
- pfdietz 4y ago
  One watt is one joule per second. What exactly do you mean by "a per-second measure"?
  - belval 4y ago
    The seconds are not a denominator, OP was doing TWh / W as if W = 1 / 3600 * Wh which is not the case.
primis 4y ago
Watts are actually a time independent measurement, note that the TWh has "hour" affixed to the end. This is 1 Tera Watt over the course of one hour, not one second. Your numbers are off by a factor of 3600. 1TWh / 20 Watt brain = 50,000,000,000 (50 Billion) Hours. 50 Billion Hours / (24h * 365.25) = 5,703,855.8 Years
- 4y ago

gnfargbl 4y ago

I can't edit my post any more, but I'm a moron: the article says about a GWh, not a TWh. So the calculation is out by a factor of 1000.

deleted 4y ago
[deleted]