5 ms·

Human mind (aka brain) has consumed little over 6000 kWh energy when it's 35. From the paper they cite algorithm kWh-PUE ---------- ------- E

Human mind (aka brain) has consumed little over 6000 kWh energy when it's 35. From the paper they cite algorithm kWh-PUE ---------- ------- ELMo 275 BERT(base) 1,507 NAS 656,347 NAS was Transformer (213M parameters) for neural architecture search 2019. They didn't have kWh numbers for GPT-2.

sbierwagen 6y ago
I don't know about that 6000 KWh number. More importantly, a human brain isn't trained from scratch at birth. It's had a few hundred million years of pretraining.
- wongarsu 6y ago
  The 6000kWh number is assuming 20W over 35 years. That might be right if you only count the brain, but then you would have to revise the neural network numbers to only count CPU and memory, which would easily halve those numbers (fans take a lot of power). If we want to account for power supply, temperature control etc. we could use the entire calorie count of a human over 35 years, leaving us with around 30000kWh (assuming a health, not-overweight human). You could now argue that that's too high (the human is doing a lot of other things). But as you pointed out it's a bit of a pointless comparison anyways as humans don't start from scratch.
  - antisthenes 6y ago
    > we could use the entire calorie count of a human over 35 years, leaving us with around 30000kWh (assuming a health, not-overweight human) That's still too low, considering you can't just have a human eating food and educating themselves in a vacuum. We need to account for energy going to and from school, shelter, heating, clothing. Basically all the necessities that enable someone to learn (and their professors) That's another 5x at a minimum.
    - sifar 6y ago
      For the NN architectures, you'll have to factor in there production, design, the energy of the humans involved and so on and on. This doesn't take you anywhere.

sbierwagen 6y ago

I don't know about that 6000 KWh number. More importantly, a human brain isn't trained from scratch at birth. It's had a few hundred million years of pretraining.

wongarsu 6y ago
The 6000kWh number is assuming 20W over 35 years. That might be right if you only count the brain, but then you would have to revise the neural network numbers to only count CPU and memory, which would easily halve those numbers (fans take a lot of power). If we want to account for power supply, temperature control etc. we could use the entire calorie count of a human over 35 years, leaving us with around 30000kWh (assuming a health, not-overweight human). You could now argue that that's too high (the human is doing a lot of other things). But as you pointed out it's a bit of a pointless comparison anyways as humans don't start from scratch.
- antisthenes 6y ago
  > we could use the entire calorie count of a human over 35 years, leaving us with around 30000kWh (assuming a health, not-overweight human) That's still too low, considering you can't just have a human eating food and educating themselves in a vacuum. We need to account for energy going to and from school, shelter, heating, clothing. Basically all the necessities that enable someone to learn (and their professors) That's another 5x at a minimum.
  - sifar 6y ago
    For the NN architectures, you'll have to factor in there production, design, the energy of the humans involved and so on and on. This doesn't take you anywhere.