3 ms·

Isn't it disconcerting that brains consume so little energy doing continuous high complexity tasks like visual processing and even resting produce vivid simulat

Isn't it disconcerting that brains consume so little energy doing continuous high complexity tasks like visual processing and even resting produce vivid simulations, while numeric calculations require so much effort for advanced brains? It's like the energy usage pattern is completely opposite between brains and computers. That should tell something about our approach to intelligence by using, maybe, the wrong tool?

drdaeman 2y ago
> brains consume so little energy doing continuous high complexity tasks Do they? While I've read that there's not a lot of difference between just chilling and thinking of some hard problem (not sure if true, though - haven't really dug to verify), thinking is thinking, so I guess that makes sense. But when e.g. a reaction time is important, I suspect, it could be quite energy-hungry. I mean, my watch sends me that I have elevated heart rate when I'm playing any fast-paced videogames, and I guess all that blood pumping surely has some biological reason (supplying more oxygen or something) and isn't just for funsies. I'm not a biologist, so this is purely based on my own observations and stuff I've heard somewhere.
- eric-hu 2y ago
  > While I've read that there's not a lot of difference between just chilling and thinking of some hard problem I find that difficult to reconcile with chess grandmaster calorie consumption. https://www.npr.org/2019/09/18/762046422/the-chess-grandmasters-diet https://www.npr.org/2019/09/18/762046422/the-chess-grandmast... > Polard, this company that tracks heart rates, monitored chess players during a tournament and found out that this 21-year-old Russian grandmaster, Mikhail Antipov, had burned 560 calories in two hours, which we found out was roughly what Roger Federer would burn in one hour of singles tennis.
  - caseyy 2y ago
    I notice this in software engineering, too. Challenging problems drain brainpower quicker. I was working on difficult SOTA stuff for the last few months, half of it was reading articles and half was implementing them in software. I’d crash by 3.30-4 pm every day unless I had an extra meal around then. Lost 5 kilos, too. So it wasn’t all going into my waistline. You can observe this, I think. Drivers too tend to make more mistakes if they don’t eat well on long drives.
interstice 2y ago
Degrees of entropy perhaps? CPU's are low entropy in that they are highly ordered and organised - firing only when specifically expected to, whereas brains are high entropy in that they get to a result by riding a fine line between randomness and order and those connections are not inherently expensive. That and billions of years of optimising for low energy switches.
- readyplayernull 2y ago
  Yeah, there could be something. Computers fight entropy to gain precision, so intelligence should benefit of fuzzyness to get the "free energy meal". But that brings a sort of uncertainty principle. We might be able to grasp what intelligence is, but like a cloud of electrons we will never be sure of its detailed state.
  - burnished 2y ago
    Think that 'grasp' in 'grasp what intelligence is' might be load bearing there.
cen4 2y ago
Babies don't come installed with those brains. They take a decade or two to pick up most skills that have any value. And after all that, the vast majority end up playing replaceable support npc roles. What's the energy consumption? To get from - why is the compass pointing north to electromagnetism, it took thousands of years and the activity of millions of brains. We see a highly misleading and compressed version of that computing process in our textbooks. As they say knowledge is inherited wealth. People over estimate how much they deserve it or have earned it.
- visarga 2y ago
  A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes about 51,395,437,500,000,000,000 tokens, about 3.4 million times more than GPT-4's training run, and 342M times more than a child needs to learn. That is to show the relative cost between imitation and discovery. Evolution is on the order of 0.3 trillion times slower than catching up by imitation.
  - defrost 2y ago
    > Humanity needed the total speech of 110B people to evolve to current level Seems a loooong stretch to say the least.
    - visarga 2y ago
      I find it interesting to estimate on tokens so we can meaningfully compare people with LLMs.
      - defrost 2y ago
        It's more the implied dependancy tree ... did Isaac Newton rest on the shoulders of 110 B people to conceive of calculus just as Liebnitz did the same? How much input did the Kalahari bushmen of that time, and their ancestors also, have to that infintesimal leap forwards?
Retric 2y ago
We don’t hyper optimize computers for energy use the way evolution did. But even still the brain uses about as much energy as 4 Apple M3 chips which isn’t nothing in terms of computer AI. Multiply that by years of training time and it doesn’t seem that crazy different IMO.
- trilbyglens 2y ago
  I think more impressive is that the energy supply system is efficient enough to run a days worth of software generation off of a ham sandwich.
jordanb 2y ago
Whenever anyone points out that what LLMs are doing is definitely not how a brain works the AI people respond with "maybe it's going to be better!"
- teaearlgraycold 2y ago
  LLMs are reversing language from text. Language is as much an extension of the brain as the retina is. The words are arbitrary but the deeper levels are closer to rubbing your neurons on a listener’s ear drum. Why would a brute force simulation be anywhere as efficient as the original hardware?
  - caseyy 2y ago
    Human language seems to originate in grokking, in natural net terms. You first understand a concept, then you speak or write about it. When you hear or read something, it isn’t retained well unless you can formulate an understanding of it. So language is a communication interface, not thinking itself. LLMs seem to create an illusion of understanding largely because we believe that understanding is a prerequisite for language — whoever uses this interface must be thinking! But that really seems to only be the case in our brains, not in LLMs. Grokking in NNs is interesting though. It’s going to be pretty wild times when we crack that nut open. If we could drive grokking on a large data set, it’s difficult to even imagine the possibilities. Who knows what we could grokk about the universe, for example. But that is still something only human brains can do regularly and reasonably well in heavily-parametrized situations. And even we get overwhelmed so easily by too many variables. I do hope ML grokking becomes much better than human understanding over time. But I don’t think we are close, sadly. We have just barely scratched the surface in toy models.
caseyy 2y ago
You also have to consider what intelligence is useful. We already have human brains in abundance, but computers can do certain things we can’t. I think LLMs are a bit of a red herring. We will discover powerful ways to use machine learning that are very different from human brains. The goal doesn’t need to be to replace them.
planckscnst 2y ago
Computers aren't evolving in an energy-scarce environment (yet) we (currently) pretty much give computers as much energy as they want. Computers are evolving in an environment where a dependent species relies on them for that species' own weaknesses.
- deleted 2y ago
  [deleted]
NoPicklez 2y ago
Well the human approach to intelligence and its tool has had approx 6 million years to improve and evolve and has created the reality we experience today, which could be really intelligent behavior or not. We don't have anything to compare it to. Think of the size of the computers we had in the 1960's and their computing power, now you have that multiple times over on your wrist. The tools we use seem to get bigger and bigger, then smaller, then bigger until a new paradigm becomes available (e.g. quantum)
burnished 2y ago
I misread your statement as stating that computers broadly were less energy efficient at the same tasks, because I took umbridge at your claim that human brains use `so little energy` (citation needed???) and forgot how to reading comprehension, so I compared training costs of a 7 year old and inference costs for an adult in terms of energy to the figures quoted in the article for the various models. I'll address what you actually said below but am keeping my original writeup because its kind of neat. I'm going to compare energy costs for training a model vs growing a brain because they are analogous. TFA lists the training costs in kWh and I'll arrive at a comparable figure by estimating the total caloric intake of a 7 year old and then multiplying that by an estimate of the proportion of energy a child's brain uses. I calculate that the lifetime caloric intake of a 7 year old is 3,139,000 dietary calroies based on a daily schedule of 1,000 for the first three years and 1,400 for the next four[1], and assuming an active child's brain uses roughly 60% of total calories[2], we end up with a lifetime total of 1,883,400 calories spent on the brain. This is 2,188 kWh. >> the energy consumption of model training is somewhere in the range of 11,000 kWh to 52,000 kWh for moderate-sized LLMs, 40,000 kWh to 60,000 kWh for more expensive image generators, and up to 5,000,000 kWh for global-scale LLMs like Llama and ChatGPT. The 7 year old isn't even finished training, and frankly, needing the same training energy as 2,285 7 year olds to create ChatGPT comes across as a bargain. Just for a bonus lets compare inference costs, assuming a human brain uses roughly 200 dietary calories and that we can count all of them as part of performing productive work gives us a budget of about 0.5 kWh (464 Wh). Going to quote the article again for inference costs >> the energy consumption of an inference operation is somewhere in the range of 1.0 Wh to 7.0 Wh for standard operations, and to 4 Wh to 7.5 Wh for more expensive jobs like image or video generation. Hundreds of text operations and 50-100 images. As a back of the napkin sketch it looks pretty close. [1] https://www.healthychildren.org/English/healthy-living/nutrition/Pages/Energy-In-Recommended-Food-Drink-Amounts-for-Children.aspx https://www.healthychildren.org/English/healthy-living/nutri... [2] https://www.livescience.com/burn-calories-brain.html https://www.livescience.com/burn-calories-brain.html ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You're sorta onto something but it has to do with architecture: human brains are composed of a vast number of sparsely connected and individually poor computational units (neurons). Computers are completely connected and highly reliable but individual units are expensive. Neurons also communicate at a much slower pace than computers can perform calculations. The reason for our unreasonable effectiveness appears rooted in how very many neurons we have and how effectively parallelized most functions of the brain are. Precision calculations are easy for computers because they are literally machines that have operations that map to simple numerical calculations that are highly accurate. Numeric calculations are not simple for human brains because nothing really maps to the process of a calculation. Individual neurons do 'integrate and fire' meaning that if they get excited enough by nearby neurons in a short enough time then they will fire and excite the neurons they touch, but it isn't a reliable process, meaning that if neuron A touches neuron B and A activating is enough to cause B to activate this time it may not be enough the next time A activate. So when we do math we use those. There are investigations into neuromorphic hardware but I'm not up to date on them anymore, the basic idea is to make computing hardware inspired by how the brain operates. Last I knew there were some interesting results but nothing resembling anything practical.
axdsk 2y ago
The right tool might be Spiking Neural Networks. Due to their sparse activation, event driven computation, and temporal coding. This all depends on how good neuromorphic chips get.

drdaeman 2y ago

> brains consume so little energy doing continuous high complexity tasks Do they? While I've read that there's not a lot of difference between just chilling and thinking of some hard problem (not sure if true, though - haven't really dug to verify), thinking is thinking, so I guess that makes sense. But when e.g. a reaction time is important, I suspect, it could be quite energy-hungry. I mean, my watch sends me that I have elevated heart rate when I'm playing any fast-paced videogames, and I guess all that blood pumping surely has some biological reason (supplying more oxygen or something) and isn't just for funsies. I'm not a biologist, so this is purely based on my own observations and stuff I've heard somewhere.

eric-hu 2y ago
> While I've read that there's not a lot of difference between just chilling and thinking of some hard problem I find that difficult to reconcile with chess grandmaster calorie consumption. https://www.npr.org/2019/09/18/762046422/the-chess-grandmasters-diet https://www.npr.org/2019/09/18/762046422/the-chess-grandmast... > Polard, this company that tracks heart rates, monitored chess players during a tournament and found out that this 21-year-old Russian grandmaster, Mikhail Antipov, had burned 560 calories in two hours, which we found out was roughly what Roger Federer would burn in one hour of singles tennis.
- caseyy 2y ago
  I notice this in software engineering, too. Challenging problems drain brainpower quicker. I was working on difficult SOTA stuff for the last few months, half of it was reading articles and half was implementing them in software. I’d crash by 3.30-4 pm every day unless I had an extra meal around then. Lost 5 kilos, too. So it wasn’t all going into my waistline. You can observe this, I think. Drivers too tend to make more mistakes if they don’t eat well on long drives.

interstice 2y ago

Degrees of entropy perhaps? CPU's are low entropy in that they are highly ordered and organised - firing only when specifically expected to, whereas brains are high entropy in that they get to a result by riding a fine line between randomness and order and those connections are not inherently expensive. That and billions of years of optimising for low energy switches.

readyplayernull 2y ago
Yeah, there could be something. Computers fight entropy to gain precision, so intelligence should benefit of fuzzyness to get the "free energy meal". But that brings a sort of uncertainty principle. We might be able to grasp what intelligence is, but like a cloud of electrons we will never be sure of its detailed state.
- burnished 2y ago
  Think that 'grasp' in 'grasp what intelligence is' might be load bearing there.

cen4 2y ago

Babies don't come installed with those brains. They take a decade or two to pick up most skills that have any value. And after all that, the vast majority end up playing replaceable support npc roles. What's the energy consumption? To get from - why is the compass pointing north to electromagnetism, it took thousands of years and the activity of millions of brains. We see a highly misleading and compressed version of that computing process in our textbooks. As they say knowledge is inherited wealth. People over estimate how much they deserve it or have earned it.

visarga 2y ago
A child learns to talk in about 150M tokens, GPT-4 needs 15T tokens. Humanity needed the total speech of 110B people to evolve to current level, which comes about 51,395,437,500,000,000,000 tokens, about 3.4 million times more than GPT-4's training run, and 342M times more than a child needs to learn. That is to show the relative cost between imitation and discovery. Evolution is on the order of 0.3 trillion times slower than catching up by imitation.
- defrost 2y ago
  > Humanity needed the total speech of 110B people to evolve to current level Seems a loooong stretch to say the least.
  - visarga 2y ago
    I find it interesting to estimate on tokens so we can meaningfully compare people with LLMs.
    - defrost 2y ago
      It's more the implied dependancy tree ... did Isaac Newton rest on the shoulders of 110 B people to conceive of calculus just as Liebnitz did the same? How much input did the Kalahari bushmen of that time, and their ancestors also, have to that infintesimal leap forwards?

Retric 2y ago

We don’t hyper optimize computers for energy use the way evolution did. But even still the brain uses about as much energy as 4 Apple M3 chips which isn’t nothing in terms of computer AI. Multiply that by years of training time and it doesn’t seem that crazy different IMO.

trilbyglens 2y ago
I think more impressive is that the energy supply system is efficient enough to run a days worth of software generation off of a ham sandwich.

jordanb 2y ago

Whenever anyone points out that what LLMs are doing is definitely not how a brain works the AI people respond with "maybe it's going to be better!"

teaearlgraycold 2y ago
LLMs are reversing language from text. Language is as much an extension of the brain as the retina is. The words are arbitrary but the deeper levels are closer to rubbing your neurons on a listener’s ear drum. Why would a brute force simulation be anywhere as efficient as the original hardware?
- caseyy 2y ago
  Human language seems to originate in grokking, in natural net terms. You first understand a concept, then you speak or write about it. When you hear or read something, it isn’t retained well unless you can formulate an understanding of it. So language is a communication interface, not thinking itself. LLMs seem to create an illusion of understanding largely because we believe that understanding is a prerequisite for language — whoever uses this interface must be thinking! But that really seems to only be the case in our brains, not in LLMs. Grokking in NNs is interesting though. It’s going to be pretty wild times when we crack that nut open. If we could drive grokking on a large data set, it’s difficult to even imagine the possibilities. Who knows what we could grokk about the universe, for example. But that is still something only human brains can do regularly and reasonably well in heavily-parametrized situations. And even we get overwhelmed so easily by too many variables. I do hope ML grokking becomes much better than human understanding over time. But I don’t think we are close, sadly. We have just barely scratched the surface in toy models.

caseyy 2y ago

You also have to consider what intelligence is useful. We already have human brains in abundance, but computers can do certain things we can’t. I think LLMs are a bit of a red herring. We will discover powerful ways to use machine learning that are very different from human brains. The goal doesn’t need to be to replace them.

planckscnst 2y ago

Computers aren't evolving in an energy-scarce environment (yet) we (currently) pretty much give computers as much energy as they want. Computers are evolving in an environment where a dependent species relies on them for that species' own weaknesses.

deleted 2y ago
[deleted]

NoPicklez 2y ago

Well the human approach to intelligence and its tool has had approx 6 million years to improve and evolve and has created the reality we experience today, which could be really intelligent behavior or not. We don't have anything to compare it to. Think of the size of the computers we had in the 1960's and their computing power, now you have that multiple times over on your wrist. The tools we use seem to get bigger and bigger, then smaller, then bigger until a new paradigm becomes available (e.g. quantum)

burnished 2y ago

I misread your statement as stating that computers broadly were less energy efficient at the same tasks, because I took umbridge at your claim that human brains use `so little energy` (citation needed???) and forgot how to reading comprehension, so I compared training costs of a 7 year old and inference costs for an adult in terms of energy to the figures quoted in the article for the various models. I'll address what you actually said below but am keeping my original writeup because its kind of neat. I'm going to compare energy costs for training a model vs growing a brain because they are analogous. TFA lists the training costs in kWh and I'll arrive at a comparable figure by estimating the total caloric intake of a 7 year old and then multiplying that by an estimate of the proportion of energy a child's brain uses. I calculate that the lifetime caloric intake of a 7 year old is 3,139,000 dietary calroies based on a daily schedule of 1,000 for the first three years and 1,400 for the next four[1], and assuming an active child's brain uses roughly 60% of total calories[2], we end up with a lifetime total of 1,883,400 calories spent on the brain. This is 2,188 kWh. >> the energy consumption of model training is somewhere in the range of 11,000 kWh to 52,000 kWh for moderate-sized LLMs, 40,000 kWh to 60,000 kWh for more expensive image generators, and up to 5,000,000 kWh for global-scale LLMs like Llama and ChatGPT. The 7 year old isn't even finished training, and frankly, needing the same training energy as 2,285 7 year olds to create ChatGPT comes across as a bargain. Just for a bonus lets compare inference costs, assuming a human brain uses roughly 200 dietary calories and that we can count all of them as part of performing productive work gives us a budget of about 0.5 kWh (464 Wh). Going to quote the article again for inference costs >> the energy consumption of an inference operation is somewhere in the range of 1.0 Wh to 7.0 Wh for standard operations, and to 4 Wh to 7.5 Wh for more expensive jobs like image or video generation. Hundreds of text operations and 50-100 images. As a back of the napkin sketch it looks pretty close. [1] https://www.healthychildren.org/English/healthy-living/nutrition/Pages/Energy-In-Recommended-Food-Drink-Amounts-for-Children.aspx https://www.healthychildren.org/English/healthy-living/nutri... [2] https://www.livescience.com/burn-calories-brain.html https://www.livescience.com/burn-calories-brain.html ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You're sorta onto something but it has to do with architecture: human brains are composed of a vast number of sparsely connected and individually poor computational units (neurons). Computers are completely connected and highly reliable but individual units are expensive. Neurons also communicate at a much slower pace than computers can perform calculations. The reason for our unreasonable effectiveness appears rooted in how very many neurons we have and how effectively parallelized most functions of the brain are. Precision calculations are easy for computers because they are literally machines that have operations that map to simple numerical calculations that are highly accurate. Numeric calculations are not simple for human brains because nothing really maps to the process of a calculation. Individual neurons do 'integrate and fire' meaning that if they get excited enough by nearby neurons in a short enough time then they will fire and excite the neurons they touch, but it isn't a reliable process, meaning that if neuron A touches neuron B and A activating is enough to cause B to activate this time it may not be enough the next time A activate. So when we do math we use those. There are investigations into neuromorphic hardware but I'm not up to date on them anymore, the basic idea is to make computing hardware inspired by how the brain operates. Last I knew there were some interesting results but nothing resembling anything practical.

axdsk 2y ago

The right tool might be Spiking Neural Networks. Due to their sparse activation, event driven computation, and temporal coding. This all depends on how good neuromorphic chips get.