3 ms·

Is this still orders of magnitude smaller than a human brain? How many? Based on current human neurons/synapses knowledge?

birriel 3y ago
2 orders of magnitude smaller, assuming 100T synaptic connections in the human brain.
- ada1981 3y ago
  It’s remarkable. I’m curious how long until people are just using training brains in a jar to compute.
  - quickthrower2 3y ago
    They already are, for the hyperhyperparameter choices.
    - mlashcorp 3y ago
      Well, not in a jar ... Right?
      - zxcnmbzxcmb 3y ago
        How would you know that you're not in a jar? :-)
      - quickthrower2 3y ago
        Open plan noisy offices are almost that.
      - elzbardico 3y ago
        Not yet, but in our last call we assured our stakeholders that we are very close to that.
- jimmySixDOF 3y ago
  In the podcast they talk about 20 Peta FLOPS as the human brain equivalent for measuring comparison.
- baq 3y ago
  And how big is the memory and language part…?
justrealist 3y ago
We really have no idea how to directly compare the two. Also, vast portions of the human brain are dedicated to the visual cortex, smelling, breathing, muscle control... things which have value to us but which don't contribute to knowledge work when evaluating how many parameters it would take to replace human knowledge work.
- vbezhenar 3y ago
  It's really interesting that human organism requires so much computational power to support live.
  - gwoolhurme 3y ago
    I think it's even more interesting that the required amount of energy to do that high computational work isn't that high. Evolution has been working on it for a long time, and some things are really inefficient but overall it does an OK job at making squishy machines.
    - tedivm 3y ago
      The human brain uses roughly 20 watts, which is really a remarkably low number. https://psychology.stackexchange.com/a/12386 https://psychology.stackexchange.com/a/12386
      - interlinked 3y ago
        That's 20% of total energy consumption.
    - loh 3y ago
      I had a good chuckle at "squishy machines". That's a really interesting way to think about it. It makes me wonder if, some day, we will be able to build "squishy machines" of our own, capable of outperforming silicon while using a tiny fraction of the energy.
  - WithinReason 3y ago
light_hue_1 3y ago
We have no idea how to estimate the computational capacity of the brain at the moment. We can make silly estimates like saying that 1 human neuron is equivalent to something in an artificial network. But this is definitely wrong, biological neurons are far more complex than this. The big problem is that we don't understand the locus of computation in the brain. What is the thing performing the meaningful unit of computation in a neuron? And what is a neuron really equivalent to? The ranges are massive. Some people say that computation is some high level property of the neuron as a whole, so they think each neuron is equivalent to just a few logic gates. These people would say that the brain has a capacity of about 1 petaFLOP/s. https://lips.cs.princeton.edu/what-is-the-computational-capacity-of-the-brain/ https://lips.cs.princeton.edu/what-is-the-computational-capa... Then there are people who think every Na, K, and Ca ion channel performs meaningful computation. They would say the brain has a capacity of 1 zettaFLOP/s. https://arxiv.org/pdf/2009.10615.pdf https://arxiv.org/pdf/2009.10615.pdf Then there are computational researchers who just want to approximate what a neuron does. Their results say that neurons are more like whole 4-8 layer artificial networks. This would place the brain well somewhere in the yottaFLOP/s range https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... And we're learning more about how complex neurons are all the time. No one thinks the picture above is accurate in any way. Then there are the extremists who think that there is something non-classical about our brains. That neurons individually or areas of the brain as a whole exploit some form of quantum computation. If they're right, we're not even remotely on the trajectory to matching brains, and very likely nothing we're doing today will ever pay off in that sense. Almost no one believes them. Let's say the brain is in the zettaFLOP/s range. That's 10^21 FLOP/s. Training GPT-3 took 10^23 FLOPS total over 34 days. 34 days has 2937600 seconds. 10^23/10^7 is about 10^16 FLOP/s. So by this back of the envelope computation the brain has about 4 orders of magnitude more capacity, or 1000x. This makes a lot of sense, they're using a pettaFLOP/s supercomputer basically which we already knew. We'll have zettaFLOP/s supercomputers soon, yottaFLOP/s, people are worried we're going to hit some fundamental physical limits before we get there. All of this is a simplification and there are problems with every one of these estimates. But, in some sense none of it means anything at all. You can have an extremely efficient algorithm that runs 1 million times faster than an extremely inefficient algorithm. Machines and brains do not run the same "software", the same algorithms. So comparing their hardware directly doesn't say anything at all.

birriel 3y ago

2 orders of magnitude smaller, assuming 100T synaptic connections in the human brain.

ada1981 3y ago
It’s remarkable. I’m curious how long until people are just using training brains in a jar to compute.
- quickthrower2 3y ago
  They already are, for the hyperhyperparameter choices.
  - mlashcorp 3y ago
    Well, not in a jar ... Right?
    - zxcnmbzxcmb 3y ago
      How would you know that you're not in a jar? :-)
    - quickthrower2 3y ago
      Open plan noisy offices are almost that.
    - elzbardico 3y ago
      Not yet, but in our last call we assured our stakeholders that we are very close to that.
jimmySixDOF 3y ago
In the podcast they talk about 20 Peta FLOPS as the human brain equivalent for measuring comparison.
baq 3y ago
And how big is the memory and language part…?

justrealist 3y ago

We really have no idea how to directly compare the two. Also, vast portions of the human brain are dedicated to the visual cortex, smelling, breathing, muscle control... things which have value to us but which don't contribute to knowledge work when evaluating how many parameters it would take to replace human knowledge work.

vbezhenar 3y ago
It's really interesting that human organism requires so much computational power to support live.
- gwoolhurme 3y ago
  I think it's even more interesting that the required amount of energy to do that high computational work isn't that high. Evolution has been working on it for a long time, and some things are really inefficient but overall it does an OK job at making squishy machines.
  - tedivm 3y ago
    The human brain uses roughly 20 watts, which is really a remarkably low number. https://psychology.stackexchange.com/a/12386 https://psychology.stackexchange.com/a/12386
    - interlinked 3y ago
      That's 20% of total energy consumption.
  - loh 3y ago
    I had a good chuckle at "squishy machines". That's a really interesting way to think about it. It makes me wonder if, some day, we will be able to build "squishy machines" of our own, capable of outperforming silicon while using a tiny fraction of the energy.
- WithinReason 3y ago

light_hue_1 3y ago

We have no idea how to estimate the computational capacity of the brain at the moment. We can make silly estimates like saying that 1 human neuron is equivalent to something in an artificial network. But this is definitely wrong, biological neurons are far more complex than this. The big problem is that we don't understand the locus of computation in the brain. What is the thing performing the meaningful unit of computation in a neuron? And what is a neuron really equivalent to? The ranges are massive. Some people say that computation is some high level property of the neuron as a whole, so they think each neuron is equivalent to just a few logic gates. These people would say that the brain has a capacity of about 1 petaFLOP/s. https://lips.cs.princeton.edu/what-is-the-computational-capacity-of-the-brain/ https://lips.cs.princeton.edu/what-is-the-computational-capa... Then there are people who think every Na, K, and Ca ion channel performs meaningful computation. They would say the brain has a capacity of 1 zettaFLOP/s. https://arxiv.org/pdf/2009.10615.pdf https://arxiv.org/pdf/2009.10615.pdf Then there are computational researchers who just want to approximate what a neuron does. Their results say that neurons are more like whole 4-8 layer artificial networks. This would place the brain well somewhere in the yottaFLOP/s range https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... And we're learning more about how complex neurons are all the time. No one thinks the picture above is accurate in any way. Then there are the extremists who think that there is something non-classical about our brains. That neurons individually or areas of the brain as a whole exploit some form of quantum computation. If they're right, we're not even remotely on the trajectory to matching brains, and very likely nothing we're doing today will ever pay off in that sense. Almost no one believes them. Let's say the brain is in the zettaFLOP/s range. That's 10^21 FLOP/s. Training GPT-3 took 10^23 FLOPS total over 34 days. 34 days has 2937600 seconds. 10^23/10^7 is about 10^16 FLOP/s. So by this back of the envelope computation the brain has about 4 orders of magnitude more capacity, or 1000x. This makes a lot of sense, they're using a pettaFLOP/s supercomputer basically which we already knew. We'll have zettaFLOP/s supercomputers soon, yottaFLOP/s, people are worried we're going to hit some fundamental physical limits before we get there. All of this is a simplification and there are problems with every one of these estimates. But, in some sense none of it means anything at all. You can have an extremely efficient algorithm that runs 1 million times faster than an extremely inefficient algorithm. Machines and brains do not run the same "software", the same algorithms. So comparing their hardware directly doesn't say anything at all.