3 ms·

It's actually the reverse. The hardware we have is way more efficient than our brains. One TPU can simulate billions of neurons a second in a tiny area. Compute

by mantap 6y ago

It's actually the reverse. The hardware we have is way more efficient than our brains. One TPU can simulate billions of neurons a second in a tiny area. Computer hardware has a significant time advantage. Electrical impulses in the brain travel very slowly. The problem is we don't understand how the brain is architected at anything other than a very coarse level. That architecture is the product of millions of years of evolution so we have some catching up to do.

d_tr 6y ago
Except that the TPU does not simulate real neurons in any way. It just performs simple arithmetic using an architecture that is suited to the ANNs we use today. If you managed to understand how the brain works and tried to run the model on a cluster of TPUs, you would need way more energy to simulate one second of brain activity than the 20J a real brain uses in a second. Only by some magical coincidence would any of the hardware we use today turn out to be efficient.
- darkerside 6y ago
  Are neurons binary in the way they operate, in that they either fire or they don't? Or are they analog in nature, where a signal can be either strong, weak, nonexistent or anywhere in between?
  - qubex 6y ago
    For starters, they use a pulse-frequency based signalling system. And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in. There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate. Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all. [1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).
    - montebicyclelo 6y ago
      > Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided.. To be fair, many lectures/books/articles/blog posts on ANNs start off by presenting a biological neuron, and saying that ANNs approximate this. E.g. https://youtu.be/uXt8qF2Zzfo?t=385 https://youtu.be/uXt8qF2Zzfo?t=385 (MIT 6.034 Artificial Intelligence, Fall 2010. 12a: Neural Nets)
      - qubex 6y ago
        I do not exempt those who make such statements just because they have authority. If anything, they should know better. I started off along much the same path (I bought and read Neural Networks: A Comprehensive Foundation (1994) by Simon Haykin) when I was in high school, read it all and thought I knew everything there was to know on the topic, and then started to read about neurobiology and was stunned to discover I was living a lie. Then I read The Computational Beauty of Nature (1998) by Gary William Flake and decided I was going to dedicate myself to understanding the complexities we brush under the rugs. I think the Hacker News/Silicon Valley/Machine Learning/Developer crowd really need to look beyond the virtuosismo of their implementations. It’s a very introverted and self-referential mindset.
    - darkerside 6y ago
      Thanks, sorry if this was a dumb question. I have little background in neurobiology. Your answer was very informative.
      - mycall 6y ago
        For more information, take a look @ https://www.humanbrainproject.eu/en/brain-simulation/ https://www.humanbrainproject.eu/en/brain-simulation/
  - IIAOPSW 6y ago
    To add on to the child comment, neurons seem to fire either as a one off pulse or at a fixed frequency for a duration of time known as a burst. There is no explicit 'input layer' or 'output layer' and signal propagation takes time so there is nothing stopping a self sustaining loop signal from forming. These are just two examples of effects which are not captured in the artificial neuron model used in ML.
pchiusano 6y ago
Switching speed is very fast with silicon, but we seem to be nowhere near the computing power of the brain, which incidentally runs on an absurdly low 20W. I think this power usage aspect is quite underappreciated. You might like this article: https://blog.piekniewski.info/2018/08/28/fun-numbers-about-the-brain/ https://blog.piekniewski.info/2018/08/28/fun-numbers-about-t...

by mantap 6y ago

d_tr 6y ago

Except that the TPU does not simulate real neurons in any way. It just performs simple arithmetic using an architecture that is suited to the ANNs we use today. If you managed to understand how the brain works and tried to run the model on a cluster of TPUs, you would need way more energy to simulate one second of brain activity than the 20J a real brain uses in a second. Only by some magical coincidence would any of the hardware we use today turn out to be efficient.

darkerside 6y ago
Are neurons binary in the way they operate, in that they either fire or they don't? Or are they analog in nature, where a signal can be either strong, weak, nonexistent or anywhere in between?
- qubex 6y ago
  For starters, they use a pulse-frequency based signalling system. And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in. There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate. Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all. [1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).
  - montebicyclelo 6y ago
    > Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided.. To be fair, many lectures/books/articles/blog posts on ANNs start off by presenting a biological neuron, and saying that ANNs approximate this. E.g. https://youtu.be/uXt8qF2Zzfo?t=385 https://youtu.be/uXt8qF2Zzfo?t=385 (MIT 6.034 Artificial Intelligence, Fall 2010. 12a: Neural Nets)
    - qubex 6y ago
      I do not exempt those who make such statements just because they have authority. If anything, they should know better. I started off along much the same path (I bought and read Neural Networks: A Comprehensive Foundation (1994) by Simon Haykin) when I was in high school, read it all and thought I knew everything there was to know on the topic, and then started to read about neurobiology and was stunned to discover I was living a lie. Then I read The Computational Beauty of Nature (1998) by Gary William Flake and decided I was going to dedicate myself to understanding the complexities we brush under the rugs. I think the Hacker News/Silicon Valley/Machine Learning/Developer crowd really need to look beyond the virtuosismo of their implementations. It’s a very introverted and self-referential mindset.
  - darkerside 6y ago
    Thanks, sorry if this was a dumb question. I have little background in neurobiology. Your answer was very informative.
    - mycall 6y ago
      For more information, take a look @ https://www.humanbrainproject.eu/en/brain-simulation/ https://www.humanbrainproject.eu/en/brain-simulation/
- IIAOPSW 6y ago
  To add on to the child comment, neurons seem to fire either as a one off pulse or at a fixed frequency for a duration of time known as a burst. There is no explicit 'input layer' or 'output layer' and signal propagation takes time so there is nothing stopping a self sustaining loop signal from forming. These are just two examples of effects which are not captured in the artificial neuron model used in ML.

pchiusano 6y ago

Switching speed is very fast with silicon, but we seem to be nowhere near the computing power of the brain, which incidentally runs on an absurdly low 20W. I think this power usage aspect is quite underappreciated. You might like this article: https://blog.piekniewski.info/2018/08/28/fun-numbers-about-the-brain/ https://blog.piekniewski.info/2018/08/28/fun-numbers-about-t...