6 ms·

I think there are two important factors to this. One is that the operating frequency of the human brain is much lower. That results in much smaller energy cons

by js8 8y ago

I think there are two important factors to this. One is that the operating frequency of the human brain is much lower. That results in much smaller energy consumption. The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key. My feeling is also that brain extensively uses probabilistic algorithms with hash encoding, conceptually similar to bloom filters, count-min, etc.

zaarn 8y ago
The brain has other differences that are probably a major influences on it's efficiency; 1) It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that. 2) It doesn't run everything all the time. Parts of the brain not in use are shut down. Modern computers do this to some extend but CPU cores don't switch off parts of the ALU not in use. The brain has much better energy saving options, only 10-30% of the brain are active at a given time.
- rodiger 8y ago
  Building on 2- this is likely not a result of saving energy but rather hyper-specialization in specific regions of the brain. Like if modern CPUs had 10-20 cores, each tuned to handle a specific task. The result is energy efficiency, but in contrast to current energy saving options, the brain can't simply "speed up" and use more power and activate more areas of the brain.
- skohan 8y ago
  > It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that. Modern GPUs aren't even close: neither in terms of degree or in kind. A GPU does a lot of linear operations in parallel, but in the brain all the "compute units" are interacting with each-other in real time.
marcyb5st 8y ago
My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen :) Let me explain: Albeit true that our brain is much slower than a modern CPU in terms of ops/s, every neuron is connected to up to 10000 other neurons[0]. And we have something around 100 billion neurons in our brains. This means that the number of synapses is between 100 and 1000 trillions. If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date. Moreover, the topology of said neural network is not a simple stack of layers, but much more complex like a super intricated residual network. Finally, the brain is not using backpropagation to update the mentioned weights/synapses' strength, but instead is based on how often said synapses fire. [0]: http://www.human-memory.net/brain_neurons.html http://www.human-memory.net/brain_neurons.html
- dnadler 8y ago
  People also gloss over the fact that there are multiple neurotransmitters involved.
- skohan 8y ago
  > If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date. This is a very bad assumption. Each synapse is a highly complex chemical computer, and its "weight" with regard to its contribution to the postsynaptic neuron's activation at any point in time depends on a multitude of factors. Those factors include dozens of interacting metabolic pathways, the interactions of various neurotransmitter systems which each have distinct behavior, and things like where the individual synapses sit relative to one another on the dendritic arbor. There's even evidence that at least in some neurons the activation pattern may influence RNA production in the neuron, creating a kind of recording which influences the behavior of the cell. Current ANN models should be considered a very low dimensional approximation of the real thing.
- jcranmer 8y ago
  > My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen Neural networks were not meant to be accurate analogues to the actual operation of the neurons in the brain. They were known to be poor matches for the actual biology, even back in 1959. The concept of the multilayer neural network was actually inspired from our knowledge of the visual cortex, rather than the lower-level fundamentals of the brain. The brain is most certainly not a neural network, as the term is understood in machine learning.
mrfusion 8y ago
What about the idea that the brain is analog?
- kowdermeister 8y ago
  The brain is analog.
  - skohan 8y ago
    It's digital and analog. At the level of the action potential, the brain certainly does seem to operate with discreet units of information. Everywhere else though is certainly analog.
    - kowdermeister 8y ago
      So if a neuron fires or not (I guess that's what you mean by action potential) can simply be translated to 1 or 0 hence it's digital?
      - skohan 8y ago
        Basically yes. That's only one small piece of the information processing of the brain, but it would in principal be possible to represent the "firing state" of a biological neural network in binary terms.
danans 8y ago
> The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key. Deep neural networks, though they use linear algebra, are definitely non linear, as a result of both their architecture and their use of non linear activation functions. Their non-linearity is part of the reason they can classify phenomena that linear techniques like simple regression cannot.

by js8 8y ago

zaarn 8y ago

The brain has other differences that are probably a major influences on it's efficiency; 1) It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that. 2) It doesn't run everything all the time. Parts of the brain not in use are shut down. Modern computers do this to some extend but CPU cores don't switch off parts of the ALU not in use. The brain has much better energy saving options, only 10-30% of the brain are active at a given time.

rodiger 8y ago
Building on 2- this is likely not a result of saving energy but rather hyper-specialization in specific regions of the brain. Like if modern CPUs had 10-20 cores, each tuned to handle a specific task. The result is energy efficiency, but in contrast to current energy saving options, the brain can't simply "speed up" and use more power and activate more areas of the brain.
skohan 8y ago
> It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that. Modern GPUs aren't even close: neither in terms of degree or in kind. A GPU does a lot of linear operations in parallel, but in the brain all the "compute units" are interacting with each-other in real time.

marcyb5st 8y ago

My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen :) Let me explain: Albeit true that our brain is much slower than a modern CPU in terms of ops/s, every neuron is connected to up to 10000 other neurons[0]. And we have something around 100 billion neurons in our brains. This means that the number of synapses is between 100 and 1000 trillions. If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date. Moreover, the topology of said neural network is not a simple stack of layers, but much more complex like a super intricated residual network. Finally, the brain is not using backpropagation to update the mentioned weights/synapses' strength, but instead is based on how often said synapses fire. [0]: http://www.human-memory.net/brain_neurons.html http://www.human-memory.net/brain_neurons.html

dnadler 8y ago
People also gloss over the fact that there are multiple neurotransmitters involved.
skohan 8y ago
> If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date. This is a very bad assumption. Each synapse is a highly complex chemical computer, and its "weight" with regard to its contribution to the postsynaptic neuron's activation at any point in time depends on a multitude of factors. Those factors include dozens of interacting metabolic pathways, the interactions of various neurotransmitter systems which each have distinct behavior, and things like where the individual synapses sit relative to one another on the dendritic arbor. There's even evidence that at least in some neurons the activation pattern may influence RNA production in the neuron, creating a kind of recording which influences the behavior of the cell. Current ANN models should be considered a very low dimensional approximation of the real thing.
jcranmer 8y ago
> My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen Neural networks were not meant to be accurate analogues to the actual operation of the neurons in the brain. They were known to be poor matches for the actual biology, even back in 1959. The concept of the multilayer neural network was actually inspired from our knowledge of the visual cortex, rather than the lower-level fundamentals of the brain. The brain is most certainly not a neural network, as the term is understood in machine learning.

mrfusion 8y ago

What about the idea that the brain is analog?

kowdermeister 8y ago
The brain is analog.
- skohan 8y ago
  It's digital and analog. At the level of the action potential, the brain certainly does seem to operate with discreet units of information. Everywhere else though is certainly analog.
  - kowdermeister 8y ago
    So if a neuron fires or not (I guess that's what you mean by action potential) can simply be translated to 1 or 0 hence it's digital?
    - skohan 8y ago
      Basically yes. That's only one small piece of the information processing of the brain, but it would in principal be possible to represent the "firing state" of a biological neural network in binary terms.

danans 8y ago

> The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key. Deep neural networks, though they use linear algebra, are definitely non linear, as a result of both their architecture and their use of non linear activation functions. Their non-linearity is part of the reason they can classify phenomena that linear techniques like simple regression cannot.