8 ms·

Knowing little about neuroscience outside of biology class, I've always wondered why neurons would be so simple as to just compute weighted sums and thresholds.

Knowing little about neuroscience outside of biology class, I've always wondered why neurons would be so simple as to just compute weighted sums and thresholds. I mean there are unicellular creatures that can do incredibly complex things, and neurons, as all cells, are ultimately the descendants of such unicellular organisms, "teamed up" into a multi-cellular creature. They have a lot of internal structure, organelles, whatnot. Modeling them as some simple switches seems a stretch. My point is, overall this doesn't seem so surprising to me as a layman. (Which doesn't mean more than just literally that. I'm not claiming laypeople should decide scientific questions from their armchairs.)

svara 7y ago
This is an intuition that many neuroscientists have shared, it's just that direct experimental evidence of 1) it happening at all in a non-trivial way and 2) it being relevant to the behavior of an animal is somewhat scarce. Plus, the sum and threshold model actually gets you pretty far, as evidenced by artificial neural networks, so there might be some resistance to adding complexity that might not be necessary. Edit: Google "dendritic computation" if you want to dig deeper.
- buboard 7y ago
  clarifying that dendritic computation has been studied for decades, this one is just a new type of dendritic spike
- sansnomme 7y ago
  Also: spiking neural networks. Doing useful stuff with them has been a work-in-progress so far though.
  - nravic 7y ago
    Yeah, spiking neural networks are a tough nut to crack. Check out Nengo if you're interested in learning more
buboard 7y ago
they are not. the current consensus in computational neuroscience is that a neuron can be equivalent to a 2-layer or 3-layer ANN with point nonlinearities. Edit: i am blocked from posting, so here are the refs: https://www.ncbi.nlm.nih.gov/pubmed/20800473 https://www.ncbi.nlm.nih.gov/pubmed/20800473 https://www.ncbi.nlm.nih.gov/pubmed/25554708 https://www.ncbi.nlm.nih.gov/pubmed/25554708 https://sci-hub.tw/https://www.sciencedirect.com/science/article/pii/S0301008214001373 https://sci-hub.tw/https://www.sciencedirect.com/science/art...
- visres 7y ago
  Do you have any references you could share on this? I'd like to read more, thanks!
  - deepnet 7y ago
    Minsky and Papert showed that computing XOR requires a 3 layer artificial neural net: Input, Hidden Layer, Output - which has 2 layers of weighted connections. This[0] research shows a single neuron can compute XOR, thus the artificial neuron has less computational power than a real one. [0] - https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_single_biological_neuron_can_compute_xor/ https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...
    - visres 7y ago
      Ah interesting thank you
    - svara 7y ago
      That's exactly what TFA is about ;)
nabla9 7y ago
Inside each sell is gene regulatory network (GRN). Active genes produce proteins. Some of those proteins increase or decrease activity of genes. GRN as a whole can be modeled as stochastic recurrent neural network. The potential for computation inside each neuron is similar to recurrent neural network.
- buboard 7y ago
  Except the clock is ticking every 30 minutes in that network. it's too slow for behavior. GRNs are relevant to long-term plasticity which is protein-dependent and has prominent modulators (e.g. CREB)
dkarras 7y ago
The idea is that a certain simplification could work enough to be valuable. It is similar with gravity isn't it? We don't have a grand unified theory of quantum mechanics, but even Newton's abstractions serves us well in certain situations. We don't know how it behaves in the micro level but we can still observe the macro effects and create an abstraction out of said behavior. And to be honest, it took us somewhere. Yes we don't have AGI or anything close to it but the products of machine learning is something we depend on every single day now.
- Filligree 7y ago
  The risk is that we've developed a formalism and ecosystem that works on entirely different principles than the brain, even if it looks similar. It still works, but it might not be a useful model to anyone studying the brain. I've yet to meet the neuroscientist who assumes it is, so perhaps that's not a problem.
  - bnegreve 7y ago
    > The risk is that we've developed a formalism and ecosystem that works on entirely different principles [...] It's a possibility, but I sometime have the feeling that people dismiss the idea that such a simple model of the brain could be enough to explain complex behaviors because they want to believe there is more to the brain. I'm sure real neurons are very complex and difficult to model, but I also believe that the real challenge is to explain how neurons interact with each other, not how they behave individually.
    - AstralStorm 7y ago
      And why do you believe the current non-intelligent ML is sufficient? As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons. Yet it is still a toy compared to a nematode.
      - bnegreve 7y ago
jcims 7y ago
Using some back of the napkin math there are approximately a hundred trillion atoms in a typical neuron. https://www.quora.com/If-a-neuron-were-a-galaxy-how-would-the-number-of-atoms-in-it-stack-up-to-the-number-of-stars https://www.quora.com/If-a-neuron-were-a-galaxy-how-would-th...
hinkley 7y ago
Around the time everyone was talking about tensegrity there was a researcher that had figured out a mechanism by which cells can sense pressure, due I think to mechanical stress on the molecules in membrane being detectable inside the cell. Proteins and lipids as proverbial nerve endings. They speculated from this that neurons rely on quantum effects, possibly using similar mechanisms, and that the surface area is involved. The arms of neurons have a lot of the surface area, don’t they?
- cyorir 7y ago
  Surface area of the neuron depends on the type of neuron. It can be a very important factor in summation of inputs and propagation of potentials, along with shape, diameter, myelination, etc.
starpilot 7y ago
Understanding of neurons goes wayy beyond that today, see https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model ANNs are abstractions that seem to work. We still don't have a full understanding of how two neurons communicate.
Fishysoup 7y ago
We always knew that neurons did way more than just integrate voltage and produce spikes. The problem with testing out what those things are has largely been technological. Only until quite recently have we been able to electrically isolate dendrites and axons for recording, and slowly it's becoming tractable to image from entire neurons with high temporal resolution using voltage-sensitive fluorescent indicators. So the field of dendritic and other sub-neuronal computation arguably in its infancy. The other things we figured out in the meantime relate to different types of neurotransmission (e.g. volume neurotransmission), active processes in axons and dendrites like vesicular trafficking, synaptic tag-and-capture (if it turns out to be true) and all kinds of weird types of plasticity. Basically a neuron's function is a lot more nuanced than the simplistic "integrate and fire" idea. Artificial neural networks (in the deep learning sense) therefore don't really have much of anything to do with biological ones function-wise. It really is just regression with a lot of nodes and layers, and maybe some bells and whistles. That doesn't mean they're not powerful in their own right, just not comparable to brain circuits and don't do the same kind of thing nor solve the same kinds of tasks.
salty_biscuits 7y ago
Computing weighted sums and thresholds gives you a universal function approximator, so that's nice. I'm more interested in how the buggers set those weights and thresholds. It is hard to imagine them having access to any gradient information, so how do they work?
- AstralStorm 7y ago
  They do have chemical gradients, but the mechanisms is not even comparable to simple backpropagation...

svara 7y ago

This is an intuition that many neuroscientists have shared, it's just that direct experimental evidence of 1) it happening at all in a non-trivial way and 2) it being relevant to the behavior of an animal is somewhat scarce. Plus, the sum and threshold model actually gets you pretty far, as evidenced by artificial neural networks, so there might be some resistance to adding complexity that might not be necessary. Edit: Google "dendritic computation" if you want to dig deeper.

buboard 7y ago
clarifying that dendritic computation has been studied for decades, this one is just a new type of dendritic spike
sansnomme 7y ago
Also: spiking neural networks. Doing useful stuff with them has been a work-in-progress so far though.
- nravic 7y ago
  Yeah, spiking neural networks are a tough nut to crack. Check out Nengo if you're interested in learning more

buboard 7y ago

they are not. the current consensus in computational neuroscience is that a neuron can be equivalent to a 2-layer or 3-layer ANN with point nonlinearities. Edit: i am blocked from posting, so here are the refs: https://www.ncbi.nlm.nih.gov/pubmed/20800473 https://www.ncbi.nlm.nih.gov/pubmed/20800473 https://www.ncbi.nlm.nih.gov/pubmed/25554708 https://www.ncbi.nlm.nih.gov/pubmed/25554708 https://sci-hub.tw/https://www.sciencedirect.com/science/article/pii/S0301008214001373 https://sci-hub.tw/https://www.sciencedirect.com/science/art...

visres 7y ago
Do you have any references you could share on this? I'd like to read more, thanks!
- deepnet 7y ago
  Minsky and Papert showed that computing XOR requires a 3 layer artificial neural net: Input, Hidden Layer, Output - which has 2 layers of weighted connections. This[0] research shows a single neuron can compute XOR, thus the artificial neuron has less computational power than a real one. [0] - https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_single_biological_neuron_can_compute_xor/ https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...
  - visres 7y ago
    Ah interesting thank you
  - svara 7y ago
    That's exactly what TFA is about ;)

nabla9 7y ago

Inside each sell is gene regulatory network (GRN). Active genes produce proteins. Some of those proteins increase or decrease activity of genes. GRN as a whole can be modeled as stochastic recurrent neural network. The potential for computation inside each neuron is similar to recurrent neural network.

buboard 7y ago
Except the clock is ticking every 30 minutes in that network. it's too slow for behavior. GRNs are relevant to long-term plasticity which is protein-dependent and has prominent modulators (e.g. CREB)

dkarras 7y ago

The idea is that a certain simplification could work enough to be valuable. It is similar with gravity isn't it? We don't have a grand unified theory of quantum mechanics, but even Newton's abstractions serves us well in certain situations. We don't know how it behaves in the micro level but we can still observe the macro effects and create an abstraction out of said behavior. And to be honest, it took us somewhere. Yes we don't have AGI or anything close to it but the products of machine learning is something we depend on every single day now.

Filligree 7y ago
The risk is that we've developed a formalism and ecosystem that works on entirely different principles than the brain, even if it looks similar. It still works, but it might not be a useful model to anyone studying the brain. I've yet to meet the neuroscientist who assumes it is, so perhaps that's not a problem.
- bnegreve 7y ago
  > The risk is that we've developed a formalism and ecosystem that works on entirely different principles [...] It's a possibility, but I sometime have the feeling that people dismiss the idea that such a simple model of the brain could be enough to explain complex behaviors because they want to believe there is more to the brain. I'm sure real neurons are very complex and difficult to model, but I also believe that the real challenge is to explain how neurons interact with each other, not how they behave individually.
  - AstralStorm 7y ago
    And why do you believe the current non-intelligent ML is sufficient? As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons. Yet it is still a toy compared to a nematode.
    - bnegreve 7y ago

jcims 7y ago

Using some back of the napkin math there are approximately a hundred trillion atoms in a typical neuron. https://www.quora.com/If-a-neuron-were-a-galaxy-how-would-the-number-of-atoms-in-it-stack-up-to-the-number-of-stars https://www.quora.com/If-a-neuron-were-a-galaxy-how-would-th...

hinkley 7y ago

Around the time everyone was talking about tensegrity there was a researcher that had figured out a mechanism by which cells can sense pressure, due I think to mechanical stress on the molecules in membrane being detectable inside the cell. Proteins and lipids as proverbial nerve endings. They speculated from this that neurons rely on quantum effects, possibly using similar mechanisms, and that the surface area is involved. The arms of neurons have a lot of the surface area, don’t they?

cyorir 7y ago
Surface area of the neuron depends on the type of neuron. It can be a very important factor in summation of inputs and propagation of potentials, along with shape, diameter, myelination, etc.

starpilot 7y ago

Understanding of neurons goes wayy beyond that today, see https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model ANNs are abstractions that seem to work. We still don't have a full understanding of how two neurons communicate.

Fishysoup 7y ago

We always knew that neurons did way more than just integrate voltage and produce spikes. The problem with testing out what those things are has largely been technological. Only until quite recently have we been able to electrically isolate dendrites and axons for recording, and slowly it's becoming tractable to image from entire neurons with high temporal resolution using voltage-sensitive fluorescent indicators. So the field of dendritic and other sub-neuronal computation arguably in its infancy. The other things we figured out in the meantime relate to different types of neurotransmission (e.g. volume neurotransmission), active processes in axons and dendrites like vesicular trafficking, synaptic tag-and-capture (if it turns out to be true) and all kinds of weird types of plasticity. Basically a neuron's function is a lot more nuanced than the simplistic "integrate and fire" idea. Artificial neural networks (in the deep learning sense) therefore don't really have much of anything to do with biological ones function-wise. It really is just regression with a lot of nodes and layers, and maybe some bells and whistles. That doesn't mean they're not powerful in their own right, just not comparable to brain circuits and don't do the same kind of thing nor solve the same kinds of tasks.

salty_biscuits 7y ago

Computing weighted sums and thresholds gives you a universal function approximator, so that's nice. I'm more interested in how the buggers set those weights and thresholds. It is hard to imagine them having access to any gradient information, so how do they work?

AstralStorm 7y ago
They do have chemical gradients, but the mechanisms is not even comparable to simple backpropagation...