3 ms·

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neuron’ will give you some idea. The original idea of approximating something like a neuron using a weighted sum (which is a fairly obvious idea, given the initial discovery that neurons become ‘activated’ and they do so in proportion to how much the neurons they are connected to are) did come from thinking about biological brains, but the mathematical building blocks are incredibly simple and are hundreds of years old, if not thousands.

naasking 3y ago
> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then that just increases the number of artificial neural network neurons needed to describe it, it does not need any computation of a different kind.
- xanderlewis 3y ago
  PowerPoint is Turing complete. Does that mean PowerPoint should be regarded as being biological or at least neuroscience-inspired?
  - naasking 3y ago
    No, but neural networks literally were inspired by biology so I'm not sure what your point is.
    - xanderlewis 3y ago
      My point is that you seem to think neurons in the sense of artificial neural networks and neurons in the human brain are equivalent because: (1) Neural networks are Turing complete, and hence can do anything brains can. [debatable anyway; We don’t know this to be the case since brains might be doing more than computation. Ask a philosopher or a cognitive scientist. Or Roger Penrose.] (2) Neural networks were very loosely inspired by the idea that the human brain is made up of interconnected nodes that ‘activate’ in proportion to how other related nodes do. I don’t think that’s nearly enough to say that they’re equivalent. For (1), we don’t yet know (and we’re not even close), and anyway: if you consider all Turing complete systems to be equivalent to the point of it being a waste of time to talk about their differences then you can say goodbye to quite a lot of work in theoretical computer science. For (2): so what? Lots of things are inspired by other things. It doesn’t make them in any sense equivalent, especially if the analogy is as weak as it is in this case. No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron. They operate on completely different principles, as we now know much better than when such things were first dreamed up.
      - naasking 3y ago
        The brain certainly could be doing super-Turing computation, but that would overturn quite a bit of physics seeing as how not even quantum computers are more powerful than Turing machines (they're just faster on some problems). Extraordinary claims and all that. As for equivalency, that depends on how that's defined. Real neurons would not feature any more computational power than Turing machines or artificial neural networks, but I never said it would be a waste of time to talk about their differences. I merely pointed out that the artificial neural network model is still sufficient, even if real neurons have more complexity. > No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron Fortunately that's not what I said. If the neuron indeed has more relevant complexity, then it wouldn't be one weighted sum = one biological neuron, but one biological neuron = a network of weighted sums, since such a network can model any function.
        3y ago
- andoando 3y ago
  And assembly is also turing complete, so if two models being both Turing completeness means they are equivalent, there would be no need for coding neural networks at all. Would you consider LLMs a different kind of computation than writing assembly code? Perhaps fundamentally they are not, but its also true that just writing more and more random assembly code isn't going to lead to an LLM.
  - naasking 3y ago
    LLMs aren't randomly generated though, they are shaped by training data. This means there would, in principle, be a comparable way to synthesize an equivalent assembly program from that same training data. The difference here is that it's just more obvious how to do this in one case than the other. My point was only that 1) neural networks are sufficient, even if real neurons have additional complexity, and 2) whatever that additional complexity, artificial neural networks can learn to reproduce it.
    - andoando 3y ago
      I understand that, what I am saying though is the fact that they can doesn't mean that they will by simply scaling their number. It still entirely depends on how they are trained/arranged, meaning it may take a completely different way of composing/glueing neurons together to stimulate any additional complexity. Its like saying a nand gate is turing complete, I put 1000000000 of them in a series, but its not doing anything, what gives, do I need to add a billion more? Just as a modeling and running a single neuron takes x amount of transistors configured in a very specific way for example, it may take y amount of neurons arranged in some very specific, unknown to model something that has extra properties. And its not clear either whether neurons are fundamentally the correct approach to reach this higher level construction than some other kind of node.
- srean 3y ago
  > This is a difference of degree not of kind Nope. Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. This is a fundamentally different phenomenon than ANNs where the output (voltage) is a squash transformed aggregated input values (voltages).
  - phkahler 3y ago
    >> Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. I thought they worked like accumulators where the spike "energy" accumulates until the output "fires". If that's the case then the artificial NNs are still an approximation of that process. I agree that this is a significant difference, but the mathematical version is still a rough approximation inspired by the biological one.
    - srean 2y ago
      Sandpile mathematics, something that is studied by computer scientists and mathematicians, would be an approximation. Its not so much the level of the spike that matters but how often they spike is what conveys the signal. The temporal behavior is supremely important. It used to be believed that the rate is all that matters, but now, no longer. There are ANN models that model these spike trains (that's what these 'avalanches' are called), these do work similar to real neurons, but they are not part of the deep neural network popularity [0,1]. Besides, backpropagation is not what goes on in the brain, its known to be biologically infeasible. So all in all the traditional ANNs are nothing like real neural networks. That's ok, aeroplanes do not fly like birds, but they do still 'fly'. [0] https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/

naasking 3y ago

> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then that just increases the number of artificial neural network neurons needed to describe it, it does not need any computation of a different kind.

xanderlewis 3y ago
PowerPoint is Turing complete. Does that mean PowerPoint should be regarded as being biological or at least neuroscience-inspired?
- naasking 3y ago
  No, but neural networks literally were inspired by biology so I'm not sure what your point is.
  - xanderlewis 3y ago
    My point is that you seem to think neurons in the sense of artificial neural networks and neurons in the human brain are equivalent because: (1) Neural networks are Turing complete, and hence can do anything brains can. [debatable anyway; We don’t know this to be the case since brains might be doing more than computation. Ask a philosopher or a cognitive scientist. Or Roger Penrose.] (2) Neural networks were very loosely inspired by the idea that the human brain is made up of interconnected nodes that ‘activate’ in proportion to how other related nodes do. I don’t think that’s nearly enough to say that they’re equivalent. For (1), we don’t yet know (and we’re not even close), and anyway: if you consider all Turing complete systems to be equivalent to the point of it being a waste of time to talk about their differences then you can say goodbye to quite a lot of work in theoretical computer science. For (2): so what? Lots of things are inspired by other things. It doesn’t make them in any sense equivalent, especially if the analogy is as weak as it is in this case. No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron. They operate on completely different principles, as we now know much better than when such things were first dreamed up.
    - naasking 3y ago
      The brain certainly could be doing super-Turing computation, but that would overturn quite a bit of physics seeing as how not even quantum computers are more powerful than Turing machines (they're just faster on some problems). Extraordinary claims and all that. As for equivalency, that depends on how that's defined. Real neurons would not feature any more computational power than Turing machines or artificial neural networks, but I never said it would be a waste of time to talk about their differences. I merely pointed out that the artificial neural network model is still sufficient, even if real neurons have more complexity. > No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron Fortunately that's not what I said. If the neuron indeed has more relevant complexity, then it wouldn't be one weighted sum = one biological neuron, but one biological neuron = a network of weighted sums, since such a network can model any function.
      - 3y ago
andoando 3y ago
And assembly is also turing complete, so if two models being both Turing completeness means they are equivalent, there would be no need for coding neural networks at all. Would you consider LLMs a different kind of computation than writing assembly code? Perhaps fundamentally they are not, but its also true that just writing more and more random assembly code isn't going to lead to an LLM.
- naasking 3y ago
  LLMs aren't randomly generated though, they are shaped by training data. This means there would, in principle, be a comparable way to synthesize an equivalent assembly program from that same training data. The difference here is that it's just more obvious how to do this in one case than the other. My point was only that 1) neural networks are sufficient, even if real neurons have additional complexity, and 2) whatever that additional complexity, artificial neural networks can learn to reproduce it.
  - andoando 3y ago
    I understand that, what I am saying though is the fact that they can doesn't mean that they will by simply scaling their number. It still entirely depends on how they are trained/arranged, meaning it may take a completely different way of composing/glueing neurons together to stimulate any additional complexity. Its like saying a nand gate is turing complete, I put 1000000000 of them in a series, but its not doing anything, what gives, do I need to add a billion more? Just as a modeling and running a single neuron takes x amount of transistors configured in a very specific way for example, it may take y amount of neurons arranged in some very specific, unknown to model something that has extra properties. And its not clear either whether neurons are fundamentally the correct approach to reach this higher level construction than some other kind of node.
srean 3y ago
> This is a difference of degree not of kind Nope. Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. This is a fundamentally different phenomenon than ANNs where the output (voltage) is a squash transformed aggregated input values (voltages).
- phkahler 3y ago
  >> Neurons in our brain operate fundamentally differently. They work by transient spikes and information is carried not by the intensity of the spike voltage, but by the frequency of spiking. I thought they worked like accumulators where the spike "energy" accumulates until the output "fires". If that's the case then the artificial NNs are still an approximation of that process. I agree that this is a significant difference, but the mathematical version is still a rough approximation inspired by the biological one.
  - srean 2y ago
    Sandpile mathematics, something that is studied by computer scientists and mathematicians, would be an approximation. Its not so much the level of the spike that matters but how often they spike is what conveys the signal. The temporal behavior is supremely important. It used to be believed that the rate is all that matters, but now, no longer. There are ANN models that model these spike trains (that's what these 'avalanches' are called), these do work similar to real neurons, but they are not part of the deep neural network popularity [0,1]. Besides, backpropagation is not what goes on in the brain, its known to be biologically infeasible. So all in all the traditional ANNs are nothing like real neural networks. That's ok, aeroplanes do not fly like birds, but they do still 'fly'. [0] https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/