3 ms·

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approxim

>> one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name And yet, artificial neural networks ARE an approximation of how biological neurons work. It is worth noting that they came out of neurobiology and not some math department - well at least in the forward direction, I'm not sure who came up with the training algorithms (probably the math folks). Should they be considered mystical? No. I would also posit that biological neurons are more efficient and probably have better learning algorithms than artificial ones today. I'm confused as to why some people seem to shun the biological equivalence of these things. In a recent thread here I learned that physical synaptic weights (in our brains) are at least partly stored in DNA or its methylation. If that isn't fascinating I'm not sure what is. Or is it more along the lines of intelligence can be reduced to a large number of simple things, and biology has given us an interesting physical implementation?

kragen 3y ago
anns originated in hypotheses about how neurobiology might work in the 01940s but diverged completely from neurobiology in the 01960s; they contain nothing we've learned about neurons in the last 50 years, and not much from before that either (they don't, for example, do hebbian learning). current anns use training methods like gradient descent with momentum and activation functions like relu which have no plausible biological realization artificial neural networks are an approximation of biological neural networks in the same way that a submarine is an approximation of a fish
xanderlewis 3y ago
As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neuron’ will give you some idea. The original idea of approximating something like a neuron using a weighted sum (which is a fairly obvious idea, given the initial discovery that neurons become ‘activated’ and they do so in proportion to how much the neurons they are connected to are) did come from thinking about biological brains, but the mathematical building blocks are incredibly simple and are hundreds of years old, if not thousands.
- naasking 3y ago
  > the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then that just increases the number of artificial neural network neurons needed to describe it, it does not need any computation of a different kind.
  - xanderlewis 3y ago
    PowerPoint is Turing complete. Does that mean PowerPoint should be regarded as being biological or at least neuroscience-inspired?
    - naasking 3y ago
      No, but neural networks literally were inspired by biology so I'm not sure what your point is.
      - xanderlewis 3y ago
        My point is that you seem to think neurons in the sense of artificial neural networks and neurons in the human brain are equivalent because: (1) Neural networks are Turing complete, and hence can do anything brains can. [debatable anyway; We don’t know this to be the case since brains might be doing more than computation. Ask a philosopher or a cognitive scientist. Or Roger Penrose.] (2) Neural networks were very loosely inspired by the idea that the human brain is made up of interconnected nodes that ‘activate’ in proportion to how other related nodes do. I don’t think that’s nearly enough to say that they’re equivalent. For (1), we don’t yet know (and we’re not even close), and anyway: if you consider all Turing complete systems to be equivalent to the point of it being a waste of time to talk about their differences then you can say goodbye to quite a lot of work in theoretical computer science. For (2): so what? Lots of things are inspired by other things. It doesn’t make them in any sense equivalent, especially if the analogy is as weak as it is in this case. No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron. They operate on completely different principles, as we now know much better than when such things were first dreamed up.
srean 3y ago
> And yet, artificial neural networks ARE an approximation of how biological neurons work For a non-vapid/non-vacuous definition of 'approximation' this is not true at all. It is well understood that (i) back-propagation is biologically infeasible in the brain (ii) output 'voltage' is a transformed weighted average of the input 'voltage' -- is not how neurons operate. (ii) is in the 'not even wrong' category. Neurons operate in terms of spikes and frequency and quiescence of spiking. If you are interested any undergrad text in neurobiology will help correct the wrong notions.
chriswarbo 3y ago
> And yet, artificial neural networks ARE an approximation of how biological neurons work. Only if you limit yourself to "sums of weighted inputs, sent through a 1D activation function". However, the parent said "differentiable primitives": these days people have built networks that contain differentiable ray-tracers, differentiable physics simulations, etc. Those seem like crazy ideas if we limit ourselves to the "neural" analogy; but are quite natural for a "composition of differentiable primitives" approach.

kragen 3y ago

anns originated in hypotheses about how neurobiology might work in the 01940s but diverged completely from neurobiology in the 01960s; they contain nothing we've learned about neurons in the last 50 years, and not much from before that either (they don't, for example, do hebbian learning). current anns use training methods like gradient descent with momentum and activation functions like relu which have no plausible biological realization artificial neural networks are an approximation of biological neural networks in the same way that a submarine is an approximation of a fish

xanderlewis 3y ago

As the commenter below mentions, the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version. The neural network version is essentially just a weighted sum, with an extra layer of shaping applied afterwards to make it nonlinear. As far as I know, we still don’t understand all of the complexity about how biological neurons work. Even skimming the Wikipedia page for ‘neuron’ will give you some idea. The original idea of approximating something like a neuron using a weighted sum (which is a fairly obvious idea, given the initial discovery that neurons become ‘activated’ and they do so in proportion to how much the neurons they are connected to are) did come from thinking about biological brains, but the mathematical building blocks are incredibly simple and are hundreds of years old, if not thousands.

naasking 3y ago
> the biological version of a neuron (i.e. a neuron) is much more complicated than the neural network version This is a difference of degree not of kind, because neural networks are Turning complete. Whatever additional complexity the neuron has can itself be modelled as a neural network. Edit: meaning, that if the greater complexity of a biological neuron is relevant to its information processing component, then that just increases the number of artificial neural network neurons needed to describe it, it does not need any computation of a different kind.
- xanderlewis 3y ago
  PowerPoint is Turing complete. Does that mean PowerPoint should be regarded as being biological or at least neuroscience-inspired?
  - naasking 3y ago
    No, but neural networks literally were inspired by biology so I'm not sure what your point is.
    - xanderlewis 3y ago
      My point is that you seem to think neurons in the sense of artificial neural networks and neurons in the human brain are equivalent because: (1) Neural networks are Turing complete, and hence can do anything brains can. [debatable anyway; We don’t know this to be the case since brains might be doing more than computation. Ask a philosopher or a cognitive scientist. Or Roger Penrose.] (2) Neural networks were very loosely inspired by the idea that the human brain is made up of interconnected nodes that ‘activate’ in proportion to how other related nodes do. I don’t think that’s nearly enough to say that they’re equivalent. For (1), we don’t yet know (and we’re not even close), and anyway: if you consider all Turing complete systems to be equivalent to the point of it being a waste of time to talk about their differences then you can say goodbye to quite a lot of work in theoretical computer science. For (2): so what? Lots of things are inspired by other things. It doesn’t make them in any sense equivalent, especially if the analogy is as weak as it is in this case. No neuroscientist thinks that a weighted sum is an adequate (or even remotely accurate) model of a real biological neuron. They operate on completely different principles, as we now know much better than when such things were first dreamed up.

srean 3y ago

> And yet, artificial neural networks ARE an approximation of how biological neurons work For a non-vapid/non-vacuous definition of 'approximation' this is not true at all. It is well understood that (i) back-propagation is biologically infeasible in the brain (ii) output 'voltage' is a transformed weighted average of the input 'voltage' -- is not how neurons operate. (ii) is in the 'not even wrong' category. Neurons operate in terms of spikes and frequency and quiescence of spiking. If you are interested any undergrad text in neurobiology will help correct the wrong notions.

chriswarbo 3y ago

> And yet, artificial neural networks ARE an approximation of how biological neurons work. Only if you limit yourself to "sums of weighted inputs, sent through a 1D activation function". However, the parent said "differentiable primitives": these days people have built networks that contain differentiable ray-tracers, differentiable physics simulations, etc. Those seem like crazy ideas if we limit ourselves to the "neural" analogy; but are quite natural for a "composition of differentiable primitives" approach.