4 ms·

Comparisons between the number of neurons/synapses in the human brain and the number of parameters in these language models are meaningless. The computational p

Comparisons between the number of neurons/synapses in the human brain and the number of parameters in these language models are meaningless. The computational power of a single real neuron is immense, whereas an artificial neuron is trivial. For example, a real neuron performs complex signal processing on input action potentials (e.g. spike train decoding), and is capable of responding very differently to many different “encodings” of those potentials (i.e. neurotransmitters). On the other hand, an artificial neuron simply performs weighted sums of the inputs and passes them into trivial activation functions (e.g. sigmoid or ReLU). To me, these analogies are like comparing an ant colony to human civilization solely based on the number of individual organisms.

lysecret 5y ago
I mean yes, but you could say take a sufficiently large set of hidden units (say 10-100) with all their parameters and they could easily represent the kinds of functions you are talking about. So the comparrisonn is off (we probably need about 100 times more parameters than the brain has neurons if that is the right measure)
- skohan 5y ago
  I'm not totally convinced that hidden units are enough to model every neuronal system. For instance what the post above is alluding to with highly specific temporal information of encoding is not so easily represented in current ANN's as far as I'm aware. And there's also a ton of metabolic processes in the neuron which affect information processing in different ways at different time scales. I would not be so confident a neuron can be effectively modeled by just scaling up node count. And even if it can, that would mean you probably need an ANN of orders-of-magnitude greater size than neuron count to simulate a human brain with any accuracy. I.e. we might be talking about multiple nodes per synapse.
FeepingCreature 5y ago
Artificial neurons aren't all that simple either- the more complex your learning algorithms, the more additional state has to be associated with each neuron during training. And it's not clear how much of the computational work of a biological neuron is housekeeping overhead.
- robbedpeter 5y ago
  The output of a biological neuron doesn't appear to be graduated. When a neuron fires, it releases a consistent signal, and it is everything downstream from the activation where dynamic learning takes place. Some connections can lead to inhibitory or excitatory signaling, reducing or increasing the voltage differentials, suppressing or reinforcing contextual signals, but the output of a neuron is a binary event. The attenuation or reinforcement of output signals occurs discretely from the activation of neurons, so the myelin can get thicker in dendritic connections, or connections can be added and pruned, or chemistry changes can affect the signal in a dynamic way. Once we figure out the algorithms encoded in the connectome/physical architecture, we'll have the opportunity to uncover the algorithms encoded in the data being transported between regions of the brain.
leecarraher 5y ago
For reference this recent article describes research that tries to nail down the exact computational ratio https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... spoiler : 5-8 layers of artificial neurons per complexity of biological neuron
- tgb 5y ago
  That's how many it takes to "represent" a neuron. But what does that mean and is it relevant? I haven't checked the original research, but if it means to model the results of a neuron, then it's not relevant (unless you're trying to do brain uploading). Two ANNs with different activation functions will need a lot of nodes to model a single node of the other network, but they have comparable computational power. We don't need to match the neurons to match the computational abilities.
  - dxuh 5y ago
    What's the theoretical basis of this though? I don't have a theoretical background in computability, but in physics and there the complexity of a system very much correlates with the number of degrees of freedom. When simulating something, you want a bijection of the state of the simulation to the state of the system you are simulating, so you need to match (or exceed) the number of degrees of freedom. So if you know how many parameters you need to simulate another system until the simulation works, you can estimate how many internal degrees of freedom it has (degrees of freedom that are relevant to whatever you are measuring of course).
nmca 5y ago
Meaningless feels like a stretch here. Your evidence supports a claim much more like "mistaken by almost two orders of magnitude" - which at a sufficient growth rate might be moot.
svantana 5y ago
You can just as easily turn the comparison around: how many neurons does it take to compute a single 32-bit float multiplication? Clearly a lot, since >99.99% of humans are enable to do it without any tools. Whereas artificial nets rely on trillions of such operations per second. Computers and brains are quite different, but I would say computers are nearer to simulating a brain than vice versa.
- raducu 5y ago
  That is irrelevant because in all the neural nets that I'm aware of, the activation functions we are using are very simple non-linear functions. We only use fp multiplication because that' the way we add up the weights to activations, nature has analogue ways of doing that.
- qualudeheart 5y ago
  I believe that to be conflating conscious thought and unconscious thought. The brain has to be able to do fairly complex physics simulations just predicting how fast you’re going somewhere or wether a projectile flying through air will hit you. Consciously doing that is extremely hard (most people can’t do that math) but thatms just because consciousness is computationally inefficient. Most brain computation is unconscious and doesn’t have that limit.
raducu 5y ago
I always thought the big advantage of human/biological neurons was that they each could have 10.000 axons/dendrites and each one of those could also interact with thousands and thousands of neurons; and new connections could be created on the fly. And I'm sure some neurons could perform complex computations themselves, but artificial neurons can also do that, but so far we haven't found that to be particularly useful and only use a bunch of simple non-linear functions.

lysecret 5y ago

I mean yes, but you could say take a sufficiently large set of hidden units (say 10-100) with all their parameters and they could easily represent the kinds of functions you are talking about. So the comparrisonn is off (we probably need about 100 times more parameters than the brain has neurons if that is the right measure)

skohan 5y ago
I'm not totally convinced that hidden units are enough to model every neuronal system. For instance what the post above is alluding to with highly specific temporal information of encoding is not so easily represented in current ANN's as far as I'm aware. And there's also a ton of metabolic processes in the neuron which affect information processing in different ways at different time scales. I would not be so confident a neuron can be effectively modeled by just scaling up node count. And even if it can, that would mean you probably need an ANN of orders-of-magnitude greater size than neuron count to simulate a human brain with any accuracy. I.e. we might be talking about multiple nodes per synapse.

FeepingCreature 5y ago

Artificial neurons aren't all that simple either- the more complex your learning algorithms, the more additional state has to be associated with each neuron during training. And it's not clear how much of the computational work of a biological neuron is housekeeping overhead.

robbedpeter 5y ago
The output of a biological neuron doesn't appear to be graduated. When a neuron fires, it releases a consistent signal, and it is everything downstream from the activation where dynamic learning takes place. Some connections can lead to inhibitory or excitatory signaling, reducing or increasing the voltage differentials, suppressing or reinforcing contextual signals, but the output of a neuron is a binary event. The attenuation or reinforcement of output signals occurs discretely from the activation of neurons, so the myelin can get thicker in dendritic connections, or connections can be added and pruned, or chemistry changes can affect the signal in a dynamic way. Once we figure out the algorithms encoded in the connectome/physical architecture, we'll have the opportunity to uncover the algorithms encoded in the data being transported between regions of the brain.

leecarraher 5y ago

For reference this recent article describes research that tries to nail down the exact computational ratio https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/ https://www.quantamagazine.org/how-computationally-complex-i... spoiler : 5-8 layers of artificial neurons per complexity of biological neuron

tgb 5y ago
That's how many it takes to "represent" a neuron. But what does that mean and is it relevant? I haven't checked the original research, but if it means to model the results of a neuron, then it's not relevant (unless you're trying to do brain uploading). Two ANNs with different activation functions will need a lot of nodes to model a single node of the other network, but they have comparable computational power. We don't need to match the neurons to match the computational abilities.
- dxuh 5y ago
  What's the theoretical basis of this though? I don't have a theoretical background in computability, but in physics and there the complexity of a system very much correlates with the number of degrees of freedom. When simulating something, you want a bijection of the state of the simulation to the state of the system you are simulating, so you need to match (or exceed) the number of degrees of freedom. So if you know how many parameters you need to simulate another system until the simulation works, you can estimate how many internal degrees of freedom it has (degrees of freedom that are relevant to whatever you are measuring of course).

nmca 5y ago

Meaningless feels like a stretch here. Your evidence supports a claim much more like "mistaken by almost two orders of magnitude" - which at a sufficient growth rate might be moot.

svantana 5y ago

You can just as easily turn the comparison around: how many neurons does it take to compute a single 32-bit float multiplication? Clearly a lot, since >99.99% of humans are enable to do it without any tools. Whereas artificial nets rely on trillions of such operations per second. Computers and brains are quite different, but I would say computers are nearer to simulating a brain than vice versa.

raducu 5y ago
That is irrelevant because in all the neural nets that I'm aware of, the activation functions we are using are very simple non-linear functions. We only use fp multiplication because that' the way we add up the weights to activations, nature has analogue ways of doing that.
qualudeheart 5y ago
I believe that to be conflating conscious thought and unconscious thought. The brain has to be able to do fairly complex physics simulations just predicting how fast you’re going somewhere or wether a projectile flying through air will hit you. Consciously doing that is extremely hard (most people can’t do that math) but thatms just because consciousness is computationally inefficient. Most brain computation is unconscious and doesn’t have that limit.

raducu 5y ago

I always thought the big advantage of human/biological neurons was that they each could have 10.000 axons/dendrites and each one of those could also interact with thousands and thousands of neurons; and new connections could be created on the fly. And I'm sure some neurons could perform complex computations themselves, but artificial neurons can also do that, but so far we haven't found that to be particularly useful and only use a bunch of simple non-linear functions.