3 ms·

> the latest LLM models have almost as many parameters as we have neurons I often see this take, but the apt comparison is between parameter and synapse count,

> the latest LLM models have almost as many parameters as we have neurons I often see this take, but the apt comparison is between parameter and synapse count, not neuron count. You should be counting hidden units rather than weights if you want to compare to neuron counts.

markhahn 2y ago
I think you should compare synapse and weight counts if you want a measure of the network's state/capacity. If you want something closer to its compute power, compare neurons and hidden units.
- RaftPeople 2y ago
  > I think you should compare synapse and weight counts if you want a measure of the network's state/capacity. If you want something closer to its compute power, compare neurons and hidden units. Even this is far too simple. 1-Astrocytes have processes (extensions) with localized calcium wave signaling as well as global cell calcium wave signaling that is involved in computation of some types of sensory inputs 2-Astrocytes detect and produce neuro-transmitters as well as glial-transmitters (and generally control/influence the synapses between neurons) 3-Neurons have tunneling nanotubes that dynamically connect cells together (on short time scales) and can transfer proteins, calcium (action potentials), etc. 4-Some types of neurons have natural resonance properties which impacts their function and processing of inputs 5-Some types of neurons learn input patterns in isolation (purkinje cells), meaning one single neuron can learn a time series of input and respond appropriately when pattern detected etc. There is a long list of interesting and unknown capabilities, it's pretty difficult to compare neurons+cells to an ANN at this point, too much unknown.
Bjartr 2y ago
To expand on that, as a point of comparison, a single neuron can have thousands of synaptic connections. So we're still a few orders of magnitude out from modeling NNs that have a degree of connectivity similar to the brain, even though the synapse counts are comparable.

markhahn 2y ago

I think you should compare synapse and weight counts if you want a measure of the network's state/capacity. If you want something closer to its compute power, compare neurons and hidden units.

RaftPeople 2y ago
> I think you should compare synapse and weight counts if you want a measure of the network's state/capacity. If you want something closer to its compute power, compare neurons and hidden units. Even this is far too simple. 1-Astrocytes have processes (extensions) with localized calcium wave signaling as well as global cell calcium wave signaling that is involved in computation of some types of sensory inputs 2-Astrocytes detect and produce neuro-transmitters as well as glial-transmitters (and generally control/influence the synapses between neurons) 3-Neurons have tunneling nanotubes that dynamically connect cells together (on short time scales) and can transfer proteins, calcium (action potentials), etc. 4-Some types of neurons have natural resonance properties which impacts their function and processing of inputs 5-Some types of neurons learn input patterns in isolation (purkinje cells), meaning one single neuron can learn a time series of input and respond appropriately when pattern detected etc. There is a long list of interesting and unknown capabilities, it's pretty difficult to compare neurons+cells to an ANN at this point, too much unknown.

Bjartr 2y ago

To expand on that, as a point of comparison, a single neuron can have thousands of synaptic connections. So we're still a few orders of magnitude out from modeling NNs that have a degree of connectivity similar to the brain, even though the synapse counts are comparable.