3 ms·

> Current hardware is easily up to the task. I don't think so. If you want to model a single synapse in full to capture all effects that might lead to "learnin

> Current hardware is easily up to the task. I don't think so. If you want to model a single synapse in full to capture all effects that might lead to "learning", you have a system of ordinary differential equations. Solving that is very hard, and solving that for 10 million neurons is impossible. On current hardware can only implement but a poor caricature of a real neuron.

carlmr 4y ago
While this is true, the complexity perspective misses something more fundamental. 1) Our brains, and moreso those of animals, come with a really good pretraining at birth. This is collective genetic knowledge of millions of generations distilled into your brain. 2) Our brains have a lot of sensors and actuators to interact with the world. We only learn by reading as adults when our brains can already do the synesthesia of translating words into thought. But even as adults, most of us learn better if we do something, write something, engage in dialog, instead of passively listening, reading, or watching. Passive data can never replicate the rich environment our brains grow up in.
- beisner 4y ago
  While true, there’s a relatively small upper bound on how many bits of information are in this pre-training. Specifically, in the form of how much information is contained in DNA, which is only a couple gigabytes.
  - lossolo 4y ago
    Stable diffusion model is around 4 gigabytes, inside that 4 gigabytes you have understanding of the whole english language model and mapping to billions of objects, people, concepts etc capable of generating from just a single sentence almost any picture in any style you imagine. Seems like a few gigabytes can hold a lot of information.
mkaic 4y ago
I think this requires the assumption that modeling the complexity of biological synapses is required for general intelligence, when we don't know that to be the case. Personally, I believe that it's not a requirement at all and that the first AGI will be strikingly non-neuromorphic. Just my two cents though.
rbanffy 4y ago
> On current hardware can only implement but a poor caricature of a real neuron. We don't need a complete physiological model for it to be useful. We don't need a perfectly accurate silicon-based mirror of a mammalian brain to outsmart ours on every task we do (and many we don't even realize we could). The challenge will be to coexist and cooperate with these completely alien intelligences that share almost nothing with ours.
- data_maan 4y ago
  I agree with your comment (but the OP point was not about which approximate model might still work)
solsane 4y ago
Real neurons are far slower (interaction is chemical vs electrical) and far less precise ( iirc something comparable to 4 - 7x less precise than 32bit float) than physical neurons.
Animats 4y ago
That problem has been overcome.[1] This is a neat result. This research started with the differential equation model of a neuron and tried to train various neural nets to get the same result to within 99%. They succeeded. Worst case took an 8-layer net with 256 elements per layer. See Fig. 4. So, 10 billion elements for a squirrel. Not that big by current standards. It's not clear that a model which tracks the biological neuron that accurately is needed. They discuss simpler models that are almost as good. Low-end mammal brains should be buildable right now. It's not a hardware limitation. [1] https://www.sciencedirect.com/science/article/pii/S0896627321005018 https://www.sciencedirect.com/science/article/pii/S089662732...

carlmr 4y ago

While this is true, the complexity perspective misses something more fundamental. 1) Our brains, and moreso those of animals, come with a really good pretraining at birth. This is collective genetic knowledge of millions of generations distilled into your brain. 2) Our brains have a lot of sensors and actuators to interact with the world. We only learn by reading as adults when our brains can already do the synesthesia of translating words into thought. But even as adults, most of us learn better if we do something, write something, engage in dialog, instead of passively listening, reading, or watching. Passive data can never replicate the rich environment our brains grow up in.

beisner 4y ago
While true, there’s a relatively small upper bound on how many bits of information are in this pre-training. Specifically, in the form of how much information is contained in DNA, which is only a couple gigabytes.
- lossolo 4y ago
  Stable diffusion model is around 4 gigabytes, inside that 4 gigabytes you have understanding of the whole english language model and mapping to billions of objects, people, concepts etc capable of generating from just a single sentence almost any picture in any style you imagine. Seems like a few gigabytes can hold a lot of information.

mkaic 4y ago

I think this requires the assumption that modeling the complexity of biological synapses is required for general intelligence, when we don't know that to be the case. Personally, I believe that it's not a requirement at all and that the first AGI will be strikingly non-neuromorphic. Just my two cents though.

rbanffy 4y ago

> On current hardware can only implement but a poor caricature of a real neuron. We don't need a complete physiological model for it to be useful. We don't need a perfectly accurate silicon-based mirror of a mammalian brain to outsmart ours on every task we do (and many we don't even realize we could). The challenge will be to coexist and cooperate with these completely alien intelligences that share almost nothing with ours.

data_maan 4y ago
I agree with your comment (but the OP point was not about which approximate model might still work)

solsane 4y ago

Real neurons are far slower (interaction is chemical vs electrical) and far less precise ( iirc something comparable to 4 - 7x less precise than 32bit float) than physical neurons.

Animats 4y ago

That problem has been overcome.[1] This is a neat result. This research started with the differential equation model of a neuron and tried to train various neural nets to get the same result to within 99%. They succeeded. Worst case took an 8-layer net with 256 elements per layer. See Fig. 4. So, 10 billion elements for a squirrel. Not that big by current standards. It's not clear that a model which tracks the biological neuron that accurately is needed. They discuss simpler models that are almost as good. Low-end mammal brains should be buildable right now. It's not a hardware limitation. [1] https://www.sciencedirect.com/science/article/pii/S0896627321005018 https://www.sciencedirect.com/science/article/pii/S089662732...