4 ms·

Here's a thought. Recently the whole connectome of the fruit fly has been mapped. 3000 neurons and 548k connections. The fruit flys brain is capable of pilotin

Here's a thought. Recently the whole connectome of the fruit fly has been mapped. 3000 neurons and 548k connections. The fruit flys brain is capable of piloting the body of the fruit fly on a precision and energy efficiency that's simply unmatched in any kind of aerial machine we have developed. And only with those many connections, half a million? Neurons are very complex computation machines that can module their response very carefully. The human brain does so much more than simply being able to process language. It's capable of one shot learning and to course correct incredibly easy. These features are very important for survivability and I would say are the "magic sauce" that we just haven't been able to replicate. So I'm actually not so sure this is simply a parameters game. There could be important structures and specialisation that we are missing. In the fruit fly there are 93 different types of neurons, in the human brain iirc it might be hundreds.

BulgarianIdiot 4y ago
A language model is also capable of one shot learning. Try it in the chat, you tell it something, once, and it incorporates it for the rest of the chat. :-) You may think I don't know what "one shot learning" means, but I do know, I'm just saying it's a problem of our perspective, not of the model's capability. Aside from it having no long term memory (by design) from the chat. Yet. Our high-level learning is absolutely nothing like how we learned as toddlers. It took us DECADES to understand complex concepts from "few shots". Only pretrained models can do that, and you're also one, in your adulthood. What you say about extra complexity from neuron types and neurotransmitters combining with one another to modulate even simple networks is true of course. But it's also true you don't need to replicate all those organic details in a NN, because they're equivalent to just a few more "regular" parameters. Think of an artificial neuron in a network as a Turing machine. A Turing machine can do everything with just some more tape, and a neural network can do everything with just some more parameters. And yet, our models show that we don't need AS many parameters as we thought. And that's exciting.
kragen 4y ago
half a million connections is about 500 times more connections than there are in a 6502, so structure clearly matters a lot

BulgarianIdiot 4y ago

A language model is also capable of one shot learning. Try it in the chat, you tell it something, once, and it incorporates it for the rest of the chat. :-) You may think I don't know what "one shot learning" means, but I do know, I'm just saying it's a problem of our perspective, not of the model's capability. Aside from it having no long term memory (by design) from the chat. Yet. Our high-level learning is absolutely nothing like how we learned as toddlers. It took us DECADES to understand complex concepts from "few shots". Only pretrained models can do that, and you're also one, in your adulthood. What you say about extra complexity from neuron types and neurotransmitters combining with one another to modulate even simple networks is true of course. But it's also true you don't need to replicate all those organic details in a NN, because they're equivalent to just a few more "regular" parameters. Think of an artificial neuron in a network as a Turing machine. A Turing machine can do everything with just some more tape, and a neural network can do everything with just some more parameters. And yet, our models show that we don't need AS many parameters as we thought. And that's exciting.

kragen 4y ago

half a million connections is about 500 times more connections than there are in a 6502, so structure clearly matters a lot