3 ms·

It’s literally simulating a neural network. How much of your 5-sense experiential memories and decades of academic book learning are you bringing to understand

It’s literally simulating a neural network. How much of your 5-sense experiential memories and decades of academic book learning are you bringing to understand my reply to your post? How many gigabytes do you think that’s equivalent to?

_false 3y ago
I love both parent post perspectives on this.
skybrian 3y ago
Jamba seems to be distributed as 21 5-gigabyte files [1] so I guess that’s another way of looking at it. [1] https://huggingface.co/ai21labs/Jamba-v0.1/tree/main https://huggingface.co/ai21labs/Jamba-v0.1/tree/main
- imtringued 3y ago
  So what? I have seen models distributed as 26x 10GB files.
richardw 3y ago
It’s kinda simulating our brains but not really. When I attempted to dig more into how neurons work I realised that it’s a massive chasm of difference. Very much worth doing if you haven’t (you might know far better then me, this is for people who don’t yet.) In terms of results: Our brains are working with 20w of power and can be trained to compete with LLM’s using a tiny fraction of the world’s data. They also have to keep you breathing and your blood pumping and manage all the dangers of catching a ball near traffic. Or skiing, or poetry, or sunsets. And they remember stuff five minutes later and don’t need a training run that takes months. We have SO many opportunities to improve the AI architecture it’s ridiculous. This is a good thing.
- reissbaker 3y ago
  To be fair most of the brain is more like a pretrained model — it isn't being trained at any point after conception to keep your blood pumping or your lungs working, it does that out of the box roughly as soon as you sprout those organs (or the minute you're born, in the case of lungs). The training process was billions of years of evolution. And, well, given fairly persistent cross-cultural cognitive biases, I expect the conscious thought parts are starting from a pretrained model, too, and all we're doing in school is finetuning ;)
- imtringued 3y ago
  People don't understand that to simulate a single neuron, you need an entire neural network. So 70 billion parameters might at best be equivalent to a million neurons but that is assuming that your neural network architecture is akin to the connections between neurons. Considering the physical sparsity, you might need even more parameters to model the connections of a biological neural network. So less than a million neurons in practice.

_false 3y ago

I love both parent post perspectives on this.

skybrian 3y ago

Jamba seems to be distributed as 21 5-gigabyte files [1] so I guess that’s another way of looking at it. [1] https://huggingface.co/ai21labs/Jamba-v0.1/tree/main https://huggingface.co/ai21labs/Jamba-v0.1/tree/main

imtringued 3y ago
So what? I have seen models distributed as 26x 10GB files.

richardw 3y ago

It’s kinda simulating our brains but not really. When I attempted to dig more into how neurons work I realised that it’s a massive chasm of difference. Very much worth doing if you haven’t (you might know far better then me, this is for people who don’t yet.) In terms of results: Our brains are working with 20w of power and can be trained to compete with LLM’s using a tiny fraction of the world’s data. They also have to keep you breathing and your blood pumping and manage all the dangers of catching a ball near traffic. Or skiing, or poetry, or sunsets. And they remember stuff five minutes later and don’t need a training run that takes months. We have SO many opportunities to improve the AI architecture it’s ridiculous. This is a good thing.

reissbaker 3y ago
To be fair most of the brain is more like a pretrained model — it isn't being trained at any point after conception to keep your blood pumping or your lungs working, it does that out of the box roughly as soon as you sprout those organs (or the minute you're born, in the case of lungs). The training process was billions of years of evolution. And, well, given fairly persistent cross-cultural cognitive biases, I expect the conscious thought parts are starting from a pretrained model, too, and all we're doing in school is finetuning ;)
imtringued 3y ago
People don't understand that to simulate a single neuron, you need an entire neural network. So 70 billion parameters might at best be equivalent to a million neurons but that is assuming that your neural network architecture is akin to the connections between neurons. Considering the physical sparsity, you might need even more parameters to model the connections of a biological neural network. So less than a million neurons in practice.