3 ms·

As large as the absolute largest models are today, they are still microscopic compared to our brains. A 1.7T param model would only store an actual total of abo

As large as the absolute largest models are today, they are still microscopic compared to our brains. A 1.7T param model would only store an actual total of about 850 GB if fully saturated (4 bits of information per weight estimated for bf16 transformers), a lot less than a human brain with 150T synapses running in full analog precision. We need to scale the current gen of models at least another 10-100x to even reach the human level of complexity, something we'll be able to do in the next two decades. And well then there's going beyond just text. Current multimodal models are basically patchwork bullshit, separately trained image/audio to text/embeddings encoders slapped onto an existing model and hoping it does something neat. Tokenization and samplers are likewise bullshit that's there to compensate for lack of training compute. Once we have enough to be able to brute force it properly with bytes in, bytes out, regardless of data format, the results should be way better.

charlie0 2y ago
Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't.
- moffkalast 2y ago
  Well once again it turns out that what is hard for people is easy for computers, and vice versa. The things we go to college for 6 years for they can (relatively) master in a week of pretraining. We are optimized to smartly kill things, eat them, and reproduce, that's what machines will beat us at last lol. Right now a human expert is still obviously better in depth, but nowhere close in breadth. Probably not for much longer though, at least on the historical time scale. And granted a lot of parts of the human brain are dedicated to specific jobs that are entirely nonexistent in a normal LLM (kinematics, vision, sound, touch, taste, smell, autonomic organ control) so the actual bit we should be comparing for just language and reasoning is way smaller. Still the brain is pretty efficient inference energy wise, it's like the ultimate mixture of experts, extreme amounts of sparse storage and most of it is not computed until needed. The router must be pretty good.
- jampekka 2y ago
  OTOH humans can e.g. walk on two feet and drive a car.
  - fragmede 2y ago
    Boston Dynamics and Waymo might not have gotten human levels of competency with those two particular tasks, but we've already got robots that are better than drunk/tired/angry humans at it, and they're getting better at it.
- Jensson 2y ago
  > Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is Until you have AGI you can't say this since until then we don't know how much the different parts costs to replace with AI systems.
- tim333 2y ago
  In terms of energy use the human brain is way more efficient than LLMs. It's a completely different hardware model - the brain may have trillions of synapses but they only fire occasionally. I agree you have to compare more on the results than number of synapses etc. It's something that gives me pause on the idea that we have to build many GW of power stations. It may be possible to get much more energy efficient AI via better algorithms.
- onemoresoop 2y ago
  > Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't. Meanwhile the best class AI is trained on the output of human intelligence. When AI can learn by itself in the same way humans do (or even more efficient ways) that's when we can say human intelligence has been surpassed. Until then it's just tools for humans to use to augment their intelligence.
throw310822 2y ago
> 150T synapses running in full analog precision Analog systems are not known for being very precise- they're noisy, signals get corrupted easily- and that's why we prefer digital ones. As soon as we had the technology, we switched everything we could- audio and video recording, telephone calls, photography, to a digital medium. This makes me wonder if the seemingly extraordinary efficiency of artificial neural networks is simply due to the precision with which they can be trained.
- Terr_ 2y ago
  With respect to brain activity, how do we know its really noise, and not just layers of meaning--or at least purpose--which we don't yet understand? If straightforward binary signaling was so universally superior, I think the worldwide network of over a quintillion ruthlessly self-replicating nanobots would be using a much more heavily after the last billion years.
  - moffkalast 2y ago
    Electrical analog systems are since there's 1001 ways to create or conduct away electricity that adds noise, chemical ones might not suffer nearly as much too. There's also an interesting bit I've observed with LLMs, quantization in range of 4-8 bits doesn't reduce performance as much as it mainly reduces consistency. If you generate an answer a bunch of times and take the average you'll end up with roughly the same result as an fp16/32 would do every time. In nature, being inconsistent is usually weighed negatively with death... so that's probably why it hasn't caught on even if it is more efficient. Or this is enough of a different abstraction that we can't draw parallels anyway.
  - planktonas 2y ago
    Or we are just the biological bootloader for further evolution of binary signalling.

charlie0 2y ago

Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't.

moffkalast 2y ago
Well once again it turns out that what is hard for people is easy for computers, and vice versa. The things we go to college for 6 years for they can (relatively) master in a week of pretraining. We are optimized to smartly kill things, eat them, and reproduce, that's what machines will beat us at last lol. Right now a human expert is still obviously better in depth, but nowhere close in breadth. Probably not for much longer though, at least on the historical time scale. And granted a lot of parts of the human brain are dedicated to specific jobs that are entirely nonexistent in a normal LLM (kinematics, vision, sound, touch, taste, smell, autonomic organ control) so the actual bit we should be comparing for just language and reasoning is way smaller. Still the brain is pretty efficient inference energy wise, it's like the ultimate mixture of experts, extreme amounts of sparse storage and most of it is not computed until needed. The router must be pretty good.
jampekka 2y ago
OTOH humans can e.g. walk on two feet and drive a car.
- fragmede 2y ago
  Boston Dynamics and Waymo might not have gotten human levels of competency with those two particular tasks, but we've already got robots that are better than drunk/tired/angry humans at it, and they're getting better at it.
Jensson 2y ago
> Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is Until you have AGI you can't say this since until then we don't know how much the different parts costs to replace with AI systems.
tim333 2y ago
In terms of energy use the human brain is way more efficient than LLMs. It's a completely different hardware model - the brain may have trillions of synapses but they only fire occasionally. I agree you have to compare more on the results than number of synapses etc. It's something that gives me pause on the idea that we have to build many GW of power stations. It may be possible to get much more energy efficient AI via better algorithms.
onemoresoop 2y ago
> Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't. Meanwhile the best class AI is trained on the output of human intelligence. When AI can learn by itself in the same way humans do (or even more efficient ways) that's when we can say human intelligence has been surpassed. Until then it's just tools for humans to use to augment their intelligence.

throw310822 2y ago

> 150T synapses running in full analog precision Analog systems are not known for being very precise- they're noisy, signals get corrupted easily- and that's why we prefer digital ones. As soon as we had the technology, we switched everything we could- audio and video recording, telephone calls, photography, to a digital medium. This makes me wonder if the seemingly extraordinary efficiency of artificial neural networks is simply due to the precision with which they can be trained.

Terr_ 2y ago
With respect to brain activity, how do we know its really noise, and not just layers of meaning--or at least purpose--which we don't yet understand? If straightforward binary signaling was so universally superior, I think the worldwide network of over a quintillion ruthlessly self-replicating nanobots would be using a much more heavily after the last billion years.
- moffkalast 2y ago
  Electrical analog systems are since there's 1001 ways to create or conduct away electricity that adds noise, chemical ones might not suffer nearly as much too. There's also an interesting bit I've observed with LLMs, quantization in range of 4-8 bits doesn't reduce performance as much as it mainly reduces consistency. If you generate an answer a bunch of times and take the average you'll end up with roughly the same result as an fp16/32 would do every time. In nature, being inconsistent is usually weighed negatively with death... so that's probably why it hasn't caught on even if it is more efficient. Or this is enough of a different abstraction that we can't draw parallels anyway.
- planktonas 2y ago
  Or we are just the biological bootloader for further evolution of binary signalling.