6 ms·
If anyone needed to be convinced that LLMs are not accurate models of the human brain they need look no further than these numbers. The smallest model under di
by lisper 4y ago
If anyone needed to be convinced that LLMs are not accurate models of the human brain they need look no further than these numbers. The smallest model under discussion requires "more books than are in the Kindle store on Amazon U.S." Humans obviously acquire language proficiency with a lot less input than that.
- flashgordon 4y agoTrue but the machinery (is the brain) that enables humans to acquire/process/understand that knowledge is a lot more complex and went through years of environmental conditioning?
- lisper 4y agoYes, but that environmental conditioning did not involve training on the contents of the Kindle library. And the results of that training are encoded in under a gigabyte. For the record, I'm not saying LLMs are not a huge step forward. They are. But they are not -- and cannot be -- the whole answer.
- GaggiX 4y agoI think the human brain is naturally predisposed to learn a language, an evolutionary bias. That helps tremendously compared to a tabula rasa.
- sgu999 4y agoIf we are going in that direction, I guess the few million years of evolution should also count
- flashgordon 4y agoNo totally. Not an ml guy by long shot. My understanding is llms are foundational models which iiuc are "base" models representing all human knowledge until some point in time. Now I'd argue that you don't have to count all of evolution but just the point from when you were born. I'd say the all the sensory experiences + knowledge input from the day a brain is activated (DOB) would be in the order of PBytes if not more?
- deleted 4y ago[deleted]
- scarmig 4y agoOne argument (that I don't necessarily buy) is that inputs from sensory perception play a major role in building a language model in humans. A human might not read TB of books to learn language, but if you put them in a sensory deprivation tank that only displayed a massive series of Unicode characters it would take forever for us to learn language. If we did the same for an ML model, perhaps those non-language training materials would help. (That said, DL architectures are obviously wildly different from how the human brain works. E.g. backprop is physically impossible.)
- ml_basics 4y agoYep totally agreed. One of the things I'm excited to see going forward is multimodal models (trained e.g. on text + video + audio + images). I'm sure there's a lot more to it than this, but maybe one factor that makes humans a lot more data efficient is the multimodal input we receive. If that's the case, imagine how much better things could get when we train with all the videos, podcasts, radio etc in the world, in addition to all the text out there!
- fatneckbeard 4y agoHelen Keller managed to do pretty well without sight or hearing.
- youssefabdelm 4y agoTouch is pretty important though...
- ttpphd 4y ago[flagged]
- undersuit 4y agoShe was brought up in respect to a human in a sensory deprivation tank. You've made the connection to LLMs.
- mindvirus 4y agoFair point, but accurate versus efficient are different. While humans do acquire language much more efficiently, these models "know" more than a human about most things - in that if you asked it to describe each of the 100 most popular subjects or books it would have no trouble doing so (in a dozen languages no less). So I don't think it's quite an apples to apples comparison.
- lisper 4y ago> these models "know" more than a human about most things So does Wikipedia. But that's not a good model of the human brain either. > I don't think it's quite an apples to apples comparison. Yes. That is exactly my point. Despite the superficially similar I/O behavior, the two systems are very different under the hood.
- fastball 4y agoI don't think anyone has argued that LLMs are effectively digital humans or digital human brains, so seems like you're strawmanning a bit here.
- krainboltgreene 4y agoYou aren't reading enough of HN/Twitter then.
- int_19h 4y agoTwitter is a madhouse, but as for HN, the closest I've seen anyone come to that here is claiming that the difference in underlying structure can/does lead to emergence of the same phenomena.
- stevenhuang 4y agoNot as strong an argument as you think as you're forgetting humans have 5 senses, are embodied, and millions of years of tuned weights from natural evolution codified into our brain structure.
- neatze 4y agoBrain is nothing like attention networks at least in terms of dynamics and individual neuron complexity, my wild ass guess; there is not enough space in DNA to store even 10% of brain connections. What weights are you referring to ?
- BobbyJo 4y agoCluster density, cluster location, eagerness to connect, distribution of cell types, etc. are all coded in DNA and contribute to creating "initial weights". Neuron "weights" need not be stored completely uncompressed. Obviously newborns need to develop and take in stimulus before they "know" anything, so the initial conditions are not sufficient, but they are obviously necessary to make the limited learning useful.
- neatze 4y agoIt always seems naive to compare neural structures and functions to artificial networks, dynamics are simply not there, even single neural would require multi layered network (if not mistaken 7 layer network, and very long training time), it is fascinating how brain self-(re)organizes through out life time from small number of examples here and there. As far my understanding goes to very large degree it is unknown how to model such dynamics, where it would be possible to start with no spiking neurons and evolve effective/stable learning behavior from small number of examples/experiences.
- stevenhuang 4y agoFor all we know it may not even be necessary to model the higher fidelity aspects of our brains for a form of intelligence to emerge. There may be many roads to intelligence. The path biology and evolution took may just be one such path.
- gumballindie 4y agoThis is a hallucination.
- kolinko 4y agoOur brains are come pre-trained for speech from a few million years of evolution. Artifitial NN are trained from scratch. A chimpanzee can listen to humans as much as it wants, and it will still not pick up much of the language.
- jxy 4y agoRight. The nature trained all the nervous systems with genetic algorithms for millennia. Individuals rather fine tuned their own with natural noise and dropoffs and natural feedback. Fine tuning our brains are rather expensive.
- est31 4y agoThere are a few innate reflexes but you lose them quickly and they are pretty basic. Our brains have way way more neurons/parameters than these models and are still not overfitting... the issue lies in the learning method.
- famouswaffles 4y agoIt's not just about reflexes though I would hardly classify reflexes as "lost quickly". That's just not true. It's deeper than that. Our brains are optimized for a lot general human functions. Learning language is one of them. There's a whole section of the brain dedicated to it and other things. There's also a lot of vital biological information encoded in DNA/RNA. We are not even close to starting from scratch.
- deleted 4y ago[deleted]
- mrshadowgoose 4y agoThat argument is utterly unconvincing, as you've conveniently left out the millions of years of "training" and "fine tuning" that has occurred through the evolutionary process.
- nl 4y agoWhich implies the brain has more than a structural bias for language; ie language and/or knowledge is somehow passed genetically. There's some reasons to suspect this is at least partially true, but to what extent is unknown and contraversial.
- marginalia_nu 4y agoIt could be argued based on the extreme difficulty we're having in teaching animals human language, even primates. We can create a semiotic system and get a chimpanzee to communicate through it, but it won't learn English no matter how hard we try.
- nl 4y agoI think that's generally accepted to be that humans have a very large language center in the brain; IE: and inductive bias towards learning language (and I think there is something about our mouth and tongue being able to make specific sounds?) It doesn't say anything about that knowledge being precoded.
- marginalia_nu 4y agoIsn't a language center by itself precoded human toward the acquisition of human language (which has co-evolved to make use of our language center)? I don't understand how it could be any other way.
- nl 4y agoExactly. But it is unclear if there is intrinsic language knowledge somehow encoded too (which is what the OP is implying: https://news.ycombinator.com/item?id=35590091 https://news.ycombinator.com/item?id=35590091)
- Dylan16807 4y agoThat depends on what you mean by "accurate". If you mean a lot of accuracy, that's obvious and doesn't really need argument. And this new fact doesn't change the argument. If you mean a more moderate amount of accuracy, this isn't proof either way. Human brains take in less text but they put a lot more processing into it.
- jacquesm 4y agoTrue, but humans have a 4B element text book downloaded into each cell and some of that stuff encodes for a machine that already has a lot of language proficiency built in from day #1. I've always wondered how much of that genetic code is 'soft' in the sense that it pre-sets certain bits in the brains that it generates. Sort of a boot-strap ROM for a human.
- mupuff1234 4y agoDoes someone claim that they are? But regardless, the architecture of the human brain doesn't have to be the only way to get to AGI (not that LLMs are necessarily the way)
- teruakohatu 4y agoYet we learning from 18 hours of ultra high resolution video per day (x2), along with 18 hours of reinforcement learning, along with 18 hours of audio and 18 hours of nerve stimulus data. This is assuming we are learning nothing during sleep, which probably isn't true. By the time a person is 21 years old, they have been trained on at least 1 petabyte of data. By two years old, about 125TB of data. It makes LLMs look quite good in comparison.
- abetusk 4y agoBy no means rigorous, an estimate for the number of neurons is around 10^10 [0] with an average number of connections estimated at around 8000 [1]. Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such). [0] https://en.wikipedia.org/wiki/Human_brain#Microanatomy https://en.wikipedia.org/wiki/Human_brain#Microanatomy [1] https://psychology.stackexchange.com/questions/7967/how-many-synapses-in-the-average-human-brain https://psychology.stackexchange.com/questions/7967/how-many...
- edulix 4y agoAdd to that that our neurons are more complex to point neurons used in typical Artificial neural-nets. A single pyramid neuron in the neocortex might be more comparable to a multilayer neural net. https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1.full.pdf https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1....
- pmoriarty 4y agoTheir communication is much more complex too, and neither they nor their connections are static. We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created. Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net. Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs. The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.
- H8crilA 4y agoWhy do they have to be accurate models? They just have to be possible to construct and have to perform. The construction method is irrelevant beyond feasibility. I don't care if my model needs an exabyte of RAM if I can just go and buy that much RAM one day.
- deleted 4y ago[deleted]
- est31 4y agoThey aren't accurate models of the human brain just as airplanes are not flapping their wings and cars don't walk on legs. Our brain's way of achieving intelligence is not the only one out there, even if LLMs aren't there yet. Yes, we use now magnitudes more oil than we used to 120 years ago. But back then oil was used for cooking and lamps, while now it is used for so many more things. Same goes for data. It is the new oil :).
- fnordpiglet 4y agoI’d note that the models don’t just acquire language facility, but the ability to use language with alacrity about almost any subject. There’s no human alive that is remotely as capable, and most I’ve met hallucinate more.
- marginalia_nu 4y agoWhile I agree with the conclusion, I don't think your argument actually supports it. Humans for the most part don't acquire language through books until they're mostly already fluent in their native language. These LLMs are also trained on arbitrary books. When acquiring new languages, we use educational materials that are specifically created to facilitate an understanding of language. Not just arbitrary books in random order.
- deleted 4y ago[deleted]