5 ms·
My body may have "ingested" it in some way but filtering starts even at the sense ("sensor") level. So there is no way I have learned from it. I didn't even hav
by depr 3y ago
My body may have "ingested" it in some way but filtering starts even at the sense ("sensor") level. So there is no way I have learned from it. I didn't even have to learn how to filter, it is part of my genetic makeup.
And even if I did, most of it would not be of the same category as what LLMs are learning; information about how a fabric feels or the sound of an ambulance doesn't teach me anything about programming or other things GPT can do. So when comparing the inputs, it makes no sense to count all the information that our senses are getting, to all the input for an LLM.
>This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning
Not really comparable
- SanderNL 3y agoI think I implied a strong connection between LLMs and how biological intelligence works, but I certainly didn’t intend to. I think you discount general knowledge acquired non-verbally too easily. The sound of an ambulance and how it moves through space and how it correlates with visual information represents an astronomical amount of information that can be generalized from handsomely. All these data streams have to be connected, they correlate and intertwine in fantastically complex ways. I think the sound of an ambulance does teach you things that eventually help you “program”. You have witnessed similar (non-)events thousands if not millions of times. Each time it was accompanied with shitloads of sensory data from all modalities, both external and internal. The base of general patterns you start out from once you are ready for language is staggering. Again not saying LLMs work like that, because they do not. All I mean to do is put their information requirements in perspective. We ingest a lot more than a bunch of books.