3 ms·

To be honest… it’s amazing it can have any understanding given the only “senses” it has are the 1024 inputs of d_model! Imagine trying to understand the world

by h_tbob 2y ago

To be honest… it’s amazing it can have any understanding given the only “senses” it has are the 1024 inputs of d_model! Imagine trying to understand the world if you were simply given books and books in a language you had never read… and you didn’t even know how to read or even talk! So it’s pretty incredible it’s got this far!

pjerem 2y ago
I mean, I’m amazed by LLMs. But what you describe is basically done by any human on Earth : you are born not knowing how to read or how to talk and after years of learning, reading all the books may give you some understanding of the world. Contrary to LLM though, human brains don’t have a virtually infinite energy supply, cannot be parallelized and have to dedicate their already scarce energy to do a lot of other things than reading books including moving in a 3D world, living in a society, feeding themselves, doing their own hardware maintenance (sleep …), pay attention not to die every single day etc etc. So, for sure, LLMs _algorithms_ are really incredible, but they are also useful only if you throw a lot of hardware and energy into them. I’d be curious to see how long you’d need to train (not use) a useful LLM with only 20W of power (which is more or less the power we estimate the brain is using to function). We can still be impressed by the results, but not really by the speed. And when you have to learn the entire written corpus in some weeks/months, speed is pretty useful.
- corimaith 2y ago
  Pretty sure the human brain is far more parallelized than a regular CPU or GPU. Our image recognition for example probably dosen't take shortcuts like convulution because of the cost of processing each "pixel", we directly do it all with those millions of eye neurons. Well, to be fair there is alot of post-processing and "magic" involved in getting the final image.
  - danielbln 2y ago
    There is soooo much post processing happening in the visual cortex. The image from the eye is crude, only in focus in a tiny area, warped and upside down. It's quite incredible how our brains create this seamless virtual reality out of that input soup.
- jjk166 2y ago
  > But what you describe is basically done by any human on Earth You don't simply learn on training data, you are taught by other humans who were in turn taught by more humans still. When you do learn genuinely new information, you know exactly how it makes sense to you, and you can share it with others in a way optimized for them to incorporate it as well. LLMs aren't just learning to speak and read, they are developing from scratch a framework for comprehending the training data thrown at them. An LLM has no concept of phonetics because it has no ears with which to hear nor a mouth with which to speak. It derived english from pulses of electricity. Leave a child in a box with no human contact and a library of books and perhaps it will ascribe meaning to the symbols written on the pages, but it will never know what they actually mean. Even if they lived a million lifetimes, and were able to recognize that the symbols appear with certain regularity and could infer some rules of the language, they would never know what concept the string "comprehension" refers to.

by h_tbob 2y ago

pjerem 2y ago

I mean, I’m amazed by LLMs. But what you describe is basically done by any human on Earth : you are born not knowing how to read or how to talk and after years of learning, reading all the books may give you some understanding of the world. Contrary to LLM though, human brains don’t have a virtually infinite energy supply, cannot be parallelized and have to dedicate their already scarce energy to do a lot of other things than reading books including moving in a 3D world, living in a society, feeding themselves, doing their own hardware maintenance (sleep …), pay attention not to die every single day etc etc. So, for sure, LLMs _algorithms_ are really incredible, but they are also useful only if you throw a lot of hardware and energy into them. I’d be curious to see how long you’d need to train (not use) a useful LLM with only 20W of power (which is more or less the power we estimate the brain is using to function). We can still be impressed by the results, but not really by the speed. And when you have to learn the entire written corpus in some weeks/months, speed is pretty useful.

corimaith 2y ago
Pretty sure the human brain is far more parallelized than a regular CPU or GPU. Our image recognition for example probably dosen't take shortcuts like convulution because of the cost of processing each "pixel", we directly do it all with those millions of eye neurons. Well, to be fair there is alot of post-processing and "magic" involved in getting the final image.
- danielbln 2y ago
  There is soooo much post processing happening in the visual cortex. The image from the eye is crude, only in focus in a tiny area, warped and upside down. It's quite incredible how our brains create this seamless virtual reality out of that input soup.
jjk166 2y ago
> But what you describe is basically done by any human on Earth You don't simply learn on training data, you are taught by other humans who were in turn taught by more humans still. When you do learn genuinely new information, you know exactly how it makes sense to you, and you can share it with others in a way optimized for them to incorporate it as well. LLMs aren't just learning to speak and read, they are developing from scratch a framework for comprehending the training data thrown at them. An LLM has no concept of phonetics because it has no ears with which to hear nor a mouth with which to speak. It derived english from pulses of electricity. Leave a child in a box with no human contact and a library of books and perhaps it will ascribe meaning to the symbols written on the pages, but it will never know what they actually mean. Even if they lived a million lifetimes, and were able to recognize that the symbols appear with certain regularity and could infer some rules of the language, they would never know what concept the string "comprehension" refers to.