7 ms·

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, langu

Okay I think I qualify. I'll bite. LeCun's argument is this: 1) You can't learn an accurate world model just from text. 2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. He and people like Hinton and Bengio have been saying for a while that there are tasks that mice can understand that an AI can't. And that even have mouse-level intelligence will be a breakthrough, but we cannot achieve that through language learning alone. A simple example from "How Large Are Lions? Inducing Distributions over Quantitative Attributes" (https://arxiv.org/abs/1906.01327 https://arxiv.org/abs/1906.01327) is this: Learning the size of objects using pure text analysis requires significant gymnastics, while vision demonstrates physical size more easily. To determine the size of a lion you'll need to read thousands of sentences about lions, or you could look at two or three pictures. LeCun isn't saying that LLMs aren't useful. He's just concerned with bigger problems, like AGI, which he believes cannot be solved purely through linguistic analysis. The energy minimization architecture is more about joint multimodal learning. (Energy minimization is a very old idea. LeCun has been on about it for a while and it's less controversial these days. Back when everyone tried to have a probabilistic interpretation of neural models, it was expensive to compute the normalization term / partition function. Energy minimization basically said: Set up a sensible loss and minimize it.)

cm2012 2y ago
This seems strongly backed up by Claude Plays Pokemon
- fewhil 2y ago
  Isn't Claude Plays Pokemon using image input in addition to text? Not that it's perfect at it (some of its most glaring mistakes are when it just doesn't seem to understand what's on the screen correctly).
  - cm2012 2y ago
    Yes but because it's trained on text and in the backend, images are converted to tokens, it is absolutely dogshit at navigation and basic puzzles. It can't figure out what Squirrels can about how to achieve goals in a maze.
    - mountainriver 2y ago
      The images are converted to an embedding space the size of token embedding space. And the model is trained on that new embedding space. A joint representation of text and images is formed. It’s not as though the image is converted to text tokens.
codenlearn 2y ago
Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentially more efficient by leveraging human-curated information. Text can describe simulations of physical environments. Models might learn physical dynamics through textual descriptions of physics, video game logs, scientific papers, etc. A sufficiently comprehensive text corpus might contain enough information to develop reasonable physical intuition without direct sensory experience. As I'm typing this there is one reality that I'm understanding, the quality and completeness of the data fundamentally determines how well an AI system will work. and with just text this is hard to achieve and a multi modal experience is a must. thank you for explaining in very simple terms where I could understand
- furyofantares 2y ago
  > Doesn't Language itself encode multimodal experiences? When communicating between two entities with similar brains who have both had many thousands of hours of similar types of sensory experiences, yeah. When I read text I have a lot more than other text to relate it to in my mind; I bring to bear my experiences as a human in the world. The author is typically aware of this and effectively exploits this fact.
- danielmarkbruce 2y ago
  > Doesn't Language itself encode multimodal experiences Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we don't have great text datasets to describe many things in enough detail, but there isn't any reason we couldn't.
  - heyjamesknight 2y ago
    There are absolutely reasons that we cannot capture the entirety—or even a proper image—of human cognition in semantic space. Cognition is not purely semantic. It is dynamic, embodied, socially distributed, culturally extended, and conscious. LLMs are great semantic heuristic machines. But they don't even have access to those other components.
throw310822 2y ago
I don't get it. 1) Yes it's true, learning from text is very hard. But LLMs are multimodal now. 2) That "size of a lion" paper is from 2019, which is a geological era from now. The SOTA was GPT2 which was barely able to spit out coherent text. 3) Have you tried asking a mouse to play chess or reason its way through some physics problem or to write some code? I'm really curious in which benchmark are mice surpassing chatgpt/ grok/ claude etc.
- nextts 2y ago
  Mice can survive, forage, reproduce. Reproduce a mammal. There is a whole load of capability not available in an LLM. An LLM is essentially a search over a compressed dataset with a tiny bit of reasoning as emergent behaviour. Because it is a parrot that is why you get "hallucinations". The search failed (like when you get a bad result in Google) or the lossy compression failed or it's reasoning failed. Obviously there is a lot of stuff the LLM can find in its searches that are reminiscent of the great intelligence of the people writing for its training data. The magic trick is impressive because when we judge a human what do we do... an exam? an interview? Someone with a perfect memory can fool many people because most people only acquire memory from tacit knowledge. Most people need to live in Paris to become fluent in French. So we see a robot that has a tiny bit of reasoning and a brilliant memory as a brilliant mind. But this is an illusion. Here is an example: User: what is the French Revolution? Agent: The French Revolution was a period of political and societal change in France which began with the Estates General of 1789 and ended with the Coup of 18 Brumaire on 9 November 1799. Many of the revolution's ideas are considered fundamental principles of liberal democracy and its values remain central to modern French political discourse. Can you spot the trick?
  - pfisch 2y ago
    When you talk to ~3 year old children they hallucinate quite a lot. Really almost nonstop when you ask them about almost anything. I'm not convinced that what LLM's are doing is that far off the beaten path from our own cognition.
    - 2y ago
hintymad 2y ago
I'm curious why their claims are controversial. It seems pretty obvious to me that LLMs sometimes generate idiotic answers because the models lack common sense, do not have ability for deductive logical reasoning, let alone the ability to induce. And the current transformer architectures plus all the post-training techniques do not do anything to build such intelligence or the world model per LeCun's words.
jcims 2y ago
Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer. Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slipping on a cold sidewalk can be explained in a clinical fashion, but an android’s association of that to the powerful dynamic response required to even attempt to recover will give a newfound association and power to the word ‘slip’.
- ninetyninenine 2y ago
  LLM is just the name. You can encode anything into the "language" including pictures video and sound.
  - pessimizer 2y ago
    I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics, but I've been too lazy to do the research to find anybody working on anything like that, or to experiment myself with it.
    - Tepix 2y ago
      When you train a neural net for Donkeycar with camera images plus the joystick commands of the driver, isn't that close to nerve impulses already?
    - inetknght 2y ago
      > I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics Neuralink. Musk warning though. For reference, see Neuralink Launch Event at 59:33 [0], and continue watching through until Musk takes over again. The technical information there is highly relevant to a multi-modal AI model with sensory input/output. https://youtu.be/r-vbh3t7WVI?t=3575 https://youtu.be/r-vbh3t7WVI?t=3575
iainctduncan 2y ago
Thanks for articulating this so well. I'm a musician and music/CS phd student, and as a jazz improvisor of advanced skill (30+ years), I'm accutely aware that there are significant areas of intelligence for which linguistic thinking is not only not good enough, but something to be avoided as much as one can (which is bloody hard sometimes). I have found it so frustrating, but hard to figure out how to counter, that the current LLM zeitgeist seems to hinge on a belief that linguistic intelligence is both necessary and sufficient for AGI.
- kadushka 2y ago
  Most modern LLMs are multimodal.
  - yahoozoo 2y ago
    Does it really matter? At the end of the day, all the modalities and their architectures boil down to matrices of numbers and statistical probability. There’s no agency, no soul.
    - kadushka 2y ago
      At the end of the day, all modalities boil down to patterns of electrical activity in your brain.
      - yahoozoo 2y ago
        The brain is the important part. The electricity just keeps it going. And it’s more than numerical matrices.
        kadushka 2y ago
        You mean soul?
        namaria 2y ago
ninetyninenine 2y ago
>1) You can't learn an accurate world model just from text. >2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. LLMs can be trained with multimodal data. Language is only tokens and pixel and sound data can be encoded into tokens. All data can be serialized. You can train this thing on data we can't even comprehend. Here's the big question. It's clear we need less data then an LLM. But I think it's because evolution has pretrained our brains for this so we have brains geared towards specific things. Like we are geared towards walking, talking, reading, in the same way a cheetah is geared towards ground speed more then it is at flight. If we placed a human and an LLM in completely unfamiliar spaces and tried to train both with data. Which will perform better? And I mean completely non familiar spaces. Like let's make it non Euclidean space and only using sonar for visualization. Something totally foreign to reality as humans know it. I honestly think the LLM will beat us in this environment. We might've succeeded already in creating AGI it's just the G is too much. It's too general so it's learning everything from scratch and it can't catch up to us. Maybe what we need is to figure out how to bias the AI to think and be biased in the way humans are biased.
- kedarkhand 2y ago
  > non Euclidean space and only using sonar for visualization Pretty good idea for a video game!
- physicsguy 2y ago
  Hugely different data too? If you think of all the neurons connected up to vision, touch, hearing, heat receptors, balance, etc. there’s a constant stream of multimodal data of different types along with constant reinforcement learning - e.g. ‘if you move your eye in this way, the scene you see changes’, ‘if you tilt your body this way your balance changes’, etc. and this runs from even before you are born, throughout your life.
- MITSardine 2y ago
  Humans are more adaptable than you think: - echolocation in blind humans https://en.wikipedia.org/wiki/Human_echolocation https://en.wikipedia.org/wiki/Human_echolocation - sight through signals sent on tongue https://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ https://www.scientificamerican.com/article/device-lets-blind... In the latter case, I recall reading the people involved ended up perceiving these signals as a "first order" sense (not consciously treated information, but on an intuitive level like hearing or vision).
petesergeant 2y ago
> LeCun isn't saying that LLMs aren't useful. He's just concerned with bigger problems, like AGI, which he believes cannot be solved purely through linguistic analysis. It feels like special pleading: surely _this_ will be the problem class that doesn’t fall to “the bitter lesson”. My intuition is that the main problem with the current architecture is that mapping into tokens causes quantization that a real brain doesn’t have, and lack of plasticity. I don’t build models, I spend 100% of my time reading and adjusting model outputs though.
satellite2 2y ago
So blind people aren't generally sentient? (I'm obviously exaggerating a bit for the sake of the argument, but the point stands. Multimodality should not be a prerequisite to AGI)
- thousand_nights 2y ago
  blind people still have other senses including touch which gives them a size reference they can compare to. you can feel physical objects to gain an understanding of their size. the LLM is more like a brain in a vat with only one sensory input - a stream of text
somenameforme 2y ago
Is that what he's arguing? My perspective on what he's arguing is that LLMs effectively rely on a probabilistic approach to the next token based on the previous. When they're wrong, which the technology all but ensures will happen with some significant degree of frequency, you get cascading errors. It's like in science where we all build upon the shoulders of giants, but if it turns out that one of those shoulders was simply wrong, somehow, then everything built on top of it would be increasingly absurd. E.g. - how the assumption of a geocentric universe inevitably leads to epicycles which leads to ever more elaborate, and plainly wrong, 'outputs.' Without any 'understanding' or knowledge of what they're saying, they will remain irreconcilably dysfunctional. Hence the typical pattern with LLMs: --- How do I do [x]? You do [a]. No that's wrong because reasons. Oh I'm sorry. You're completely right. Thanks for correcting me. I'll keep that in mind. You do [b]. No that's also wrong because reasons. Oh I'm sorry. You're completely right. Thanks for correcting me. I'll keep that in mind. You do [a]. FML --- More advanced systems might add a c or a d, but it's just more noise before repeating the same pattern. Deep Seek's more visible (and lengthy) reasoning demonstrates this perhaps the most clearly. It just can't stop coming back to the same wrong (but statistically probable) answer and so ping-ponging off that (which it at least acknowledges is wrong due to user input) makes up basically the entirety of its reasoning phase.
- gsf_emergency_2 2y ago
  on "stochastic parrots" Table stakes for sentience: knowing when the best answer is not good enough.. try prompting LLMs with that.. It's related to LeCun's (and Ravid's) subtle question I mentioned in passing below: To Compress Or Not To Compress? (For even a vast majority of Humans, except tacitly, that is not a question!)
- tmaly 2y ago
  Right now, humans still have enough practice thinking to point out the errors, but what happens when humanity becomes increasingly dependent on LLMs to do this thinking?
gsf_emergency_2 2y ago
usual disclaimer: you decide on your own whether I'm an insider or not :) where LeCun might be prescient should intersect with the nemesis SCHMIDHUBER. They can't both be wrong, I suppose?! It's only "tangentially" related to energy minimization, technically speaking :) connection to multimodalities is spot-on. https://www.mdpi.com/1099-4300/26/3/252 https://www.mdpi.com/1099-4300/26/3/252 To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review With Ravid, double-handedly blue-flag MDPI! Sunmarized for the layman (propaganda?) https://archive.is/https://nyudatascience.medium.com/how-should-self-supervised-learning-models-represent-their-data-8d9ca64ff8d9 https://archive.is/https://nyudatascience.medium.com/how-sho... >When asked about practical applications and areas where these insights might be immediately used, Shwartz-Ziv highlighted the potential in multi-modalities and tabula Imho, best take I've seen on this thread (irony: literal energy minimization) https://news.ycombinator.com/item?id=43367126 https://news.ycombinator.com/item?id=43367126 Of course, this would make Google/OpenAI/DeepSeek wrong by two whole levels (both architecturally and conceptually)
PeterStuer 2y ago
Late 80's and 90's had the 'nouvelle AI' movement that argued embodiment was required for grounding the system into the shared world model. Without it symbols would be ungrounded and never achieve open world consistency. So unlike their knowledge system predecessors, a bit derogatory refered to as GOFAI (good old fashioned AI), nAI hawked back to cybernetics and multi layered dynamical systems rather than having explicit internal symbolic models. Braitenberg rather than blocksworld so to speak. Seems like we are back for another turn of the wheel in this aspect.
- implmntatio 2y ago
  > grounding the system into the shared world model before we fix certain things [..., 'corruption', Ponzi schemes, deliberate impediment of information flow to population segments and social classes, among other things, ... and a chain of command in hierarchies that are build on all that] is impossible. Why do smart people not talk about this at all? The least engineers and smart people should do is picking these fights for real. It's just a few interest groups, not all of them. I understand a certain balance is necessary in order to keep some systems from tipping over, aka "this is humanity, silly, this is who we are", but we are far from the point of efficient friction and it's only because "smart people" like LeCun et al are not picking those fights. How the hell do you expect to ground an ()AI in a world where elected ignorance amplifies bias and fallacies for power and profit while the literal shit is hitting all the fans via intended and unintended side effects? Any embodied AI will pretend until there is no way to deny that the smartest, brightest and the productive don't care about the system in any way but are just running algorithmically while ignoring what should not be ignored - should as in, an AI should be aligned with humanities interests and should be grounded into the shared world model.
  - PeterStuer 2y ago
    I hear you, but while you can have many layers of semantic obfuscation, no amount of sophistry will allow you to smash your face unharmed through a concrete wall. Reality is a hard mistress. In absense of being able to sense reality, post modernism can run truly unchecked.
brulard 2y ago
I don't know about telling better the size from a picture. I can imagine seeing 2 pictures of the moon. One is extreme telephoto showing moon next to a building and it looks real big. Then there would be another image where moon is a tiny speckle in the sky. How big is the moon? I would rather understand a text: "its radius is x km".
- veidr 2y ago
  I think the example is simplified to make its point efficiently, but also: the moon is something whose size would very likely be precisely explained in texts about it. While some hunting journals might brag about the weight of a lion that was killed, or whatever, most texts that I can recall reading about lions basically assumed you already know roughly how big a lion is; which indeed I learned from pictures as a pre-literate child. A good, precise spec is better that a few pictures, sure; the random text content of whatever training set you can scrape together, perhaps not (?)
- nonameiguess 2y ago
  Reading "its radius is x km" would mean nothing to you if you'd never experienced spatial extent directly, whether that be visually or just by moving through space and existing in it. You'd need to do exactly what is being said in the paper, read about thousands of other roughly spherical objects and their radii. At some point, you'd get a decent sense of relative sizes. On the other hand, if you ever simply see a meter stick, any statement that something measures a particular multiple or fraction of that you can already understand, without ever needing to learn the size of anything else.
qoez 2y ago
"LeCun has been on about it for a while and it's less controversial these days." Funny how that sentence could have been used 15 years ago too when he was right about persevering through neural network scepticism.
mcculley 2y ago
Are people born both blind and deaf incapable of true learning?
- grayhatter 2y ago
  There's a whole fork lore/meme around how hard it is, in America culture... But given blidness and deafness is an impediment to acquiring language, more than anything else, I'd say that's the exact opposite of the conclusions from the comment you're replying to. But yes, depending on where you set the bar for "true learning" being blind and deaf would prevent it. I assume you're asking if vision and sound are required for learning, the answer I assume is no. Those were just chosen because we've already invented cameras and microphones. Haptics are less common, and thus less talked about.
eli_gottlieb 2y ago
>(Energy minimization is a very old idea. LeCun has been on about it for a while and it's less controversial these days. Back when everyone tried to have a probabilistic interpretation of neural models, it was expensive to compute the normalization term / partition function. Energy minimization basically said: Set up a sensible loss and minimize it.) Ehhhh, energy-based models are trained via contrastive divergence, not just minimizing a simple loss averaged over the training data.

cm2012 2y ago

This seems strongly backed up by Claude Plays Pokemon

fewhil 2y ago
Isn't Claude Plays Pokemon using image input in addition to text? Not that it's perfect at it (some of its most glaring mistakes are when it just doesn't seem to understand what's on the screen correctly).
- cm2012 2y ago
  Yes but because it's trained on text and in the backend, images are converted to tokens, it is absolutely dogshit at navigation and basic puzzles. It can't figure out what Squirrels can about how to achieve goals in a maze.
  - mountainriver 2y ago
    The images are converted to an embedding space the size of token embedding space. And the model is trained on that new embedding space. A joint representation of text and images is formed. It’s not as though the image is converted to text tokens.

codenlearn 2y ago

Doesn't Language itself encode multimodal experiences? Let's take this case write when we write text, we have the skill and opportunity to encode the visual, tactile, and other sensory experiences into words. and the fact is llm's trained on massive text corpora are indirectly learning from human multimodal experiences translated into language. This might be less direct than firsthand sensory experience, but potentially more efficient by leveraging human-curated information. Text can describe simulations of physical environments. Models might learn physical dynamics through textual descriptions of physics, video game logs, scientific papers, etc. A sufficiently comprehensive text corpus might contain enough information to develop reasonable physical intuition without direct sensory experience. As I'm typing this there is one reality that I'm understanding, the quality and completeness of the data fundamentally determines how well an AI system will work. and with just text this is hard to achieve and a multi modal experience is a must. thank you for explaining in very simple terms where I could understand

furyofantares 2y ago
> Doesn't Language itself encode multimodal experiences? When communicating between two entities with similar brains who have both had many thousands of hours of similar types of sensory experiences, yeah. When I read text I have a lot more than other text to relate it to in my mind; I bring to bear my experiences as a human in the world. The author is typically aware of this and effectively exploits this fact.
danielmarkbruce 2y ago
> Doesn't Language itself encode multimodal experiences Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we don't have great text datasets to describe many things in enough detail, but there isn't any reason we couldn't.
- heyjamesknight 2y ago
  There are absolutely reasons that we cannot capture the entirety—or even a proper image—of human cognition in semantic space. Cognition is not purely semantic. It is dynamic, embodied, socially distributed, culturally extended, and conscious. LLMs are great semantic heuristic machines. But they don't even have access to those other components.

throw310822 2y ago

I don't get it. 1) Yes it's true, learning from text is very hard. But LLMs are multimodal now. 2) That "size of a lion" paper is from 2019, which is a geological era from now. The SOTA was GPT2 which was barely able to spit out coherent text. 3) Have you tried asking a mouse to play chess or reason its way through some physics problem or to write some code? I'm really curious in which benchmark are mice surpassing chatgpt/ grok/ claude etc.

nextts 2y ago
Mice can survive, forage, reproduce. Reproduce a mammal. There is a whole load of capability not available in an LLM. An LLM is essentially a search over a compressed dataset with a tiny bit of reasoning as emergent behaviour. Because it is a parrot that is why you get "hallucinations". The search failed (like when you get a bad result in Google) or the lossy compression failed or it's reasoning failed. Obviously there is a lot of stuff the LLM can find in its searches that are reminiscent of the great intelligence of the people writing for its training data. The magic trick is impressive because when we judge a human what do we do... an exam? an interview? Someone with a perfect memory can fool many people because most people only acquire memory from tacit knowledge. Most people need to live in Paris to become fluent in French. So we see a robot that has a tiny bit of reasoning and a brilliant memory as a brilliant mind. But this is an illusion. Here is an example: User: what is the French Revolution? Agent: The French Revolution was a period of political and societal change in France which began with the Estates General of 1789 and ended with the Coup of 18 Brumaire on 9 November 1799. Many of the revolution's ideas are considered fundamental principles of liberal democracy and its values remain central to modern French political discourse. Can you spot the trick?
- pfisch 2y ago
  When you talk to ~3 year old children they hallucinate quite a lot. Really almost nonstop when you ask them about almost anything. I'm not convinced that what LLM's are doing is that far off the beaten path from our own cognition.
  - 2y ago

hintymad 2y ago

I'm curious why their claims are controversial. It seems pretty obvious to me that LLMs sometimes generate idiotic answers because the models lack common sense, do not have ability for deductive logical reasoning, let alone the ability to induce. And the current transformer architectures plus all the post-training techniques do not do anything to build such intelligence or the world model per LeCun's words.

jcims 2y ago

Over the last few years I’ve become exceedingly aware at how insufficient language really is. It feels like a 2D plane and no matter how many projections you attempt to create from it, they are ultimately limited in the fidelity of the information transfer. Just a lay opinion here but to me each mode of input creates a new, largely orthogonal dimension for the network to grow into. The experience of your heel slipping on a cold sidewalk can be explained in a clinical fashion, but an android’s association of that to the powerful dynamic response required to even attempt to recover will give a newfound association and power to the word ‘slip’.

ninetyninenine 2y ago
LLM is just the name. You can encode anything into the "language" including pictures video and sound.
- pessimizer 2y ago
  I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics, but I've been too lazy to do the research to find anybody working on anything like that, or to experiment myself with it.
  - Tepix 2y ago
    When you train a neural net for Donkeycar with camera images plus the joystick commands of the driver, isn't that close to nerve impulses already?
  - inetknght 2y ago
    > I've always been wondering if anyone is working on using nerve impulses. My first thought when transformers came around was if they could be used for prosthetics Neuralink. Musk warning though. For reference, see Neuralink Launch Event at 59:33 [0], and continue watching through until Musk takes over again. The technical information there is highly relevant to a multi-modal AI model with sensory input/output. https://youtu.be/r-vbh3t7WVI?t=3575 https://youtu.be/r-vbh3t7WVI?t=3575

iainctduncan 2y ago

Thanks for articulating this so well. I'm a musician and music/CS phd student, and as a jazz improvisor of advanced skill (30+ years), I'm accutely aware that there are significant areas of intelligence for which linguistic thinking is not only not good enough, but something to be avoided as much as one can (which is bloody hard sometimes). I have found it so frustrating, but hard to figure out how to counter, that the current LLM zeitgeist seems to hinge on a belief that linguistic intelligence is both necessary and sufficient for AGI.

kadushka 2y ago
Most modern LLMs are multimodal.
- yahoozoo 2y ago
  Does it really matter? At the end of the day, all the modalities and their architectures boil down to matrices of numbers and statistical probability. There’s no agency, no soul.
  - kadushka 2y ago
    At the end of the day, all modalities boil down to patterns of electrical activity in your brain.
    - yahoozoo 2y ago
      The brain is the important part. The electricity just keeps it going. And it’s more than numerical matrices.
      - kadushka 2y ago
        You mean soul?
        namaria 2y ago

ninetyninenine 2y ago

>1) You can't learn an accurate world model just from text. >2) Multimodal learning (vision, language, etc) and interaction with the environment is crucial for true learning. LLMs can be trained with multimodal data. Language is only tokens and pixel and sound data can be encoded into tokens. All data can be serialized. You can train this thing on data we can't even comprehend. Here's the big question. It's clear we need less data then an LLM. But I think it's because evolution has pretrained our brains for this so we have brains geared towards specific things. Like we are geared towards walking, talking, reading, in the same way a cheetah is geared towards ground speed more then it is at flight. If we placed a human and an LLM in completely unfamiliar spaces and tried to train both with data. Which will perform better? And I mean completely non familiar spaces. Like let's make it non Euclidean space and only using sonar for visualization. Something totally foreign to reality as humans know it. I honestly think the LLM will beat us in this environment. We might've succeeded already in creating AGI it's just the G is too much. It's too general so it's learning everything from scratch and it can't catch up to us. Maybe what we need is to figure out how to bias the AI to think and be biased in the way humans are biased.

kedarkhand 2y ago
> non Euclidean space and only using sonar for visualization Pretty good idea for a video game!
physicsguy 2y ago
Hugely different data too? If you think of all the neurons connected up to vision, touch, hearing, heat receptors, balance, etc. there’s a constant stream of multimodal data of different types along with constant reinforcement learning - e.g. ‘if you move your eye in this way, the scene you see changes’, ‘if you tilt your body this way your balance changes’, etc. and this runs from even before you are born, throughout your life.
MITSardine 2y ago
Humans are more adaptable than you think: - echolocation in blind humans https://en.wikipedia.org/wiki/Human_echolocation https://en.wikipedia.org/wiki/Human_echolocation - sight through signals sent on tongue https://www.scientificamerican.com/article/device-lets-blind-see-with-tongues/ https://www.scientificamerican.com/article/device-lets-blind... In the latter case, I recall reading the people involved ended up perceiving these signals as a "first order" sense (not consciously treated information, but on an intuitive level like hearing or vision).

petesergeant 2y ago

> LeCun isn't saying that LLMs aren't useful. He's just concerned with bigger problems, like AGI, which he believes cannot be solved purely through linguistic analysis. It feels like special pleading: surely _this_ will be the problem class that doesn’t fall to “the bitter lesson”. My intuition is that the main problem with the current architecture is that mapping into tokens causes quantization that a real brain doesn’t have, and lack of plasticity. I don’t build models, I spend 100% of my time reading and adjusting model outputs though.

satellite2 2y ago

So blind people aren't generally sentient? (I'm obviously exaggerating a bit for the sake of the argument, but the point stands. Multimodality should not be a prerequisite to AGI)

thousand_nights 2y ago
blind people still have other senses including touch which gives them a size reference they can compare to. you can feel physical objects to gain an understanding of their size. the LLM is more like a brain in a vat with only one sensory input - a stream of text

somenameforme 2y ago

Is that what he's arguing? My perspective on what he's arguing is that LLMs effectively rely on a probabilistic approach to the next token based on the previous. When they're wrong, which the technology all but ensures will happen with some significant degree of frequency, you get cascading errors. It's like in science where we all build upon the shoulders of giants, but if it turns out that one of those shoulders was simply wrong, somehow, then everything built on top of it would be increasingly absurd. E.g. - how the assumption of a geocentric universe inevitably leads to epicycles which leads to ever more elaborate, and plainly wrong, 'outputs.' Without any 'understanding' or knowledge of what they're saying, they will remain irreconcilably dysfunctional. Hence the typical pattern with LLMs: --- How do I do [x]? You do [a]. No that's wrong because reasons. Oh I'm sorry. You're completely right. Thanks for correcting me. I'll keep that in mind. You do [b]. No that's also wrong because reasons. Oh I'm sorry. You're completely right. Thanks for correcting me. I'll keep that in mind. You do [a]. FML --- More advanced systems might add a c or a d, but it's just more noise before repeating the same pattern. Deep Seek's more visible (and lengthy) reasoning demonstrates this perhaps the most clearly. It just can't stop coming back to the same wrong (but statistically probable) answer and so ping-ponging off that (which it at least acknowledges is wrong due to user input) makes up basically the entirety of its reasoning phase.

gsf_emergency_2 2y ago
on "stochastic parrots" Table stakes for sentience: knowing when the best answer is not good enough.. try prompting LLMs with that.. It's related to LeCun's (and Ravid's) subtle question I mentioned in passing below: To Compress Or Not To Compress? (For even a vast majority of Humans, except tacitly, that is not a question!)
tmaly 2y ago
Right now, humans still have enough practice thinking to point out the errors, but what happens when humanity becomes increasingly dependent on LLMs to do this thinking?

gsf_emergency_2 2y ago

usual disclaimer: you decide on your own whether I'm an insider or not :) where LeCun might be prescient should intersect with the nemesis SCHMIDHUBER. They can't both be wrong, I suppose?! It's only "tangentially" related to energy minimization, technically speaking :) connection to multimodalities is spot-on. https://www.mdpi.com/1099-4300/26/3/252 https://www.mdpi.com/1099-4300/26/3/252 To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review With Ravid, double-handedly blue-flag MDPI! Sunmarized for the layman (propaganda?) https://archive.is/https://nyudatascience.medium.com/how-should-self-supervised-learning-models-represent-their-data-8d9ca64ff8d9 https://archive.is/https://nyudatascience.medium.com/how-sho... >When asked about practical applications and areas where these insights might be immediately used, Shwartz-Ziv highlighted the potential in multi-modalities and tabula Imho, best take I've seen on this thread (irony: literal energy minimization) https://news.ycombinator.com/item?id=43367126 https://news.ycombinator.com/item?id=43367126 Of course, this would make Google/OpenAI/DeepSeek wrong by two whole levels (both architecturally and conceptually)

PeterStuer 2y ago

Late 80's and 90's had the 'nouvelle AI' movement that argued embodiment was required for grounding the system into the shared world model. Without it symbols would be ungrounded and never achieve open world consistency. So unlike their knowledge system predecessors, a bit derogatory refered to as GOFAI (good old fashioned AI), nAI hawked back to cybernetics and multi layered dynamical systems rather than having explicit internal symbolic models. Braitenberg rather than blocksworld so to speak. Seems like we are back for another turn of the wheel in this aspect.

implmntatio 2y ago
> grounding the system into the shared world model before we fix certain things [..., 'corruption', Ponzi schemes, deliberate impediment of information flow to population segments and social classes, among other things, ... and a chain of command in hierarchies that are build on all that] is impossible. Why do smart people not talk about this at all? The least engineers and smart people should do is picking these fights for real. It's just a few interest groups, not all of them. I understand a certain balance is necessary in order to keep some systems from tipping over, aka "this is humanity, silly, this is who we are", but we are far from the point of efficient friction and it's only because "smart people" like LeCun et al are not picking those fights. How the hell do you expect to ground an ()AI in a world where elected ignorance amplifies bias and fallacies for power and profit while the literal shit is hitting all the fans via intended and unintended side effects? Any embodied AI will pretend until there is no way to deny that the smartest, brightest and the productive don't care about the system in any way but are just running algorithmically while ignoring what should not be ignored - should as in, an AI should be aligned with humanities interests and should be grounded into the shared world model.
- PeterStuer 2y ago
  I hear you, but while you can have many layers of semantic obfuscation, no amount of sophistry will allow you to smash your face unharmed through a concrete wall. Reality is a hard mistress. In absense of being able to sense reality, post modernism can run truly unchecked.

brulard 2y ago

I don't know about telling better the size from a picture. I can imagine seeing 2 pictures of the moon. One is extreme telephoto showing moon next to a building and it looks real big. Then there would be another image where moon is a tiny speckle in the sky. How big is the moon? I would rather understand a text: "its radius is x km".

veidr 2y ago
I think the example is simplified to make its point efficiently, but also: the moon is something whose size would very likely be precisely explained in texts about it. While some hunting journals might brag about the weight of a lion that was killed, or whatever, most texts that I can recall reading about lions basically assumed you already know roughly how big a lion is; which indeed I learned from pictures as a pre-literate child. A good, precise spec is better that a few pictures, sure; the random text content of whatever training set you can scrape together, perhaps not (?)
nonameiguess 2y ago
Reading "its radius is x km" would mean nothing to you if you'd never experienced spatial extent directly, whether that be visually or just by moving through space and existing in it. You'd need to do exactly what is being said in the paper, read about thousands of other roughly spherical objects and their radii. At some point, you'd get a decent sense of relative sizes. On the other hand, if you ever simply see a meter stick, any statement that something measures a particular multiple or fraction of that you can already understand, without ever needing to learn the size of anything else.

qoez 2y ago

"LeCun has been on about it for a while and it's less controversial these days." Funny how that sentence could have been used 15 years ago too when he was right about persevering through neural network scepticism.

mcculley 2y ago

Are people born both blind and deaf incapable of true learning?

grayhatter 2y ago
There's a whole fork lore/meme around how hard it is, in America culture... But given blidness and deafness is an impediment to acquiring language, more than anything else, I'd say that's the exact opposite of the conclusions from the comment you're replying to. But yes, depending on where you set the bar for "true learning" being blind and deaf would prevent it. I assume you're asking if vision and sound are required for learning, the answer I assume is no. Those were just chosen because we've already invented cameras and microphones. Haptics are less common, and thus less talked about.

eli_gottlieb 2y ago

>(Energy minimization is a very old idea. LeCun has been on about it for a while and it's less controversial these days. Back when everyone tried to have a probabilistic interpretation of neural models, it was expensive to compute the normalization term / partition function. Energy minimization basically said: Set up a sensible loss and minimize it.) Ehhhh, energy-based models are trained via contrastive divergence, not just minimizing a simple loss averaged over the training data.