4 ms·

He also said other things about LLMs that turned out to be either wrong or easily bypassed with some glue. While I understand where he comes from, and that his

He also said other things about LLMs that turned out to be either wrong or easily bypassed with some glue. While I understand where he comes from, and that his stance is pure research-y theory driven, at the end of the day his positions were wrong. Previously, he very publicly and strongly said: a) LLMs can't do math. They trick us in poetry but that's subjective. They can't do objective math. b) they can't plan c) by the very nature of autoregressive arch, errors compound. So the longer you go in your generation, the higher the error rate. so at long contexts the answers become utter garbage. All of these were proven wrong, 1-2 years later. "a" at the core (gold at IMO), "b" w/ software glue and "c" with better training regimes. I'm not interested in the will it won't it debates about AGI, I'm happy with what we have now, and I think these things are good enough now, for several usecases. But it's important to note when people making strong claims get them wrong. Again, I think I get where he's coming from, but the public stances aren't the place to get into the deep research minutia. That being said, I hope he gets to find whatever it is that he's looking for, and wish him success in his endeavours. Between him, Fei Fei Li and Ilya, something cool has to come out of the small shops. Heck, I'm even rooting for the "let's commoditise lora training" that Mira's startup seems to go for.

ilaksh 1y ago
That's true but I also think despite being wrong about the capabilities of LLMs, LeCun has been right in that variations of LLMs are not an appropriate target for long term research that aims to significantly advance AI. Especially at the level of Meta. I think transformers have been proven to be general purpose, but that doesn't mean that we can't use new fundamental approaches. To me it's obvious that researchers are acting like sheep as they always do. He's trying to come up with a real innovation. LeCun has seen how new paradigms have taken over. Variations of LLMs are not the type of new paradigm that serious researches should be aiming for. I wonder if there can be a unification of spatial-temporal representations and language. I am guessing diffusion video generators already achieve this in some way. But I wonder if new techniques can improve the efficiency and capabilities. I assume the Nested Learning stuff is pretty relevant. Although I've never totally grokked transformers and LLMs, I always felt that MoE was the right direction and besides having a strong mapping or unified view of spatial and language info, there also should somehow be the capability of representing information in a non-sequential way. We really use sequences because we can only speak or hear one sound at a time. Information in general isn't particularly sequential, so I doubt that's an ideal representation. So I guess I am kind of variations of transformers myself to be honest. But besides being able to convert between sequential discrete representations and less discrete non-sequential representations (maybe you have tokens but every token has a scalar attached), there should be lots of tokenizations, maybe for each expert. Then you have experts that specialize in combining and translating between different scalar-token tokenizations. Like automatically clustering problems or world model artifacts or something and automatically encoding DSLs for each sub problem. I wish I really understood machine learning.
tonii141 1y ago
a) Still true: vanilla LLMs can’t do math, they pattern-match unless you bolt on tools. b) Still true: next-token prediction isn’t planning. c) Still true: error accumulation is mitigated, not eliminated. Long-context quality still relies on retrieval, checks, and verifiers. Yann’s claims were about LLMs as LLMs. With tooling, you can work around limits, but the core point stands.
- NitpickLawyer 1y ago
  a) no, gemini 2.5 was shown to "win" gold w/o tools. - https://arxiv.org/html/2507.15855v1 https://arxiv.org/html/2507.15855v1 b) reductionism isn't worth our time. Planning works in the real world, today. (try any agentic tool like cc/codex/whatever). And if you're set on the purist view, there's mounting evidence from anthropic that there is planning in the core of an LLM. c) so ... not true? Long context works today. This is simply moving goalposts and nothing more. X can't do Y -> well, here they are doing Y -> well, not like that.
  - tonii141 1y ago
    a) That "no-tools" win depends on prompt orchestration which can still be categorized as tooling. b) Next-token training doesn’t magically grant inner long-horizon planners.. c) Long context ≠ robust at any length. Degradation with scale remains. Not moving goalposts, just keeping terms precise.
    - ACCount37 1y ago
      My man, you're literally moving all the goalposts as we speak. It's not just "long context" - you demand "infinite context" and "any length" now. Even humans don't have that. "No tools" is no longer enough - what, do you demand "no prompts" now too? Having LLMs decompose tasks and prompt each other the way humans do is suddenly a no-no?
      - tonii141 1y ago
        I’m not demanding anything, I’m pointing out that performance tends to degrade as context scales, which follows from current LLM architectures as autoregressive models. In that sense, Yann was right.
HarHarVeryFunny 1y ago
> So the longer you go in your generation, the higher the error rate. so at long contexts the answers become utter garbage. Not totally wrong. They can self-correct, but it seems context rot will eventually set in.

ilaksh 1y ago

That's true but I also think despite being wrong about the capabilities of LLMs, LeCun has been right in that variations of LLMs are not an appropriate target for long term research that aims to significantly advance AI. Especially at the level of Meta. I think transformers have been proven to be general purpose, but that doesn't mean that we can't use new fundamental approaches. To me it's obvious that researchers are acting like sheep as they always do. He's trying to come up with a real innovation. LeCun has seen how new paradigms have taken over. Variations of LLMs are not the type of new paradigm that serious researches should be aiming for. I wonder if there can be a unification of spatial-temporal representations and language. I am guessing diffusion video generators already achieve this in some way. But I wonder if new techniques can improve the efficiency and capabilities. I assume the Nested Learning stuff is pretty relevant. Although I've never totally grokked transformers and LLMs, I always felt that MoE was the right direction and besides having a strong mapping or unified view of spatial and language info, there also should somehow be the capability of representing information in a non-sequential way. We really use sequences because we can only speak or hear one sound at a time. Information in general isn't particularly sequential, so I doubt that's an ideal representation. So I guess I am kind of variations of transformers myself to be honest. But besides being able to convert between sequential discrete representations and less discrete non-sequential representations (maybe you have tokens but every token has a scalar attached), there should be lots of tokenizations, maybe for each expert. Then you have experts that specialize in combining and translating between different scalar-token tokenizations. Like automatically clustering problems or world model artifacts or something and automatically encoding DSLs for each sub problem. I wish I really understood machine learning.

tonii141 1y ago

a) Still true: vanilla LLMs can’t do math, they pattern-match unless you bolt on tools. b) Still true: next-token prediction isn’t planning. c) Still true: error accumulation is mitigated, not eliminated. Long-context quality still relies on retrieval, checks, and verifiers. Yann’s claims were about LLMs as LLMs. With tooling, you can work around limits, but the core point stands.

NitpickLawyer 1y ago
a) no, gemini 2.5 was shown to "win" gold w/o tools. - https://arxiv.org/html/2507.15855v1 https://arxiv.org/html/2507.15855v1 b) reductionism isn't worth our time. Planning works in the real world, today. (try any agentic tool like cc/codex/whatever). And if you're set on the purist view, there's mounting evidence from anthropic that there is planning in the core of an LLM. c) so ... not true? Long context works today. This is simply moving goalposts and nothing more. X can't do Y -> well, here they are doing Y -> well, not like that.
- tonii141 1y ago
  a) That "no-tools" win depends on prompt orchestration which can still be categorized as tooling. b) Next-token training doesn’t magically grant inner long-horizon planners.. c) Long context ≠ robust at any length. Degradation with scale remains. Not moving goalposts, just keeping terms precise.
  - ACCount37 1y ago
    My man, you're literally moving all the goalposts as we speak. It's not just "long context" - you demand "infinite context" and "any length" now. Even humans don't have that. "No tools" is no longer enough - what, do you demand "no prompts" now too? Having LLMs decompose tasks and prompt each other the way humans do is suddenly a no-no?
    - tonii141 1y ago
      I’m not demanding anything, I’m pointing out that performance tends to degrade as context scales, which follows from current LLM architectures as autoregressive models. In that sense, Yann was right.

HarHarVeryFunny 1y ago

> So the longer you go in your generation, the higher the error rate. so at long contexts the answers become utter garbage. Not totally wrong. They can self-correct, but it seems context rot will eventually set in.