3 ms·

I think that LeCun has correctly identified that LLM is only one type of intelligence and that AGI/AMI needs to combine multiple other types … hierarchical goal

I think that LeCun has correctly identified that LLM is only one type of intelligence and that AGI/AMI needs to combine multiple other types … hierarchical goal setting, attention/focus management, and so on. Seems that he is able to garner support for his ideas and to make progress at the leading edge - yes a little bit hard to take the “I know better” style, but then many innovations are driven by narcissism.

ACCount37 1y ago
There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best. For how "naive" transformer LLMs seem, they sure set a high bar. Saying "I know better" is quite easy. Backing that up is really hard.
- cindyllm 1y ago
  [dead]
- librasteve 1y ago
  excellent point
- littlestymaar 1y ago
  > There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best. That's kind of an awkward timing to say that, as alternative to transformers have flourished over the past few weeks (Qwen3-Next, Granite 4). But IIRC Le Cun's criticism applies to more than just transformers and to next-token predictors as a whole.
  - ACCount37 1y ago
    Both are still transformer LLMs at their core, and perform as such. They don't show a massive jump in capabilities over your average transformer. Improvements in long context efficiency sure are nice, and I do think that trying to combine transformers with architectures that aren't cursed with O(n^2) on sequence length is a promising approach. But it's promising as an incremental improvement, not a breakthrough that completely redefines the way AIs are made, the way transformer LLMs themselves did.
    - littlestymaar 1y ago
      > They don't show a massive jump in capabilities over your average transformer Long context is a massive capability improvement. > But it's promising as an incremental improvement, not a breakthrough that completely redefines the way AIs are made, the way transformer LLMs themselves did. Transformers themselves were an incremental improvement over RNN with attention, and in terms of capabilities they weren't immediately superior to their predecessor. What changed the game was that they were vastly cheaper to train which allowed to train massive models on phenomenal amounts of data. Linear attention models being much more compute-efficient than transformers on longer context may result in a similar breakthrough. It's very hard to tell in advance what will be a marginal improvement and what will be a game changer.

ACCount37 1y ago

There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best. For how "naive" transformer LLMs seem, they sure set a high bar. Saying "I know better" is quite easy. Backing that up is really hard.

cindyllm 1y ago
[dead]
librasteve 1y ago
excellent point
littlestymaar 1y ago
> There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best. That's kind of an awkward timing to say that, as alternative to transformers have flourished over the past few weeks (Qwen3-Next, Granite 4). But IIRC Le Cun's criticism applies to more than just transformers and to next-token predictors as a whole.
- ACCount37 1y ago
  Both are still transformer LLMs at their core, and perform as such. They don't show a massive jump in capabilities over your average transformer. Improvements in long context efficiency sure are nice, and I do think that trying to combine transformers with architectures that aren't cursed with O(n^2) on sequence length is a promising approach. But it's promising as an incremental improvement, not a breakthrough that completely redefines the way AIs are made, the way transformer LLMs themselves did.
  - littlestymaar 1y ago
    > They don't show a massive jump in capabilities over your average transformer Long context is a massive capability improvement. > But it's promising as an incremental improvement, not a breakthrough that completely redefines the way AIs are made, the way transformer LLMs themselves did. Transformers themselves were an incremental improvement over RNN with attention, and in terms of capabilities they weren't immediately superior to their predecessor. What changed the game was that they were vastly cheaper to train which allowed to train massive models on phenomenal amounts of data. Linear attention models being much more compute-efficient than transformers on longer context may result in a similar breakthrough. It's very hard to tell in advance what will be a marginal improvement and what will be a game changer.