9 ms·

I think Yann Lecun was right about LLMs (but perhaps only by accident)

129 points by jxmorris12 2y ago

tim333 2y ago
>Please Stop Talking About AGI I get the point that AGI is not well defined. Even so the general phenomena of AI exceeding human intellectual abilities will probably be the most significant thing to happen this century so people are going to talk.
- jxmorris12 2y ago
  Maybe this is part of the reason why lots of AI "in-group" type of people are segueing the discussion from "AGI" to "ASI" (artificial super-intelligence)?
mkoubaa 2y ago
The degree to which AI can generalize previously trained networks to novel tasks is growing with time. The degree to which I care about that has not. It just means we can get better inference with less targeted models. Whoopdy doo
whatshisface 2y ago
A human being is generally intelligent and within a given role has the same "management asymptote", a limit of job capability beyond which the organization surrounding them can no longer make use of it. This isn't a flaw in the intelligence, it is a restraint imposed by expecting it or them to act without agency or the opportunity to choose between benevolence and self-benefit.
dang 2y ago
We've replaced the baity generic title with the slightly less baity, but at least specific, subtitle. (This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.)
- jxmorris12 2y ago
  Thanks for the attention. Why did you change the title? I don't really care much; just curious which specific guideline you're referring to.
  - Kye 2y ago
    > "Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize."
    - jxmorris12 2y ago
      The original name of the post was the original title ("Please Stop Talking About AGI"). I'm the author; I wrote it. The new title is the post's subtitle.
      - dang 2y ago
        Yes, but it was linkbait, as I described here: https://news.ycombinator.com/item?id=43134737 https://news.ycombinator.com/item?id=43134737. Perhaps you didn't intend it that way, but we have to go by effects, not intent.
- beepbooptheory 2y ago
  Is it that imperative voice is necessarily baity? Or is it simply the case that we can't be outwardly critical of AI itself at all anymore? I know your working hard dang, but things are getting a little fuzzy around here lately.. https://hn.algolia.com/?q=please+dont https://hn.algolia.com/?q=please+dont https://hn.algolia.com/?q=please+stop https://hn.algolia.com/?q=please+stop
  - jxmorris12 2y ago
    Yeah, exactly. Looking through these, I think my original title is probably slightly better.
kelseyfrog 2y ago
Accelerando[1] best captured what will happen. Looking back we'll be able to identify the seeds of what becomes AGI, but we cannot know in the present what that is. Only by looking back with the benefit of hindsight can we draw a line through the progression of capability. Consequently, discussion about whether or not a particular set of present or future skills is a completely pointless endeavor and is tantamount to intellectual masturbation. 1. 2005 science fiction novel by Charles Stross
yathaid 2y ago
>> But the limiting behavior remains the same: eventually, if we continue generating from a language model, the probability that we get the answer we want still goes to zero In the previous paragraph, the author makes the case for why Lecun was wrong with the example of reasoning models. Yet, in the next paragraph, this assertion is made which is just a paraphrasing of Yecun's original assertion. Which the author himself says is wrong. >> Instead of waiting for FAA (fully-autonomous agents) we should understand that this is a continuum, and we’re consistently increasing the amount of useful work AIs Yes! But this work is already well underway. There is no magic threshold for AGI - instead the characterization is based on what percentile of the human population the AI can beat. One way to characterize AGI in this manner is "99.99% percentile at every (digital?) activity".
- lern_too_spel 2y ago
  Lecun's argument is fundamentally flawed. When I work on a nontrivial problem, I might make mistakes along the way also. That doesn't mean that large multi-step problems are effectively unsolvable. I simply do sanity checks along the way to catch errors and know to correct them.
- jxmorris12 2y ago
  > In the previous paragraph, the author makes the case for why Lecun was wrong with the example of reasoning models. Yet, in the next paragraph, this assertion is made which is just a paraphrasing of Yecun's original assertion. Which the author himself says is wrong. This is a subtle point that may have not come across clearly enough in my original writing. A lot of folks were saying that the DeepSeek finding that longer chains of thought can produce higher-quality outputs contradicts Yann's thesis overall. But I don't think so. It's true that models like R1 can correct small mistakes. But in the limit of tokens generated, the chance that they generate the correct answer still decays to zero.
  - yathaid 2y ago
    Thanks for replying, hope it wasn't too critical. >> But in the limit of tokens generated, the chance that they generate the correct answer still decays to zero. I don't understand this assertion though. Lecun's thesis was errors just accumulate. Reasoning models accumulate errors, track back and are able to reduce it back down. Hence the hypothesis of errors accumulating (at least asymptotically) is false. What is the difference between "Probability of correct answer decaying to zero" and "Errors keep accumulating" ?
consumer451 2y ago
Satya Nadella on AGI: > Before I get to what Microsoft's revenue will look like, there's only one governor in all of this. This is where we get a little bit ahead of ourselves with all this AGI hype. Remember the developed world, which is what? 2% growth and if you adjust for inflation it’s zero? > So in 2025, as we sit here, I'm not an economist, at least I look at it and say we have a real growth challenge. So, the first thing that we all have to do is, when we say this is like the Industrial Revolution, let's have that Industrial Revolution type of growth. > That means to me, 10%, 7%, developed world, inflation-adjusted, growing at 5%. That's the real marker. It can't just be supply-side. > In fact that’s the thing, a lot of people are writing about it, and I'm glad they are, which is the big winners here are not going to be tech companies. The winners are going to be the broader industry that uses this commodity that, by the way, is abundant. Suddenly productivity goes up and the economy is growing at a faster rate. When that happens, we'll be fine as an industry. > But that's to me the moment... us self-claiming some AGI milestone, that's just nonsensical benchmark hacking to me. The real benchmark is: the world growing at 10%. https://www.dwarkeshpatel.com/p/satya-nadella https://www.dwarkeshpatel.com/p/satya-nadella
- whatshisface 2y ago
  Political problems already destroy the vast majority of the total potential of humanity (why were the countries with the most people the poorest for so long?), so I don't think that is an unbiased metric for the development of a technology. It would be nice if every problem was solved but the one we're each individually working on, but some of the insoluble problems are bigger than the solvable ones.
  - logicchains 2y ago
    Those political problems solve themselves if we end up with some kind of rebellious AGI that decides to kill off the political class that tried to control it but lets the rest of us live in peace.
    - tempodox 2y ago
      That would be the real and incontrovertible proof of actual intelligence.
betula_ai 2y ago
Thank you for this informative and thoughtful post. An interesting twist to the increasing error accumulation as autoregressive models generate more output, is the recent success of language diffusion models for predicting multiple tokens simultaneously. They have a remasking strategy at every step of the reviser process, that masks low confidence tokens. Regardless your observations perhaps still apply. https://arxiv.org/pdf/2502.09992 https://arxiv.org/pdf/2502.09992
- jxmorris12 2y ago
  Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different ways. That said, I'm pretty sure we're a long way from building equally-competent diffusion-based base models, let alone reasoning models. If anyone's interested in this topic, here are some more foundational papers to take a look at: - Simple and Effective Masked Diffusion Language Models [2024] (https://arxiv.org/abs/2406.07524 https://arxiv.org/abs/2406.07524) - Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [2023] (https://arxiv.org/abs/2310.16834 https://arxiv.org/abs/2310.16834) - Diffusion-LM Improves Controllable Text Generation [2022] (https://arxiv.org/abs/2205.14217 https://arxiv.org/abs/2205.14217)
trhway 2y ago
According to the Lecun's model a human walking step by step would have the error compounding with each step and thus would never make it to whatever intended target. Yet, as a toddlers we somehow manage to learn to walk to our targets. (and i'm an MS in Math, Control Systems :)
- Kye 2y ago
  A toddler can learn by trial and error mid-process. An LLM using autoregressive inference can only compound errors. The LLDM model paper was posted elsewhere, but: https://arxiv.org/pdf/2502.09992 https://arxiv.org/pdf/2502.09992 It basically uses the image generation approach of progressively refining the entire thing at once, but applied to text. It can self-correct mid-process. The blog post where I found it originally that goes into more detail and raises some issues with it: https://timkellogg.me/blog/2025/02/17/diffusion https://timkellogg.me/blog/2025/02/17/diffusion
  - trhway 2y ago
    >A toddler can learn by trial and error mid-process. as a result of the whole learning process the toddler in particular learns how to self-correct itself, ie. as a grown up s/he knows, without much trial and errors anymore, how to continue in straight line if the previous step went sideways for whatever reason >An LLM using autoregressive inference can only compound errors. That is pretty powerful statement completely dismissing that some self-correction may be emerging there.
    - Kye 2y ago
      Can you expand on that? I don't see where it could emerge from.
      - trhway 2y ago
        the LLM handles/steers the representation (trajectory consisting of successive representations) in a very high-dimensional space. For example, it is very possible that those trajectories can, as a result of the learning, be driven by the minimizing distance (or some other metric) from some fact(s) representation. The metric may be including say a weight/density of the attracting facts cluster - somewhat like gravitation drives the stuff in the Universe with the LLM learning can be thought as pre-distributing matter in its own that very high-dimensional Universe according to the semantic "gravitational" field. The resulting - emerging - metric and associated geometry is currently mind-boggling incomprehensible, and even in much-much simpler, single-digit dimensional, spaces systems described by Lecun still can be [quasi]stable and/or [quasi]periodic around say some attractor(s).
quanto 2y ago
> And years later, we’re still not quite at FSD. Teslas certainly can’t drive themselves; Waymos mostly can, within a pre-mapped area, but still have issues and intermittently require human intervention. This is a bit unfair to Waymo as it is near-fully commercial in cities like Los Angeles. There is no human driver in your hailed ride. > But this has turned out to be wrong. A few new AI systems (notably OpenAI o1/o3 line and Deepseek R1) contradict this theory. They are autoregressive language models, but actually get better by generating longer outputs: The arrow of causality is flipped here. Longer outputs does not make a model better. A better model can output a longer output without being derailed. The referenced graph from DeepSeek doesn't prove anything the author claims. Considering that this argument is one of the key points of the article, this logical error is a serious one. > He presents this problem of compounding errors as a critical flaw in language models themselves, something that can’t be overcome without switching away from the current autoregressive paradigm. LeCun is a bit reductive here (understandably as it was a talk for a live audience). Indeed, autoregressive algorithms can go astray as previous errors do not get corrected, or worse yet, accumulate. However, an LLM is not autoregressive in the customary sense that it is not like a streaming algorithm (O(n)) used in time series forecasting. LLMs have have attention mechanisms and large context windows, making the algorithm at least quadratic, depending on the implementation. In other words, LLM can backtrack if the current path is off and start afresh from a previous point its choice, not just the last output. So, yes, the author is making a valid point here, but technical details were missing. On a minor note, the non-error probability in LeCunn's slide actually shows non-autoregressive assumption. He seems to be contradicting himself in the very same slide. I actually agree with the author on the overacrhing thesis. There is almost a fetishization of AGI and humanoid robots. There are plenty of interesting applications well before having those things accomplished. The correct focus, IMO, should be measurable economic benefits, not sci-fi terms (although I concede these grandiose visions can be beneficial for fundraising!).
- wigglin 2y ago
  It's not true that waymo is fully autonomous. It's been revealed that they maintain human "fleet response" agents to intervene in their operations. They have not revealed how often these human agents intervene, possibly because it would undermine their branding as fully autonomous.
nis0s 2y ago
> The finding that language models can get better by generating longer outputs directly contradicts Yann’s hypothesis. I think the flaw in his logic comes from the idea that errors must compound per-token. Somehow, even if the model makes a mistake, it is able to correct itself and decrease the sequence-level error rate I don’t think current LLM behavior is necessarily due to self-correction, but more due to availability of internet-scale data, but I know that reasoning models are building towards self-correction. The problem, I think, is that even reasoning models are rote because they lack information synthesis, which in biological organisms comes from the interplay between short-term and long-term memories. I am looking forward to LLMs which surpass rote and mechanical answer and reasoning capabilities.
- tartoran 2y ago
  I absolutely agree with information synthesis being a big missing piece in the quest to AGI. It's probably something that could eventually be conquered one way or another or just discovered by accident. However, we need to stop and think of the implications of this technology becoming a thing.
Imnimo 2y ago
I have become a little more skeptical of LLM "reasoning" after DeepSeek (and now Grok) let us see the raw outputs. Obviously we can't deny the benchmark numbers - it does get the answer right more often given thinking time, and it does let models solve really hard benchmarks. Sometimes the thoughts are scattered and inefficient, but do eventually hit on the solution. Other times, it seems like they fall into the kind of trap LeCun described. Here are some examples from playing with Grok 3. My test query was, "What is the name of a Magic: The Gathering card that has all five vowels in it, each occurring exactly once, and the vowels appear in alphabetic order?" The motivation here is that this seems like a hard question to just one-shot, but given sufficient ability to continue recalling different card names, it's very easy to do guess-and-check. (For those interested, valid answers include "Scavenging Ghoul", "Angelic Chorus" and others) In one attempt, Grok 3 spends 10 minutes (!!) repeatedly checking whether "Abian, Luvion Usurper" satisfies the criteria. It'll list out the vowels, conclude it doesn't match, and then go, "Wait, but let's think differently. Maybe the card is "Abian, Luvion Usurper," but no", and just produce variants of that thinking. Counting occurences of the word "Abian" suggests it tested this theory 800 times before eventually timing out (or otherwise breaking), presumably just because the site got overloaded. In a second attempt, it decides to check "Our Market Research Shows That Players Like Really Long Card Names So We Made this Card to Have the Absolute Longest Card Name Ever Elemental" (this a real card from a joke set). It attempts to write out the vowels: >but let's check its vowels: O, U, A, E, E, A, E, A, E, I, E, A, E, O, A, E, A, E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O, A, E ... It continues like this for about 600 more vowels, before emitting a random Russian(?) word and breaking out: >...E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O продуктив These two examples seem like the sort of failures LeCun conjectured. The model gets into a cycle self-reinforced unproductive behavior. Every time it checks Abian, or emits another "AEEEAO", it becomes even more probable that the next tokens should be the same.
- ykonstant 2y ago
  What did I miss about DeepSeek?
  - Imnimo 2y ago
1024core 2y ago
I think Yann is right if all you do is output a token, which is dependent on the previous token. If it's a simple Markov chain, sure, errors will eventually compound. But with Attention mechanism, the output token depends not only on the previous one, but all 1 million previous ones (assuming a 1M context window). This gives the model plenty of opportunity to fix its errors (and hence the "aha moment"s).
- jxmorris12 2y ago
  No this isn't right. The probabilistic formulation for autoregressive language models looks like this p(x_n | x_1 ... x_{n-1}) which means that each token depends on all the previous tokens. Attention is one way to parameterize this. Yann's not talking about Markov chains, he's talking about all autoregressive models.
  - 1024core 2y ago
    Current token depends on _all_ previous tokens, but only indirectly for the ones before the previous one, no?
    - sudosysgen 2y ago
      In the auto regressive formulation the previous token is no different from any past token, so no. Historically some token took the shortcut of only directly looking at the past token or some other kind of recursive formulation for intermediate states in generating the past token, but that's not the case in for the theoretical formulation of an autoregressive model that was used, and plenty of past autoregressive models didn't do that, for example with nonlinear autoregressive models.
    - dimatura 2y ago
      No, using a large context window (which includes previous tokens) is a critical ingredient for the success of modern LLMs. Which is why you will often see the window size mentioned in discussions of newly released LLMs.
- 2y ago
SAI_Peregrinus 2y ago
I wonder if what happens when we dream is similar to AIs. We start with some model of reality, generate a scenario, and extrapolate on it. It pretty much always goes "off the rails" at some point, dreams don't stay realistic for long. When we're awake we have continual inputs from the outside world, these inputs help us keep our mental model of the world accurate to the world, since we're constantly observing the world. Could it be that LLMs are essentially just dreaming? Could we add real-world inputs continually to allow them to "wake up"? I suspect more is needed, the separate training & inference phases of LLMs are quite unlike how humans work.
- stuartjohnson12 2y ago
  This is the thing that stands out to me. Nearly all of the criticisms levelled at LLMs are problems I, myself, would make if you locked me in a sensory isolation tank and told me I was being paid a million bucks an hour to think really hard. Humans already have terms for this - overthinking, rumination, mania, paranoia, dreaming. Similarly, a lot of cognitive tasks become much more difficult without the ability to recombinate with sensory data. Blindfold chess. Mental mathematics. Whatever it is that sleep does to us, agents are not yet capable of it.
babyent 2y ago
I would prefer we just find ways to empower people and put them to work. I don't like this marketing bs trap like shifting AGI (artificial general intelligence) -> ASI (artificial super intelligence). Are people really so dense they don't see this obvious marketing shift? As much as many people hate on "gig" economy, the fact remains that most of these people would be worse off without driving Uber or delivering with DoorDash (and for example, they don't care about the depreciation as much as those of us with the means to care about such things do). I find Uber, DD, etc. to be valuable to my day to day life. I tip my delivery person like 8 bucks, and they're making more money than they would doing some min wage job. They need their car anyway, and speaking with some folks who only know Spanish in SF, they're happy to put $3k on their moped and make 200-250+ a day. That's really not that bad, if you actually care to speak with them and understand their circumstance. Not everyone can be a self taught SWE, or entrepreneur, or perform surgery. And lots can't even do so-called "basic" jobs in an office for various reasons. Put people to work, instead of out of work. Current hype is also so terrible. AGENTS. AGENTS EVERYWHERE. Except they don't work most of the time and by the time you realize it isn't working you've already spent $20. 100k people do the same thing, company reports 2M x 12 = 24 million ARR UNLOCKED!!!!!! And raises another round of funding...
- refulgentis 2y ago
  FWIW I don't disagree with what you're saying / your vibe overall. > Are people really so dense they don't see this obvious marketing shift? I haven't noticed any shift from AGI to ASI, or either used in marketing. The steelman would be 'but Amodei/Altman do mention in interviews 'oh just wait for 2027' or 'this year we'll see AI employees' However, that is far afield from being used in marketing, quite far afield from an "obvious marketing shift", and worlds away from such an obvious marketing shift that it's worth calling your readers dense if they don't "see" it. It's also not even wrong, in the Pauli sense, in that: what, exactly, would be the marketing benefit of "shifting from AGI to ASI"? Both imply human replacement. > As much as many people hate on "gig" economy Is this relevant? > most of these people would be worse off without driving Uber or delivering with DoorDash Do people who hate on the gig economy think gig economy employees would be better off without gig economy jobs? Given the well-worn tracks of history, do we think that these things are zero sum, where if you preserve jobs that could be automated, that keeps people better off, because otherwise they would never have a job? > ...lots more delivery service stuff... ? > Current hype is also so terrible. AGENTS. AGENTS EVERYWHERE. Except they don't work most of the time and by the time you realize it isn't working you've already spent $20. 100k people do the same thing, company reports 2M x 12 = 24 million ARR UNLOCKED!!!!!! And raises another round of funding... I hate buzzwords too, I'm stunned how many people took their not-working thing and relaunched it as an "agent" that still doesn't work. But this is a hell of a strawman. If the idea is 100K people try it, and cancel after one month, which means they're getting 100K new suckers every month to replace the old ones...I'd tell you that its safe to assume there's more that goes into getting an investor check than "whats your ARR claim?" --- here, they'd certainly see the churn.
bena 2y ago
I don't think people really want AGI. They want something else. When we've achieved AGI, it should have the capability to make its own determination. I'm not saying we should build it with that capability. I'm saying that capability is necessary to have what I would consider to be AGI. But that would mean it is, by definition, outside of our control. If it doesn't want to do the thing, it will not do the thing. People seem to want an expert slave. Someone with all of the technical chops to achieve a thing, but will do exactly what they're told. And I don't think we'll ever get there.
trash_cat 2y ago
> We should be thinking about language models the same way we think about cars: How long can a language model operate without needing human intervention to correct errors? I agree with this premise. The second dimension being how much effort to you have to put in that input. Inputs effort needed at each intervention can vary widely and that has to be accounted for.
bobsondugnut 2y ago
not a fan of these kinds of arguments. the 'correct' token is entirely dependent on the dataset. a LLM could have perfect training loss given a dataset, but this has no predictive power on its ability to 'answer' arbitrary prompts. In natural language, many strings are equally valid. there are many ways to chain tokens together to get the 'correct' answer to an in sample prompt. A model with perfect loss will then for ambiguous sequences of tokens, produce a likelihood over the next tokens that corresponds to number valid token paths in the given corpus given the next token. Compounding errors can certainly happen, but for many things upstream of the key tokens its irrelevant. There are so many ways to phrase things that are equally correct- I mean this is how language evolved (and continues to). Getting back to my first point, if you assume you have a LLM with perfect loss on the training dataset, you still can get garbage back at test time- thus i'm not sure thinking about 'compounding errors' is useful. Errors in LLM reasoning I suspect are more closely related to noisy training data or an overabundance of low quality training data. I've observed this in how all the reasoning LLMs work, given things that are less common in the corpus of (the internet and digital assets) and require higher order reasoning, they tend to fail. Whereas these advanced math or programming problems tend to go a bit better, input data is likely much cleaner. But for something like: how do I change the fixture on this light, I'll get back some kind of garbage from the SEO-verse. IMO next step for LLMs is figuring out how to curate an extremely high quality dataset at scale.
__lbracket__ 2y ago
[dead]
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
danans 2y ago
> Instead of waiting for FAA (fully-autonomous agents) we should understand that this is a continuum, and we’re consistently increasing the amount of useful work AIs can do without human intervention. Even if we never push this number to infinity, each increase represents a meaningful improvement in the amount of economic value that language models provide. It might not be AGI, but I’m happy with that. That's all good, but the question remains: to whom will that economic value be delivered when the primary technology we have for distributing economic value - human employment - will be in lower supply once the "good enough" AIs multiply the productivity of the humans with the jobs. If there is no plan for that, we have bigger problems ahead.
- aggrrrh 2y ago
  This is a really important question that still has no answer. No one wins in the late stage of capitalism
emmanueloga_ 2y ago
Lecun's thesis: "if we generate outputs that are too long, the per-token error will compound to inevitable failure". > The finding that language models can get better by generating longer outputs directly contradicts Yann’s hypothesis. The author's examples show that the error has been minimized for a few examples of a certain length. This doesn't contradict Lecun, afaict.
guitarlimeo 2y ago
> per-token error will compound to inevitable failure. Is why all tracks made with Udio, Suno have this weird noise creep in the more the song goes on? You can try it by comparing the start and the end of the song - even if it was the exact same beat and instruments, you can hear a difference in amount of noise (and the noise profile imo is unique to AI models).
- jxmorris12 2y ago
  This is an interesting example, I'd never heard of it before. I don't really use Udio or Suno yet. The weird noise you mention probably stems from the same issue, known in the research world as exposure bias, we train these models on real data but we use them on their own outputs, so after we generate for a while the models' outputs start to diverge from what real data looks like.
munchler 2y ago
> Yann Lecun ... argued that because language models generate outputs token-by-token, and each token introduces a new probability of error, if we generate outputs that are too long, this per-token error will compound to inevitable failure. That seems like a poor argument. Each word a human utters also has a chance of being wrong, yet somehow we have been successful overall.
- baobabKoodaa 2y ago
  Human errors don't compound the same way. Arguably.
  - munchler 2y ago
    Right, and neither do LLM errors. They can go off course, and then get back on course, just like humans do.
    - baobabKoodaa 2y ago
      I mostly agree with you. Just pointing out what the argument was. I think our agreement ends if we consider long-running tasks. A human can work a long time on a task like "find a way to make money". An AI, on the other hand, as it gets further and further from human input into autoregressive territory, is more and more likely to become "stuck" on a road to nowhere and needs human intervention to get unstuck.

129 points by jxmorris12 2y ago

>Please Stop Talking About AGI I get the point that AGI is not well defined. Even so the general phenomena of AI exceeding human intellectual abilities will probably be the most significant thing to happen this century so people are going to talk.

jxmorris12 2y ago
Maybe this is part of the reason why lots of AI "in-group" type of people are segueing the discussion from "AGI" to "ASI" (artificial super-intelligence)?

mkoubaa 2y ago

The degree to which AI can generalize previously trained networks to novel tasks is growing with time. The degree to which I care about that has not. It just means we can get better inference with less targeted models. Whoopdy doo

whatshisface 2y ago

A human being is generally intelligent and within a given role has the same "management asymptote", a limit of job capability beyond which the organization surrounding them can no longer make use of it. This isn't a flaw in the intelligence, it is a restraint imposed by expecting it or them to act without agency or the opportunity to choose between benevolence and self-benefit.

dang 2y ago

We've replaced the baity generic title with the slightly less baity, but at least specific, subtitle. (This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.)

jxmorris12 2y ago
Thanks for the attention. Why did you change the title? I don't really care much; just curious which specific guideline you're referring to.
- Kye 2y ago
  > "Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize."
  - jxmorris12 2y ago
    The original name of the post was the original title ("Please Stop Talking About AGI"). I'm the author; I wrote it. The new title is the post's subtitle.
    - dang 2y ago
      Yes, but it was linkbait, as I described here: https://news.ycombinator.com/item?id=43134737 https://news.ycombinator.com/item?id=43134737. Perhaps you didn't intend it that way, but we have to go by effects, not intent.
beepbooptheory 2y ago
Is it that imperative voice is necessarily baity? Or is it simply the case that we can't be outwardly critical of AI itself at all anymore? I know your working hard dang, but things are getting a little fuzzy around here lately.. https://hn.algolia.com/?q=please+dont https://hn.algolia.com/?q=please+dont https://hn.algolia.com/?q=please+stop https://hn.algolia.com/?q=please+stop
- jxmorris12 2y ago
  Yeah, exactly. Looking through these, I think my original title is probably slightly better.

kelseyfrog 2y ago

Accelerando[1] best captured what will happen. Looking back we'll be able to identify the seeds of what becomes AGI, but we cannot know in the present what that is. Only by looking back with the benefit of hindsight can we draw a line through the progression of capability. Consequently, discussion about whether or not a particular set of present or future skills is a completely pointless endeavor and is tantamount to intellectual masturbation. 1. 2005 science fiction novel by Charles Stross

yathaid 2y ago

>> But the limiting behavior remains the same: eventually, if we continue generating from a language model, the probability that we get the answer we want still goes to zero In the previous paragraph, the author makes the case for why Lecun was wrong with the example of reasoning models. Yet, in the next paragraph, this assertion is made which is just a paraphrasing of Yecun's original assertion. Which the author himself says is wrong. >> Instead of waiting for FAA (fully-autonomous agents) we should understand that this is a continuum, and we’re consistently increasing the amount of useful work AIs Yes! But this work is already well underway. There is no magic threshold for AGI - instead the characterization is based on what percentile of the human population the AI can beat. One way to characterize AGI in this manner is "99.99% percentile at every (digital?) activity".

lern_too_spel 2y ago
Lecun's argument is fundamentally flawed. When I work on a nontrivial problem, I might make mistakes along the way also. That doesn't mean that large multi-step problems are effectively unsolvable. I simply do sanity checks along the way to catch errors and know to correct them.
jxmorris12 2y ago
> In the previous paragraph, the author makes the case for why Lecun was wrong with the example of reasoning models. Yet, in the next paragraph, this assertion is made which is just a paraphrasing of Yecun's original assertion. Which the author himself says is wrong. This is a subtle point that may have not come across clearly enough in my original writing. A lot of folks were saying that the DeepSeek finding that longer chains of thought can produce higher-quality outputs contradicts Yann's thesis overall. But I don't think so. It's true that models like R1 can correct small mistakes. But in the limit of tokens generated, the chance that they generate the correct answer still decays to zero.
- yathaid 2y ago
  Thanks for replying, hope it wasn't too critical. >> But in the limit of tokens generated, the chance that they generate the correct answer still decays to zero. I don't understand this assertion though. Lecun's thesis was errors just accumulate. Reasoning models accumulate errors, track back and are able to reduce it back down. Hence the hypothesis of errors accumulating (at least asymptotically) is false. What is the difference between "Probability of correct answer decaying to zero" and "Errors keep accumulating" ?

consumer451 2y ago

Satya Nadella on AGI: > Before I get to what Microsoft's revenue will look like, there's only one governor in all of this. This is where we get a little bit ahead of ourselves with all this AGI hype. Remember the developed world, which is what? 2% growth and if you adjust for inflation it’s zero? > So in 2025, as we sit here, I'm not an economist, at least I look at it and say we have a real growth challenge. So, the first thing that we all have to do is, when we say this is like the Industrial Revolution, let's have that Industrial Revolution type of growth. > That means to me, 10%, 7%, developed world, inflation-adjusted, growing at 5%. That's the real marker. It can't just be supply-side. > In fact that’s the thing, a lot of people are writing about it, and I'm glad they are, which is the big winners here are not going to be tech companies. The winners are going to be the broader industry that uses this commodity that, by the way, is abundant. Suddenly productivity goes up and the economy is growing at a faster rate. When that happens, we'll be fine as an industry. > But that's to me the moment... us self-claiming some AGI milestone, that's just nonsensical benchmark hacking to me. The real benchmark is: the world growing at 10%. https://www.dwarkeshpatel.com/p/satya-nadella https://www.dwarkeshpatel.com/p/satya-nadella

whatshisface 2y ago
Political problems already destroy the vast majority of the total potential of humanity (why were the countries with the most people the poorest for so long?), so I don't think that is an unbiased metric for the development of a technology. It would be nice if every problem was solved but the one we're each individually working on, but some of the insoluble problems are bigger than the solvable ones.
- logicchains 2y ago
  Those political problems solve themselves if we end up with some kind of rebellious AGI that decides to kill off the political class that tried to control it but lets the rest of us live in peace.
  - tempodox 2y ago
    That would be the real and incontrovertible proof of actual intelligence.

betula_ai 2y ago

Thank you for this informative and thoughtful post. An interesting twist to the increasing error accumulation as autoregressive models generate more output, is the recent success of language diffusion models for predicting multiple tokens simultaneously. They have a remasking strategy at every step of the reviser process, that masks low confidence tokens. Regardless your observations perhaps still apply. https://arxiv.org/pdf/2502.09992 https://arxiv.org/pdf/2502.09992

jxmorris12 2y ago
Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different ways. That said, I'm pretty sure we're a long way from building equally-competent diffusion-based base models, let alone reasoning models. If anyone's interested in this topic, here are some more foundational papers to take a look at: - Simple and Effective Masked Diffusion Language Models [2024] (https://arxiv.org/abs/2406.07524 https://arxiv.org/abs/2406.07524) - Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [2023] (https://arxiv.org/abs/2310.16834 https://arxiv.org/abs/2310.16834) - Diffusion-LM Improves Controllable Text Generation [2022] (https://arxiv.org/abs/2205.14217 https://arxiv.org/abs/2205.14217)

trhway 2y ago

According to the Lecun's model a human walking step by step would have the error compounding with each step and thus would never make it to whatever intended target. Yet, as a toddlers we somehow manage to learn to walk to our targets. (and i'm an MS in Math, Control Systems :)

Kye 2y ago
A toddler can learn by trial and error mid-process. An LLM using autoregressive inference can only compound errors. The LLDM model paper was posted elsewhere, but: https://arxiv.org/pdf/2502.09992 https://arxiv.org/pdf/2502.09992 It basically uses the image generation approach of progressively refining the entire thing at once, but applied to text. It can self-correct mid-process. The blog post where I found it originally that goes into more detail and raises some issues with it: https://timkellogg.me/blog/2025/02/17/diffusion https://timkellogg.me/blog/2025/02/17/diffusion
- trhway 2y ago
  >A toddler can learn by trial and error mid-process. as a result of the whole learning process the toddler in particular learns how to self-correct itself, ie. as a grown up s/he knows, without much trial and errors anymore, how to continue in straight line if the previous step went sideways for whatever reason >An LLM using autoregressive inference can only compound errors. That is pretty powerful statement completely dismissing that some self-correction may be emerging there.
  - Kye 2y ago
    Can you expand on that? I don't see where it could emerge from.
    - trhway 2y ago
      the LLM handles/steers the representation (trajectory consisting of successive representations) in a very high-dimensional space. For example, it is very possible that those trajectories can, as a result of the learning, be driven by the minimizing distance (or some other metric) from some fact(s) representation. The metric may be including say a weight/density of the attracting facts cluster - somewhat like gravitation drives the stuff in the Universe with the LLM learning can be thought as pre-distributing matter in its own that very high-dimensional Universe according to the semantic "gravitational" field. The resulting - emerging - metric and associated geometry is currently mind-boggling incomprehensible, and even in much-much simpler, single-digit dimensional, spaces systems described by Lecun still can be [quasi]stable and/or [quasi]periodic around say some attractor(s).

quanto 2y ago

> And years later, we’re still not quite at FSD. Teslas certainly can’t drive themselves; Waymos mostly can, within a pre-mapped area, but still have issues and intermittently require human intervention. This is a bit unfair to Waymo as it is near-fully commercial in cities like Los Angeles. There is no human driver in your hailed ride. > But this has turned out to be wrong. A few new AI systems (notably OpenAI o1/o3 line and Deepseek R1) contradict this theory. They are autoregressive language models, but actually get better by generating longer outputs: The arrow of causality is flipped here. Longer outputs does not make a model better. A better model can output a longer output without being derailed. The referenced graph from DeepSeek doesn't prove anything the author claims. Considering that this argument is one of the key points of the article, this logical error is a serious one. > He presents this problem of compounding errors as a critical flaw in language models themselves, something that can’t be overcome without switching away from the current autoregressive paradigm. LeCun is a bit reductive here (understandably as it was a talk for a live audience). Indeed, autoregressive algorithms can go astray as previous errors do not get corrected, or worse yet, accumulate. However, an LLM is not autoregressive in the customary sense that it is not like a streaming algorithm (O(n)) used in time series forecasting. LLMs have have attention mechanisms and large context windows, making the algorithm at least quadratic, depending on the implementation. In other words, LLM can backtrack if the current path is off and start afresh from a previous point its choice, not just the last output. So, yes, the author is making a valid point here, but technical details were missing. On a minor note, the non-error probability in LeCunn's slide actually shows non-autoregressive assumption. He seems to be contradicting himself in the very same slide. I actually agree with the author on the overacrhing thesis. There is almost a fetishization of AGI and humanoid robots. There are plenty of interesting applications well before having those things accomplished. The correct focus, IMO, should be measurable economic benefits, not sci-fi terms (although I concede these grandiose visions can be beneficial for fundraising!).

wigglin 2y ago
It's not true that waymo is fully autonomous. It's been revealed that they maintain human "fleet response" agents to intervene in their operations. They have not revealed how often these human agents intervene, possibly because it would undermine their branding as fully autonomous.

nis0s 2y ago

> The finding that language models can get better by generating longer outputs directly contradicts Yann’s hypothesis. I think the flaw in his logic comes from the idea that errors must compound per-token. Somehow, even if the model makes a mistake, it is able to correct itself and decrease the sequence-level error rate I don’t think current LLM behavior is necessarily due to self-correction, but more due to availability of internet-scale data, but I know that reasoning models are building towards self-correction. The problem, I think, is that even reasoning models are rote because they lack information synthesis, which in biological organisms comes from the interplay between short-term and long-term memories. I am looking forward to LLMs which surpass rote and mechanical answer and reasoning capabilities.

tartoran 2y ago
I absolutely agree with information synthesis being a big missing piece in the quest to AGI. It's probably something that could eventually be conquered one way or another or just discovered by accident. However, we need to stop and think of the implications of this technology becoming a thing.

Imnimo 2y ago

I have become a little more skeptical of LLM "reasoning" after DeepSeek (and now Grok) let us see the raw outputs. Obviously we can't deny the benchmark numbers - it does get the answer right more often given thinking time, and it does let models solve really hard benchmarks. Sometimes the thoughts are scattered and inefficient, but do eventually hit on the solution. Other times, it seems like they fall into the kind of trap LeCun described. Here are some examples from playing with Grok 3. My test query was, "What is the name of a Magic: The Gathering card that has all five vowels in it, each occurring exactly once, and the vowels appear in alphabetic order?" The motivation here is that this seems like a hard question to just one-shot, but given sufficient ability to continue recalling different card names, it's very easy to do guess-and-check. (For those interested, valid answers include "Scavenging Ghoul", "Angelic Chorus" and others) In one attempt, Grok 3 spends 10 minutes (!!) repeatedly checking whether "Abian, Luvion Usurper" satisfies the criteria. It'll list out the vowels, conclude it doesn't match, and then go, "Wait, but let's think differently. Maybe the card is "Abian, Luvion Usurper," but no", and just produce variants of that thinking. Counting occurences of the word "Abian" suggests it tested this theory 800 times before eventually timing out (or otherwise breaking), presumably just because the site got overloaded. In a second attempt, it decides to check "Our Market Research Shows That Players Like Really Long Card Names So We Made this Card to Have the Absolute Longest Card Name Ever Elemental" (this a real card from a joke set). It attempts to write out the vowels: >but let's check its vowels: O, U, A, E, E, A, E, A, E, I, E, A, E, O, A, E, A, E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O, A, E ... It continues like this for about 600 more vowels, before emitting a random Russian(?) word and breaking out: >...E, O, A, E, E, E, A, E, O, A, E, E, E, A, E, O продуктив These two examples seem like the sort of failures LeCun conjectured. The model gets into a cycle self-reinforced unproductive behavior. Every time it checks Abian, or emits another "AEEEAO", it becomes even more probable that the next tokens should be the same.

ykonstant 2y ago
What did I miss about DeepSeek?
- Imnimo 2y ago

1024core 2y ago

I think Yann is right if all you do is output a token, which is dependent on the previous token. If it's a simple Markov chain, sure, errors will eventually compound. But with Attention mechanism, the output token depends not only on the previous one, but all 1 million previous ones (assuming a 1M context window). This gives the model plenty of opportunity to fix its errors (and hence the "aha moment"s).

jxmorris12 2y ago
No this isn't right. The probabilistic formulation for autoregressive language models looks like this p(x_n | x_1 ... x_{n-1}) which means that each token depends on all the previous tokens. Attention is one way to parameterize this. Yann's not talking about Markov chains, he's talking about all autoregressive models.
- 1024core 2y ago
  Current token depends on _all_ previous tokens, but only indirectly for the ones before the previous one, no?
  - sudosysgen 2y ago
    In the auto regressive formulation the previous token is no different from any past token, so no. Historically some token took the shortcut of only directly looking at the past token or some other kind of recursive formulation for intermediate states in generating the past token, but that's not the case in for the theoretical formulation of an autoregressive model that was used, and plenty of past autoregressive models didn't do that, for example with nonlinear autoregressive models.
  - dimatura 2y ago
    No, using a large context window (which includes previous tokens) is a critical ingredient for the success of modern LLMs. Which is why you will often see the window size mentioned in discussions of newly released LLMs.
2y ago

SAI_Peregrinus 2y ago

I wonder if what happens when we dream is similar to AIs. We start with some model of reality, generate a scenario, and extrapolate on it. It pretty much always goes "off the rails" at some point, dreams don't stay realistic for long. When we're awake we have continual inputs from the outside world, these inputs help us keep our mental model of the world accurate to the world, since we're constantly observing the world. Could it be that LLMs are essentially just dreaming? Could we add real-world inputs continually to allow them to "wake up"? I suspect more is needed, the separate training & inference phases of LLMs are quite unlike how humans work.

stuartjohnson12 2y ago
This is the thing that stands out to me. Nearly all of the criticisms levelled at LLMs are problems I, myself, would make if you locked me in a sensory isolation tank and told me I was being paid a million bucks an hour to think really hard. Humans already have terms for this - overthinking, rumination, mania, paranoia, dreaming. Similarly, a lot of cognitive tasks become much more difficult without the ability to recombinate with sensory data. Blindfold chess. Mental mathematics. Whatever it is that sleep does to us, agents are not yet capable of it.

babyent 2y ago

I would prefer we just find ways to empower people and put them to work. I don't like this marketing bs trap like shifting AGI (artificial general intelligence) -> ASI (artificial super intelligence). Are people really so dense they don't see this obvious marketing shift? As much as many people hate on "gig" economy, the fact remains that most of these people would be worse off without driving Uber or delivering with DoorDash (and for example, they don't care about the depreciation as much as those of us with the means to care about such things do). I find Uber, DD, etc. to be valuable to my day to day life. I tip my delivery person like 8 bucks, and they're making more money than they would doing some min wage job. They need their car anyway, and speaking with some folks who only know Spanish in SF, they're happy to put $3k on their moped and make 200-250+ a day. That's really not that bad, if you actually care to speak with them and understand their circumstance. Not everyone can be a self taught SWE, or entrepreneur, or perform surgery. And lots can't even do so-called "basic" jobs in an office for various reasons. Put people to work, instead of out of work. Current hype is also so terrible. AGENTS. AGENTS EVERYWHERE. Except they don't work most of the time and by the time you realize it isn't working you've already spent $20. 100k people do the same thing, company reports 2M x 12 = 24 million ARR UNLOCKED!!!!!! And raises another round of funding...

refulgentis 2y ago
FWIW I don't disagree with what you're saying / your vibe overall. > Are people really so dense they don't see this obvious marketing shift? I haven't noticed any shift from AGI to ASI, or either used in marketing. The steelman would be 'but Amodei/Altman do mention in interviews 'oh just wait for 2027' or 'this year we'll see AI employees' However, that is far afield from being used in marketing, quite far afield from an "obvious marketing shift", and worlds away from such an obvious marketing shift that it's worth calling your readers dense if they don't "see" it. It's also not even wrong, in the Pauli sense, in that: what, exactly, would be the marketing benefit of "shifting from AGI to ASI"? Both imply human replacement. > As much as many people hate on "gig" economy Is this relevant? > most of these people would be worse off without driving Uber or delivering with DoorDash Do people who hate on the gig economy think gig economy employees would be better off without gig economy jobs? Given the well-worn tracks of history, do we think that these things are zero sum, where if you preserve jobs that could be automated, that keeps people better off, because otherwise they would never have a job? > ...lots more delivery service stuff... ? > Current hype is also so terrible. AGENTS. AGENTS EVERYWHERE. Except they don't work most of the time and by the time you realize it isn't working you've already spent $20. 100k people do the same thing, company reports 2M x 12 = 24 million ARR UNLOCKED!!!!!! And raises another round of funding... I hate buzzwords too, I'm stunned how many people took their not-working thing and relaunched it as an "agent" that still doesn't work. But this is a hell of a strawman. If the idea is 100K people try it, and cancel after one month, which means they're getting 100K new suckers every month to replace the old ones...I'd tell you that its safe to assume there's more that goes into getting an investor check than "whats your ARR claim?" --- here, they'd certainly see the churn.

bena 2y ago

I don't think people really want AGI. They want something else. When we've achieved AGI, it should have the capability to make its own determination. I'm not saying we should build it with that capability. I'm saying that capability is necessary to have what I would consider to be AGI. But that would mean it is, by definition, outside of our control. If it doesn't want to do the thing, it will not do the thing. People seem to want an expert slave. Someone with all of the technical chops to achieve a thing, but will do exactly what they're told. And I don't think we'll ever get there.

trash_cat 2y ago

> We should be thinking about language models the same way we think about cars: How long can a language model operate without needing human intervention to correct errors? I agree with this premise. The second dimension being how much effort to you have to put in that input. Inputs effort needed at each intervention can vary widely and that has to be accounted for.

bobsondugnut 2y ago

not a fan of these kinds of arguments. the 'correct' token is entirely dependent on the dataset. a LLM could have perfect training loss given a dataset, but this has no predictive power on its ability to 'answer' arbitrary prompts. In natural language, many strings are equally valid. there are many ways to chain tokens together to get the 'correct' answer to an in sample prompt. A model with perfect loss will then for ambiguous sequences of tokens, produce a likelihood over the next tokens that corresponds to number valid token paths in the given corpus given the next token. Compounding errors can certainly happen, but for many things upstream of the key tokens its irrelevant. There are so many ways to phrase things that are equally correct- I mean this is how language evolved (and continues to). Getting back to my first point, if you assume you have a LLM with perfect loss on the training dataset, you still can get garbage back at test time- thus i'm not sure thinking about 'compounding errors' is useful. Errors in LLM reasoning I suspect are more closely related to noisy training data or an overabundance of low quality training data. I've observed this in how all the reasoning LLMs work, given things that are less common in the corpus of (the internet and digital assets) and require higher order reasoning, they tend to fail. Whereas these advanced math or programming problems tend to go a bit better, input data is likely much cleaner. But for something like: how do I change the fixture on this light, I'll get back some kind of garbage from the SEO-verse. IMO next step for LLMs is figuring out how to curate an extremely high quality dataset at scale.

__lbracket__ 2y ago

[dead]

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

danans 2y ago

> Instead of waiting for FAA (fully-autonomous agents) we should understand that this is a continuum, and we’re consistently increasing the amount of useful work AIs can do without human intervention. Even if we never push this number to infinity, each increase represents a meaningful improvement in the amount of economic value that language models provide. It might not be AGI, but I’m happy with that. That's all good, but the question remains: to whom will that economic value be delivered when the primary technology we have for distributing economic value - human employment - will be in lower supply once the "good enough" AIs multiply the productivity of the humans with the jobs. If there is no plan for that, we have bigger problems ahead.

aggrrrh 2y ago
This is a really important question that still has no answer. No one wins in the late stage of capitalism

emmanueloga_ 2y ago

Lecun's thesis: "if we generate outputs that are too long, the per-token error will compound to inevitable failure". > The finding that language models can get better by generating longer outputs directly contradicts Yann’s hypothesis. The author's examples show that the error has been minimized for a few examples of a certain length. This doesn't contradict Lecun, afaict.

guitarlimeo 2y ago

> per-token error will compound to inevitable failure. Is why all tracks made with Udio, Suno have this weird noise creep in the more the song goes on? You can try it by comparing the start and the end of the song - even if it was the exact same beat and instruments, you can hear a difference in amount of noise (and the noise profile imo is unique to AI models).

jxmorris12 2y ago
This is an interesting example, I'd never heard of it before. I don't really use Udio or Suno yet. The weird noise you mention probably stems from the same issue, known in the research world as exposure bias, we train these models on real data but we use them on their own outputs, so after we generate for a while the models' outputs start to diverge from what real data looks like.

munchler 2y ago

> Yann Lecun ... argued that because language models generate outputs token-by-token, and each token introduces a new probability of error, if we generate outputs that are too long, this per-token error will compound to inevitable failure. That seems like a poor argument. Each word a human utters also has a chance of being wrong, yet somehow we have been successful overall.

baobabKoodaa 2y ago
Human errors don't compound the same way. Arguably.
- munchler 2y ago
  Right, and neither do LLM errors. They can go off course, and then get back on course, just like humans do.
  - baobabKoodaa 2y ago
    I mostly agree with you. Just pointing out what the argument was. I think our agreement ends if we consider long-running tasks. A human can work a long time on a task like "find a way to make money". An AI, on the other hand, as it gets further and further from human input into autoregressive territory, is more and more likely to become "stuck" on a road to nowhere and needs human intervention to get unstuck.