9 ms·

I think we've actually had capable AIs for long enough now to see that this kind of exponential advance to AGI in 2 years is extremely unlikely. The AI we have

I think we've actually had capable AIs for long enough now to see that this kind of exponential advance to AGI in 2 years is extremely unlikely. The AI we have today isn't radically different from the AI we had in 2023. They are much better at the thing they are good at, and there are some new capabilities that are big, but they are still fundamentally next-token predictors. They still fail at larger scope longer term tasks in mostly the same way, and they are still much worse at learning from small amounts of data than humans. Despite their ability to write decent code, we haven't seen the signs of a runaway singularity as some thought was likely. I see people saying that these kinds of things are happening behind closed doors, but I haven't seen any convincing evidence of it, and there is enormous propensity for AI speculation to run rampant.

byearthithatius 2y ago
Disagree. We know it _can_ learn out of distribution capabilities based on similarities to other distributions. Like the TikZ Unicorn[1] (which was not in training data anywhere) or my code (which has variable names and methods/ideas probably not seen 1:1 in training). IMO this out of distribution learning is all we need to scale to AGI. Sure there are still issues, it doesn't always know which distribution to pick from. Neither do we, hence car crashes. [1]: https://arxiv.org/pdf/2303.12712 https://arxiv.org/pdf/2303.12712 or on YT https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c
benlivengood 2y ago
METR [0] explicitly measures the progress on long term tasks; it's as steep a sigmoid as the other progress at the moment with no inflection yet. As others have pointed out in other threads RLHF has progressed beyond next-token prediction and modern models are modeling concepts [1]. [0] https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... [1] https://www.anthropic.com/news/tracing-thoughts-language-model https://www.anthropic.com/news/tracing-thoughts-language-mod...
- Fraterkes 2y ago
  The METR graph proposes a 6 year trend, based largely on 4 datapoints before 2024. I get that it is hard to do analyses since were in uncharted territory, and I personally find a lot of the AI stuff impressive, but this just doesn't strike me as great statistics.
  - benlivengood 2y ago
    I agree that we don't have any good statistical models for this. If AI development were that predictable we'd likely already be past a singularity of some sort or in a very long winter just by reverse-engineering what makes the statistical model tick.
- Vegenoid 2y ago
  At the risk of coming off like a dolt and being super incorrect: I don't put much stock into these metrics when it comes to predicting AGI. Even if the trend of "length of task an AI can reliably do doubles every 7 months" continues, as they say that means we're years away from AI that can complete tasks that take humans weeks or months. I'm skeptical that the doubling trend will continue into that timescale, I think there is a qualitative difference between tasks that take weeks or months and tasks that take minutes or hours, a difference that is not reflected by simple quantity. I think many people responsible for hiring engineers are keenly aware of this distinction, because of their experience attempting to choose good engineers based on how they perform in task-driven technical interviews that last only hours. Intelligence as humans have it seems like a "know it when you see it" thing to me, and metrics that attempt to define and compare it will always be looking at only a narrow slice of the whole picture. To put it simply, the gut feeling I get based on my interactions with current AI, and how it is has developed over the past couple of years, is that AI is missing key elements of general intelligence at its core. While there's more lots more room for its current approaches to get better, I think there will be something different needed for AGI. I'm not an expert, just a human.
  - benlivengood 2y ago
    > I think there is a qualitative difference between tasks that take weeks or months and tasks that take minutes or hours, a difference that is not reflected by simple quantity. I'd label that difference as long-term planning plus executive function, and wherever that overlaps with or includes delegation. Most long-term projects are not done by a single human and so delegation almost always plays a big part. To delegate, tasks must be broken down in useful ways. To break down tasks a holistic model of the goal is needed where compartmentalization of components can be identified. I think a lot of those individual elements are within reach of current model architectures but they are likely out of distribution. How many gantt charts and project plans and project manager meetings are in the pretraining datasets? My guess is few; rarely published internal artifacts. Books and articles touch on the concepts but I think the models learn best from the raw data; they can probably tell you very well all of the steps of good project management because the descriptions are all over the place. The actual doing of it is farther toward the tail of the distribution.
jug 2y ago
> there are some new capabilities that are big, but they are still fundamentally next-token predictors Anthropic recently released research where they saw how when Claude attempted to compose poetry, it didn't simply predict token by token and "react" to when it thought it might need a rhyme and then looked at its context to think of something appropriate, but actually saw several tokens ahead and adjusted for where it'd likely end up, ahead of time. Anthropic also says this adds to evidence seen elsewhere that language models seem to sometimes "plan ahead". Please check out the section "Planning in poems" here; it's pretty interesting! https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...
- percentcer 2y ago
  Isn't this just a form of next token prediction? i.e. you'll keep your options open for a potential rhyme if you select words that have many associated rhyming pairs, and you'll further keep your options open if you focus on broad topics over niche
  - throwuxiytayq 2y ago
    In the same way that human brains are just predicting the next muscle contraction.
    - alfalfasprout 2y ago
      Except that's not how it works...
      - Workaccount2 2y ago
        To be fair, we don't actually know how the human mind works. The most sure things we know is that it is a physical system, and that does feel like something to be one of these systems.
      - ToValueFunfetti 2y ago
        It may well be: https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding
ComplexSystems 2y ago
> They are much better at the thing they are good at, and there are some new capabilities that are big, but they are still fundamentally next-token predictors. I don't really get this. Are you saying autoregressive LLMs won't qualify as AGI, by definition? What about diffusion models, like Mercury? Does it really matter how inference is done if the result is the same?
- Vegenoid 2y ago
  > Are you saying autoregressive LLMs won't qualify as AGI, by definition? No, I am speculating that they will not reach capabilities that qualify them as AGI.
  - otabdeveloper4 2y ago
    They will, we just need meatspace people to become dumber and more predictable. Making huge strides on that front, actually. (In no small part due to LLMs themselves, yeah.)
uejfiweun 2y ago
Isn't the brain kind of just a predictor as well, just a more complicated one? Instead of predicting and emitting tokens, we're predicting future outcomes and emitting muscle movements. Which is obviously different in a sense but I don't think you can write off the entire paradigm as a dead end just because the medium is different.
boznz 2y ago
> we haven't seen the signs of a runaway singularity as some thought was likely. The signs are not there but while we may not be on an exponential curve (which would be difficult to see), we are definitely on a steep upward one which may get steeper or may fizzle out if LLM's can only reach human level 'intelligence' but not surpass it. Original article was a fun read though and 360,000 words shorter than my very similar fiction novel :-)
- grey-area 2y ago
  LLMs don’t have any sort of intelligence at present, they have a large corpus of data and can produce modified copies of it.
  - boznz 2y ago
    Agree, the "intelligence" part is definitely the missing link in all this, however humans are smart cookies, and can see there's a gap, so I expect someone, (not necessarily a major player,) will eventually figure "it" out.
  - EMIRELADERO 2y ago
    While certainly not human-level intelligence, I don't see how you could say they don't have any sort of it. There's clearly generalization there. What would you say is the threshold?
    - dangus 2y ago
      Seems like you’d have to prove the inverse. The threshold would be “produce anything that isn’t identical or a minor transfiguration of input training data.” In my experience my AI assistant in my code editor can’t do a damn thing that isn’t widely documented and sometimes botches tasks that are thoroughly documented (such as hallucinating parameters names that don’t exist). I can witness this when I reach the edge of common use cases where extending beyond the documentation requires following an implication. For example, AI can’t seem to understand how to help me in any way with Terraform dynamic credentials because the documentation is very sparse, and it is not part of almost any blog posts or examples online. My definition the variable is populated dynamically and real aren’t shown anywhere. I get a lot of irrelevant nonsense suggestions on how to fix it. AI is a great “amazing search engine” and it can string together combinations of logic that already exist in documentation and examples while changing some names here and there, but what looks like true understanding really is just token prediction. IMO the massive amount of training data is making the man behind the curtain look way better than he is.
killerstorm 2y ago
False. We got from ~0% on SWE-bench to 63%. It's a huge increase of capability in 2 years. It's like saying that both a baby who can make a few steps and an adult have capability of "walking". It's just wrong.
OrangeMusic 2y ago
They still can't tell how many Rs in Strawberry
- senordevnyc 2y ago
  This is obviously false. Even 4o and o3-mini can do this.
sqw3rl 2y ago
the argument in the paper seems to be that coding ability is what leads to the tipping point. Eventually human-level (then superhuman) coders augment the AI research process until an AI research agent is developed, and it's exponential from there. We know they are developing more advanced models, and we know they're secretive about it, but how advanced?... ¯\_(ツ)_/¯

byearthithatius 2y ago

Disagree. We know it _can_ learn out of distribution capabilities based on similarities to other distributions. Like the TikZ Unicorn[1] (which was not in training data anywhere) or my code (which has variable names and methods/ideas probably not seen 1:1 in training). IMO this out of distribution learning is all we need to scale to AGI. Sure there are still issues, it doesn't always know which distribution to pick from. Neither do we, hence car crashes. [1]: https://arxiv.org/pdf/2303.12712 https://arxiv.org/pdf/2303.12712 or on YT https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c

benlivengood 2y ago

METR [0] explicitly measures the progress on long term tasks; it's as steep a sigmoid as the other progress at the moment with no inflection yet. As others have pointed out in other threads RLHF has progressed beyond next-token prediction and modern models are modeling concepts [1]. [0] https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... [1] https://www.anthropic.com/news/tracing-thoughts-language-model https://www.anthropic.com/news/tracing-thoughts-language-mod...

Fraterkes 2y ago
The METR graph proposes a 6 year trend, based largely on 4 datapoints before 2024. I get that it is hard to do analyses since were in uncharted territory, and I personally find a lot of the AI stuff impressive, but this just doesn't strike me as great statistics.
- benlivengood 2y ago
  I agree that we don't have any good statistical models for this. If AI development were that predictable we'd likely already be past a singularity of some sort or in a very long winter just by reverse-engineering what makes the statistical model tick.
Vegenoid 2y ago
At the risk of coming off like a dolt and being super incorrect: I don't put much stock into these metrics when it comes to predicting AGI. Even if the trend of "length of task an AI can reliably do doubles every 7 months" continues, as they say that means we're years away from AI that can complete tasks that take humans weeks or months. I'm skeptical that the doubling trend will continue into that timescale, I think there is a qualitative difference between tasks that take weeks or months and tasks that take minutes or hours, a difference that is not reflected by simple quantity. I think many people responsible for hiring engineers are keenly aware of this distinction, because of their experience attempting to choose good engineers based on how they perform in task-driven technical interviews that last only hours. Intelligence as humans have it seems like a "know it when you see it" thing to me, and metrics that attempt to define and compare it will always be looking at only a narrow slice of the whole picture. To put it simply, the gut feeling I get based on my interactions with current AI, and how it is has developed over the past couple of years, is that AI is missing key elements of general intelligence at its core. While there's more lots more room for its current approaches to get better, I think there will be something different needed for AGI. I'm not an expert, just a human.
- benlivengood 2y ago
  > I think there is a qualitative difference between tasks that take weeks or months and tasks that take minutes or hours, a difference that is not reflected by simple quantity. I'd label that difference as long-term planning plus executive function, and wherever that overlaps with or includes delegation. Most long-term projects are not done by a single human and so delegation almost always plays a big part. To delegate, tasks must be broken down in useful ways. To break down tasks a holistic model of the goal is needed where compartmentalization of components can be identified. I think a lot of those individual elements are within reach of current model architectures but they are likely out of distribution. How many gantt charts and project plans and project manager meetings are in the pretraining datasets? My guess is few; rarely published internal artifacts. Books and articles touch on the concepts but I think the models learn best from the raw data; they can probably tell you very well all of the steps of good project management because the descriptions are all over the place. The actual doing of it is farther toward the tail of the distribution.

jug 2y ago

> there are some new capabilities that are big, but they are still fundamentally next-token predictors Anthropic recently released research where they saw how when Claude attempted to compose poetry, it didn't simply predict token by token and "react" to when it thought it might need a rhyme and then looked at its context to think of something appropriate, but actually saw several tokens ahead and adjusted for where it'd likely end up, ahead of time. Anthropic also says this adds to evidence seen elsewhere that language models seem to sometimes "plan ahead". Please check out the section "Planning in poems" here; it's pretty interesting! https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...

percentcer 2y ago
Isn't this just a form of next token prediction? i.e. you'll keep your options open for a potential rhyme if you select words that have many associated rhyming pairs, and you'll further keep your options open if you focus on broad topics over niche
- throwuxiytayq 2y ago
  In the same way that human brains are just predicting the next muscle contraction.
  - alfalfasprout 2y ago
    Except that's not how it works...
    - Workaccount2 2y ago
      To be fair, we don't actually know how the human mind works. The most sure things we know is that it is a physical system, and that does feel like something to be one of these systems.
    - ToValueFunfetti 2y ago
      It may well be: https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding

ComplexSystems 2y ago

> They are much better at the thing they are good at, and there are some new capabilities that are big, but they are still fundamentally next-token predictors. I don't really get this. Are you saying autoregressive LLMs won't qualify as AGI, by definition? What about diffusion models, like Mercury? Does it really matter how inference is done if the result is the same?

Vegenoid 2y ago
> Are you saying autoregressive LLMs won't qualify as AGI, by definition? No, I am speculating that they will not reach capabilities that qualify them as AGI.
- otabdeveloper4 2y ago
  They will, we just need meatspace people to become dumber and more predictable. Making huge strides on that front, actually. (In no small part due to LLMs themselves, yeah.)

uejfiweun 2y ago

Isn't the brain kind of just a predictor as well, just a more complicated one? Instead of predicting and emitting tokens, we're predicting future outcomes and emitting muscle movements. Which is obviously different in a sense but I don't think you can write off the entire paradigm as a dead end just because the medium is different.

boznz 2y ago

> we haven't seen the signs of a runaway singularity as some thought was likely. The signs are not there but while we may not be on an exponential curve (which would be difficult to see), we are definitely on a steep upward one which may get steeper or may fizzle out if LLM's can only reach human level 'intelligence' but not surpass it. Original article was a fun read though and 360,000 words shorter than my very similar fiction novel :-)

grey-area 2y ago
LLMs don’t have any sort of intelligence at present, they have a large corpus of data and can produce modified copies of it.
- boznz 2y ago
  Agree, the "intelligence" part is definitely the missing link in all this, however humans are smart cookies, and can see there's a gap, so I expect someone, (not necessarily a major player,) will eventually figure "it" out.
- EMIRELADERO 2y ago
  While certainly not human-level intelligence, I don't see how you could say they don't have any sort of it. There's clearly generalization there. What would you say is the threshold?
  - dangus 2y ago
    Seems like you’d have to prove the inverse. The threshold would be “produce anything that isn’t identical or a minor transfiguration of input training data.” In my experience my AI assistant in my code editor can’t do a damn thing that isn’t widely documented and sometimes botches tasks that are thoroughly documented (such as hallucinating parameters names that don’t exist). I can witness this when I reach the edge of common use cases where extending beyond the documentation requires following an implication. For example, AI can’t seem to understand how to help me in any way with Terraform dynamic credentials because the documentation is very sparse, and it is not part of almost any blog posts or examples online. My definition the variable is populated dynamically and real aren’t shown anywhere. I get a lot of irrelevant nonsense suggestions on how to fix it. AI is a great “amazing search engine” and it can string together combinations of logic that already exist in documentation and examples while changing some names here and there, but what looks like true understanding really is just token prediction. IMO the massive amount of training data is making the man behind the curtain look way better than he is.

killerstorm 2y ago

False. We got from ~0% on SWE-bench to 63%. It's a huge increase of capability in 2 years. It's like saying that both a baby who can make a few steps and an adult have capability of "walking". It's just wrong.

OrangeMusic 2y ago

They still can't tell how many Rs in Strawberry

senordevnyc 2y ago
This is obviously false. Even 4o and o3-mini can do this.

sqw3rl 2y ago

the argument in the paper seems to be that coding ability is what leads to the tipping point. Eventually human-level (then superhuman) coders augment the AI research process until an AI research agent is developed, and it's exponential from there. We know they are developing more advanced models, and we know they're secretive about it, but how advanced?... ¯\_(ツ)_/¯