6 ms·

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 19

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.

tehjoker 7mo ago
Part of the issue there is that the data quantity prior to 1905 is a small drop in the bucket compared to the internet era even though the logical rigor is up to par.
- antupis 7mo ago
  Humans need way less data. Just compare Waymo to average 16 year-old with car.
  - cellis 7mo ago
    A 16 year old has been training for almost 16 years to drive a car. I would argue the opposite: Waymo’s / Specific AIs need far less data than humans. Humans can generalize their training, but they definitely need a LOT of training!
    - jimbokun 7mo ago
      No 16 year old has practiced driving a car for 16 years.
      - Dansvidania 7mo ago
        If you see gaining fine motor control, understanding pictographic language […] as a prerequisite to driving a car, then yes, all of them are
        krige 7mo ago
        That's an exaggeration. Nobody is trained to read STOP signs for 16 years, a few months top. And Waymo doesn't need to coordinate a four-limbed, 20-digited, one-headed body to operate a car.
        danielEM 7mo ago
        Well, I also think that there is a lot that we process 'in background' and learn on beforehand in order to learn how to drive and then drive. I think the most 'fair' would be to figure out absolute lowest age of kids that would allow them to perform well on streets behind steering wheel.
        Dansvidania 7mo ago
        i am not making a point that it is, I am rather expanding on the possible perspective in which 16 years of training produce a human driver. That being said, you don't really need training to understand a STOP sign by the time you are required to, its pretty damn clear, it being one of the simpler signs. But you do get a lot of "cultural training" so to speak.
      - krisoft 7mo ago
        They were practicing object recognition, movement tracking and prediction, self-localisation, visual odometry fused with porpiroception and the vestibular system, and movement controls for 16 years before they even sit behind a steering wheel though.
    - noduerme 7mo ago
      When humans, or dogs or cats for that matter, react to novel situations they encounter, when they appear to generalize or synthesize prior diverse experience into a novel reaction, that new experience and new reaction feeds directly back into their mental model and alters it on the fly. It doesn't just tack on a new memory. New experience and new information back-propagates constantly adjusting the weights and meanings of prior memories. This is a more multi-dimensional alteration than simply re-training a model to come up with a new right answer... it also exposes to the human mental model all the potential flaws in all the previous answers which may have been sufficiently correct before. This is why, for example, a 30 year old can lose control of a car on an icy road and then suddenly, in the span of half a second before crashing, remember a time they intentionally drifted a car on the street when they were 16 and reflect on how stupid they were. In the human or animal mental model, all events are recalled by other things, and all are constantly adapting, even adapting past things. The tokens we take in and process are not words, nor spatial artifacts. We read a whole model as a token, and our output is a vector of weighted models that we somewhat trust and somewhat discard. Meeting a new person, you will compare all their apparent models to the ones you know: Facial models, audio models, language models, political models. You ingest their vector of models as tokens and attempt to compare them to your own existing ones, while updating yours at the same time. Only once our thoughts have arranged those competing models we hold in some kind of hierarchy do we poll those models for which ones are appropriate to synthesize words or actions from.
      - tomrod 7mo ago
        In a word, JEPA?
        noduerme 7mo ago
        No. Not at all like that. I said: >> nor spatial artifacts I meant visual patterns, too. You're thinking about what I said on too granular a level. JEPA is visual, based ultimately on pixels. The tokens may be digested from pixels until they're as large as whole recognizable objects, but the tokens are not whole mental models themselves. Here's an example of humans evaluating competing mental models as tokens: You see a car, it's white, it's got some blood stains on the door, and it's traveling towards a red light at 90 miles an hour in a 30 mph residential zone, while you're about to make a left turn. A human foot is dangling from the trunk. You refer to several mental models you have about high speed chases, drug cartels in the area, murders, etc. You compare these models to determine the next action the car might take. What were the tokens in this scenario? The color of the car, the pixels of blood, the speed, the traffic pattern? Or whole models of understanding behavior where you had to choose between a normal driver's behavior and that of someone with a dead body fleeing a crime scene?
- jerf 7mo ago
  Yet the humans of the time, a small number of the smartest ones, did it, and on much less training data than we throw at LLMs today. If LLMs have shown us anything it is that AGI or super-human AI isn't on some line, where you either reach it or don't. It's a much higher dimensional concept. LLMs are still, at their core, language models, the term is no lie. Humans have language models in their brains, too. We even know what happens if they end up disconnected from the rest of the brain because there are some unfortunate people who have experienced that for various reasons. There's a few things that can happen, the most interesting of which is when they emit grammatically-correct sentences with no meaning in them. Like, "My green carpet is eating on the corner." If we consider LLMs as a hypertrophied langauge model, they are blatently, grotesquely superhuman on that dimension. LLMs are way better at not just emitting grammatically-correct content but content with facts in them, related to other facts. On the other hand, a human language model doesn't require the entire freaking Internet to be poured through it, multiple times (!), in order to start functioning. It works on multiple orders of magnitude less input. The "is this AGI" argument is going to continue swirling in circles for the forseeable future because "is this AGI" is not on a line. In some dimensions, current LLMs are astonishingly superhuman. Find me a polyglot who is truly fluent in 20 languages and I'll show you someone who isn't also conversant with PhD-level topics in a dozen fields. And yet at the same time, they are clearly sub-human in that we do hugely more with our input data then they do, and they have certain characteristic holes in their cognition that are stubbornly refusing to go away, and I don't expect they will. I expect there to be some sort of AI breakthrough at some point that will allow them to both fix some of those cognitive holes, and also, train with vastly less data. No idea what it is, no idea when it will be, but really, is the proposition "LLMs will not be the final manifestation of AI capability for all time" really all that bizarre a claim? I will go out on a limb and say I suspect it's either only one more step the size of "Attention is All You Need", or at most two. It's just hard to know when they'll occur.
radlad 7mo ago
I'm not sure - with tool calling, AI can both fetch and create new context.
- 0xbadcafebee 7mo ago
  It still can't learn. It would need to create content, experiment with it, make observations, then re-train its model on that observation, and repeat that indefinitely at full speed. That won't work on a timescale useful to a human. Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. So we're hyper-tweaking LLMs to make them more useful faster, in the hopes that that will make us more money. Which it does. But it doesn't make you an AGI.
  - charcircuit 7mo ago
    It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.
    - otabdeveloper4 7mo ago
      > they update their memories Their contexts, not their memories. An LLM context is like 100k tokens. That's a fruit fly, not AGI.
      - charcircuit 7mo ago
        A human can't keep 100k tokens active in their mind at the same time. We just need a place to store them and tools to query it. You could have exabytes of memories that the AI could use.
        otabdeveloper4 7mo ago
deleted 7mo ago
[deleted]
crazy5sheep 7mo ago
The 1905 thought experiment actually cuts both ways. Did humans "invent" the airplane? We watched birds fly for thousands of years — that's training data. The Wright brothers didn't conjure flight from pure reasoning, they synthesized patterns from nature, prior failed attempts, and physics they'd absorbed. Show me any human invention and I'll show you the training data behind it. Take the wheel. Even that wasn't invented from nothing — rolling logs, round stones, the shape of the sun. The "invention" was recognizing a pattern already present in the physical world and abstracting it. Still training data, just physical and sensory rather than textual. And that's actually the most honest critique of current LLMs — not that they're architecturally incapable, but that they're missing a data modality. Humans have embodied training data. You don't just read about gravity, you've felt it your whole life. You don't just know fire is hot, you've been near one. That physical grounding gives human cognition a richness that pure text can't fully capture — yet. Einstein is the same story. He stood on Faraday, Maxwell, Lorentz, and Riemann. General Relativity was an extraordinary synthesis — not a creation from void. If that's the bar for "real" intelligence, most humans don't clear it either. The uncomfortable truth is that human cognition and LLMs aren't categorically different. Everything you've ever "thought" comes from what you've seen, heard, and experienced. That's training data. The brain is a pattern-recognition and synthesis machine, and the attention mechanism in transformers is arguably our best computational model of how associative reasoning actually works. So the question isn't whether LLMs can invent from nothing — nothing does that, not even us. Are there still gaps? Sure. Data quality, training methods, physical grounding — these are real problems. But they're engineering problems, not fundamental walls. And we're already moving in that direction — robots learning from physical interaction, multimodal models connecting vision and language, reinforcement learning from real-world feedback. The brain didn't get smart because it has some magic ingredient. It got smart because it had millions of years of rich, embodied, high-stakes training data. We're just earlier in that journey with AI. The foundation is already there — AGI isn't a question of if anymore, it's a question of execution.
- drw85 7mo ago
  Nice ChatGPT answer. Put some real thought and data in it too.
xdennis 7mo ago
> Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. AGI just means human level intelligence. I couldn't come up with General Relativity. That doesn't mean I don't have general intelligence. I don't understand why people are moving the goalposts.
- nurettin 7mo ago
  > AGI just means human level intelligence. It seems more like people haven't decided on what the goal post is. If AGI is just another human, that's pretty underwhelming. That's why people are imagining something that surpasses humans by heaps and bounds in terms of reasoning, leading to wondrous new discoveries.
  - regularfry 7mo ago
    "Just another human" would be outright astonishing if it landed.
    - nurettin 7mo ago
      Yeah as a dad I can tell you it gets old quickly.
      - regularfry 7mo ago
        "Just another human, but one you can switch off at the wall" is both better and terrifyingly worse.
- 0xbadcafebee 7mo ago
  A 4 year old is currently more capable than LLMs (I'm not making this up, ask Yann LeCun). You're going to need it to reach at least "adult" level to be general intelligence.
- tomrod 7mo ago
  I'd argue they are clarifying the goalposts with aplomb.
TiredOfLife 7mo ago
> Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. Same thing is true for humans.
- tomrod 7mo ago
  We did?
foxglacier 7mo ago
That's an assertion, not a thought experiment. You can't logically reach the conclusion ("It won't") by thinking about it. But it doesn't sound so grand if you say "The assertion I use constantly to explain this".
- mold_aid 7mo ago
  To be fair, the post being replied to is arguing by assertion as well. "The core ideas are all there" is pure if-you-say-so stuff.
joefourier 7mo ago
When did AGI start meaning ASI? LLMs are artificial general intelligence, as per the Wikipedia definition: > generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming Even GPT-3 could meet that bar.
- bornfreddy 7mo ago
  Wtf? Once it was AI. Then the models started passing the Turing test and calling themselves AI, so we started using AGI to say "truly intelligent machines". Now, as per the definition you quoted, apparently even GPT-3 is AGI, so we now have to use "ASI" to mean "intelligent, but artificial"? I think I'll just keep using AI and then explain to anyone who uses that term that there is no "I" in today's LLMs, and they shouldn't use this term for some years at least. And that when they can, we will have a big problem.
  - joefourier 7mo ago
    What's your definition of intelligence? If you exclude LLMs, you might have to exclude quite a few humans as well.
    - ezst 7mo ago
      LLMs are artificial intelligence illusion engines, they only "reason" as far as there's an already made answer in their dataset that they can retrieve and eventually tweak (when things go best). Take them where there's no training data and give them the new axioms to solve your specific problem and see them fail with incorrect gibberish provided as confident answer. Humans of any level of intelligence wouldn't behave like that.
canjobear 7mo ago
It's not obvious why it wouldn't, especially if it gets to read Poincaré and Riemann.

tehjoker 7mo ago

Part of the issue there is that the data quantity prior to 1905 is a small drop in the bucket compared to the internet era even though the logical rigor is up to par.

antupis 7mo ago
Humans need way less data. Just compare Waymo to average 16 year-old with car.
- cellis 7mo ago
  A 16 year old has been training for almost 16 years to drive a car. I would argue the opposite: Waymo’s / Specific AIs need far less data than humans. Humans can generalize their training, but they definitely need a LOT of training!
  - jimbokun 7mo ago
    No 16 year old has practiced driving a car for 16 years.
    - Dansvidania 7mo ago
      If you see gaining fine motor control, understanding pictographic language […] as a prerequisite to driving a car, then yes, all of them are
      - krige 7mo ago
        That's an exaggeration. Nobody is trained to read STOP signs for 16 years, a few months top. And Waymo doesn't need to coordinate a four-limbed, 20-digited, one-headed body to operate a car.
        danielEM 7mo ago
        Well, I also think that there is a lot that we process 'in background' and learn on beforehand in order to learn how to drive and then drive. I think the most 'fair' would be to figure out absolute lowest age of kids that would allow them to perform well on streets behind steering wheel.
        Dansvidania 7mo ago
        i am not making a point that it is, I am rather expanding on the possible perspective in which 16 years of training produce a human driver. That being said, you don't really need training to understand a STOP sign by the time you are required to, its pretty damn clear, it being one of the simpler signs. But you do get a lot of "cultural training" so to speak.
    - krisoft 7mo ago
      They were practicing object recognition, movement tracking and prediction, self-localisation, visual odometry fused with porpiroception and the vestibular system, and movement controls for 16 years before they even sit behind a steering wheel though.
  - noduerme 7mo ago
    When humans, or dogs or cats for that matter, react to novel situations they encounter, when they appear to generalize or synthesize prior diverse experience into a novel reaction, that new experience and new reaction feeds directly back into their mental model and alters it on the fly. It doesn't just tack on a new memory. New experience and new information back-propagates constantly adjusting the weights and meanings of prior memories. This is a more multi-dimensional alteration than simply re-training a model to come up with a new right answer... it also exposes to the human mental model all the potential flaws in all the previous answers which may have been sufficiently correct before. This is why, for example, a 30 year old can lose control of a car on an icy road and then suddenly, in the span of half a second before crashing, remember a time they intentionally drifted a car on the street when they were 16 and reflect on how stupid they were. In the human or animal mental model, all events are recalled by other things, and all are constantly adapting, even adapting past things. The tokens we take in and process are not words, nor spatial artifacts. We read a whole model as a token, and our output is a vector of weighted models that we somewhat trust and somewhat discard. Meeting a new person, you will compare all their apparent models to the ones you know: Facial models, audio models, language models, political models. You ingest their vector of models as tokens and attempt to compare them to your own existing ones, while updating yours at the same time. Only once our thoughts have arranged those competing models we hold in some kind of hierarchy do we poll those models for which ones are appropriate to synthesize words or actions from.
    - tomrod 7mo ago
      In a word, JEPA?
      - noduerme 7mo ago
        No. Not at all like that. I said: >> nor spatial artifacts I meant visual patterns, too. You're thinking about what I said on too granular a level. JEPA is visual, based ultimately on pixels. The tokens may be digested from pixels until they're as large as whole recognizable objects, but the tokens are not whole mental models themselves. Here's an example of humans evaluating competing mental models as tokens: You see a car, it's white, it's got some blood stains on the door, and it's traveling towards a red light at 90 miles an hour in a 30 mph residential zone, while you're about to make a left turn. A human foot is dangling from the trunk. You refer to several mental models you have about high speed chases, drug cartels in the area, murders, etc. You compare these models to determine the next action the car might take. What were the tokens in this scenario? The color of the car, the pixels of blood, the speed, the traffic pattern? Or whole models of understanding behavior where you had to choose between a normal driver's behavior and that of someone with a dead body fleeing a crime scene?
jerf 7mo ago
Yet the humans of the time, a small number of the smartest ones, did it, and on much less training data than we throw at LLMs today. If LLMs have shown us anything it is that AGI or super-human AI isn't on some line, where you either reach it or don't. It's a much higher dimensional concept. LLMs are still, at their core, language models, the term is no lie. Humans have language models in their brains, too. We even know what happens if they end up disconnected from the rest of the brain because there are some unfortunate people who have experienced that for various reasons. There's a few things that can happen, the most interesting of which is when they emit grammatically-correct sentences with no meaning in them. Like, "My green carpet is eating on the corner." If we consider LLMs as a hypertrophied langauge model, they are blatently, grotesquely superhuman on that dimension. LLMs are way better at not just emitting grammatically-correct content but content with facts in them, related to other facts. On the other hand, a human language model doesn't require the entire freaking Internet to be poured through it, multiple times (!), in order to start functioning. It works on multiple orders of magnitude less input. The "is this AGI" argument is going to continue swirling in circles for the forseeable future because "is this AGI" is not on a line. In some dimensions, current LLMs are astonishingly superhuman. Find me a polyglot who is truly fluent in 20 languages and I'll show you someone who isn't also conversant with PhD-level topics in a dozen fields. And yet at the same time, they are clearly sub-human in that we do hugely more with our input data then they do, and they have certain characteristic holes in their cognition that are stubbornly refusing to go away, and I don't expect they will. I expect there to be some sort of AI breakthrough at some point that will allow them to both fix some of those cognitive holes, and also, train with vastly less data. No idea what it is, no idea when it will be, but really, is the proposition "LLMs will not be the final manifestation of AI capability for all time" really all that bizarre a claim? I will go out on a limb and say I suspect it's either only one more step the size of "Attention is All You Need", or at most two. It's just hard to know when they'll occur.

radlad 7mo ago

I'm not sure - with tool calling, AI can both fetch and create new context.

0xbadcafebee 7mo ago
It still can't learn. It would need to create content, experiment with it, make observations, then re-train its model on that observation, and repeat that indefinitely at full speed. That won't work on a timescale useful to a human. Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. So we're hyper-tweaking LLMs to make them more useful faster, in the hopes that that will make us more money. Which it does. But it doesn't make you an AGI.
- charcircuit 7mo ago
  It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.
  - otabdeveloper4 7mo ago
    > they update their memories Their contexts, not their memories. An LLM context is like 100k tokens. That's a fruit fly, not AGI.
    - charcircuit 7mo ago
      A human can't keep 100k tokens active in their mind at the same time. We just need a place to store them and tools to query it. You could have exabytes of memories that the AI could use.
      - otabdeveloper4 7mo ago

deleted 7mo ago

[deleted]

crazy5sheep 7mo ago

The 1905 thought experiment actually cuts both ways. Did humans "invent" the airplane? We watched birds fly for thousands of years — that's training data. The Wright brothers didn't conjure flight from pure reasoning, they synthesized patterns from nature, prior failed attempts, and physics they'd absorbed. Show me any human invention and I'll show you the training data behind it. Take the wheel. Even that wasn't invented from nothing — rolling logs, round stones, the shape of the sun. The "invention" was recognizing a pattern already present in the physical world and abstracting it. Still training data, just physical and sensory rather than textual. And that's actually the most honest critique of current LLMs — not that they're architecturally incapable, but that they're missing a data modality. Humans have embodied training data. You don't just read about gravity, you've felt it your whole life. You don't just know fire is hot, you've been near one. That physical grounding gives human cognition a richness that pure text can't fully capture — yet. Einstein is the same story. He stood on Faraday, Maxwell, Lorentz, and Riemann. General Relativity was an extraordinary synthesis — not a creation from void. If that's the bar for "real" intelligence, most humans don't clear it either. The uncomfortable truth is that human cognition and LLMs aren't categorically different. Everything you've ever "thought" comes from what you've seen, heard, and experienced. That's training data. The brain is a pattern-recognition and synthesis machine, and the attention mechanism in transformers is arguably our best computational model of how associative reasoning actually works. So the question isn't whether LLMs can invent from nothing — nothing does that, not even us. Are there still gaps? Sure. Data quality, training methods, physical grounding — these are real problems. But they're engineering problems, not fundamental walls. And we're already moving in that direction — robots learning from physical interaction, multimodal models connecting vision and language, reinforcement learning from real-world feedback. The brain didn't get smart because it has some magic ingredient. It got smart because it had millions of years of rich, embodied, high-stakes training data. We're just earlier in that journey with AI. The foundation is already there — AGI isn't a question of if anymore, it's a question of execution.

drw85 7mo ago
Nice ChatGPT answer. Put some real thought and data in it too.

xdennis 7mo ago

> Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. AGI just means human level intelligence. I couldn't come up with General Relativity. That doesn't mean I don't have general intelligence. I don't understand why people are moving the goalposts.

nurettin 7mo ago
> AGI just means human level intelligence. It seems more like people haven't decided on what the goal post is. If AGI is just another human, that's pretty underwhelming. That's why people are imagining something that surpasses humans by heaps and bounds in terms of reasoning, leading to wondrous new discoveries.
- regularfry 7mo ago
  "Just another human" would be outright astonishing if it landed.
  - nurettin 7mo ago
    Yeah as a dad I can tell you it gets old quickly.
    - regularfry 7mo ago
      "Just another human, but one you can switch off at the wall" is both better and terrifyingly worse.
0xbadcafebee 7mo ago
A 4 year old is currently more capable than LLMs (I'm not making this up, ask Yann LeCun). You're going to need it to reach at least "adult" level to be general intelligence.
tomrod 7mo ago
I'd argue they are clarifying the goalposts with aplomb.

TiredOfLife 7mo ago

> Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. Same thing is true for humans.

tomrod 7mo ago
We did?

foxglacier 7mo ago

That's an assertion, not a thought experiment. You can't logically reach the conclusion ("It won't") by thinking about it. But it doesn't sound so grand if you say "The assertion I use constantly to explain this".

mold_aid 7mo ago
To be fair, the post being replied to is arguing by assertion as well. "The core ideas are all there" is pure if-you-say-so stuff.

joefourier 7mo ago

When did AGI start meaning ASI? LLMs are artificial general intelligence, as per the Wikipedia definition: > generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming Even GPT-3 could meet that bar.

bornfreddy 7mo ago
Wtf? Once it was AI. Then the models started passing the Turing test and calling themselves AI, so we started using AGI to say "truly intelligent machines". Now, as per the definition you quoted, apparently even GPT-3 is AGI, so we now have to use "ASI" to mean "intelligent, but artificial"? I think I'll just keep using AI and then explain to anyone who uses that term that there is no "I" in today's LLMs, and they shouldn't use this term for some years at least. And that when they can, we will have a big problem.
- joefourier 7mo ago
  What's your definition of intelligence? If you exclude LLMs, you might have to exclude quite a few humans as well.
  - ezst 7mo ago
    LLMs are artificial intelligence illusion engines, they only "reason" as far as there's an already made answer in their dataset that they can retrieve and eventually tweak (when things go best). Take them where there's no training data and give them the new axioms to solve your specific problem and see them fail with incorrect gibberish provided as confident answer. Humans of any level of intelligence wouldn't behave like that.

canjobear 7mo ago

It's not obvious why it wouldn't, especially if it gets to read Poincaré and Riemann.