4 ms·

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely. Personally I like to remind people that thes

by simonw 3mo ago

Some people use it to demystify, but a whole lot of people seem to be using it to dismiss the technology entirely. Personally I like to remind people that these things are next-token predictors, but then emphasize how truly astonishing the results we can get out of sufficiently advanced next-token predictors are.

ameliaquining 3mo ago
I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.
- simonw 3mo ago
  I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text. It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
  - dgellow 3mo ago
    While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)
  - ameliaquining 3mo ago
    I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.) I recommend this explanation: https://www.astralcodexten.com/p/next-token-predictor-is-an-ais-job https://www.astralcodexten.com/p/next-token-predictor-is-an-...
    - dgellow 3mo ago
      You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
      - ameliaquining 3mo ago
        Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)
        slopinthebag 3mo ago
        I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates. So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc. Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.
        ToValueFunfetti 3mo ago
        I would think that qualia is the element of that list that actually matters here- "Is there a way that it feels to be an LLM?" is, I expect, the underlying question being asked within "Is an LLM conscious?". If LLMs could only experience the timeless, nonlocalized color blue, they'd still be conscious. The presence of positive and negative valence qualia is additionally relevant if someone is getting at whether they have moral worth, but it's secondary. But qualia are not directly measurable and the rest of the list only matters if those features are necessary for qualia, which we can't decide without such measurements or at least a strong theoretical model.
  - deleted 3mo ago
    [deleted]
  - Forgeties79 3mo ago
    > It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next. At the risk of sounding overly flippant, all world domination has been achieved by some person(s) figuring out which word should come next. Words quite literally = action when it comes to LLM’s with tools access
    - KPGv2 3mo ago
      > At the risk of seeming X, statement that overwhelmingly demonstrates X-ness. It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accurately forecast effective rhetoric is...indescribably divorced from reality. At the risk of seeming like a jerk.
      - notathrowaway51 3mo ago
        "Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.
        KPGv2 3mo ago
        I suggest you re-read what I wrote. In particular the part where I said there was no large corpus to perform this prediction back then. After you edit, I'll address your point about how military strategy is nothing but skill with words.
        Forgeties79 3mo ago
        > After you edit, I'll address your point This has to be the most insufferable thing I’ve read/heard in weeks. Are you being serious right now?
  - CamperBob2 3mo ago
    It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next. The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged. The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"
    - amenhotep 3mo ago
      LLMs self-evidently have no experience when not inferencing, seems like the most consequential difference to me. If we are next token predictors then we are next token predictors that are inferencing at every waking moment and arguably much of our sleeping moments too; when we train an LLM that can inference as much as we do and remain coherent then I'll be more worried about whether it might have experience.
wren6991 3mo ago
The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
- simonw 3mo ago
  Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one. You can get into RL as part of explaining why it's so unnervingly good at picking a next token.
  - wren6991 3mo ago
    That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing. I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834 https://arxiv.org/pdf/2508.09834): > The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e ...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
    - didibus 3mo ago
      > which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it. Isn't G a learned probability?
slopinthebag 3mo ago
I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.

by simonw 3mo ago

ameliaquining 3mo ago

I'm curious, what are you hoping to convey by reminding people that LLMs are next-token predictors? They are, of course, but most people without an AI background won't fully understand what that means, so I assume you're using it at least partly as a proxy for something else.

simonw 3mo ago
I think understanding how this stuff works is really important. For technical people it gives them a useful starting point for understanding it all. For less technical people it's crucial to help them understand that it's not some weird new magical science-fiction AI - it's still computer programs that turn text into numbers and do stuff with the numbers and turn those back into text. It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.
- dgellow 3mo ago
  While you’re active in this thread, I just want to say thank you for all your writing, you’re such a reliable source of sanity in that crazy new world :)
- ameliaquining 3mo ago
  I don't think the next-token-predictor thing should increase anyone's confidence that LLMs aren't conscious or can't escape the control of their operators. A very closely analogous argument would "prove" that humans aren't conscious or can't do [insert task here] either. (No, I'm not saying that any of this is true of today's LLMs, I'm saying this particular argument doesn't work.) I recommend this explanation: https://www.astralcodexten.com/p/next-token-predictor-is-an-ais-job https://www.astralcodexten.com/p/next-token-predictor-is-an-...
  - dgellow 3mo ago
    You can say that for any argument regarding consciousness, because we don’t have an actual, all encompassing definition of what consciousness is. In general I don’t think comparison with humans makes much sense, we should be able to discuss LLMs without always falling back to “but what about humans” (sorry for the caricature)
    - ameliaquining 3mo ago
      Shouldn't that imply that agnosticism is the proper view, rather than asserting that something is impossible on a next-token-predictor architecture? (Note: I don't actually think the consciousness question is the most important one in the near term. Where I think this line of reasoning gets really dangerous is when people use it to assert that LLMs can't or won't engage in certain behaviors no matter much they advance; this doesn't have anything to do with consciousness.)
      - slopinthebag 3mo ago
        I think you can reliably assert that X != Y without having a complete definition of Y, as long as you can identify at least one property or condition that Y possesses which X violates. So for consciousness and LLMs it could be Qualia, lack of semantic understanding, lack of continuity in time, lack of a high degree of integrated causal feedback, etc. Or perhaps those are just features of human consciousness but not integral to consciousness as a whole. To me this then implies panpsychism to some degree, which I'm alright with too.
        ToValueFunfetti 3mo ago
        I would think that qualia is the element of that list that actually matters here- "Is there a way that it feels to be an LLM?" is, I expect, the underlying question being asked within "Is an LLM conscious?". If LLMs could only experience the timeless, nonlocalized color blue, they'd still be conscious. The presence of positive and negative valence qualia is additionally relevant if someone is getting at whether they have moral worth, but it's secondary. But qualia are not directly measurable and the rest of the list only matters if those features are necessary for qualia, which we can't decide without such measurements or at least a strong theoretical model.
- deleted 3mo ago
  [deleted]
- Forgeties79 3mo ago
  > It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next. At the risk of sounding overly flippant, all world domination has been achieved by some person(s) figuring out which word should come next. Words quite literally = action when it comes to LLM’s with tools access
  - KPGv2 3mo ago
    > At the risk of seeming X, statement that overwhelmingly demonstrates X-ness. It's exhausting to even consider where to begin addressing the assertion that good leadership is just predicting the next word to say. Especially considering the corpus available to most great leaders in history was extremely small. To think Hannibal's military campaigns were just because he'd read like ten books in his life and could accurately forecast effective rhetoric is...indescribably divorced from reality. At the risk of seeming like a jerk.
    - notathrowaway51 3mo ago
      "Competent strategy" is just as good a target for a word predictor to optimize as "effective rhetoric" is.
      - KPGv2 3mo ago
        I suggest you re-read what I wrote. In particular the part where I said there was no large corpus to perform this prediction back then. After you edit, I'll address your point about how military strategy is nothing but skill with words.
        Forgeties79 3mo ago
        > After you edit, I'll address your point This has to be the most insufferable thing I’ve read/heard in weeks. Are you being serious right now?
- CamperBob2 3mo ago
  It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next. The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged. The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"
  - amenhotep 3mo ago
    LLMs self-evidently have no experience when not inferencing, seems like the most consequential difference to me. If we are next token predictors then we are next token predictors that are inferencing at every waking moment and arguably much of our sleeping moments too; when we train an LLM that can inference as much as we do and remain coherent then I'll be more worried about whether it might have experience.

wren6991 3mo ago

The "next-token predictor" framing is also a bit shaky. It's an accurate description of pre-training, where next-token prediction is a useful learning objective to force the model to learn higher-level representations. It's wildly misleading for a model put through an RL post-training campaign. The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.

simonw 3mo ago
Right, but it's still useful to think of these models in terms of next-tokens because it helps explain that they look at every token that came before and use that to put out the next one. You can get into RL as part of explaining why it's so unnervingly good at picking a next token.
- wren6991 3mo ago
  That's true. The fact that an LLM is a pure function of (all previous tokens) -> (next token), with internal state like KV cache only existing for optimisation purposes, is pretty mind-blowing. I guess it was more the "predictor" part I had issue with. There's a tendency to reach for statistical or probabilistic terminology to describe things that aren't usefully understood in those terms. For example in the "Speed Always Wins" LLM technical survey (https://arxiv.org/pdf/2508.09834 https://arxiv.org/pdf/2508.09834): > The gate is a crucial component to bring sparsity in MoE models. For a batch of input token representations X ∈ RT×D, the gate function G determines the probabilities of dispatching token xi to each expert e ...which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it.
  - didibus 3mo ago
    > which is nonsense: the gate simply, directly, selects the experts. There's nothing probabilistic about it. Isn't G a learned probability?

slopinthebag 3mo ago

I think this is smart, it seems to me that the best way to use these models its to approach them as a next-token predictor instead of an intelligent entity. That's how I've gotten the best results from them and allows me to avoid some of the pitfalls people fall into by anthropromising them.