8 ms·

In the last example (the riddle)--I generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen

by sejje 2y ago

In the last example (the riddle)--I generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen it already. I would do the same thing, I think. It's too well-known. The variation doesn't read like a riddle at all, so it's confusing even to me as a human. I can't find the riddle part. Maybe the AI is confused, too. I think it makes an okay assumption. I guess it would be nice if the AI asked a follow up question like "are you sure you wrote down the riddle correctly?", and I think it could if instructed to, but right now they don't generally do that on their own.

Jensson 2y ago
> generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen it already. LLMs doesn't assume, its a text completer. It sees something that looks almost like a well known problem and it will complete with that well known problem, its a problem specific to being a text completer that is hard to get around.
- simonw 2y ago
  These newer "reasoning" LLMs really don't feel like pure text completers any more.
  - jordemort 2y ago
    And yet
  - gavinray 2y ago
    Is it not physically impossible for LLM's to be anything but "plausible text completion"? Neural Networks as I understand them are universal function approximators. In terms of text, that means they're trained to output what they believe to be the "most probably correct" sequence of text. An LLM has no idea that it is "conversing", or "answering" -- it relates some series of symbolic inputs to another series of probabilistic symbolic outputs, aye?
    - int_19h 2y ago
      At this point you need to actually define what it means for an LLM to "have an idea".
  - Borealid 2y ago
    What your parent poster said is nonetheless true, regardless of how it feels to you. Getting text from an LLM is a process of iteratively attempting to find a likely next token given the preceding ones. If you give an LLM "The rain in Spain falls" the single most likely next token is "mainly", and you'll see that one proportionately more than any other. If you give an LLM "Find an unorthodox completion for the sentence 'The rain in Spain falls'", the most likely next token is something other than "mainly" because the tokens in "unorthodox" are more likely to appear before text that otherwise bucks statistical trends. If you give the LLM "blarghl unorthodox babble The rain in Spain" it's likely the results are similar to the second one but less likely to be coherent (because text obeying grammatical rules is more likely to follow other text also obeying those same rules). In any of the three cases, the LLM is predicting text, not "parsing" or "understanding" a prompt. The fact it will respond similarly to a well-formed and unreasonably-formed prompt is evidence of this. It's theoretically possible to engineer a string of complete gibberish tokens that will prompt the LLM to recite song lyrics, or answer questions about mathemtical formulae. Those strings of gibberish are just difficult to discover.
    - Workaccount2 2y ago
      The problem is showing that humans aren't just doing next word prediction too.
      - Borealid 2y ago
        I don't see that as a problem. I don't particularly care how human intelligence works; what matters is what an LLM is capable of doing and what a human is capable of doing. If those two sets of accomplishments are the same there's no point arguing about differences in means or terms. Right now humans can build better LLMs but nobody has come up with an LLM that can build better LLMs.
        baq 2y ago
        That’s literally the definition of takeoff, when it starts it gets us to singularity in a decade and there’s no publicly available evidence that it’s started… emphasis on publicly available.
- monkpit 2y ago
  This take really misses a key part of implementation of these LLMs and I’ve been struggling to put my finger on it. In every LLM thread someone chimes in with “it’s just a statistical token predictor”. I feel this misses the point and I think it dismisses attention heads and transformers, and that’s what sits weird with me every time I see this kind of take. There _is_ an assumption being made within the model at runtime. Assumption, confusion, uncertainty - one camp might argue that none of these exist in the LLM. But doesn’t the implementation constantly make assumptions? And what even IS your definition of “assumption” that’s not being met here? Edit: I guess my point, overall, is: what’s even the purpose of making this distinction anymore? It derails the discussion in a way that’s not insightful or productive.
  - Jensson 2y ago
    > I feel this misses the point and I think it dismisses attention heads and transformers Those just makes it better at completing the text, but for very common riddles those tools still gets easily overruled by pretty simple text completion logic since the weights for those will be so extremely strong. The point is that if you understand its a text completer then its easy to understand why it fails at these. To fix these properly you need to make it no longer try to complete text, and that is hard to do without breaking it.
- wongarsu 2y ago
  If you have the model output a chain of thought, whether it's a reasoning model or you prompt a "normal" model to do so, you will see examples of the model going "user said X, but did they mean Y? Y makes more sense, I will assume Y". Sometimes stretched over multiple paragraphs, consuming the entire reasoning budget for that prompt. Discussing whether models can "reason" or "think" is a popular debate topic on here, but I think we can all at least agree that they do something that at least resembles "reasoning" and "assumptions" from our human point of view. And if in its chain-of-thought it decides your prompt is wrong it will go ahead answering what it assumes is the right prompt
- sejje 2y ago
  > it's a text completer Yes, and it can express its assumptions in text. Ask it to make some assumptions, like about a stack for a programming task, and it will. Whether or not the mechanism behind it feels like real thinking to you, it can definitely do this.
  - wobfan 2y ago
    If you call putting text together that reads like an assumption, then yes. But it cannot express assumption, as it is not assuming. It is completing text, like OP said.
    - ToValueFunfetti 2y ago
      It's trained to complete text, but it does so by constructing internal circuitry during training. We don't have enough transparency into that circuitry or the human brain's to positively assert that it doesn't assume. But I'd wager it's there; assuming is not a particularly impressive or computationally intense operation. There's a tendency to bundle all of human consciousness into the definitions of our cognitive components, but I would argue that, eg., a branch predictor is meeting the bar for any sane definition of 'assume'.
- famouswaffles 2y ago
  Text Completion is just the objective function. It's not descriptive and says nothing about how the models complete text. Why people hang on this word, I'll never understand. When you wrote your comment, you were completing text. The problem you've just described is a problem with humans as well. LLMs are assuming all the time. Maybe you would like to call it another word, but it is happening.
  - codr7 2y ago
    With a plan, aiming for something, that's the difference.
    - famouswaffles 2y ago
      Again, you are only describing the how here, not the what (text completion). Also, LLMs absolutely 'plan' and 'aim for something' in the process of completing text. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...
      - namaria 2y ago
        Yeah this paper is great fodder for the LLM pixel dust argument. They use a replacement model. It isn't even observing the LLM itself but a different architecture model. And it is very liberal with interpreting the patterns of activations seen in the replacement model with flowery language. It also include some very relevant caveats, such as: "Our cross-layer transcoder is trained to mimic the activations of the underlying model at each layer. However, even when it accurately reconstructs the model’s activations, there is no guarantee that it does so via the same mechanisms." https://transformer-circuits.pub/2025/attribution-graphs/methods.html https://transformer-circuits.pub/2025/attribution-graphs/met... So basically the whole exercise might or might not be valid. But it generates some pretty interactive graphics and a nice blog post to reinforce the anthropomorphization discourse
        famouswaffles 2y ago
- chairdoor 2y ago
  "Assume" can just be a proxy term for "text completion that contains an assumption," especially considering that we don't have enough concrete details about human cognition to know for sure that we aren't doing the same thing.
moffkalast 2y ago
Yeah you need specific instruct training for that sort of thing, Claude Opus being one of the rare examples that does such a sensibility check quite often and even admits when it doesn't know something. These days it's all about confidently bullshitting on benchmarks and overfitting on common riddles to make pointless numbers go up. The more impressive models get on paper, the more rubbish they are in practice.
- VHRanger 2y ago
  Do you have an example or two of a query that opus does well that others fail at?
- pants2 2y ago
  Gemini 2.5 is actually pretty good at this. It's the only model ever to tell me "no" to a request in Cursor. I asked it to add websocket support for my app and it responded like, "looks like you're using long polling now. That's actually better and simpler. Lets leave it how it is." I was genuinely amazed.
furyofantares 2y ago
I don't really mind using analogies for LLMs "assuming" things or being "confused" too much. I think there really is _some_ value to such analogies. However I gotta take issue with using those analogies when "it's trained for text completion and the punchline to this riddle is surely in its training data a lot" is a perfectly good explanation. I guess I would also add that the answer is well-aligned with RLHF-values. I wouldn't go for an explanation that requires squishy analogies when the stuff we know about these things seems completely adequate.
Skunkleton 2y ago
https://kagi.com/assistant/3752c5f9-bf5c-4a43-bada-b3eccbe946de https://kagi.com/assistant/3752c5f9-bf5c-4a43-bada-b3eccbe94... You should be able to click left right on the prompt to see different responses. Sonnet 3.7 with extended thinking notices the issue, and then chooses to totally ignore it with no explanation. From Claude for those who don’t want to click: Wait, I notice a difference from the traditional riddle. In this version, the surgeon says "I can operate on this boy" (affirmative) rather than "I can't operate on this boy" (negative). This changes the nature of the puzzle somewhat. If the surgeon is saying they CAN operate, then we need to explain why this is surprising or seemingly impossible, but actually possible. The traditional answer would still apply: the surgeon is the boy's mother.
valenterry 2y ago
> I generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen it already. Just not enough training data I suppose. Were it really smart then it would "understand" the situation and clarify: "I assume you are asking me that popular riddle - the answer is X". At least after OPs first question a human would usually respond like that.

by sejje 2y ago

Jensson 2y ago

> generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen it already. LLMs doesn't assume, its a text completer. It sees something that looks almost like a well known problem and it will complete with that well known problem, its a problem specific to being a text completer that is hard to get around.

simonw 2y ago
These newer "reasoning" LLMs really don't feel like pure text completers any more.
- jordemort 2y ago
  And yet
- gavinray 2y ago
  Is it not physically impossible for LLM's to be anything but "plausible text completion"? Neural Networks as I understand them are universal function approximators. In terms of text, that means they're trained to output what they believe to be the "most probably correct" sequence of text. An LLM has no idea that it is "conversing", or "answering" -- it relates some series of symbolic inputs to another series of probabilistic symbolic outputs, aye?
  - int_19h 2y ago
    At this point you need to actually define what it means for an LLM to "have an idea".
- Borealid 2y ago
  What your parent poster said is nonetheless true, regardless of how it feels to you. Getting text from an LLM is a process of iteratively attempting to find a likely next token given the preceding ones. If you give an LLM "The rain in Spain falls" the single most likely next token is "mainly", and you'll see that one proportionately more than any other. If you give an LLM "Find an unorthodox completion for the sentence 'The rain in Spain falls'", the most likely next token is something other than "mainly" because the tokens in "unorthodox" are more likely to appear before text that otherwise bucks statistical trends. If you give the LLM "blarghl unorthodox babble The rain in Spain" it's likely the results are similar to the second one but less likely to be coherent (because text obeying grammatical rules is more likely to follow other text also obeying those same rules). In any of the three cases, the LLM is predicting text, not "parsing" or "understanding" a prompt. The fact it will respond similarly to a well-formed and unreasonably-formed prompt is evidence of this. It's theoretically possible to engineer a string of complete gibberish tokens that will prompt the LLM to recite song lyrics, or answer questions about mathemtical formulae. Those strings of gibberish are just difficult to discover.
  - Workaccount2 2y ago
    The problem is showing that humans aren't just doing next word prediction too.
    - Borealid 2y ago
      I don't see that as a problem. I don't particularly care how human intelligence works; what matters is what an LLM is capable of doing and what a human is capable of doing. If those two sets of accomplishments are the same there's no point arguing about differences in means or terms. Right now humans can build better LLMs but nobody has come up with an LLM that can build better LLMs.
      - baq 2y ago
        That’s literally the definition of takeoff, when it starts it gets us to singularity in a decade and there’s no publicly available evidence that it’s started… emphasis on publicly available.
monkpit 2y ago
This take really misses a key part of implementation of these LLMs and I’ve been struggling to put my finger on it. In every LLM thread someone chimes in with “it’s just a statistical token predictor”. I feel this misses the point and I think it dismisses attention heads and transformers, and that’s what sits weird with me every time I see this kind of take. There _is_ an assumption being made within the model at runtime. Assumption, confusion, uncertainty - one camp might argue that none of these exist in the LLM. But doesn’t the implementation constantly make assumptions? And what even IS your definition of “assumption” that’s not being met here? Edit: I guess my point, overall, is: what’s even the purpose of making this distinction anymore? It derails the discussion in a way that’s not insightful or productive.
- Jensson 2y ago
  > I feel this misses the point and I think it dismisses attention heads and transformers Those just makes it better at completing the text, but for very common riddles those tools still gets easily overruled by pretty simple text completion logic since the weights for those will be so extremely strong. The point is that if you understand its a text completer then its easy to understand why it fails at these. To fix these properly you need to make it no longer try to complete text, and that is hard to do without breaking it.
wongarsu 2y ago
If you have the model output a chain of thought, whether it's a reasoning model or you prompt a "normal" model to do so, you will see examples of the model going "user said X, but did they mean Y? Y makes more sense, I will assume Y". Sometimes stretched over multiple paragraphs, consuming the entire reasoning budget for that prompt. Discussing whether models can "reason" or "think" is a popular debate topic on here, but I think we can all at least agree that they do something that at least resembles "reasoning" and "assumptions" from our human point of view. And if in its chain-of-thought it decides your prompt is wrong it will go ahead answering what it assumes is the right prompt
sejje 2y ago
> it's a text completer Yes, and it can express its assumptions in text. Ask it to make some assumptions, like about a stack for a programming task, and it will. Whether or not the mechanism behind it feels like real thinking to you, it can definitely do this.
- wobfan 2y ago
  If you call putting text together that reads like an assumption, then yes. But it cannot express assumption, as it is not assuming. It is completing text, like OP said.
  - ToValueFunfetti 2y ago
    It's trained to complete text, but it does so by constructing internal circuitry during training. We don't have enough transparency into that circuitry or the human brain's to positively assert that it doesn't assume. But I'd wager it's there; assuming is not a particularly impressive or computationally intense operation. There's a tendency to bundle all of human consciousness into the definitions of our cognitive components, but I would argue that, eg., a branch predictor is meeting the bar for any sane definition of 'assume'.
famouswaffles 2y ago
Text Completion is just the objective function. It's not descriptive and says nothing about how the models complete text. Why people hang on this word, I'll never understand. When you wrote your comment, you were completing text. The problem you've just described is a problem with humans as well. LLMs are assuming all the time. Maybe you would like to call it another word, but it is happening.
- codr7 2y ago
  With a plan, aiming for something, that's the difference.
  - famouswaffles 2y ago
    Again, you are only describing the how here, not the what (text completion). Also, LLMs absolutely 'plan' and 'aim for something' in the process of completing text. https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...
    - namaria 2y ago
      Yeah this paper is great fodder for the LLM pixel dust argument. They use a replacement model. It isn't even observing the LLM itself but a different architecture model. And it is very liberal with interpreting the patterns of activations seen in the replacement model with flowery language. It also include some very relevant caveats, such as: "Our cross-layer transcoder is trained to mimic the activations of the underlying model at each layer. However, even when it accurately reconstructs the model’s activations, there is no guarantee that it does so via the same mechanisms." https://transformer-circuits.pub/2025/attribution-graphs/methods.html https://transformer-circuits.pub/2025/attribution-graphs/met... So basically the whole exercise might or might not be valid. But it generates some pretty interactive graphics and a nice blog post to reinforce the anthropomorphization discourse
      - famouswaffles 2y ago
chairdoor 2y ago
"Assume" can just be a proxy term for "text completion that contains an assumption," especially considering that we don't have enough concrete details about human cognition to know for sure that we aren't doing the same thing.

moffkalast 2y ago

Yeah you need specific instruct training for that sort of thing, Claude Opus being one of the rare examples that does such a sensibility check quite often and even admits when it doesn't know something. These days it's all about confidently bullshitting on benchmarks and overfitting on common riddles to make pointless numbers go up. The more impressive models get on paper, the more rubbish they are in practice.

VHRanger 2y ago
Do you have an example or two of a query that opus does well that others fail at?
pants2 2y ago
Gemini 2.5 is actually pretty good at this. It's the only model ever to tell me "no" to a request in Cursor. I asked it to add websocket support for my app and it responded like, "looks like you're using long polling now. That's actually better and simpler. Lets leave it how it is." I was genuinely amazed.

furyofantares 2y ago

I don't really mind using analogies for LLMs "assuming" things or being "confused" too much. I think there really is _some_ value to such analogies. However I gotta take issue with using those analogies when "it's trained for text completion and the punchline to this riddle is surely in its training data a lot" is a perfectly good explanation. I guess I would also add that the answer is well-aligned with RLHF-values. I wouldn't go for an explanation that requires squishy analogies when the stuff we know about these things seems completely adequate.

Skunkleton 2y ago

https://kagi.com/assistant/3752c5f9-bf5c-4a43-bada-b3eccbe946de https://kagi.com/assistant/3752c5f9-bf5c-4a43-bada-b3eccbe94... You should be able to click left right on the prompt to see different responses. Sonnet 3.7 with extended thinking notices the issue, and then chooses to totally ignore it with no explanation. From Claude for those who don’t want to click: Wait, I notice a difference from the traditional riddle. In this version, the surgeon says "I can operate on this boy" (affirmative) rather than "I can't operate on this boy" (negative). This changes the nature of the puzzle somewhat. If the surgeon is saying they CAN operate, then we need to explain why this is surprising or seemingly impossible, but actually possible. The traditional answer would still apply: the surgeon is the boy's mother.

valenterry 2y ago

> I generally assume the AI isn't misreading, rather that it assumes you couldn't give it the riddle correctly, but it has seen it already. Just not enough training data I suppose. Were it really smart then it would "understand" the situation and clarify: "I assume you are asking me that popular riddle - the answer is X". At least after OPs first question a human would usually respond like that.