3 ms·

I don't think this proves that the LLM is just "pattern matcher". Human makes similar mistakes too, especially when under time pressure (similar to non-reasonin

I don't think this proves that the LLM is just "pattern matcher". Human makes similar mistakes too, especially when under time pressure (similar to non-reasoning model that needs to "use system one" to generate answer on one go). This is further evident that if you specifically ask the models to pay attention to traps, or just ask follow up question "are you sure?", then they usually can get it right.

jsheard 2y ago
You're saying that humans perform worse on problems that are slightly different than previously published forms of the same problem? To be clear we are only talking about changing variable names and constants here.
- exe34 2y ago
  Often yes, because we assume we already know the answer and jump to the conclusion. At least those of us with ADHD do.
  - zeroonetwothree 2y ago
    Not really true for Putnam problems since you have to write a proof. You literally can’t just jump to a conclusion and succeed.
- Lerc 2y ago
  That is the principle behind the game 'Simon says'
  - fldskfjdslkfj 2y ago
    'Simon says' is about reaction time and pressure.
  - chairhairair 2y ago
    No, it’s not at all. This is all getting so tiresome.
  - zeroonetwothree 2y ago
    That’s a very silly analogy. A more realistic analogy would be do humans perform better on computing 37x41 or 87x91 (with showing the work)?
    - Lerc 2y ago
      It was not an analogy at all. It was an simplified example of the idea that a slight change in a pattern can induce error in humans. It seems some people disagree that that is what the game "Simon Says" is about. I feel like they might play a vastly simplified version of the game that I am familiar with. There was a recent episode of Game Changer based on this which is an excellent example of how the game leader should attempt to induce errors by making a change that does not get correctly accounted for.
deleted 2y ago
[deleted]

jsheard 2y ago

You're saying that humans perform worse on problems that are slightly different than previously published forms of the same problem? To be clear we are only talking about changing variable names and constants here.

exe34 2y ago
Often yes, because we assume we already know the answer and jump to the conclusion. At least those of us with ADHD do.
- zeroonetwothree 2y ago
  Not really true for Putnam problems since you have to write a proof. You literally can’t just jump to a conclusion and succeed.
Lerc 2y ago
That is the principle behind the game 'Simon says'
- fldskfjdslkfj 2y ago
  'Simon says' is about reaction time and pressure.
- chairhairair 2y ago
  No, it’s not at all. This is all getting so tiresome.
- zeroonetwothree 2y ago
  That’s a very silly analogy. A more realistic analogy would be do humans perform better on computing 37x41 or 87x91 (with showing the work)?
  - Lerc 2y ago
    It was not an analogy at all. It was an simplified example of the idea that a slight change in a pattern can induce error in humans. It seems some people disagree that that is what the game "Simon Says" is about. I feel like they might play a vastly simplified version of the game that I am familiar with. There was a recent episode of Game Changer based on this which is an excellent example of how the game leader should attempt to induce errors by making a change that does not get correctly accounted for.

deleted 2y ago

[deleted]