3 ms·

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday. The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if mo

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday. The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if model was not trained on this). [1] https://github.com/kagisearch/llm-chess-puzzles https://github.com/kagisearch/llm-chess-puzzles

alexop 1y ago
Oh cool, I wonder how good 03 will be. While using 03, I noticed something funny: sometimes I gave it a screenshot without any position data. It ended up using Python and spent 10 minutes just trying to figure out where the figures were exactly.
Gimpei 1y ago
Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.
- globnomulous 1y ago
  And then there's the further question of where we draw the line in ourselves. One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare. He's a world-renowned expert but says he can count on one hand the number of times in his life that he felt he was thinking rather than remembering and reorganizing what he already knew. That's not to say "are you remembering or reasoning" means the same thing when applied to humans vs when it's applied to LLMs.
  - bluefirebrand 1y ago
    > One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare Probably should listen to psychologists and neuroscientists about this, not philosophers tbh
    - globnomulous 1y ago
      When it comes to describing what the brain physically does, sure. When it comes to naming and interpreting, the philosophers are the ones to trust.
Hammershaft 1y ago
It's getting incredibly difficult to find anything on the internet that these models weren't trained on, which is why recent llm tests have used so much secrecy and only shows a few sample questions.

alexop 1y ago

Oh cool, I wonder how good 03 will be. While using 03, I noticed something funny: sometimes I gave it a screenshot without any position data. It ended up using Python and spent 10 minutes just trying to figure out where the figures were exactly.

Gimpei 1y ago

Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.

globnomulous 1y ago
And then there's the further question of where we draw the line in ourselves. One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare. He's a world-renowned expert but says he can count on one hand the number of times in his life that he felt he was thinking rather than remembering and reorganizing what he already knew. That's not to say "are you remembering or reasoning" means the same thing when applied to humans vs when it's applied to LLMs.
- bluefirebrand 1y ago
  > One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare Probably should listen to psychologists and neuroscientists about this, not philosophers tbh
  - globnomulous 1y ago
    When it comes to describing what the brain physically does, sure. When it comes to naming and interpreting, the philosophers are the ones to trust.

Hammershaft 1y ago

It's getting incredibly difficult to find anything on the internet that these models weren't trained on, which is why recent llm tests have used so much secrecy and only shows a few sample questions.