4 ms·

Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who

Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.

MichaelNolan 11d ago
I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
- shric 11d ago
  > so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
  - traes 10d ago
    This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
    - fahrvrgnugen 10d ago
      I feel like every position is in the database so you could just lookup the most popular move for an arbitrary elo and that's the bot.
      - shric 10d ago
        That would only work for the first few (from around 10 to 20 typically depending on how close people stick to opening book) moves. Conservatively there are well over 10 to the 30 positions likely to show up in realistic games. There are of the order of 10 to the 10 or so games recorded. Thus well under one in a trillion positions are "known".
        fahrvrgnugen 7d ago
        It's much smaller than that. You would be unlikely to find yourself in a novel position after 40 moves even if you were trying.
        traes 7d ago
        This is simply blatant misinformation. If you play a game online on lichess and go to the analysis board you can find when your game becomes novel. It will be within 20 turns unless you are intentionally following a known opening. In fact it will likely become unique within 10-15 turns.
        fahrvrgnugen 6d ago
        It's not my experience at all. If you find yourself in a novel position within 10-15 moves it's likely a resignable one. Edit: maybe you don't understand what i mean by novel position. I mean any position that has never been reached in the billions of lichess games, including bullet games among beginners. Also yes I will concede that it's possible to make "quiet" moves. pawn nudges that barely affect anything. If you're doing those you're not 1500 ELO. You're intentionally trying to throw wrenches and I just don't see why an ELO bot even needs to bother with nonsense like that. This is supposed to be for fun / training!

Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.

MichaelNolan 11d ago

I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.

shric 11d ago
> so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
- traes 10d ago
  This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
  - fahrvrgnugen 10d ago
    I feel like every position is in the database so you could just lookup the most popular move for an arbitrary elo and that's the bot.
    - shric 10d ago
      That would only work for the first few (from around 10 to 20 typically depending on how close people stick to opening book) moves. Conservatively there are well over 10 to the 30 positions likely to show up in realistic games. There are of the order of 10 to the 10 or so games recorded. Thus well under one in a trillion positions are "known".
      - fahrvrgnugen 7d ago
        It's much smaller than that. You would be unlikely to find yourself in a novel position after 40 moves even if you were trying.
        traes 7d ago
        This is simply blatant misinformation. If you play a game online on lichess and go to the analysis board you can find when your game becomes novel. It will be within 20 turns unless you are intentionally following a known opening. In fact it will likely become unique within 10-15 turns.
        fahrvrgnugen 6d ago
        It's not my experience at all. If you find yourself in a novel position within 10-15 moves it's likely a resignable one. Edit: maybe you don't understand what i mean by novel position. I mean any position that has never been reached in the billions of lichess games, including bullet games among beginners. Also yes I will concede that it's possible to make "quiet" moves. pawn nudges that barely affect anything. If you're doing those you're not 1500 ELO. You're intentionally trying to throw wrenches and I just don't see why an ELO bot even needs to bother with nonsense like that. This is supposed to be for fun / training!