3 ms·

wow I actually did something similar recently and no LLM could win and the centipawn loss was always going through the roof (sort of). I created a leaderboard b

by fsndz 2y ago

wow I actually did something similar recently and no LLM could win and the centipawn loss was always going through the roof (sort of). I created a leaderboard based on it. https://www.lycee.ai/blog/what-happens-when-llms-play-chess https://www.lycee.ai/blog/what-happens-when-llms-play-chess I am very surprised by the perf of got-3.5-turbo-instruct. Beating stockfish ? I will have to run the experiment with that model to check that out

fsndz 2y ago
PS: I ran and as suspected got-3.5-turbo-instruct does not beat stockfish, it is not even close "Final Results: gpt-3.5-turbo-instruct: Wins=0, Losses=6, Draws=0, Rating=1500.00 stockfish: Wins=6, Losses=0, Draws=0, Rating=1500.00" https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?sid=e621216c-2501-45cd-b1f4-abd42e525608 https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?...
- janalsncm 2y ago
  > I always had the LLM play as white against Stockfish—a standard chess AI—on the lowest difficulty setting I think the author was comparing against Stockfish at a lower skill level (roughly, the number of nodes explored in a move).
  - fsndz 2y ago
    Did the same and gpt-3.5-turbo-instruct still lost all the games. maybe a diff in stockfish version ? I am using stockfish 16
    - janalsncm 2y ago
      Huh. Honestly, your answer makes more sense, LLMs shouldn’t be good at chess, and this anomaly looks more like a bug. Maybe the author should share his code so it can be replicated.
- deleted 2y ago
  [deleted]
- tedsanders 2y ago
  Your issue is that the performance of these models at chess is incredibly sensitive to the prompt. If you have gpt-3.5-turbo-instruction complete a PGN transcript, then you'll see performance in the 1800 Elo range. If you ask in English or diagram the board, you'll see vastly degraded performance. Unlike people, how you ask the question really really affects the output quality.

by fsndz 2y ago

fsndz 2y ago

PS: I ran and as suspected got-3.5-turbo-instruct does not beat stockfish, it is not even close "Final Results: gpt-3.5-turbo-instruct: Wins=0, Losses=6, Draws=0, Rating=1500.00 stockfish: Wins=6, Losses=0, Draws=0, Rating=1500.00" https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?sid=e621216c-2501-45cd-b1f4-abd42e525608 https://www.loom.com/share/870ea03197b3471eaf7e26e9b17e1754?...

janalsncm 2y ago
> I always had the LLM play as white against Stockfish—a standard chess AI—on the lowest difficulty setting I think the author was comparing against Stockfish at a lower skill level (roughly, the number of nodes explored in a move).
- fsndz 2y ago
  Did the same and gpt-3.5-turbo-instruct still lost all the games. maybe a diff in stockfish version ? I am using stockfish 16
  - janalsncm 2y ago
    Huh. Honestly, your answer makes more sense, LLMs shouldn’t be good at chess, and this anomaly looks more like a bug. Maybe the author should share his code so it can be replicated.
deleted 2y ago
[deleted]
tedsanders 2y ago
Your issue is that the performance of these models at chess is incredibly sensitive to the prompt. If you have gpt-3.5-turbo-instruction complete a PGN transcript, then you'll see performance in the 1800 Elo range. If you ask in English or diagram the board, you'll see vastly degraded performance. Unlike people, how you ask the question really really affects the output quality.