3 ms·
We don't need to come up with a new board game. How about a board game that has been written about extensively for hundreds of years LLMs can't consistently wi
by bigfishrunning 8mo ago
We don't need to come up with a new board game. How about a board game that has been written about extensively for hundreds of years
LLMs can't consistently win at chess
https://www.nicowesterdale.com/blog/why-llms-cant-play-chess https://www.nicowesterdale.com/blog/why-llms-cant-play-chess
Now, some of the best chess engines in the world are Neural Networks, but general purpose LLMs are consistently bad at chess.
As far as "LLM's don't have understanding", that is axiomatically true by the nature of how they're implemented. A bunch of matrix multiplies resulting in a high-dimensional array of tokens does not think; this has been written about extensively. They are really good for generating language that looks plausible; some of that plausable-looking language happens to be true.
- simianwords 8mo agofalse, chess ELO is pretty good https://maxim-saplin.github.io/llm_chess/ https://maxim-saplin.github.io/llm_chess/ ets not cherry pick and actually see benchmarks please. i would say even ~1000 elo means that it can reason better than the average human.
- bigfishrunning 8mo agoIf you look at the "workflow" section of that page, they had to add a bunch of scaffolding around telling the model what moves are legal -- an llm can't keep enough context to know how to play chess; only to choose an advantageous move from a given list. But feel free to "cherry pick".
- simianwords 8mo agowhy do you think this falsifies that it can't reason?
- simianwords 8mo agoi ran the benchmark without the valid moves tool as well as the three mistakes grace and gpt-5.4 holds well. it can achieve 1000 ELO which is much higher than my own. this clearly tells me that GPT is good at chess, at least better than a normal person who has played ~30-40 games in their life.