At the risk of stating the obvious, DUH. There was a relatively comprehensive article around benching LLMs to play chess that measured even the SOTA models at

At the risk of stating the obvious, DUH. There was a relatively comprehensive article around benching LLMs to play chess that measured even the SOTA models at around a mediocre 1000 ELO as compared to Carlsen who is rated at ~2850. https://maxim-saplin.github.io/llm_chess https://maxim-saplin.github.io/llm_chess