3 ms·

>I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some

>I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. Just so you know, LLMs can play chess just fine. Like this isn't some inability. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c...

unclebucknasty 3y ago
From the README: >Per move, a model gets 5 illegal moves before forced resignation of the round. >Most of gpt-4's losses were due to illegal moves, so it may be possible to come up with a prompt to have gpt-4 correct illegal moves and improve its score. My experience exactly with gpt4. I first solved by keeping a list of illegal moves and subsequently prompting to move again and explicitly exclude those moves. Idea was to constrain gpt as little as possible, but that became inefficient. So I began to prompt for each move with the complete set of legal moves to choose from. In any case, at a certain point it becomes somewhat random when the goal is just to produce a legal move. But, it's interesting that producing legal moves is a challenge for gpt4, especially given that it's such a discrete problem.
bongodongobob 3y ago
The github you linked shows ~1200, not 1800 which is orders of magnitude difference. The other is fine tuning, which is out of scope. A 1200 has something like a 1% chance of beating an 1800. 1200 is sort of you know the rules and maybe have a general idea of an openings, make weak tactical moves mid game and poor concept of endgame. It's just too "deep" for an LLM and a poor standard regardless. As an avid chess and GPT person, I've tried to get it to play decent chess but can't. Neither of your links support it, but if you've got anything else, send it.
- famouswaffles 3y ago
  >The github you linked shows ~1200, not 1800 which is orders of magnitude difference. 1200 is the Elo of the small gpt model the github owner trained. 1800 is the Elo of gpt-3.5-turbo-instruct. https://twitter.com/GrantSlatton/status/1703913578036904431?t=UGkUvLXCXWu8ema3YH548A&s=19 https://twitter.com/GrantSlatton/status/1703913578036904431?... >As an avid chess and GPT person, I've tried to get it to play decent chess but can't. You are not using the model that can play chess. It can only be accessed by Open AI's playground under the Completions endpoint. https://platform.openai.com/playground?mode=complete&model=gpt-3.5-turbo-instruct https://platform.openai.com/playground?mode=complete&model=g...
  - bongodongobob 3y ago
    Very interesting, I will check this out, thank you.

unclebucknasty 3y ago

From the README: >Per move, a model gets 5 illegal moves before forced resignation of the round. >Most of gpt-4's losses were due to illegal moves, so it may be possible to come up with a prompt to have gpt-4 correct illegal moves and improve its score. My experience exactly with gpt4. I first solved by keeping a list of illegal moves and subsequently prompting to move again and explicitly exclude those moves. Idea was to constrain gpt as little as possible, but that became inefficient. So I began to prompt for each move with the complete set of legal moves to choose from. In any case, at a certain point it becomes somewhat random when the goal is just to produce a legal move. But, it's interesting that producing legal moves is a challenge for gpt4, especially given that it's such a discrete problem.

bongodongobob 3y ago

The github you linked shows ~1200, not 1800 which is orders of magnitude difference. The other is fine tuning, which is out of scope. A 1200 has something like a 1% chance of beating an 1800. 1200 is sort of you know the rules and maybe have a general idea of an openings, make weak tactical moves mid game and poor concept of endgame. It's just too "deep" for an LLM and a poor standard regardless. As an avid chess and GPT person, I've tried to get it to play decent chess but can't. Neither of your links support it, but if you've got anything else, send it.

famouswaffles 3y ago
>The github you linked shows ~1200, not 1800 which is orders of magnitude difference. 1200 is the Elo of the small gpt model the github owner trained. 1800 is the Elo of gpt-3.5-turbo-instruct. https://twitter.com/GrantSlatton/status/1703913578036904431?t=UGkUvLXCXWu8ema3YH548A&s=19 https://twitter.com/GrantSlatton/status/1703913578036904431?... >As an avid chess and GPT person, I've tried to get it to play decent chess but can't. You are not using the model that can play chess. It can only be accessed by Open AI's playground under the Completions endpoint. https://platform.openai.com/playground?mode=complete&model=gpt-3.5-turbo-instruct https://platform.openai.com/playground?mode=complete&model=g...
- bongodongobob 3y ago
  Very interesting, I will check this out, thank you.