3 ms·
> Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even
by simianwords 7mo ago
> Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording.
I'm positive that they are perfectly fine and will a pretty good job. Did you actually try it?
- CamperBob2 7mo agoEh, I can see their point, I think. The models can restate the rules differently, I'm sure, but it sounds like the GP is saying that LLMs can't tell whether the rules are well-balanced. It would be interesting to see some example problems along those lines. Design some games with complex rules, including one or two of the most subtle game-wrecking bugs you can think of, and ask the models if they can spot them. In fact that sounds more interesting the more I think about it. Intensive RL on that sort of thing might generalize in... let's say useful ways.
- simianwords 7mo agoI would love to see examples but I think we won’t. I’m happy to be proven wrong that an llm will do worse than a fairly smart human (without prior experience in the board game).