4 ms·

I see what you're saying. I believe both players are not playing that close to optimally, but it is hard to prove this without a more optimal AI. Quackle's AI d

I see what you're saying. I believe both players are not playing that close to optimally, but it is hard to prove this without a more optimal AI. Quackle's AI does not account for many factors, and we can find many situations where it makes the wrong move, so we know it's not playing optimally.

dane-pgp 5y ago
In order to judge whether these "many situations" are significant, we need to judge how big an effect the randomness of tile drawing is. To pick an extreme example, let's say that the first player to get an "unlucky" rack gets stuck in loop that somehow keeps them at a disadvantage, one which perhaps snowballs. As an analogy, imagine a game of chess where each player rolls a die at the start of each turn, and if they roll a 6 they have to lose a pawn of their choice. It might be possible for a chess grandmaster to beat a chess computer under that rule set nearly 50% of the time, even with the human making sub-optimal moves in "many situations". Remember, in a perfectly random game, 50% of the time it doesn't matter if you make a mistake, because you were going to lose anyway. (I don't just mean there are psychological factors of players getting lazy when they know they're unlikely to win, I mean that these mistakes don't affect the outcome, so we're talking about very small statistical effects here). Anyway, your experience with Scrabble games probably gives you some good intuition for how big an effect luck could possibly have, and I'm happy to accept an assertion that 500 games would be enough to show a statistically measurable lead for a truly perfect player over both the top human and top AI players. It would still be nice to get some data on the effect of randomness, though, for example by measuring the difference in total score between an AI that gets a standard random draw of tiles each turn, and an AI that always draws some set of tiles in the n-th decile of expected score.

dane-pgp 5y ago

In order to judge whether these "many situations" are significant, we need to judge how big an effect the randomness of tile drawing is. To pick an extreme example, let's say that the first player to get an "unlucky" rack gets stuck in loop that somehow keeps them at a disadvantage, one which perhaps snowballs. As an analogy, imagine a game of chess where each player rolls a die at the start of each turn, and if they roll a 6 they have to lose a pawn of their choice. It might be possible for a chess grandmaster to beat a chess computer under that rule set nearly 50% of the time, even with the human making sub-optimal moves in "many situations". Remember, in a perfectly random game, 50% of the time it doesn't matter if you make a mistake, because you were going to lose anyway. (I don't just mean there are psychological factors of players getting lazy when they know they're unlikely to win, I mean that these mistakes don't affect the outcome, so we're talking about very small statistical effects here). Anyway, your experience with Scrabble games probably gives you some good intuition for how big an effect luck could possibly have, and I'm happy to accept an assertion that 500 games would be enough to show a statistically measurable lead for a truly perfect player over both the top human and top AI players. It would still be nice to get some data on the effect of randomness, though, for example by measuring the difference in total score between an AI that gets a standard random draw of tiles each turn, and an AI that always draws some set of tiles in the n-th decile of expected score.