3 ms·

Apparently I massively misremembered the move rate. 5 (potentially less) illegal moves in 8205. So 1 in ~1650. Uh..oops. https://github.com/adamkarvonen/chess_

Apparently I massively misremembered the move rate. 5 (potentially less) illegal moves in 8205. So 1 in ~1650. Uh..oops. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval A plane is comparable to a bird because they both fly even if they do it very differently. Replace plane with bee and it's the same thing. GPT is obviously comparable to a human regardless of how similar whatever internal processes it uses are. I've never once said GPT was playing chess exactly the same way as a human. I wouldn't know. It is playing it though and it does understand the game.

danShumway 3y ago
> A plane is comparable to a bird because they both fly even if they do it very differently. Replace plane with bee and it's the same thing. I think this is exactly where we're differing. I would not say that a bird and a plane should be compared in that way. Definitely not a bird and a bee. They don't even have the same number of wings, the mechanisms and capabilities diverge very quickly. The outcome is similar, but the methodology is extremely different, and that difference in methodology matters because it allows us to make predictions and to reason about when the output between the two systems is going to subtly differ -- which it does, GPT and humans don't perform identically on all tasks and we don't struggle at the same things. There are things birds can do while flying that planes can't do, and vice-versa. I think you're setting yourself up for failure if you engage with a system only looking at its outputs and not considering how its processes influence those outputs. > I've never once said GPT was playing chess exactly the same way as a human. I wouldn't know. My assertion is, you should know. You don't have to know exactly how GPT works or how humans work to look at both and say, "huh, these things exhibit very different strengths and weaknesses, and those strengths and weaknesses fit into consistent patterns, almost as if they're using separate strategies to accomplish tasks." If you look at a bee and an airplane both flying, you don't have to know how either of them fly to realize that they aren't flying the same way.

danShumway 3y ago

> A plane is comparable to a bird because they both fly even if they do it very differently. Replace plane with bee and it's the same thing. I think this is exactly where we're differing. I would not say that a bird and a plane should be compared in that way. Definitely not a bird and a bee. They don't even have the same number of wings, the mechanisms and capabilities diverge very quickly. The outcome is similar, but the methodology is extremely different, and that difference in methodology matters because it allows us to make predictions and to reason about when the output between the two systems is going to subtly differ -- which it does, GPT and humans don't perform identically on all tasks and we don't struggle at the same things. There are things birds can do while flying that planes can't do, and vice-versa. I think you're setting yourself up for failure if you engage with a system only looking at its outputs and not considering how its processes influence those outputs. > I've never once said GPT was playing chess exactly the same way as a human. I wouldn't know. My assertion is, you should know. You don't have to know exactly how GPT works or how humans work to look at both and say, "huh, these things exhibit very different strengths and weaknesses, and those strengths and weaknesses fit into consistent patterns, almost as if they're using separate strategies to accomplish tasks." If you look at a bee and an airplane both flying, you don't have to know how either of them fly to realize that they aren't flying the same way.