3 ms·

The blog doesn’t say anything about how this “decision quality” metric is calculated… but presumably it’s using very similar Go evaluation functions to the ones

The blog doesn’t say anything about how this “decision quality” metric is calculated… but presumably it’s using very similar Go evaluation functions to the ones used in the superhuman AI players, right? I think it’s highly unsurprising that humans would improve by that metric — they’re learning from the machine, so of course the machine likes it. Also, most things in life are not two-player zero-sum games where you can construct an evaluation function and build a “decision quality” metric out of it. So I’m not sure what the takeaway should be in those cases.

SonOfLilit 3y ago
Computers are so much better than humans at Go that the metric for board evaluation applies better to human games than it does to computer games. Just as I'm better at evaluating code written by GPT than code written by senior developers. Otherwise you'd see players who don't train with computers winning in tournaments against those who do.
- intuitionist 3y ago
  Yeah, I don’t think the metric is wrong or bad, just that it’s not telling us anything special. Or maybe it’s telling us something about Go AIs (that the “insights” they have are human-comprehensible) but it’s not at all clear that this fully generalizes.
  - visarga 3y ago
    Of course it doesn't work that easy in other fields. Basically the go board is an environment, the the AI model learns by creating its experiences in the environment. This can be applied to other kinds of environment, such as code execution, simulation, video games, human-AI chat rooms or robots. But each environment has its own complexity and searching for good strategies can take a long time. It's the same with scientific research, got to validate the theory in the world.
  - SonOfLilit 3y ago
    It sounds from your analysis that "better as evaluated by AI" would be true even if it wasn't really objectively better. All I'm saying is that yes, it does mean objectively better in this case.
omoikane 3y ago
"Decision quality" appears to be average difference in winning probability between player's move versus a move by Alpha Go (Leela Zero). It's on page 16 of: https://doi.org/10.2139/ssrn.3893835 https://doi.org/10.2139/ssrn.3893835 Where it says "Measuring the quality of moves". I found this via citation #31 of: https://arxiv.org/abs/2311.11388 https://arxiv.org/abs/2311.11388 Which was referenced by the second graph in the blog post (they link to Nature, but the paper at Arxiv appears to be the same).

SonOfLilit 3y ago

Computers are so much better than humans at Go that the metric for board evaluation applies better to human games than it does to computer games. Just as I'm better at evaluating code written by GPT than code written by senior developers. Otherwise you'd see players who don't train with computers winning in tournaments against those who do.

intuitionist 3y ago
Yeah, I don’t think the metric is wrong or bad, just that it’s not telling us anything special. Or maybe it’s telling us something about Go AIs (that the “insights” they have are human-comprehensible) but it’s not at all clear that this fully generalizes.
- visarga 3y ago
  Of course it doesn't work that easy in other fields. Basically the go board is an environment, the the AI model learns by creating its experiences in the environment. This can be applied to other kinds of environment, such as code execution, simulation, video games, human-AI chat rooms or robots. But each environment has its own complexity and searching for good strategies can take a long time. It's the same with scientific research, got to validate the theory in the world.
- SonOfLilit 3y ago
  It sounds from your analysis that "better as evaluated by AI" would be true even if it wasn't really objectively better. All I'm saying is that yes, it does mean objectively better in this case.

omoikane 3y ago

"Decision quality" appears to be average difference in winning probability between player's move versus a move by Alpha Go (Leela Zero). It's on page 16 of: https://doi.org/10.2139/ssrn.3893835 https://doi.org/10.2139/ssrn.3893835 Where it says "Measuring the quality of moves". I found this via citation #31 of: https://arxiv.org/abs/2311.11388 https://arxiv.org/abs/2311.11388 Which was referenced by the second graph in the blog post (they link to Nature, but the paper at Arxiv appears to be the same).