7 ms·

Sounds like when you teach a neural network chess, it ends up learning many of the same concepts that we do. To me, this is a great illustration that superhuma

Sounds like when you teach a neural network chess, it ends up learning many of the same concepts that we do. To me, this is a great illustration that superhuman intelligence is not magic. When a superhuman AI plays chess, the moves usually make sense to an expert. And even if they don't immediately make sense, they usually make sense once the expert plays out some lines to see what happens. Superhuman AIs have crushed humans at chess not by discovering new, wildly counterintuitive openings that dumb humans missed (it actually plays many of the same openings). Rather, it just does what humans do - win pieces, keep your king safe, attack, gain space, restrict your opponent's moves - but better and more consistently. This paper builds on that concept and finds that not only are a superhuman AI's moves usually understandable by experts, but even some of the superhuman AI's internal representations are understandable by experts (!). In the 5th century BCE, Greek philosophers thought the Earth was a sphere. Although they were eventually improved upon by Newton's ellipsoid, they still had arrived at the right concept. And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding still builds upon the same concepts. Thinking broadly, like, suppose we invent some superhuman AGI for $100B in 2100. And then we ask it how to optimize our paperclip production. I don't think its recommendations will be magic. Paperclip production is a fairly well understood problem: you bring in some metal, you reshape it, you send it out. It's probably gonna have much of the same advice we've already figured out: invest in good equipment, take advantage of economies of scale, choose cheap but effective metals, put factories near sources of energy and labor and raw materials, ship out paperclips in a hierarchical network with caches along the way to supply unanticipated demand, etc. Like, an AGI's suggestions may well be better than a management consultant's, but it's not like it's going to wave a magic wand and flip the laws of physics. Rather, I expect it to win within the same frameworks we've developed, but by doing things better and more consistently than humans.

barry-cotter 5y ago
> Rather, it just does what [less successful people] do - ... - but better and more consistently. Reminds me of comparisons between Renaissance and other quant funds. If you get enough better it looks like what you're doing is magic until you analyse what was done and see they have better systems and people. Sometimes, magic is just someone spending more time on something than anyone else might reasonably expect - Teller
- belter 5y ago
  Shame they could not extend their better people and systems to all their customers: "Renaissance Clients Exit After Firm’s Anemic Run of Results" https://www.bloomberg.com/news/articles/2021-02-08/renaissance-clients-pull-out-after-firm-s-rotten-run-of-results https://www.bloomberg.com/news/articles/2021-02-08/renaissan...
noxvilleza 5y ago
> When a superhuman AI plays chess, the moves usually make sense to an expert. It's really interesting to look at very high level players analyzing specific AI moves. Many they understand after a short while, others take longer - and some they don't understand directly, but might have an inkling as to why it's good. My feeling is that many high level analysts would rather just automatically defer to the AI's analysis rather than challenging it - so there is in some sense a lack of feedback to correct such situations. Even in the 1997 Deep Blue vs Kasparov match, game #1 Deep Blue apparently was a bugged position so it moved randomly (a legal random move) - but because Kasparov believed it was correct he felt the analysis was way beyond him so he conceded.
Galanwe 5y ago
> When a superhuman AI plays chess, the moves usually make sense to an expert. And even if they don't immediately make sense, they usually make sense once the expert plays out some lines to see what happens. Do you actually play chess? Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM like Nakamura or Carlsen. It's actually the most reliable and used way to know whether someone is cheating or not, if the moves don't make sense to high level players, then it's very probably someone using an engine. > And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts. That is very far from being a truth that lots of chess experts would agree on. Most chess players at high level reason with similar, well established concepts, such as tempo, positions, baits, color dominance, etc. Super AIs don't use these concepts, and it's obvious when you see their moves. They often break tempo abruptly, disregard color dominance, enter what looks like dangerous positions,etc.
- sudosysgen 5y ago
  You can find out why an engine is making a move though. Hikaru and Magnus won't immediately understand why a move was played but they eventually will as they analyze the game.
  - kruxigt 5y ago
    Yeah since chess involves relatively few moves in total it's pretty clear that almost every move should make sense in hindsight since it has to lead to something that makes sense (like capturing a piece and/or improving positioning in a way understandable to GM:s) in relatively few moves.
- hackinthebochs 5y ago
  Stockfish prior to adopting a neural network evaluation engine made a lot of questionable-looking moves. But it makes many more natural looking moves after being upgraded to use the NN evaluation engine. Broadly speaking, NN based engines make more human looking moves compared to rule-based evaluations[1], with the human-expert accuracy going up as the human rating increases. [1]https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf (scroll down for the graphs)
29athrowaway 5y ago
In the first match of the Lee Sedol / Alpha Go jubango, many moves were initially interpreted as slow, or simply mistakes. Human players try to maximize score difference, Alpha Go tries to maximize the probability of winning. This means, between playing a risky move that makes more territory, and a safe move that makes less territory, Alpha Go prefers the safer move. To experts, those move came across as "slow", inefficient or suboptimal. But they turn out to be the superior moves.
- AceyMan 5y ago
  The difference in strategy makes sense when you consider humans know we are likely to make some variable (suboptimal) choices and so our strategy includes building a buffer to guard against the inevitable gaff. GAI/NN doesn't 'fear' variability and so it's OK to ride a razor thin margin all the way to the victory. [edit: rephrase]
csee 5y ago
But could there be a class of games whose solutions are so complex that we can't understand them, yet an AGI would? We designed chess to be easily understood by humans. There's a cap on the conceptual complexity that fits within our capabilities. So it's not surprising that we can understand AI solutions that merely asymptote to that fixed cap.
ajuc 5y ago
Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the universe rules that makes this impossible we simply don't know how to do it (yet).
- zwegner 5y ago
  Or even without using any such new technology, it could at least take advantage of the fact that almost all human communication is mediated by computers, and coerce virtually the entire human race into helping it. I think the GP wasn't being nearly creative enough when thinking about the possibilities/dangers of AGI.
- notahacker 5y ago
  Sure, and God could smite all unbelievers, turn water into wine and bring about the Rapture: there are no limits to what purely hypothetical omniscient beings can do. But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
  - ajuc 5y ago
    The assumption was > suppose we invent some superhuman AGI for $100B in 2100 not > AI systems humans actually know how to build and obtain useful results from You're talking about specialized expert system that is possible today. BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe. Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine. We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
snarf21 5y ago
Serious question: I haven't done the research but I've always felt that in abstracts like Chess and Go that a larger part of the skill is in "reading the board". Humans are leveraging our pattern matching but we can get tired or miss even one little thing. The AI never misses, ever. I've always wondered how a human would do against an AI if the human was given a list of the top 50 potential moves in a randomized order to consider before making their move. Would the human be on more level playing field in terms of reading?
- PeterisP 5y ago
  There never are 50 reasonable possible moves in Chess, but with top 3 it would be an advantage. Perhaps it's illustrative to consider the chess variant "hand and brain" played by teams of two people, one ("brain") picking the piece to be moved, and the other ("hand") choosing where to move it; the relationship is complex, but in general having a strong "brain" limit your possible moves to just a single piece would improve the chances of a weaker "hand" above what they could do on their own.
ucha 5y ago
After watching some of the games AlphaZero played against Stockfish, I have to say I don't really agree that the moves make sense to a grandmaster... AlphaZero has this uncanny ability to sacrifice many pawns to restrict the movement of the opponent's pieces, something that you might see in advanced play but nowhere near the extent you see it with AlphaZero. Many more of the moves it plays are hard to make sense of compared to a classic engine such as Stockfish. See GothamChess' very entertaining analysis: https://www.youtube.com/watch?v=8dT6CR9_6l4 https://www.youtube.com/watch?v=8dT6CR9_6l4
themeiguoren 5y ago
The fallacy here is that we can only measure whether AZ is playing human strategies, by comparing against our own known space of strategies. Sure, we human have developed good heuristics the AI has also found, but we can’t know if it has better/more subtle concepts that we haven’t discovered, exactly because we haven’t discovered them yet. Who is to say “mobility” is even a coherent top level concept compared to some unpronounceable abstraction the AI has figured out?

barry-cotter 5y ago

> Rather, it just does what [less successful people] do - ... - but better and more consistently. Reminds me of comparisons between Renaissance and other quant funds. If you get enough better it looks like what you're doing is magic until you analyse what was done and see they have better systems and people. Sometimes, magic is just someone spending more time on something than anyone else might reasonably expect - Teller

belter 5y ago
Shame they could not extend their better people and systems to all their customers: "Renaissance Clients Exit After Firm’s Anemic Run of Results" https://www.bloomberg.com/news/articles/2021-02-08/renaissance-clients-pull-out-after-firm-s-rotten-run-of-results https://www.bloomberg.com/news/articles/2021-02-08/renaissan...

noxvilleza 5y ago

> When a superhuman AI plays chess, the moves usually make sense to an expert. It's really interesting to look at very high level players analyzing specific AI moves. Many they understand after a short while, others take longer - and some they don't understand directly, but might have an inkling as to why it's good. My feeling is that many high level analysts would rather just automatically defer to the AI's analysis rather than challenging it - so there is in some sense a lack of feedback to correct such situations. Even in the 1997 Deep Blue vs Kasparov match, game #1 Deep Blue apparently was a bugged position so it moved randomly (a legal random move) - but because Kasparov believed it was correct he felt the analysis was way beyond him so he conceded.

Galanwe 5y ago

> When a superhuman AI plays chess, the moves usually make sense to an expert. And even if they don't immediately make sense, they usually make sense once the expert plays out some lines to see what happens. Do you actually play chess? Half the moves made by an AI at 3500+ ELO (e.g. Stockfish 13+) is completely alien to even a human super GM like Nakamura or Carlsen. It's actually the most reliable and used way to know whether someone is cheating or not, if the moves don't make sense to high level players, then it's very probably someone using an engine. > And here I think we see the same thing: although a superhuman AI 'understands' chess better than human experts, its understanding is still leverages the same concepts. That is very far from being a truth that lots of chess experts would agree on. Most chess players at high level reason with similar, well established concepts, such as tempo, positions, baits, color dominance, etc. Super AIs don't use these concepts, and it's obvious when you see their moves. They often break tempo abruptly, disregard color dominance, enter what looks like dangerous positions,etc.

sudosysgen 5y ago
You can find out why an engine is making a move though. Hikaru and Magnus won't immediately understand why a move was played but they eventually will as they analyze the game.
- kruxigt 5y ago
  Yeah since chess involves relatively few moves in total it's pretty clear that almost every move should make sense in hindsight since it has to lead to something that makes sense (like capturing a piece and/or improving positioning in a way understandable to GM:s) in relatively few moves.
hackinthebochs 5y ago
Stockfish prior to adopting a neural network evaluation engine made a lot of questionable-looking moves. But it makes many more natural looking moves after being upgraded to use the NN evaluation engine. Broadly speaking, NN based engines make more human looking moves compared to rule-based evaluations[1], with the human-expert accuracy going up as the human rating increases. [1]https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf https://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf (scroll down for the graphs)

29athrowaway 5y ago

In the first match of the Lee Sedol / Alpha Go jubango, many moves were initially interpreted as slow, or simply mistakes. Human players try to maximize score difference, Alpha Go tries to maximize the probability of winning. This means, between playing a risky move that makes more territory, and a safe move that makes less territory, Alpha Go prefers the safer move. To experts, those move came across as "slow", inefficient or suboptimal. But they turn out to be the superior moves.

AceyMan 5y ago
The difference in strategy makes sense when you consider humans know we are likely to make some variable (suboptimal) choices and so our strategy includes building a buffer to guard against the inevitable gaff. GAI/NN doesn't 'fear' variability and so it's OK to ride a razor thin margin all the way to the victory. [edit: rephrase]

csee 5y ago

But could there be a class of games whose solutions are so complex that we can't understand them, yet an AGI would? We designed chess to be easily understood by humans. There's a cap on the conceptual complexity that fits within our capabilities. So it's not surprising that we can understand AI solutions that merely asymptote to that fixed cap.

ajuc 5y ago

Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the universe rules that makes this impossible we simply don't know how to do it (yet).

zwegner 5y ago
Or even without using any such new technology, it could at least take advantage of the fact that almost all human communication is mediated by computers, and coerce virtually the entire human race into helping it. I think the GP wasn't being nearly creative enough when thinking about the possibilities/dangers of AGI.
notahacker 5y ago
Sure, and God could smite all unbelievers, turn water into wine and bring about the Rapture: there are no limits to what purely hypothetical omniscient beings can do. But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
- ajuc 5y ago
  The assumption was > suppose we invent some superhuman AGI for $100B in 2100 not > AI systems humans actually know how to build and obtain useful results from You're talking about specialized expert system that is possible today. BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe. Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine. We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.

snarf21 5y ago

Serious question: I haven't done the research but I've always felt that in abstracts like Chess and Go that a larger part of the skill is in "reading the board". Humans are leveraging our pattern matching but we can get tired or miss even one little thing. The AI never misses, ever. I've always wondered how a human would do against an AI if the human was given a list of the top 50 potential moves in a randomized order to consider before making their move. Would the human be on more level playing field in terms of reading?

PeterisP 5y ago
There never are 50 reasonable possible moves in Chess, but with top 3 it would be an advantage. Perhaps it's illustrative to consider the chess variant "hand and brain" played by teams of two people, one ("brain") picking the piece to be moved, and the other ("hand") choosing where to move it; the relationship is complex, but in general having a strong "brain" limit your possible moves to just a single piece would improve the chances of a weaker "hand" above what they could do on their own.

ucha 5y ago

After watching some of the games AlphaZero played against Stockfish, I have to say I don't really agree that the moves make sense to a grandmaster... AlphaZero has this uncanny ability to sacrifice many pawns to restrict the movement of the opponent's pieces, something that you might see in advanced play but nowhere near the extent you see it with AlphaZero. Many more of the moves it plays are hard to make sense of compared to a classic engine such as Stockfish. See GothamChess' very entertaining analysis: https://www.youtube.com/watch?v=8dT6CR9_6l4 https://www.youtube.com/watch?v=8dT6CR9_6l4

themeiguoren 5y ago

The fallacy here is that we can only measure whether AZ is playing human strategies, by comparing against our own known space of strategies. Sure, we human have developed good heuristics the AI has also found, but we can’t know if it has better/more subtle concepts that we haven’t discovered, exactly because we haven’t discovered them yet. Who is to say “mobility” is even a coherent top level concept compared to some unpronounceable abstraction the AI has figured out?