4 ms·

>>> NetHack benefits ‘strategic’ play — good play often involves executing a series of actions with a well-defined, expressible sub-objective, eg: “Find Sokoban

>>> NetHack benefits ‘strategic’ play — good play often involves executing a series of actions with a well-defined, expressible sub-objective, eg: “Find Sokoban” or “Apply a Unicorn Horn to Cure Poison”. Symbolic bots found it easy to define ‘strategy’-like subroutines and to decide when to deploy them based on rich, human-legible representations of the game state. This feels akin to Classic Game AI vs Modern AI debates that happen all the time. And even in 2022, with desktop GPU capabilities nearing supercomputing levels, it feels like rules-based, goal-oriented planning still dominates NPC & Enemy AI in games. The question really becomes, why did symbolic ai research die when it's so effective at specialization? Rather, the research obsession is solving generalization. And the problems always seem to stem around the black box itself, the lack of "human-legible representations" of its latent spaces ;)

jerf 4y ago
"The question really becomes, why did symbolic ai research die when it's so effective at specialization?" I think we want "AI" that scales faster than humans can implement. Game "AI" looks nothing like machine learning and looks a lot more like some programmer just smashing out some code and trying it out for a while. Some autotuning can be done, of course, but it's largely a program being written. Who has time for that?
gwern 4y ago
Because Nethack and video games remove most of the reasons that GOFAI failed; for example, everything hard about perception from pixels is removed when you have a little parsed text grid of cells presented to you or are hooked into the game engine and query objects directly. (The real world is not made of a clean little grid of discrete high-level objects like 'dragon' or 'black jelly'.) Moravec's paradox again: humans find perception easy but things like Sokoban hard, while GOFAI approaches find Sokoban so trivial that it's common to use it (or Sudoku) as a toy problem introduction to constraint solving. Nethack literally has levels which are just Sokoban and which are important to solve; GOFAI can push the boulders around perfectly, even though it would be unable to recognize a photograph of 'a boulder' or use the word 'boulder' in a story. Then you have the extensive hand-engineering of expert knowledge which goes into ascension agents and the symbolic winners, above and beyond merely plugging in a Sokoban solver. There are increasing experiments in making DRL agents exploit or initialize from pretrained language models (https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2204.01691#google https://arxiv.org/abs/2204.01691#google https://ai.stanford.edu/blog/DrRepair/ https://ai.stanford.edu/blog/DrRepair/ https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2204.00598 https://arxiv.org/abs/2204.00598 come to mind) or reading manuals (https://arxiv.org/abs/1401.5390 https://arxiv.org/abs/1401.5390 all the way back in 2012!), and of course, a DRL agent can learn a tremendous amount without actually doing any playing by offline and off-policy and imitation learning, but while it is exciting and things like Gato look like the future, there is a long way to go from feeding in a dump of the Nethack wiki which mentions offhandedly "you can do X" to an agent recognizing an opportunity for X in the wild and executing it. (Which is something that symbolic approaches also don't come anywhere close to doing, because they just cheat by the capability being given to them by hand-engineering rather than having to autonomously read, understand, and apply.)
- gwern 4y ago
  Probably the biggest thing for Nethack DRL will be when datasets of human games allow for imitation-learning initialization, rather than learning from tabula rasa (which even humans can't do). I asked them about that and apparently the existing telnet server datasets aren't usable but they have been working on getting usable game logs in the future: https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_we_are_facebook_ai_researchs_nethack_learning/h9pb7p9/ https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_w... As sessions pile up fast from all the active players, I expect at some point they'll be able to dump 50 or 100k human games, including by expert players, and that'll lead to a huge boost for DRL perf. And especially with those logged games, language models with retrieval will be useful: https://arxiv.org/abs/2206.05314#deepmind https://arxiv.org/abs/2206.05314#deepmind
Bjartr 4y ago
> rules-based, goal-oriented planning still dominates NPC & Enemy AI in games. I think this is largely because the goal of most game AI is not to win per se, but rather to provide an engaging experience to the player. To that end, a game designer wants to be able to purposefully craft the experience, and explicit rule/goal based systems allow that.
- amalcon 4y ago
  Also because rule-based AI is typically much less computationally expensive to execute than ML-based AI. This matters in games because they often have other things to do with the available computational resources on the target hardware.
- munificent 4y ago
  And, in particular, most players enjoy AI that is just smart enough to be challenging, but no smarter. Players feel OK losing to an enemy player that outguns them or gets lucky, but very few enjoy feeling stupid because they were outsmarted by a program that had fewer resources than them but played better.

jerf 4y ago

"The question really becomes, why did symbolic ai research die when it's so effective at specialization?" I think we want "AI" that scales faster than humans can implement. Game "AI" looks nothing like machine learning and looks a lot more like some programmer just smashing out some code and trying it out for a while. Some autotuning can be done, of course, but it's largely a program being written. Who has time for that?

gwern 4y ago

Because Nethack and video games remove most of the reasons that GOFAI failed; for example, everything hard about perception from pixels is removed when you have a little parsed text grid of cells presented to you or are hooked into the game engine and query objects directly. (The real world is not made of a clean little grid of discrete high-level objects like 'dragon' or 'black jelly'.) Moravec's paradox again: humans find perception easy but things like Sokoban hard, while GOFAI approaches find Sokoban so trivial that it's common to use it (or Sudoku) as a toy problem introduction to constraint solving. Nethack literally has levels which are just Sokoban and which are important to solve; GOFAI can push the boulders around perfectly, even though it would be unable to recognize a photograph of 'a boulder' or use the word 'boulder' in a story. Then you have the extensive hand-engineering of expert knowledge which goes into ascension agents and the symbolic winners, above and beyond merely plugging in a Sokoban solver. There are increasing experiments in making DRL agents exploit or initialize from pretrained language models (https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2204.01691#google https://arxiv.org/abs/2204.01691#google https://ai.stanford.edu/blog/DrRepair/ https://ai.stanford.edu/blog/DrRepair/ https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2204.00598 https://arxiv.org/abs/2204.00598 come to mind) or reading manuals (https://arxiv.org/abs/1401.5390 https://arxiv.org/abs/1401.5390 all the way back in 2012!), and of course, a DRL agent can learn a tremendous amount without actually doing any playing by offline and off-policy and imitation learning, but while it is exciting and things like Gato look like the future, there is a long way to go from feeding in a dump of the Nethack wiki which mentions offhandedly "you can do X" to an agent recognizing an opportunity for X in the wild and executing it. (Which is something that symbolic approaches also don't come anywhere close to doing, because they just cheat by the capability being given to them by hand-engineering rather than having to autonomously read, understand, and apply.)

gwern 4y ago
Probably the biggest thing for Nethack DRL will be when datasets of human games allow for imitation-learning initialization, rather than learning from tabula rasa (which even humans can't do). I asked them about that and apparently the existing telnet server datasets aren't usable but they have been working on getting usable game logs in the future: https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_we_are_facebook_ai_researchs_nethack_learning/h9pb7p9/ https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_w... As sessions pile up fast from all the active players, I expect at some point they'll be able to dump 50 or 100k human games, including by expert players, and that'll lead to a huge boost for DRL perf. And especially with those logged games, language models with retrieval will be useful: https://arxiv.org/abs/2206.05314#deepmind https://arxiv.org/abs/2206.05314#deepmind

Bjartr 4y ago

> rules-based, goal-oriented planning still dominates NPC & Enemy AI in games. I think this is largely because the goal of most game AI is not to win per se, but rather to provide an engaging experience to the player. To that end, a game designer wants to be able to purposefully craft the experience, and explicit rule/goal based systems allow that.

amalcon 4y ago
Also because rule-based AI is typically much less computationally expensive to execute than ML-based AI. This matters in games because they often have other things to do with the available computational resources on the target hardware.
munificent 4y ago
And, in particular, most players enjoy AI that is just smart enough to be challenging, but no smarter. Players feel OK losing to an enemy player that outguns them or gets lucky, but very few enjoy feeling stupid because they were outsmarted by a program that had fewer resources than them but played better.