3 ms·
Because Nethack and video games remove most of the reasons that GOFAI failed; for example, everything hard about perception from pixels is removed when you have
by gwern 4y ago
Because Nethack and video games remove most of the reasons that GOFAI failed; for example, everything hard about perception from pixels is removed when you have a little parsed text grid of cells presented to you or are hooked into the game engine and query objects directly. (The real world is not made of a clean little grid of discrete high-level objects like 'dragon' or 'black jelly'.)
Moravec's paradox again: humans find perception easy but things like Sokoban hard, while GOFAI approaches find Sokoban so trivial that it's common to use it (or Sudoku) as a toy problem introduction to constraint solving. Nethack literally has levels which are just Sokoban and which are important to solve; GOFAI can push the boulders around perfectly, even though it would be unable to recognize a photograph of 'a boulder' or use the word 'boulder' in a story.
Then you have the extensive hand-engineering of expert knowledge which goes into ascension agents and the symbolic winners, above and beyond merely plugging in a Sokoban solver. There are increasing experiments in making DRL agents exploit or initialize from pretrained language models (https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2201.12122 https://arxiv.org/abs/2204.01691#google https://arxiv.org/abs/2204.01691#google https://ai.stanford.edu/blog/DrRepair/ https://ai.stanford.edu/blog/DrRepair/ https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2005.07648#google https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2009.03393 https://arxiv.org/abs/2204.00598 https://arxiv.org/abs/2204.00598 come to mind) or reading manuals (https://arxiv.org/abs/1401.5390 https://arxiv.org/abs/1401.5390 all the way back in 2012!), and of course, a DRL agent can learn a tremendous amount without actually doing any playing by offline and off-policy and imitation learning, but while it is exciting and things like Gato look like the future, there is a long way to go from feeding in a dump of the Nethack wiki which mentions offhandedly "you can do X" to an agent recognizing an opportunity for X in the wild and executing it. (Which is something that symbolic approaches also don't come anywhere close to doing, because they just cheat by the capability being given to them by hand-engineering rather than having to autonomously read, understand, and apply.)
- gwern 4y agoProbably the biggest thing for Nethack DRL will be when datasets of human games allow for imitation-learning initialization, rather than learning from tabula rasa (which even humans can't do). I asked them about that and apparently the existing telnet server datasets aren't usable but they have been working on getting usable game logs in the future: https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_we_are_facebook_ai_researchs_nethack_learning/h9pb7p9/ https://www.reddit.com/r/MachineLearning/comments/p88v9w/d_w... As sessions pile up fast from all the active players, I expect at some point they'll be able to dump 50 or 100k human games, including by expert players, and that'll lead to a huge boost for DRL perf. And especially with those logged games, language models with retrieval will be useful: https://arxiv.org/abs/2206.05314#deepmind https://arxiv.org/abs/2206.05314#deepmind