9 ms·
MuZero: Mastering Go, chess, shogi and Atari without rules
- jedharris 6y agoSame topic as a year ago, but deserves much more examination than it got then.
- MasterScrat 6y agoYeah it's basically the exact same thing as in Nov 2019 right? They're just hyping up their Nature publication. Or did I miss something?
- ignoranceprior 6y agoWhoa, this is extremely impressive. Quotes from the BBC article: > "For the first time, we actually have a system which is able to build its own understanding of how the world works, and use that understanding to do this kind of sophisticated look-ahead planning that you've previously seen for games like chess. > "[It] can start from nothing, and just through trial and error both discover the rules of the world and use those rules to achieve kind of superhuman performance." > [...] MuZero is effectively able to squeeze out more insight from less data than had been possible before, explained Dr Silver. https://www.bbc.com/news/technology-55403473 https://www.bbc.com/news/technology-55403473 It seems like we're getting much closer to artificial general intelligence from two directions: reinforcement learning (such as MuZero), and sequence prediction (such as GPT-3 and iGPT). Very interesting times to be in the AI field.
- bko 6y agoI've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and pushed back against the end to end approach that doesn't make any preconceived assumptions about the environment. As an example he takes red lights. They're clearly human engineered signals, so it makes sense to have a module that can explicitly determine the signal as opposed to learning the behavior But that's true about other games as well and end to end methods still outperform. Which makes me ask, is end to end learning an inevitability for self driving as well or is this the one domain special due to complexity or other aspects?
- philipkglass 6y agoA machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems that it would need an extremely high fidelity virtual environment to learn in. The high fidelity requirement would necessitate a lot of up-front investment in trying to get the simulations right. You might spend a whole career just trying to build a drivable Virtual Philadelphia as challenging as the real thing. The details would also make it much more expensive to run training sessions at high multiples of real time. Given those factors, I'm not surprised that self-driving vehicle experiments just use real environments and don't try to learn the fundamental rules from scratch. But it's an interesting point that these choices may make it harder for agents to keep improving.
- bko 6y agoThere's a middle ground too which alpha go leveraged which bootstrapped learning from actual human gameplay predicting user actions. That's what comma.ai does and AlphaGo still performed significantly better than explicit rules or higher level abstractions. That and a mix of simulations might yield better results
- ars 6y agoIf you could make a virtual environment that was that good, you would have already solved the self driving problem. I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.
- thomasahle 6y ago>> "[It] can start from nothing, and just through trial and error both discover the rules of the world Unless they've changed a lot of things since the original paper, this is a bit exaggerated. MuZero learns what moves are allowed in a given position/situation, but it still needs to know a finite overall set of possible actions. E.g. for chess, it isn't told which fourty moves a available at each point in its search tree, but it still knows to only consider 64x64 discrete options.
- tromp 6y ago> only consider 64x64 discrete options It wouldn't know that moving its King from e1 to g1 must be accompanied by moving its rook from h1 to f1. Or that moving a white pawn from e5 to d6 must in some cases be accompanied by removing the black pawn on e6. I guess the environment does these in response. That doesn't suffice for moving its pawn from b7 to b8 must be accompanied by replacing said pawn by some other piece.
- chongli 6y agoWhat does it mean to “not be given the rules”? If you set a child down in front of a chess board with the pieces nearby and they are not aware of the rules, I doubt they’d ever figure out how to play even a single correct game of chess. Heck, the child may decide to put the pieces in their mouth or dress them up as make belief characters. Without any concept of the rules you have no way of even knowing that you’ve set up the pieces for a legal starting position, never mind executing a legal move to open the game. This is really bizarre.
- ignoranceprior 6y agoThis is explained in Appendix A of the paper ("Comparison to AlphaZero"): https://arxiv.org/pdf/1911.08265.pdf https://arxiv.org/pdf/1911.08265.pdf Basically, AlphaZero was provided with a simulator that was able to distinguish legal and illegal moves and determine which future game states would be wins or losses. This was used to generate the search tree of possible states and actions. MuZero doesn't have access to a simulator, it only has access to its direct environment. MuZero excludes actions that are immediately illegal, which solves the problem you mention in your penultimate paragraph, but it needs to learn the game's dynamics in order to determine which future moves and states are possible.
- thomasahle 6y agoThe point is this > AlphaZero used the set of legal actions obtained from the simulator to mask the policy network at interior nodes. MuZero does not perform any masking within the search tree, but only masks legal actions at the root of the search tree where the set of available actions is directly observed. The policy network rapidly learns to exclude actions that are unavailable, simply because they are never selected. MuZero still masks legal moves, but only at the root. All its parts are eventually trained on the output of its root, and so learn the legal moves. The justify this root level masking by how the Atari will only allow you to perform legal moves, while a weak enough player may consider illegal moves while planning in your head. The main thing that's slightly "hidden under the rug" is that for "masking" to make sense in the first place, MuZero needs to know a set of all moves that may be legal at some point in the games.
- kovek 6y agoAmazing! Anyone has ideas on how to: 1. Bet on AGI, 2. Encourage AGI? I have a strong belief that it could grow and I’d like to contribute (and join the development)
- mellosouls 6y agoIt's not obvious this has much to do with AGI in the sense of human level sentience.
- ve55 6y agoI think it definitely does. Is this AGI, or even close to it? No. But is it in the direction of AGI, or otherwise some small building block of it, that when worked on for decades, may contribute? Definitely
- ska 6y agoIt's still not that obvious. There has been a lot of interesting stuff in this current iteration of "AI", but the overall approach could still end up being a dead end with respect to AGI itself. It's an old discussion, and while a few of the deep learning results are really impressive I don't think any of them have fundamentally changed that discussion, yet.
- xiphias2 6y ago
- deegles 6y agoIs there a way to learn Go from scratch using these AIs? I wonder if it would pay off in the long run to be fully trained by one.
- devindotcom 6y agoI think it would just smoke you from the outset. As far as I know it doesn't have a structured intelligence it can scale back - it would make the optimal move every time, destroying you like it destroyed top-tier players. I tried learning Go a little while back but hit a wall. Was thinking about trying this more gamified option: https://www.wolfeystudios.com/TheConquestOfGo.html https://www.wolfeystudios.com/TheConquestOfGo.html
- plants 6y agoMy rudimentary understanding of most reinforcement learning systems is that there is an "probability of optimality" associated with each action. Wouldn't there be a way to make the AI take the Nth optimal move or vary the degree of optimality with each move?
- kadoban 6y agoFor these you are able to pick out worse moves than the optimal from them, but that's not actually the same thing as "play like a beginner, okay now play like an intermediate". These things are still openish questions and if nothing else there's a lot of room for improvement in tools to help you learn and review games.
- _hark 6y agoYou can make KataGo play moves that keep the score roughly even since it has a trained score head, e.g. kataJigo [1]. This will keep the game even to your level, a nice way to train. [1] https://github.com/sanderland/katrain#ais https://github.com/sanderland/katrain#ais
- 6y ago
- aleem 6y agoPoint this thing at the stock market and see how that game plays out.
- patagurbon 6y agoIt's not so clear to me how you would train a reinforcement learning agent for the stock market. You have historic data for prices etc. But that's more of a supervised learning thing. You could set it loose on one of those realtime market simulators, but the agents actions wouldn't have any impact on the simulation right?
- hnracer 6y agoThere's two problems in markets, price prediction and execution (ie what to do with your prediction). The former is a supervised learning problem but the latter is an action space problem ie an RL problem. Although nobody in industry has gotten any RL methods too work, they overfit to the incredibly small data sets.
- hnracer 6y agoMany very smart people have tried and failed. State of the art remains very basic supervised models with hand engineered features. In the markets, data is permanently scarce, so these methods don't work well. In the RL problems that DeepMind is solving, data is literally unlimited, and that's the problem space that these methods have been designed for.
- bshanks 6y agoThis is celebrating their Nature publication today. Here is the preprint: https://deepmind.com/research/publications/Mastering-Atari-Go-Chess-and-Shogi-by-Planning-with-a-Learned-Model https://deepmind.com/research/publications/Mastering-Atari-G... https://arxiv.org/pdf/1911.08265.pdf https://arxiv.org/pdf/1911.08265.pdf
- ArtWomb 6y agoCurious if MuZero was ever unleashed upon the ProcGen Benchmark? Where immediate rewards are sparse. But once the underlying generators are "solved". They can be readily exploited ;) https://www.amazon.science/blog/neurips-reinforcement-learning-challenge-winners-announced https://www.amazon.science/blog/neurips-reinforcement-learni...
- mark_l_watson 6y agoI watched the Alpha Go vs. Lee Sedol games live. Big fan. That said, I think Deep Mind should go all in for solving practical real world problems.
- syntheticmindai 6y agoThey have used muzero to do video compression and saved 5% of bits. Source: david silver wired.co.uk interview
- Jabbles 6y agoI couldn't find that source, please elaborate.
- tobessebot 6y agoI mean they pretty much solved protein folding this year...
- mark_l_watson 6y ago+1 Yes, that is impressive and practical.
- syntaxing 6y agoAs impractical as the idea is, reinforcement learning is so damn fun. I highly recommend others to play around with it. I originally was using the famous fork of OpenAi baseline, stable baseline but had issues with tuning with Optuna. I recently stumbled across Ray from Berkley [1] and it has a newer and fancier built-in hyper-parameter tuner. Even as a hardware engineer that's only a software hobbyist can make the computer play some atari games. I think my next step is to try to make my own Super Mario agent. [1] https://docs.ray.io/en/latest/index.html https://docs.ray.io/en/latest/index.html
- nightcracker 6y agoEh, I'd say it's fun if you have a couple thousand TPUs lying around. If you're just messing around with 1 GPU and a desktop PC you should be happy to get Atari breakout to work.
- syntaxing 6y agoDefinitely the worse part of RL is that it takes so long to train. But it works surprisingly well on Google Colab or equivalent.
- confuseshrink 6y agoThe published hyperparameters are usually ridiculously conservative, for the simple games like breakout and pong you can usually converge in far fewer frames than in the papers.
- nightcracker 6y agoI have tried reproducing the papers, with mixed success. I do not share your sentiment.
- webmaven 6y agoI may be missing something, but it seems that what is being described is a neural net architecture that can be trained on any of several games to get impressive results for those game. NOT that one neural net can be trained to play all of the games. So, while this is an interesting result and makes using the same architecture for specific applications easier and a bit more plug-and-play with little to no modification of the code, what it accomplishes is reducing the effort required by a software engineer or researcher on adapting the software before training even begins, but it pretty much still requires the same amount of training. What this doesn't seem to do is allow the same trained network to be applied to multiple tasks (which I think most of the AGI comments are assuming), and it certainly doesn't generalize anything among the games it is trained on.
- vermilingua 6y agoDeepMind seems to be building the Wintermute to OpenAI’s Neuromancer. Where’s Turing?
- willowwonder45 6y agoCurious as to what is "Turing" in this context?
- vermilingua 6y agoIn Neuromancer (William Gibson's 1984 genre-defining cyberpunk novel), the Turing Police enforce laws prohibiting the creation of any superintelligent AI. I don't want to spoil anything, so we'll leave it there.
- skybrian 6y agoThe full Atari game list (Appendix I) is interesting. It's not better at every game, and scores a zero on Pitfall and Montezuma's Revenge.
- lacker 6y agoYou might be interested to read https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark https://deepmind.com/blog/article/Agent57-Outperforming-the-... which has an alternative approach that does better on those games. Basically those games involve lots of “exploration” to find the winning states, so you need some algorithm that is incentivized to explore through a large state space even when it hasn’t seen any reward there.
- iamcreasy 6y agoWould it be able to play(and win) Among Us as an imposter or we are still far away from that?
- lacker 6y agoFar away - the Atari games tested do not include multiplayer logic or any communication with other agents.
- Buttons840 6y agoI'd still be most impressed to see an AI beat the top Civilization players. No mechanical advantage since it's turn based, but there are several different types of decisions to make beyond just "move a piece". AIs haven't yet conquered such environments. It would also give the gaming industry a kick in the pants to start making better AIs.
- dwohnitmok 6y agoI am fairly confident a team of DeepMind's calibre could put together an AI in fairly short order that would demolish top-level Civilization players. Despite my confidence, I still would love to see such a thing made. DeepMind made a good effort with AlphaStar at building an AI that could compete with top-level humans in Starcraft. It wasn't superhuman; it could still be consistently beaten by the absolute best Starcraft players, especially as Zerg or Terran. However, as Protoss, AlphaStar was truly a pro-level player. I'm somewhat surprised DeepMind didn't go further and try to optimize AlphaStar to truly be superhuman. I'm not sure if that indicates a fundamental limitation of their approach or whether it was a shift in approach. This was with successively refined limitations on AI action speeds that caused AlphaStar to really rely on strategy and tactics rather than brute force speed. Regardless, real-time strategy games feel much more difficult than turn-based strategy games to develop a good AI for. Just being able to split things into discrete turns seems like a massive simplification.
- asbund 6y agoYea, wake me up when these models understand rule of physics and rule of law
- ngcc_hk 6y agoWe live only once. Could this uniqueness meant some of life decision must be done without repeating the case billion of time, which obviously s impossible. I read qm. But does this actually useful for partial info. This is also another life situation where you never have full information. I still wonder about the intelligence.