36 ms·
AlphaGo Zero: Learning from scratch
- titzer 9y agoThis is pretty incredible, especially the power dissipation results. Only 4 TPUs? Humans are toast.
- skykooler 9y agoWhat is a TPU?
- red75prime 9y agohttps://en.wikipedia.org/wiki/Tensor_processing_unit https://en.wikipedia.org/wiki/Tensor_processing_unit
- skykooler 9y agoThanks!
- Symmetry 9y agoThat's still 10 times as much energy as a human body or 100 times as much as a human brain. But yeah, it's not like they're throwing a datacenter at this.
- davidkuhta 9y ago> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. So technically this version has lost every game it's ever won. Jokes aside, it's pretty interesting to note that they were able to combine the "policy" and "value" networks. Good SO answers on the difference (https://datascience.stackexchange.com/questions/10932/difference-between-alphagos-policy-network-and-value-network https://datascience.stackexchange.com/questions/10932/differ...) > accumulating thousands of years of human knowledge during a period of just a few days It'd be interesting for what this would mean when things like a neural lace become a reality. As an aside, anyone have any other links or references to others investigating learning algorithms with a 'tabula rasa' approach?
- milofeynman 9y agohttps://blog.openai.com/ https://blog.openai.com/ I believe their DotA 2 AI uses that approach
- mappingbabeljc 9y agoCorrect - we saw a similar phenomenon of rapid capability gain via self-play in our Dota 2 work: https://blog.openai.com/more-on-dota-2/ https://blog.openai.com/more-on-dota-2/
- davidkuhta 9y agoThanks!
- apetresc 9y ago> So technically this version has lost every game it's ever won. No, they've also played it against AlphaGo Lee and AlphaGo Master. The SGFs are available at: https://www.nature.com/nature/journal/v550/n7676/extref/nature24270-s2.zip https://www.nature.com/nature/journal/v550/n7676/extref/natu...
- davidkuhta 9y agoI meant that tongue-in-cheek based on the "by playing games against itself" during training. Nonetheless, thanks for clarifying that in case it's unclear for others (and for the SGFs).
- greysphere 9y agoTD-gammon is a well known version of this technique (with 2 ply lookahead, vs a 1600 deep mcts) https://en.m.wikipedia.org/wiki/TD-Gammon https://en.m.wikipedia.org/wiki/TD-Gammon Temporal difference learning was previously consider weak at 'tactical' games, ie ones with gamestates that require long chains of precise moves to improve position (like many checkmate scenarios in chess) . For anyone more familiar with this technique, is it clear how the mcts/checkpoint system overcomes this? How sensative is the system to the tuning params for those parts of the alg. Like is Go a particularly good candidate because of the ~400 play positions resulting in a (relatively) small tree seach requirement? (I kinda cant believe im saying that go has 'a small search tree'!) We us td learning for the ai in our game Race for the Galaxy, so it's neat to hear about possible avenues for improvement!
- Jyaif 9y ago"It uses one neural network rather than two." and "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This is amazing! The technology they came up with must be super generic.
- bluetwo 9y agoIt sounds like that's what they are going for. Minimal tuning, generally adaptable. Very interesting.
- samfriedman 9y agoSuper, super impressive work. I'd love to see how hard it is to apply the architecture to other games/problems that work well with self-play.
- gordon_freeman 9y agoI will be interested to see what kind of algorithms they have used to allow AlphaGo to learn from its own moves. Are these pretty generics algos or are these very customized and specific ones that only apply to AlphaGo and the game of Go?
- Scaevolus 9y agoThey have a new reinforcement learning algorithm that should be generically applicable to anything where a long sequence of moves results in a specifically gradable outcome. > The neural network in AlphaGo Zero is trained from games of selfplay by a novel reinforcement learning algorithm. In each position s, an MCTS search is executed, guided by the neural network fθ. The MCTS search outputs probabilities π of playing each move. These search probabilities usually select much stronger moves than the raw move probabilities p of the neural network fθ(s); MCTS may therefore be viewed as a powerful policy improvement operator. Self-play with search—using the improved MCTS-based policy to select each move, then using the game winner z as a sample of the value—may be viewed as a powerful policy evaluation operator. The main idea of our reinforcement learning algorithm is to use these search operators repeatedly in a policy iteration procedure: the neural network’s parameters are updated to make the move probabilities and value (p, v)= fθ(s) more closely match the improved search probabilities and selfplay winner (π, z); these new parameters are used in the next iteration of self-play to make the search even stronger.
- dsp1234 9y agoComparing the top player's ELO with Zero's ELO (assuming numbers are accurate, etc): Your rating: 3664 Opponent's rating: 5000 Probability of winning: 0.000456879355457417 So 1 in 2,200 games... ouch
- mda 9y agoI don't think you can apply this to alphago. I think probability for a human to beat alphago now is zero. Lee Sedol's single victory is the first and the last.
- VikingCoder 9y agoI disagree. This is precisely what the ELO predicts, and it has been pretty accurate over time - it's a good metric.
- mda 9y agoFor humans yes. For humans against machines? I don't think so. Can any human beat a modern chess computer? The chance is zero.
- VikingCoder 9y agoZero means zero, yes? Alpha particles can flip bits and cause erratic behavior, can they not? "[The probability of] at least one bit error in 4 gigabytes of memory at sea level on planet Earth in 72 hours is over 95%"
- mda 9y agoOh hey maybe they use ECC? Are we really arguing this? Pedantry on a weird level.
- Invictus0 9y agoIs this evidence of a broader leap forward in machine learning, or are these advancements domain-specific? In other words, could these innovations be applied to other fields and applications?
- hmate9 9y agoI haven't read the paper yet but "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features." This sound super generic, so yes.
- eanzenberg 9y agoIt's further advancement of reinforcement learning which is notoriously difficult to train into usefulness.
- habitue 9y agoI think the fact that it's no longer using Monte Carlo tree search is a huge step forward in the generalizability of the technique. But go is still - a perfect information game - with a relatively small input size (vs. arbitrary computer vision) - cheap to simulate - discrete action space - deterministic This isn't to take away from the magnitude of the achievement, but the nature of the problem itself makes the result less applicable to many tasks we might want to use RL for.
- abecedarius 9y agoMath research shares those qualities, except small input size if you include the body of all the already-known theorems as an input. I don't know if we'll see much smarter proof assistants soon, but it doesn't seem absurd to me as a possible development.
- dane-pgp 9y agoActually, just converting all the already-known theorems into a form that can be computationally verified (not just convince a skilled human) would be an interesting starting point. This would really help the metamath project, and perhaps make peer review of mathematical research papers easier: http://us.metamath.org/mpeuni/mmset.html http://us.metamath.org/mpeuni/mmset.html
- ericb 9y agoI'm reminded of Eliezer Yudkowski's article "There is no fire Alarm for Artificial General Intelligence." Is this smoke? https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/ Yes, this is not an AGI. But the hockey-stick takeoff from defeats some players, to defeats an undefeated world-champion, to defeats the version of itself that beat the world champion 100% of the time is nuts. If this happens in other domains, like finance, health, paper clip collection, the word singularity is really well chosen--we can't see past this.
- deleted 9y ago[deleted]
- panic 9y agoThe core technique of AlphaGo is using tree search as a "policy improvement operator". Tree search doesn't work on most real-world tasks: the "game state" is too complex, there are too many choices, it's hard to predict the full effect of any choice you might make, and there often isn't even a "win" or "lose" state which would let you stop your self-play.
- habitue 9y agoThis version explicitly does not use tree search.
- panic 9y agoMCTS means "Monte-Carlo Tree Search". It's the core of the algorithm. The big difference is that it doesn't use rollouts, or random play: it chooses where to expand the tree based only on the neural network.
- cjbprime 9y agoNo, 'habitue is correct. This new blog post says that the new software no longer does game readouts and just uses the neural net.
- curtisgiddings 9y agoAnyone happen to have a link (or can PM one to me) to a copy that isn't behind a paywall?
- jvolkman 9y agoFrom the Deepmind blog post: https://deepmind.com/documents/119/agz_unformatted_nature.pdf https://deepmind.com/documents/119/agz_unformatted_nature.pd...
- deleted 9y ago[deleted]
- octoploid 9y agohttps://sci-hub.cc/https://www.nature.com/nature/journal/v550/n7676/full/nature24270.html https://sci-hub.cc/https://www.nature.com/nature/journal/v55...
- pmontra 9y agoThey already answered you but go to the bottom of the Nature page for a link to the game records (Supplementary Information).
- MrZipf 9y agoI love what DeepMind is doing, but find their publication choices bizarre. Sure Nature has the occasional ML / NN paper, but it's not a top journal for AI / ML / CS and it makes getting hold of papers awkward and it doesn't seem in the spirit of most CS research.
- narvind 9y agoHow I wish Marvin Minsky would have stayed alive for one more year and seen this. He would have been so happy!
- bradleybuda 9y agoIn the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened. http://www.catb.org/jargon/html/koans.html http://www.catb.org/jargon/html/koans.html
- tedunangst 9y agoSo Sussman was right the first time?
- zardo 9y agoA random net has some random preconception. That doesn't mean it's a bad idea to try random preconceptions.
- farresito 9y agoI think he was cryopreserved, so he surely will be surprised once they wake him up in the future, assuming cryonics really works.
- thefalcon 9y agoI'd certainly be surprised if I ever woke up from being cryopreserved. Which isn't to say that I'd object to the process if I had the disposable income and an understanding/cooperative family support structure, which I do not.
- galkk 9y agoWhy don't use the same approach for chess? It's very interesting to see if it is able to handle much more advanced and tuned engines that exist for chess, game with considerable much more complicated rules?
- Florin_Andrei 9y agoI think chess is less compelling because, in a sense, it is a "solved problem" - superhuman AI chess players already exist. And chess, while it does have more complex base rules, has a much lower combinatorial complexity than Go.
- EvgeniyZh 9y agoWell, I'd love to see NN solution beating top chess engines. It might also introduce novelty to the game, just as regular engines did
- feelix 9y agoIt'd be particularly useful to have a chess bot that can play badly in the same way a human does. The problem with the current chess bots is that they play badly, badly. They choose a terrible random mistake to make every few moves, while some of their other moves are brilliant. They cannot accurately mimic beginner or intermediate level players.
- Florin_Andrei 9y agoIn a sense, that would be like replicating the human brain's functionality, including the bugs and limitations.
- thefalcon 9y agoThis seems like something DeepMind could create, given the incentive. They were able to train AlphaGo to predict human moves in Go at a very high accuracy (obviously not with AlphaGo Zero, but the inferior human-predictive version is how they determined that AGZ is playing qualitatively differently).
- nyxtom 9y agoSo this is fun: "AlphaGo Zero is the program described in this paper. It learns from self-play reinforcement learning, starting from random initial weights, without using rollouts, with no human supervision, and using only the raw board history as input features. It uses just a single machine in the Google Cloud with 4 TPUs (AlphaGo Zero could also be distributed but we chose to use the simplest possible search algorithm)."
- VikingCoder 9y agoSingle machine? Stunning.
- tasuki 9y agoAbsolutely amazing. I'm dying to see the game records!
- tasuki 9y agoSpoiler, they're in the paper: https://deepmind.com/documents/119/agz_unformatted_nature.pdf https://deepmind.com/documents/119/agz_unformatted_nature.pd...
- weavie 9y agoYou can find them here : http://www.alphago-games.com http://www.alphago-games.com
- Radim 9y agoWeird how they anthropomorphized and capitalized "her Final Form" there. Creepy!
- panic 9y agoIs AlphaGo Zero the first Go program without special code to read ladders? I'm curious how a pure neural net can read them, given how non-local they are.
- thefalcon 9y agoThe concept of locality is nothing but a human weakness in Go, the best AI must read the whole board with every move. EDIT: From the paper: "Surprisingly, shicho (“ladder” capture sequences that may span the whole board) – one of the first elements of Go knowledge learned by humans – were only understood by AlphaGo Zero much later in training" I'm surprised by the author's use of the word "Surprisingly" here.
- gizmo686 9y ago>Surprisingly, shicho (‘ladder’ capture sequences that may span the whole board)—one of the first elements of Go knowledge learned by humans—were only understood by AlphaGo Zero much later in training. [0] [0] https://www.nature.com/nature/journal/v550/n7676/full/nature24270.html https://www.nature.com/nature/journal/v550/n7676/full/nature...
- waqf 9y agoAlphaGo is still based around layers of 3×3 local convolutions. That represents a strong assumption about locality in the network design. I would expect AlphaGo to perform poorly on the game "Go with the vertices randomly permuted".
- Palptine 9y agoWell they didn't use inception. If their inception units have, say, a 7x7 conv, then ladders will probably be found much earlier.
- alexbeloi 9y agoVery impressive, the original implementation relied a lot on feature engineering. I'm surprised they're able to prevent a self-play equilibrium with such a simple loss function. It's sort of like they are using auxiliary outputs but instead of using them to fit features, they are fitting to multiple ways of arriving at 'best play', through predicting value (SL) and predicting probability for best outcome (RL). In principle, they're doing the same thing but in practice it seems like they are making up for each others shortcomings (e.g. self-play equilibrium with RL).
- aeleos 9y agoThe fact that they only used self play with no outside input here is really interesting. I wonder if this system produced more new styles of play. While I am not that familiar with Go, I know in some of the other articles they talk about things like Chinese starts that are specific to certain cultures. I wonder if the fact that it had no outside reinforcement made it produce movements that we have already seen that are somehow inherent to the game, or if it produced many more new moves that were a result of it learning without any possibility of cultural interference. According to the article it did invent some unconventional and creative moves, but I also wonder how much it rediscovered. I also wonder how much it’s style of play changes if it were re trained, due to the random start that it is given. Maybe that would produce something like seeds for procedurally generated worlds in games. Like if they could find a seed for a Chinese or japanese players, or ones that more aggressive styles. This is some pretty cool work and may open up even more doors for pure reinforcement learning
- shmageggy 9y agoThe key part from the paper: > To assess the merits of self-play reinforcement learning, compared to learning from human data, we trained a second neural network (using the same architecture) to predict expert moves in the KGS Server data set; this achieved state-of-the-art prediction accuracy compared to pre vious work 12,30–33 (see Extended Data Tables 1 and 2 for current and previous results, respectively). Supervised learning achieved a better initial performance, and was better at predicting human professional moves (Fig. 3). Notably, although supervised learning achieved higher move prediction accuracy, the self-learned player performed much better overall, defeating the human-trained player within the first 24 h of training. This suggests that AlphaGo Zero may be learning a strategy that is qualitatively different to human play.
- aeleos 9y agoThat is really interesting. Given a neural network that solely exist to play Go, one that is influenced by the human mind is limited compared to the exact same set of neurons that doesn't have that influence. EDIT: changed a set of neurons to neural network per andbbergers comments
- braymundo 9y agoI wonder if it could be applied to SC2 (since they already started the research: https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment https://deepmind.com/blog/deepmind-and-blizzard-open-starcra...)
- zardo 9y agoAs is, no. There are too many possible actions and too little time between decisions. I wouldn't discount it entirely though, some sort of clustering of actions may be able to reduce continuous action spaces to a manageable branching factor.
- VanillaCafe 9y agoSo in terms of training, they went from nothing to "Go Singularity" in about a month? Impressive.
- polskibus 9y agoI remember reading ages ago in Scientific American about a much more interesting (and useful) AI application of this technique. Genetic algorithms were used to evolve new, more efficient variants of existing electronic circuits. I dug it up - it was: https://www.scientificamerican.com/magazine/sa/2003/02-01/#article-magnetars https://www.scientificamerican.com/magazine/sa/2003/02-01/#a... Article "Evolving inventions". I have no idea if there is an open-access version anywhere. As far as I remember, that approach led to some patents, because some of the inventions were better than existing solutions. One of the examples in the article was a low-pass filter (I dont remember if AI version was actually better or worse than human-made). The essential element of this approach was that in electronics (as in go) there exist a well defined set of rules, that allows researchers to build a simulation engine with optimization/evaluation function that the AI targets by itself, without supervision. It's great to see that this approach is still alive, although in my humble opinion, application in electronics is much more interesting than Go.
- tmzt 9y agoSomebody needs to dig this up and apply it to an open FPGA toolchain like ICEstorm. The other SA article on this was The Darwin Chip which I think went into more detail. One of the limitations was the lack of documentation for the actual bitstream.
- qrv3w 9y agoNature is actually hosting it without a paywall: https://www.nature.com/scientificamerican/journal/v288/n2/pdf/scientificamerican0203-52.pdf https://www.nature.com/scientificamerican/journal/v288/n2/pd...
- foobaw 9y agoWould love to see some gameplay!
- thefalcon 9y agoHere's a self-play game: AGZ v AGZ - http://eidogo.com/#u2UdsDFJ http://eidogo.com/#u2UdsDFJ
- nicklovescode 9y agoAnyone know why they trained on TPUs? My understanding is that the main benefit of a TPU is inference. Is this not true?
- postnihilism 9y agoThe first generation TPUs were focused on inference. Second generation TPUs are focused on both training and inference. https://www.blog.google/topics/google-cloud/google-cloud-offer-tpus-machine-learning/ https://www.blog.google/topics/google-cloud/google-cloud-off...
- deleted 9y ago[deleted]
- shmageggy 9y agoLooks like the performance improvement comes from two key ingredients: 1) Using Residual networks instead of normal convolutional layers 2) Using a smarter policy training loss that uses the full information from a MCTS at each move. In the previous version, I believe they just ran the policy network to the end of the game and used a very weak {0, 1} reinforcement signal over all of the moves played. Here, it looks like they use each run of MCTS to provide a fully supervised signal over all moves it explores.
- mtremsal 9y agoHow is it different to apply the loss on each actual move at the end of the game VS on each rollout (which is itself a tiny game)? Does it help reinforce learning towards the end game as shorter rollouts are needed? Is the more accurate information then propagated to earlier moves as well?
- gwern 9y agoI think the difference is that under 1/0 policy gradient loss, it gets feedback only on the actual chosen move. Under MCTS-rollouts-each-move, it gets feedback on every move on the board whether its value estimate was slightly too high or low plus the ultimate outcome of the 1 move it did make.
- RoboTeddy 9y agoAlso (3) training a dual policy & value network that can benefit from a single shared representation of the game
- smdz 9y agoDeep reinforcement learning is interesting and has plenty of potential. But highlighting AlphaGo as an example of reinforcement learning is like undermining the concepts of reinforcement learning.
- ionforce 9y agoHow so?
- deleted 9y ago[deleted]
- hetspookjee 9y agoAmazing results, though I am somewhat frightened by how generic this model is and how it achieved such amazing results. I can't help but think that these same techniques can be used to learn how humans react in certain situaties and how they can be, very subtely, be worked to think in a certain way - one that fits the agenda of whatever party is behind it. With the mass surveillance that is Google it's quite doable to test for human reactions on certain things. They got the tools to execute a certain plan and evaluate the effectiveness. Ofcourse it can also go in a benelovent way: like what kind of policy will benefit the most people? (semantics of 'benefiting' aside) I atleast certainly hope these kinds of generic algorithms will be used to generate effective, meaningful policies that truly help the people. Still a far away future but one that gets closer by the day.
- SubiculumCode 9y agoI'd only worry if it can outperform humans when there are not rules per se. That is, if I put a queen down in the GO board, and start knocking off stones, moving three times a turn, then take a lighter and burn the go board, the AI responds by decapitating my head.
- goatlover 9y agoHa! I do wonder about using a board game where the rules periodically change in simple ways at random. A human could easily adapt to the rule changes while playing and adjust their strategy accordingly. Would a Deep Learning algorithm be able to do this? If we keep the board and pieces digital, then the board could change shape, the pieces could change color indicating a random association with a rule change, and what not.
- sidcool 9y agoI would be willing to spend an entire lifetime to perfectly understand how this algorithm works. Currently I can barely write Djikstra's algorithm.
- matthewtoast 9y ago> The system starts off with a neural network that knows nothing about the game of Go. It then plays games against itself I might have missed this, but: Where are the actual rules of Go encoded? Mustn't there be some enumeration of what constitutes "capturing," how the win condition of the game is calculated, correct?
- klipt 9y agoYes, I assume you start with an objective function ("winning Go") that includes the rules.
- dane-pgp 9y agoSurely if they are really starting with "zero", then all the AI is given is the arrangement of stones on the board (which starts empty) with the opportunity to select a position for its next stone after its opponent has placed one, until the game is over. (Let's assume that there is another piece of software responsible for determining when the game has finished, and who has won). As such, the only "rules" the AI needs are that it can only place one stone at a time, only in an empty position, and only when it is not the opponent's turn. To start with "less than zero", though, it would be interesting to see them give the AI a 3D simulation of a room with a simulated Go board and a simulated stone, and give the AI a fixed amount of time for it to have its turn. Just by using the pixel data from a simulated camera, it could learn to use a simulated arm to place the simulated stone on the board in a legal position. The reward function would just have to say, at the end of each allotted time period, whether a legal move had been made or not, and the AI could bootstrap up from that.
- ionforce 9y agoIf you give a list of "possible next states" from any given state, that should encompass all the rules. If there's a rule that's preventing you from doing something, it's not a possible state. By nothing they mean hints about what constitutes "good Go strategy". But it implicitly knows all of the rules of go.
- hzhou321 9y agoHow does that timeline have any meaning?
- Tepix 9y agoIt does if you know that they used 4 TPUs.
- Aron 9y agoOne idea occurs to me is to now evolve the Go game itself in a direction that adds more challenges for an AI to solve, and then solve those problems. How about being able to handle different and randomized board shapes? How about being allowed to say one move the opponent cannot take when you play a piece? It would be interesting to keep track of what variations the algorithm handles well automatically, and which it falls flat on, etc.
- lozenge 9y agoArimaa was a chess inspired game intended to be difficult for computers. It "fell" in 2015.
- JoeDaDude 9y agoLike Arimaa, some other games were (at least partially) designed to be hard for computers: Havannah [1] and Octi [2]. Havannah has since been defeated by the machines. Octi remains unchallenged, but that is probably due to its obscurity. [1]. https://en.wikipedia.org/wiki/Havannah#Computer_Havannah https://en.wikipedia.org/wiki/Havannah#Computer_Havannah [2]. https://news.yale.edu/1999/06/01/successor-chess-new-game-strategy-be-demonstrated-yale-june-11 https://news.yale.edu/1999/06/01/successor-chess-new-game-st...
- partycoder 9y agoVery impressive. I wonder if the reduction in computing costs has to do with pruning the resulting trained networks.
- grondilu 9y ago> If similar techniques can be applied to other structured problems, such as protein folding, reducing energy consumption or searching for revolutionary new materials, Protein folding sounds like a nice idea for their next challenge.
- thefalcon 9y agohttps://www.bloomberg.com/news/articles/2017-10-18/deepmind-s-superpowerful-ai-sets-its-sights-on-drug-discovery https://www.bloomberg.com/news/articles/2017-10-18/deepmind-... Indeed.
- yazr 9y agoDid they publish any papers on AlphaGo Master ? AFAIK the 2016 Nature paper is Alpha Go Lee. And now we skipped to AlphaGo Zero.
- gwern 9y agoNo. Skipped straight to Zero from AG Fan Hui (not AG Lee).
- conistonwater 9y agoExtended data figure 2 contains something really cool: how long it takes for AG0 to discover a joseki, and how long it takes it to later discard it as non-jokeki. So you could, in theory, evaluate a human joseki by plotting AG0's probability of playing it against training time (or against its Elo rating). What's also cool is that the differences that cause it stop playing a joseki must be really minute, but it can see them anyway.
- wwarner 9y agoI would love to know how this adversarial training doesn't end up overfitting. Or, put another way, I'd love to see another piece of software (or even a human being?) exploit Alpha Go's overfitted strategy.
- thefalcon 9y agoA great resource for "watching" AlphaGo's games: http://www.alphago-games.com http://www.alphago-games.com
- tim333 9y agoSlightly scary how it went from zero to superhuman play in three days. I wonder if general AI will go that way one day.
- goatlover 9y agoGeneral AI relative to an individual human, or billions of humans? The sum total of human beings, or organizations of humans is superhuman relative to an individual. We've had superhuman organizations for millennia. I'm not sure how much general AI will be different, other than the large scale automation of jobs which would happen. As Rodney Brooks pointed out, all technology happens within a context, not a vacuum. A general AI will come to exist in a world with a lot of other superhuman capabilities already in existence.
- nothis 9y agoOne of the more interesting things the success of "starting with zero" suggests is that the idea that some mystical "human consciousness" is the end goal for AI might be laughable in the long term. AI might just casually bypass human consciousness, say "oh, hi!" and wave us goodbye a day later. Also, a factor of 7 billion "happens" in computer science. This is getting rather creepy to think of, even if it's still science fiction. At this point, I could see a computer that out-thinks humanity within decades. What would it think? What would we even do with its findings? Would we understand it? Would it understand itself? Would it know how to manipulate us?
- VikingCoder 9y agoIn those three days, it made X moves. If you multiply X by the amount of time it takes, on average, for a human to make a move... How many human lifetimes did Zero take to get to superhuman?
- indescions_2017 9y agoWhat's fascinating (and admittedly somewhat worrying) about Self-play is that an agent can accidentally become adept at tasks other than intended via transfer learning. The "wrestling spiders" in OpenAI's demo quickly mastered the art of Sumo Wrestling. And whatever skills they learned in resisting an opposing force to stay standing on a platform, were immediately applicable to myriad different domains. In this case, being subject to hurricane force winds, and not as any normal spider may, be hurled into the sky! It's more difficult to see how Go playing skills can translate to other domains. But for tasks in robotics, cybersecurity or fintech the power of self-play trained transfer learning becomes more apparent.
- visarga 9y agoIt is clear that these "self-play" scenarios depend on simulation - unless there is an appropriate stage for self play to take place on, there can be no play. The question is - how do we stand with simulation for robotics, self driving cars, etc. My bet is that simulation is going to be the crowning jewel in the AI field, replacing static datasets and supervised learning with "dynamic datasets" and rewards. It would help with data sparsity as well (where can you find an image of a donkey riding an elephant for the new ImageNet? - but you can sim that or any possible combination). Not to mention that humans are fallen head over heels with simulation as well - VR headsets and games in general. I see a great future for simulation with both AI and humans. It will be our common learning/playing/research sandbox.
- seanwilson 9y agoHow do they evaluate that AlphaGo Zero is better than the previous AlphaGos? By playing them against each other? Or playing AlphaGo Zero against humans?
- auggierose 9y agoThe same way they make AlphaGo Zero better, by playing against other AlphaGos.
- seanwilson 9y agoBut what if the other AlphaGos are blind to some tactic that humans could take advantage of? Not saying this is likely given how the Lee Sedol match and others went but I'm curious how they come up with the rankings.
- auggierose 9y agoAt the end of the paper they describe how the come up with ELO rankings, and that in order to avoid some bias due to self-play only they include the results of the AlphaGo's versus Fan, Sedol, etc.
- andai 9y agoSerious question: could it be possible, even in theory, to translate a neural network to a high level programing language? So we can see what it's doing?
- sobellian 9y agoThe catch is that this isn't quite zero human knowledge, since the tree search algorithm is a human discovery, and not one that came easily to humans. It also massively cuts down on the search space for an appropriate policy function. That means that this setup isn't necessarily general. How applicable is MCTS to games with asymmetric information, a la Starcraft? What about games that can't quite be modeled with an alternating turn-based game tree like bughouse?
- anomnomidas 9y agoThere's a Dota 2 bot by OpenAI that played games with itself and managed to beat a lot of pros in the scene. It's still SF mid only no runes and some restricted items, but it shows that there is also potential for Starcraft. https://blog.openai.com/dota-2/ https://blog.openai.com/dota-2/
- zardo 9y agoMaybe you know something we don't... "We’re not ready to talk about agent internals" What makes you think it uses a tree search?
- tyler_larson 9y agoThat's not quite what they're talking about WRT zero human knowledge. The problem is that there's no intrinsic scoring system for Go, nothing specific to maximize, so it's difficult to tell a computer whether a given outcome is "good" or "bad". So early versions of AlphaGo used a collection of human-played Go games to get an idea of what constitutes "good" and what is "bad", so it can then train its model to predict whether a move will make things better or worse. This new system forgoes that step, and instead has the model play itself starting at random and looking for patterns that end up winning games. It's as if you gave the rules to the game of Go to a culture that's never heard of it before, and they evolved their own play style entirely in isolation. Their result is a model that is better than the one that was developed with human influence, and that's the interesting bit.
- sobellian 9y ago
- tc 9y agoWhen things will start getting interesting is when we figure out how to get move simulation and search into the network itself, rather than programming that on the outside. As far as I know, no-one has even the faintest idea of how to do that. We have an existence proof that this should be possible. The networks are great at perception and snap-prediction. Anything a human can do in 200ms is fair game. And with clever engineering, we can make magic happen by iterating or integrating those things. But it's after that first 200ms that humans get really intelligent. When we can come up with an architecture that lets the networks themselves start simulating possibilities, backtracking, deciding when to answer now or to think more -- when the network owns the loop -- then it will get interesting.
- miketery 9y agoWhat do you mean by move simulation?
- baq 9y agoI think he means that the NN somehow learns MCTS without it being coded in explicitly.
- derefr 9y ago> We have an existence proof that this should be possible. Not guaranteed. The human brain has diffusion signalling (i.e. neurotransmitters passing out of the synaptic cleft, into a neighbouring one, and activating a receptor on some other spacially-local axon as a result.) And one of those signalling molecules is thought to represent, in its intensity, a confidence-interval bias adjustment (i.e. a pruning bias factor for MCTS.) So the brain's MCTS-equivalent process may rely on some extra-graphical properties of the brain-as-embodied-meat-thing.
- red75prime 9y agoThat will be a couple of additional terms in activation function. Or am I missing something?
- Abishek_Muthian 9y agoAlgorithms > Big data in this particular machine learning scenario makes it very impressive.
- gfodor 9y agoI'm wondering if once one of these algorithms comes along that has been perfected if it is going to "burn in" the domain it was built for as the target of problem reductions, similar to 8086 assembly or the qwerty keyboard living on today despite them being ancient relics. For example, after this result it seems if you can reduce your problem domain onto Go (or a similarly structured game) you now have a way to create a superhuman solver. It may just be easier to do that then try to even figure out how to design and tune a new network. I could imagine waking up in 10 years being confused at why all software efforts in the AI space are focused on just figuring out clever ways to map real problems onto a hodgepodge of seemingly random "toy" domains like Go and Chess and Starcraft. Hell, maybe the Starcraft bot will immortalize Starcraft in a way the game never would have been able to if it becomes a good reduction target for a lot of domains. It kind of reminds me of how SVMs were "abused" by twisting non-linear domains into them via kernel methods, or by proving the NP-equivalence of a problem by reducing it onto 3-SAT, or how ImageNet's weights are being re-purposed for other image oriented prediction tasks.
- isolli 9y agoWhat is the analogy with the QWERTY keyboard?
- zodiac 9y agoI think the only thing about go that enables this technique is "turn-based perfect information 0-sum game".
- hmate9 9y agoAlso it has fairly limited input. A real world problem may have much more possible inputs at any time step as opposed to placing just 1 stone
- colllectorof 9y agoIn many domains mapping the problem to a tree search already gives you a superhuman solver or at least a passable solver. Problem mapping is what most of modern AI research is about. That's how the field was redefined in recent years. Just like Vladimir Vapnik says[1], it's becoming more engineering than science. (And sometimes more software alchemy than engineering.) [1] https://www.youtube.com/watch?v=5mvfpSdWsOo https://www.youtube.com/watch?v=5mvfpSdWsOo "Brute Force and Intelligent Paradigms of Learning"
- COil 9y agoWhat if a top human player could train like AlphaGO Zero during billions of games and record everything?
- dng88 9y agoIt is impossible to try surgeon on patient or give them 1 million different drugs and see whether it is better or worst. You can for some domain ... like GO!
- infinity0 9y agoOn the bright side, it means the several thousand years of humans playing Go, we were actually going "in the right direction" in terms of optimal strategy, despite not having reduced the game down into provably-optimal mathematical theorems.
- VikingCoder 9y agoI sometimes think about boats. We built damn good boats, even before we knew anything formal about fluid dynamics, or even AIR.
- Dowwie 9y agoCan self-improving AI predict behaviors in emergent environments? Will they?
- zardo 9y agoIs Go strategy not an emergent property of the rules of Go?
- auggierose 9y agoThis is such an impressive result, and so general, I bet many people (including me) wish they knew exactly how to duplicate this result. It would be great if they created an online course that explained all algorithms in detail right up to the creation of AlphaGo Zero itself. The paper gives the impression that it shouldn't be too hard for them to create such a course.
- yters 9y agoIs there anywhere to see the games? I'm curious if the AI is superior to humans or just human trained AI. It'd also be interesting to see the source, but that is apparently not being released for some reason.
- sillysaurus3 9y agoIt'd also be interesting to see the source, but that is apparently not being released for some reason. Last time this was brought up, someone implied it's closed source so as not to boost the Chinese competitor.
- yters 9y agoIf all their methods for success are in the paper the Chinese competitor can just copy that.
- sillysaurus3 9y agoTheoretically, yes. But anyone who has tried to implement scientific papers will tell you it's far harder. The papers often lack critical details and implementation hacks: all the little rough edges that go into making a production system work. They also lack context in many cases, so you spend more time reverse engineering the paper than figuring out how to make it work.
- yters 9y agoIt seems odd they make a big deal about tabula rasa, but won't release the source for verification.
- mrcactu5 9y agothis is quite an achievement that it plays accurately. Does this have implementations for Strategy? No two games are alike, yet we are able to learn from our experience. Human's players be somehow comparing local positions from various games and deciding (sometimes wrongly) that they can be played similarly. Can it recommend not just individual positions, but general modes of play for classes of positions?
- naveen99 9y agoWould be nice if there was an open source attempt at an alpha go clone on a 9x9 board, so it could be run on commodity hardware and maybe trained in more reasonable time. Also would be interesting to see if human would still win on a 190x190 or some arbitrary size board against alphagozero trained appropriately.
- pedrohorta 9y agoI think that we cannot full understand the implications of that in our present/near future. The use of tecnhology will make the difference. There is a small book in amazon called something like Alpha Go Zero - 10 prophecies for the world. Interesting point of view of the use of it.
- SkyMarshal 9y agoOne thing interesting here is how close the best humans appear to be to the asymptotic theoretical max skill level.