14 ms·
I'm one of the authors of the bot, AMA
by noambrown 7y ago
I'm one of the authors of the bot, AMA
- eries 7y agoCan you share some about what strategies the bot prefers and how these compare with common professional human strategies?
- noambrown 7y agoWe talk about this a bit in the paper. Based on the feedback from the pros, the bot seems to "donk bet" (call and then bet on the next round) much more than human pros do. It also randomizes between multiple bet sizes, including very large bet sizes, while humans stick to just one or two sizes depending on the situation.
- MFLoon 7y agoWhen human players donk bet it's almost always a weak player employing an extremely exploitable strategy, whereas pros almost never do it because the metagame has evolved around the presumption that nobody ever donk bets. I'd love to see what the bot's balanced GTO donking strategy looks like.
- newfangle 7y agoDonk bets exist in the meta, ie when the turn is extremely good for your range but is horrible for your opponent. ( if you have a fd on the flop and it hits on the turn you can overbet the pot on the turn with your bluffs and foushes then just go all in on the river) if they have top pair its pretty hard to play against that
- MFLoon 7y agoOh, sure. I more meant flop donk bets; I guess it doesn't specify which street the donking was happening.
- russelldavis 7y agoThe same logic can apply to flop donk bets. Some flops favor the donking player's range more than their opponent.
- MFLoon 7y agoYea I'm not saying it's impossible to devise an unexploitable flop donking strategy. I think the reason thinking players generally don't is because of the complexity of adding significantly more branches early in the game tree - basically going from 3 (check-{fold,call,raise}) to 6 (those 3 plus donk-{fold,call,raise}).
- splonk 7y agoThe other issue is that increasing the number of branches also decreases the number of hands that go into each bucket, to the point where it might not be effective any more without being able to randomize the branch choice for specific threshold hands. Most pros I know just have a hard cutoff for each branch and don't worry too much if they're slightly out of balance, but smaller bucket sizes could magnify errors. If you have 31 combos for one action when you're supposed to have 30.5, then whatever, but if you have 6 when it should be 5.5, that could become a problem faster.
- splonk 7y agoIt's basically been true along every step of the the poker bot evolution (HU limit, HU NL, and 6-max NL) that the bots donk a lot more than the humans. 10 years ago you could find pros arguing that donking in any situation is always wrong. That's been shifting for years, but still not to the level that the bots do it. My personal belief is that the "no-donk" strategy is an adaptation by fallible human minds to reduce the branching on the decision tree to something tractable.
- patrickfreed 7y agoYour personal belief is likely correct. Balancing a donking range is incredibly difficult for humans and doing so perfectly likely yields only a very small EV bonus over just always checking. For humans it makes a whole lot of sense to reduce the branching in a case like that whereas for computers it doesn't really matter. Another good example is varying continuation betting sizes. A true GTO strategy would mix in a number of different sizings (and I'm sure the bots adapted to do this), but you only sacrifice a very tiny amount of EV by basically betting the same size every time. Doing the latter limits humans risk for making errors which is far more valuable than squeezing out .05bb/100 more by varying the sizes.
- icelancer 7y agoIf true in cash games, it is funny since it is a not uncommon strategy in high-level tournament play to control pot size.
- TheChosenZygote 7y agoIs there a way to see the EV the bot is calculating when it's deciding between checking and donk betting? When you place these spots in solvers, they actually advocate for a significant amount of donk betting on certain boards, but pros don't do it because the EV is marginal and it's better for pros to simplify their strategy so they make less mistakes. If you have a flop donk bet strategy, you also have to develop a corresponding turn and river strategy, which makes it extremely difficult.
- hoerzu 7y agoThe progress you have made in this research field is amazing. What do you think will be next step or where do you the the future of your research?
- noambrown 7y agoThanks! I think going beyond two-player/team zero-sum games is really important. This was a first step, but it's definitely not the last. I'm hoping to continue in this direction, and maybe start looking at interactions involving the potential for cooperation in addition to competition.
- umanwizard 7y agoCongrats! As soon as I saw the title I thought “I wonder if this is the project Noam works on...”
- noambrown 7y agoThanks!
- haburka 7y agoWhy did you optimize for using less cpus? Was it a happy accident or a goal?
- noambrown 7y agoA little bit of both. We didn't think we needed the extra computing power. And we really wanted to convey how cheap it is to make a superstrong poker AI with these latest algorithms.
- samfriedman 7y agoAre there any ethical considerations relating to the prospect of use of this bot for cheating in real-money games? Either from your internal team or after public replication?
- noambrown 7y agoWe're really focused on advancing the fundamental AI aspect. We're not here to kill poker. The popular poker sites have quite sophisticated anti-bot measures, but it's true that this is an arms race.
- mensetmanusman 7y agoThere are no ethical reasons why a game like poker must exist. In fact, poker gives a false sense of hope to the thousands of gambling addicts that enter casinos. It is a fun game, but there are an unlimited potential number of fun games..
- kzzzznot 7y ago1 ethical reason it ‘must exist’ is that it is a man-made game that some people enjoy without causing harm to themselves or others. Not quite sure what you’re suggesting, but “banning poker” is not going to solve the problem of gambling addiction.
- srkigo 7y agoI saw people who were going occasionally to casino without problems because nothing makes you lose and tilt so much as poker. I witnessed poker destroying families and people more than other games. There are people who don't like other casino games but lose heaps on poker and before they started poker their lives had more quality and meaning. I don't play other casino games but poker had a really bad influence on my life and the lives of people around me. Also, majority of money from poker comes from the players, not from the viewers and sponsors like in other sports, like football, baseball etc.
- spenczar5 7y agoThe article makes it sound like the AI is trained by evaluating results of decisions it makes on a per-hand basis. Is there any sense in which the AI learns about strategies that depend upon multiple hands? I’m thinking of bluffing/detecting bluffs and identifying recent patterns, which is something human poker players talk about.
- noambrown 7y agoThe bot handles each hand independently. How the players play in one hand does not affect how the bot plays in future hands at all. That said, it did train by playing against itself (before the experiment against the humans began).
- kyberias 7y agoInteresting. Does this mean that it cannot adjust to human players "switching gears"? Isn't this a huge leak?
- rjldev 7y agoIt’s not a leak, it just means it cant beat the opponent for the maximum it could by playing the exploitative counter strategy vs their tendencies. Instead it just plays gto which will win against any given non gto strategy, though not for as much as the exploitative counter strategy. Playing an exploitative strategy however leaves you open for exploitation and this goes back and forth until the players converge onto gto, assuming the players are (very) good. t. former poker pro
- DennisP 7y agoAre any papers available yet? Is the bot going for game-theory-optimal play, or trying to exploit weaknesses in other players?
- noambrown 7y agoThe paper is here: https://science.sciencemag.org/content/early/2019/07/10/science.aay2400 https://science.sciencemag.org/content/early/2019/07/10/scie... It's going for game-theory-optimal play. It doesn't adapt to its opponents' observed weaknesses. But I think it's cool to show that you don't need to adapt to opponent weaknesses to win at poker at the highest levels. You just need to not have any weaknesses yourself.
- timClicks 7y agoYeah that is a really interesting insight. I presume that also makes optimization much simpler. The rules are fixed. Opponents are not.
- gfody 7y ago> you don't need to adapt to opponent weaknesses to win at poker at the highest levels that may be true for limit poker, but in a no-limit tournament the best this bot could do is not lose. as the pressure increases with the blinds and the players are forced to bluff and call bluffs how does this bot avoid folding itself to death from a run of bad cards? I could see this bot doing well at cashing but I don't see how it could consistently place 1st the way the top human players do.
- MFLoon 7y agoI couldn't find it confirmed in the primary or secondary article, but I would bet the bot is just playing cash at a fixed stack depth rather than a tournament; just like in the wild, bots are much more of a problem in online cash than online tournaments. Dynamically adjusting strategies by stack depth, number of players, and pay jumps, would probably be several orders of magnitude more complex.
- ProAm 7y agoWhat's the name of the bot? Please say its Poker McPokerface
- pogopop77 7y agoVery impressive. If my understanding of how the AI works is correct, it is using a pre-computed strategy developed by playing trillions of hands, but it is not dynamically updating that during game play, nor building any kind of profiles of opponents. I wonder if by playing against it many times, human opponents could discern any tendencies they could exploit. Especially if the pre-computed strategy remains static.
- noambrown 7y agoWe played 10,000 hands of poker over the course of 12 days in the 5 humans + 1 AI experiment, and 5,000 hands per player in the 1 human + 5 AI's experiment. That's a good amount of time for a player to find a weakness in the system. There's no indication that any of the players found any weaknesses. In fact, the methods we use are designed from the ground up to minimize exploitability. That's a really important property to have for an AI system that is actually deployed in the real world.
- andr3w321 7y agoAny chance of the code being released or a cepheus style answer key being provided? http://poker.srv.ualberta.ca/strategy http://poker.srv.ualberta.ca/strategy
- noambrown 7y agoI don't think the poker world would be happy with us if we did that. Heads-up limit hold'em isn't really played professionally anymore, but six-player no-limit hold'em is very popular.
- Avamander 7y agoAs other commenters have said, I do too think you're delaying the inevitable but releasing now would mean you get credited with the first free solution.
- andr3w321 7y agoIt depends who you ask. I think it's inevitable that it's released one day. By not releasing you're just delaying it. All the top high stakes players already have solvers that they've spent a lot of money developing and studying privately. They would definitely be upset with you, but by releasing the code you are democratizing the information to all the midstakes pros who want to study but don't have the resources to pay developers and solve the game privately.
- floodyberry- 7y agoIf you're already using programs to help you, I don't see how you can be upset if someone else is cheating better than you are.
- ewhauser421 7y agoHave you thought about open sourcing the non-AI pieces? It would be great for other researchers so they wouldn’t have to build the poker pieces from scratch
- waynecochran 7y agoKnowing when to bluff often depends on the psychology of the opponent, but since it trained playing itself it doesn't seem that knowing when to bluff would be learned. Did it bluff very often?
- femto113 7y agoAt the highest levels of play psychological factors are pretty minimal. Before a showdown which cards you actually hold aren't particularly material, as the only information you convey is through your bids. This means if you predict that you're more likely to win a hand by bidding (and inducing a fold) than by calling and going to a showdown it makes mathematical sense to "bluff". I'm sure AIs have no trouble learning that fact.
- elcomet 7y agoThe issue is that you don't know exactly the probability of your opponent folding. This is psychology.
- NhanH 7y agoThe probability of the opponent folding doesn't matter. The goal of bluffing in modern games is so that optimal players are indifferent in their decision (no matter how they play, you can't lose money). And because this is a zero sum game, if you can't lose money then you win if the opponent makes mistake. You only need to know the probability of the opponent folding so that you can deviate from the theoretical optimal strategy to win even more money if they are a biased player
- waynecochran 7y agoI'll have to go back and watch Data playing poker on Star Trek NG -- what do sci fi writers think of this.
- noambrown 7y agoThe bot does bluff, and in fact it learns from self-play that bluffing is (sometimes) the optimal thing to do. At the end of the day, bluffing is simply betting when you have a weak hand. The bot learns from experience that when it bets with a weak hand, the opponent (another copy of itself) sometimes folds and it makes more money than if it hadn't bet. The bot doesn't view it as deceptive or dishonest. It just views it as the action that makes it the most money. Of course, a key part of bluffing is getting the probabilities right. You can't always bluff and you can't never bluff, because that would make you too predictable. But our self-play and search algorithms are designed to get those probabilities right.
- snarf21 7y agoHow do you think these same pros would do in a follow-up match? As described in the article, the bot put players off their game with much more varied betting and with donks. Do you think the margin would decrease as players are exposed to these strategies? Players face mental fatigue and have so over-learned their existing strategies that it takes time to adapt new strategies and even more time for those new strategies to become second-nature. It reminds me of sports in a way. Teams start running a new wrinkle of offense in the NFL like the wildcat and it takes a few seasons for teams to instinctively know how to play defense correctly against that option.
- noambrown 7y agoIn the paper we include a graph of performance over the course of the 10,000-hand 5 humans + 1 AI experiment that was played over 12 days. There's no indication that the bot's performance decreased over time (there is a temporary downward blip in the middle, but that's likely just variance). Based on discussions with pros, it sounds like they didn't find any weaknesses and they didn't seem to think they'd find any given more time.
- TheChosenZygote 7y agoI think it would be hard for the pros to find exploits against the bot, but they could definitely lose less. When using solvers, pros generally only input a couple of sizings for bets, and avoid 2x+ pot sizings, which from the video it seemed like the bot used at much higher frequencies than other pros.
- asdfman123 7y agoI'm not great at poker, but I did play a decent amount and I know a lot of my strategy involves probing for other people's weaknesses and shifting my strategy mid game to throw people off. I feel like a lot of trained ML models have a lot laughable weaknesses, but perhaps they've been trained on every game they're well prepared for any tomfoolery.
- 7y ago
- ewhauser421 7y agoNeal - super interesting stuff. Couple of questions: 1) What were the reasons for choosing 6-handed play (assuming logistical and costs)? It would be interesting to see how the bot’s strategy would differ in a full ring game. 2) Are there any plans to commercialize the bot as a tool for training human players?
- zone411 7y ago6-handed is a very common format online.
- noambrown 7y ago1) The goal was to show convincingly that we could handle multi-player poker. The exact number of players was kind of arbitrary. We chose six-player because that's the most common/popular format. Considering training the 6-player bot would cost less than $150 on a cloud computing service, I think it's safe to say these techniques would all work fine in other formats. 2) I'm quite happy working on fundamental AI research and plan to continue in that direction.
- baq 7y agoWhen do you solve bridge? :)
- canistel 7y agoIt is in a way disappointing that this question gets so little attention, and yet, it might be the most significant. If a bot can false-card - if it can discern the strategy that the opponents have in mind, and deliberately mislead them to its own advantage - we have a real world AI. However, skills of computer bridge programs remain at club level standards.
- clavalle 7y agoWhat table information does the bot take into account? Position? Other player's stack size? >Regardless of which hand Pluribus is actually holding, it will first calculate how it would act with every possible hand . Is this information used to form an idea of what other players might be holding based on how the other player acts and how closely that action matches Pluribus's 'what if' action?
- throwamay1241 7y agoNo, it's to mask actions. If you bet big with monsters and check with air 100% of the time, you opponent knows when to fold and bet. iirc, the frequency of bets in that spot is roughly equivalent to the frequency of times you're definitely in front of your opponent in that particular spot, but not always with the hands that are beating your opponent. The concept is called Game Theory Optimal (GTO) and it's pretty popular in higher stakes games.
- bluetwo 7y agoFirst of all, I laughed at the 20-second average per game in self-play, since I ran into the same thing and have been trying to speed up the algorithm but haven't been able to get it faster (without throwing more hardware at it). Second, I haven't read everything, but I believe you are playing a cash-game and not tournament-style. Is that correct? If that is the case, any chance you will be doing a tourney-style version? [For those who don't play, in cash, a dollar is a dollar. In Tourney play, the top 2 or 3 players get paid out, so all dollars are not equal, as your strategy changes when you have only a few chips left (avoid risky bets that would knock you out) or when you are chip leader (take risky bets when they are cheap to push around your opponents).] Also, curious how much poker you folks play in the lab for "research".
- wallawe 7y ago> In Tourney play, the top 2 or 3 players get paid out Or top 2 or 3 thousand... depends on the tournament but it's usually the top 15% ish.
- bluetwo 7y agoTrue, I am thinking "sit and go" tournament where you would have 6 players like in this research.
- icelancer 7y agoIs there much to do here? ICM bots have this space covered pretty effectively.
- Phillipharryt 7y agoBut ICM is only a model that helps you evaluate information in the tournament, players will use it often to cap their bets or as a tipping point on a call, but I've never seen it used as a complete basis of play.
- noambrown 7y agoWe're doing cash games in this experiment. At the end of the day, this is about advancing AI, not about making a poker bot. Going from two-player to multi-player has important implications for AI beyond just poker. I don't think the same is true for cash game vs tournament. There's a cash game almost every night at the FBNY office! I don't usually play though -- I'm not nearly as good as the bot.
- throwamay1241 7y agoWho were the pros? Are they credible endbosses? Seth Davies works at RIO which deserves respect but I've never heard of the others except Chris Ferguson who I doubt is a very good player by todays standards (or human being, for that matter), but I've never heard of the others when I do know the likes of LLinusLove (iirc, the king of 6max), Polk and Phil Ganford. Is 10,000 hands really considered a good enough sample? Most people consider 100k hands w/ a 4bb winrate to be an acceptable other math aside. However, as your opponent and yourself play with equal skill, variance increases to the point where regs refuse to sit each other.
- ayemeng 7y agoJimmy Chou, Jason Les, Dong Kim are affiliated with Doug Polk. It is an interesting point that these are pros but their specialities are either tournament or heads up. The current 6 max pros are LLinusLove, Otb_RedBaron, TrueTeller.
- noambrown 7y agoLLinusLove was one of the players. Chris Ferguson was in one of the 5 AI's + 1 Human experiment but not the 5 Humans + 1 AI experiment. We used AIVAT to reduce variance, which reduces the number of samples we need by roughly a factor of 10: https://poker.cs.ualberta.ca/publications/aaai18-burch-aivat.pdf https://poker.cs.ualberta.ca/publications/aaai18-burch-aivat...
- deleted 7y ago[deleted]
- splonk 7y agoThey're credible enough. I'd like the sample sizes to be bigger as well but they're enough to verify that even if the bot got lucky over the sample size, it's close enough that it doesn't really matter. Add a bit more compute, optimize some algorithms a little, and you'd make up the difference. The real point is that they have a technique that scales to 6-max, and whether it's 97% or 99% is kind of immaterial in the grand scheme of things. FWIW, they did some variance reduction techniques that dramatically reduce the number of hands needed to be confident in your results, so the number of hands may be bigger than you think. e.g. the results of 10k HU hands have much higher variance than the results of 10k HU hands where everyone just collects their EV once they're all in.
- anbop 7y agoIs this publicly available? How can I use it?
- n3k5 7y agoWhat took you so long? I mean not the Pluribus team specifically, but Poker AI researchers in general. The desire to master this sort of game has inspired the development of entire branches of mathematics. Computers are better at maths than humans. They're less prone to hazardous cognitive biases (gambler's fallacy etc.) and can put on an excellent poker face. As a layperson who's rather ignorant about both no-limit Texas hold 'em and applicable AI techniques, my intuition would tell me that super-human hold 'em should have been achieved before super-human Go. Apparently your software requires way less CPU power than AlphaGo/AlphaZero, which seems to support my hypothesis. What am I missing? Bonus questions in case you have the time and inclination to oblige: What does this mean for people who like to play on-line Poker for real money? Could you recommend some literature (white papers/books/lecture series/whatever) to someone interested in writing an AI (running on potato-grade hardware) for a niche "draft and pass" card game (e.g. Sushi Go!) as a recreational programming exercise?
- noambrown 7y agoI think it took the community a while to come up with the right algorithms. So much of early AI research was focused on beating humans at chess and later Go. But those techniques don't directly carry over to an imperfect-information game like poker. The challenge of hidden information was kind of neglected by the AI community. This line of research really has its origins in the game theory community actually (which is why the notation is completely different from reinforcement learning). Fortunately, these techniques now work really really well for poker. It's now quite inexpensive to make a superhuman poker bot.
- smallgovt 7y agoThis is super interesting! What steps would you recommend a professional poker player take in order to use AI to improve his/her personal poker skills?
- mfwebser 7y agoWhat sort of defense applications could this sort of technology be used for? The last line of the Facebook blog post sparked curiosity.
- dillonmckay 7y agoWhen will you test it with 10 total players in a game?
- noambrown 7y agoThe number of players is kind of arbitrary given the techniques we're using. We chose 6 because that's the most popular/common format for poker. I don't think there's any scientific value in also doing 10.
- dillonmckay 7y agoI am obviously a human, not a bot, but in my experience playing poker, it seems much more likely for me to be successful, personally in a 6 player game, whereas a 10 player game, I never seem to do well.
- doctorpangloss 7y agoDo you want to do a Hearthstone / CCG bot? I have an engine and testers for you.
- hmate9 7y agoWhat was the most challenging part about implementing this?
- noambrown 7y agoHonestly, probably debugging. Training this thing is very cheap, but the variance in poker is huge (even with the best variance-reduction techniques) so it takes a very long time to tell whether one version is better than another version (or better than a human).
- ryandrake 7y agoInteresting that the conventional wisdom of never open limping emerged as confirmed through self-play. What other general poker “best practices” were either confirmed or upended through this research?
- vagab0nd 7y agoVery interesting results. From the paper it sounds like the algorithms you used are very similar to Libratus (pre-solved blueprint + subgame solving). What change made it so that the computation requirement is much lower now?
- noambrown 7y agoThere were several improvements but the most important was the depth-limited search. Libratus would always search to the end of the game. But that's not necessarily feasible in a game as complex as six-player poker. With these new algorithms, we don't need to go to the end of the game. Instead, we can stop at some arbitrary depth limit (as is done in chess and Go AI's). That drastically reduces the amount of compute needed.
- tc 7y agoCould you perhaps speak to some of the engineering details that the paper glosses over. E.g.: - Are the action and information abstraction procedures hand-engineered or learned in some manner? - How does it decide how many bets to consider in a particular situation? - Is there anything interesting going on with how the strategy is compressed in memory? - How do you decide in the first betting round if a bet is far enough off-tree that online search is needed? - When searching beyond leaf nodes, how did you choose how far to bias the strategies toward calling, raising, and folding? - After it calculates how it would act with every possible hand, how does it use that to balance its strategy while taking into account the hand it is actually holding? - In general, how much do these kind of engineering details and hyperparameters matter to your results and to the efficiency of training? How much time did you spend on this? Roughly how many lines of code are important for making this work? - Why does this training method work so well on CPUs vs GPUs? Do you think there are any lessons here that might improve training efficiency for 2-player perfect-information systems such as AlphaZero?
- noambrown 7y agoWe tried to make the paper as accessible as possible. A lot of these questions are covered in the supplementary material (along with pseudocode). - Are the action and information abstraction procedures hand-engineered or learned in some manner? - How does it decide how many bets to consider in a particular situation? The information abstraction is determined by k-means clustering on certain features. There wasn't much thought put into the action abstraction because it turns out the exact sizes you use don't matter that much as long as the bot has enough options to choose from. We basically just did 0.25x pot, 0.5x pot, 1x pot, etc. The number of sizes varied depending on the situation. - Is there anything interesting going on with how the strategy is compressed in memory? Nope. - How do you decide in the first betting round if a bet is far enough off-tree that online search is needed? We set a threshold at $100. - When searching beyond leaf nodes, how did you choose how far to bias the strategies toward calling, raising, and folding? In each case, we multiplied by the biased action's probability by a factor of 5 and renormalized. In theory it doesn't really matter what the factor is. - After it calculates how it would act with every possible hand, how does it use that to balance its strategy while taking into account the hand it is actually holding? This comes out naturally from our use of Linear Counterfactual Regret Minimization in the search space. It's covered in more detail in the supplementary material - In general, how much do these kind of engineering details and hyperparameters matter to your results and to the efficiency of training? How much time did you spend on this? Roughly how many lines of code are important for making this work? I think it's all pretty robust to the choice of parameters, but we didn't do extensive testing to see. While these bots are quite easy to train, the variance is so high in poker that getting meaningful experimental results is relatively quite computationally expensive. - Why does this training method work so well on CPUs vs GPUs? Do you think there are any lessons here that might improve training efficiency for 2-player perfect-information systems such as AlphaZero? I think the key is that the search algorithm is picking up so much of the slack that we don't really need to train an amazing precomputed strategy. If we weren't using search, it would probably be infeasible to generate a strong 6-player poker AI. Search was also critical for previous AI benchmark victories like chess and Go.
- RivieraKid 7y agoCan you share more details about the abstraction? The paper is kind of vague on it. How does it decide if it should use 1 or 14 bet values? Is it a perfect recall abstraction? How many information sets are there?
- noambrown 7y agoWe give more details on this in the supplementary material.
- w_s_l 7y agowill you release the source code?
- noambrown 7y agoOur goal is to make the research as accessible as possible to the AI community, so we include descriptions of the algorithms and pseudocode in the supplementary material. However, in part due to the potential negative impact this code could have on online poker, we're not releasing the code itself.
- downandout 7y agoWhile you are not releasing the code to the general public, some people who worked on it obviously have access to it and someone will likely use it in the wild. The potential profits are astronomical - Rob Reitzen solved limit hold 'em and made what is rumored to be over $100 million hiring women to play online poker using his system from his house in Beverly Hills [1]. Did you guys set any rules as to whether or not members of the team that worked on this are allowed to use it? [1] https://www.cigaraficionado.com/index.php/article/robotic-poker-19354 https://www.cigaraficionado.com/index.php/article/robotic-po...
- yalogin 7y agoFor someone not in the AI field, can you explain why AI is needed and an elaborate code with conditional blocks is not enough? Where does AI fit in with a poker game.
- b_tterc_p 7y agoConditional blocks would work, but it would be an impossibly detailed and granular tree to setup. The AI component simply helps you arrive at the decisions to create the complex tree.
- b_tterc_p 7y agoThe bot does not seem to consider previous hands in its decisions. That is to say, it does not consider who it is playing against. Should this affect how we perceive the bot as “strategic” or not? Bots that play purely mathematically optimally on expected value aren’t effective or interesting. But it feels like this is playing on just a much higher order expected value. It feels like a more down to earth version of the sci fi super human running impossible differential equations to predict exactly what you will do given knowledge that he knows what you know what he knows... etc. ad Infinitum. But since it doesn’t actually consider the person it’s predicting, it may simply be a really really good approximation of the game theoretic dominant strategy. At what complexity of game and hidden information should we feel like the bot can’t win by running a lookup table?
- noambrown 7y agoThe bot bluffs, and understands that when its opponent bets it might be a bluff. I would consider that to be strategic behavior. The fact that its strategy is determined by a mathematical process doesn't change that in my opinion.
- slg 7y ago"Strategic" is probably the wrong word, but I think there is a valid question here regarding the approach the AI is taking. One of the key things for a good poker player is having the ability to adapt and adjust their strategy depending on how others at the table are playing. Sometimes you can have the exact same cards in the exact same position and in one game it is smart to fold and in another game it is smart to raise. From the description in the article, it doesn't appear that this AI takes those ebbs and flows into consideration. Instead it seems to play "purely mathematically optimally on expected value" that was honed through trillions of simulations. There is a cliche about how poker is about playing your opponents and not the cards. Is this AI is only focusing on its cards and ignoring its opponents?
- noambrown 7y agoThe AI doesn't adapt to the opponents, and that's still an interesting challenge for AI research. That said, at the end of the day, it was making quite a bit of money playing against elite human pros. I think that suggests the cliche is, at least in part, wrong.
- isaacg 7y agoIn your Science paper, you mention playing 1H-5AI against 2 human players: Chris Ferguson and Darren Elias. In your blog post you also mention playing 1H-5AI against Linus Loelinger, who was within standard error of even money. Why did Linus not make it into the Science paper?
- noambrown 7y agoThat took place after the final version of the Science paper was submitted. It would have been nice to include but it takes a while to do those experiments and we didn't feel it was worth delaying the publication process for it.
- splonk 7y agoI haven't finished digging through the paper and the supplement yet, but I'm curious about how many hands were multiway to the flop (and whether the percentages differ significantly between 1H/5AI and 5H/1AI). I'd guess that it's a pretty small fraction of the total hands, and I'm wondering what the performance is like in those particular cases.
- noambrown 7y agoI don't have the exact percentages but I think it's less than 10%. It's not really possible to measure the bot's performance just in specific situations, but my feeling is the bot performs relatively well in these situations. Multi-way flops were basically impossible to do in a reasonable amount of time for past AI's. Our new search techniques make these situations feasible to figure out in seconds.
- splonk 7y agoCheers, thanks. One of the reasons I asked about 1H/5AI vs 5H/1AI is that historically the new bots for a given form of poker have played a bit wider than the accepted wisdom of the time, so I was curious if there were relatively more multiway pots with 5AI than with 5H.
- noambrown 7y agoThe pros described the bot's preflop strategy as very sensible, so I think it's unlikely there were more multiway pots with 5 AI's.
- tasubotadas 7y agoCongrants on the bot! I have a few basic questions. I would like to implement my own generic game bot (discrete states). Are there any universal approaches? Is MCMC sampling good enough to start? My initial idea was to do importance sampling on some utility/score function. Also, I am looking into poker game solvers - what would be a good place to start? What's the simplest algorithm? Thanks
- ikeboy 7y agoAny plans to make money using this in online games?
- noambrown 7y agoNo, I don't have any plans to do that. This is really about advancing fundamental AI research.
- o_p 7y agoDoes it beat poker by reaching Nash eq (where you cant make profit and no one can profit from you) or exploits opponents weakness to seek profit ?
- noambrown 7y agoIt doesn't exploit its opponents' weaknesses. Its focus was on not having any weaknesses that its opponents could exploit. However, the algorithms are not guaranteed to converge to a Nash equilibrium in this setting because it's not a two-player zero-sum game (and in either case, it's not clear that playing a Nash equilibrium would provide much benefit in this setting).
- Jach 7y agoWas Judea Pearl's work relevant for the counterfactual regret minimization, or is there some other basis? I've added CR to the list of things to look into later but skimming the paper it was exciting to think advances are being made using causal theory...
- noambrown 7y agoThe CFR algorithm is actually somewhat similar to Q-learning, but the connection is difficult to see because the algorithms came out of different communities, so the notation is all different.
- abstract7 7y agoHow many games did the bot beat the same 5 players? And how many games were played?
- noambrown 7y agoWe played 10,000 hands of poker in the 5 humans + 1 AI experiment. The number of hands won isn't a useful metric in poker. If you win only 10% of your hands and make $1,000 on those hands, while losing only $1 on the other 90% of hands, then you're a winning player. The bot won at a rate of 4.8 bb/100 ($4.8 per hand if the blinds are $50/$100). This is considered a large win rate by professionals.
- badfrog 7y ago> This is considered a large win rate by professionals. It depends on context. 4.8bb/100 is quite good for high-level online play, but wouldn't be enough to make a living at live poker. The biggest game that runs on a regular basis in most areas is $5/10. At ~33 hands per hour, that's 1.6bb or $16 an hour. And I'd assume there was no rake in your game? That would take a big chunk out of the rate.
- GregoryPerry 7y agoFor one player... $16/hr X (VM|microservice thread) could become astronomical profit.
- badfrog 7y agoThat's why it's a good rate online where you can play multiple tables. Unlimited VMs don't help you in a live casino.
- stevespang 7y agoNoam, I know you can't discuss the all the specific proprietary details of the algorithm, I just graduated Comp. Sci UT Austin and I'm totally captivated by any and all of the more technical details of the algorithm . . .
- grizzles 7y agoAny chance you could put Libratus / Pluribus online for people like me to try to beat it?
- noambrown 7y agoUnfortunately we don't have any plans to do that currently.
- codefiddler 7y agoAre all the hands posted online somewhere for analysis. I would be very interested!
- logical42 7y agoAny chance you’ll consider releasing the hand history of the session?
- skater 7y agothey're in the extra data section of the science mag article. formatting is terrible for importing into hand history viewers, so i'm trying to get a friend to re-format
- meuk 7y agoCongratulations on the win! Can you recommend any papers, blog(post)s, or books for the interested layman? (I am currently scanning though the facebook post, which is great, but personally I am looking for something more technical).
- cambaceres 7y agoWhat are the names of the poker pros the AI beat?
- cambaceres 7y agoAre all the hands available to the public?
- noambrown 7y agoThe hand logs from the 5 humans + 1 AI experiment are included in the supplementary material of the Science paper.
- cambaceres 7y agoThey are missing the stack sizes of the players. Would love to have logs that include that info!
- estomagordo 7y agoDo you expect the human players to play at the best of their ability when they're not playing for actual real money?
- noambrown 7y agoThere was real money at stake in this experiment. The pros were guaranteed $0.40 per hand just for participating, but that could increase to $1.60 per hand depending on how well they did. To answer your question, no, I don't think human players would play at their best when not playing for actual money.
- estomagordo 7y agoSorry, I meant for way less than they typically play.
- confidantlake 7y agoWhy would you choose Chris Ferguson to participate? Don't you know his terrible history?
- XalvinX 7y agomaybe the fact that he hold a computer science degree figures into his involvement on some level? regarding his terrible history, was that ever really proven or admitted to? and if it was, why is he not in jail (or dead, for that matter...)?? my not-very-informed take on that situation was more like he was in the wrong place at the wrong time, black friday came along, and the money dried up really fast making for some unpaid debts for the company... perhaps i don't know the full story but still, he didn't seem to ever be charged with any wrongdoing, which seems odd to say the least
- kapurs151 7y agoI'm very late to this post, so not sure if you're still around. What are your thoughts on a poker tournament for bots? Do you think it could turn into a successful product? I've always wanted to build an online poker/chess game that was designed from the ground up for bots (everything being accessible through an API), but have always worried that someone with more computational resources or the best bot would win consistently. Is it an idea you've thought about?
- darse 7y agoA hearty congratulations, Noam, on finishing another chapter of the story i opened in the early 1990s... Another person asked "What took you so long?", and i had the same question. :) I really thought this milestone would be achieved fairly soon after i left the field in 2007. However, breakthroughs require a researcher with the right amount of reflectiveness, insight, and determination. Well done.