7 ms·

How I built a poker-bot

77 points by run4yourlives 19y ago

Flemlord 19y ago
I was involved in a fairly elaborate pokerbot effort. Believe me, the mechanics of playing the game are the easy part. The hard part is writing the poker logic. Check this out if you want an intro to the "basic" concepts: Lokibot: http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html The bot described in that paper lost about 8c a hand at the $1 tables. After extensive modifications, we got it to the point where it was losing about 2c a hand. This when Bush made poker illegal in the US, and we killed the project.
goofygrin 19y ago
awesome article. I wonder mattmaroon's thoughts on this.
nanexcool 19y ago
Can't wait to read the rest of the series. Promises to have a bit of everything: OCR, screen-scraping, rules engine, WinAPI hooks, DLL injection, and that's just part one!
hollywoodcole 19y ago
Really Awesome, now I will apply to WoW.
- henning 19y ago
  Convincing others that your goldfarming bot is actually human seems like a highly challenging variant of the Turing test.
  - Hexstream 19y ago
    Not sure it's so hard, first you introduce some randomness and inefficiencies in the gathering process so that it's more credible that it's a fallible human that's doing it. And as a second crucial step, you hook groups of bots into a "monitoring computer", at which an actual human sits and responds "No, I'm not a bot lol." when the supposedly fatal question arrives from an admin. You can even outsource this to the third world.
    - henning 19y ago
      You've never played WoW, have you? "Randomness" is a dead giveaway of a bot. Other players will grief you, report you, and you will be banned quickly. Just the computer vision aspect of it alone would be amazingly hard. Efficient goldfarming (e.g., with a frost mage that can take on 15-30 monsters at once) involves carefully coordinating a complicated series of actions where timing is critical and reacting quickly and intelligently when things go wrong is the difference between living and dying. It involves making complicated judgments about relative positioning, reasoning about surroundings, cause and effect, and other things that require human-level intelligence. If you can make a bot that can seek out and kill monsters in a non-trivial manner without dying all the time there's no reason you couldn't program a robot to replace a wide variety of menial real-world jobs. The consequences would be tremendous and using it to make $20/day off the Internet would be retarded because you could license the technology for billions.
      - dangoldin 19y ago
        You are taking this a bit too seriously, I think. Although I do agree that with that technology you are better off licensing it.
      - Harkins 19y ago
        The point of bots is that they don't have to be efficient, just marginally effective and able to run with limited or no supervision. The solution to inefficient bots is easy: create more instances.
        TrevorJ 19y ago
        but therein lies the rub: it's easy for the administrators of the game to realize that no real person in the right mind would sit in the same location for 3 days and go after the same creatures over and over.
        Hexstream 19y ago
        Make the bot change locations and target creatures from time to time?... The point is that whatever heuristics are discovered to pinpoint bots won't be effective very long because of arms-race dynamics.
  - TrevorJ 19y ago
    Not if you use a hunter class character. They have the reputation of being a bit slow :-)
ssharp 19y ago
His rules engine doesn't appear to trend other players at all. There are good tools out there that gather data on players and with enough data, you can pinpoint a player fairly well. It also shows his bot playing no-limit tables which seems like a horrible idea for a bot. There is a lot more to take into consideration in no-limit poker and I'm not sure a bot that only takes 3 or 4 things into consideration for every move could ever win at worthwhile stakes. If your bot can play 14 limit tables at a time with even the slightest % edge, you're going to make money. However, if you added more advanced player profiling algorithms into your bot, you're edge would likely increase and allow you to set it up at higher stakes.
- mattmaroon 19y ago
  I don't think I agree with you on no-limit vs. limit. There's a pretty simple strategy you can use to be a marginal winner at low stakes no-limit (or especially no-limit sit and goes). It' called short stacking, and essentially involves buying in for the minimum and playing mainly all-in or fold (with some preflop calling). You have to leave the table as soon as you get past 1.5x the minimum buy-in though. It was mainly done on Party Poker, which was the only site dumb enough to let you cash out and then return to the table for the min again. Any other site either made you bring back what you left with or wait an hour. On a site with enough tables in action, though, even that would not be a limitation. A bot could do fairly well at it. Also I can tell you from first hand experience that breaking even at low stakes NL sit and goes can be done with an AI consisting of nothing but if/then statments. I did this with WinHoldem back in the days before it was banned from everywhere. I wasn't able to replicate that success with limit. The main problem with WinHoldem was that it didn't join or leave tables on its own (also, it would often bug out when the tournament got heads up), which meant that the time spent managing it made it less profitable than just playing myself. That's why we tried to build a bot ourselves. Maybe I'll blog about that some time.
  - sanswork 19y ago
    The reason limit is easier is just that there is less variables to account for much like it's easier to write a bot to play poker with fewer cards. While I will agree sit and gos are all strategy that strategy is dependent on putting your opponent on reasonable ranges which a bot like this won't do. Bots and winning bots are possible(stick most casual people up against the UofA bots for example), but I find it hard to believe this one is one of them.
gruseom 19y ago
A few years ago, I wrote a program that would screen-scrape to read all cards showing and then do a Monte Carlo simulation to randomly complete the hand and compute what % of times my cards would win (along with pot odds). It took a second or so to do the simulation, so I could use it to make decisions in real time. It was a fun project, but I soon stopped using it. Your odds against random cards are not at all your actual odds, because most players fold weaker cards. What matters most is who you're playing against. Anyway, the article, although it describes well the techniques for programmatically interacting with a poker client, is majorly misleading because this is not the hard part of making a bot. The hard part is figuring out whether, and how much, to bet. He alludes to this as "the million dollar question", but then omits to even mention it under "What skills will I need to write a bot?" Perhaps he'll address this in later articles? If you look at the last screen shot, it's clear that he built a UI for an end-user to specify the actual logic for poker play. ("Call if it has not been raised, I'm later than 7th", etc.) His "bot" doesn't actually make decisions. It collects rules from an end-user, calls those to decide what to do, and passes the result on to the poker client. In other words, the hard part of the job - writing a program to play poker - has been passed to somebody else. Maybe that somebody else is the author, creating the algorithm through his UI... but then why not just code it? There are serious researchers who have made credible bots (he links to them), bots which would (and, for all we know, do) make money on poker sites. But when I looked at it, the literature was not nearly detailed enough to code up their algorithms. (Also, they were working with simplified versions of poker, since full no-limit holdem is a hard nut to crack). What they didn't use were the "extensible rules-based systems" which this author claims "are a lot more powerful than you think". Count me skeptical. Good players eat such systems for snacks. Maybe they make money in low-stakes games against bad players, but I'd like to see evidence. In short: never mind the dll injection, show us your bankroll over 100,000 hands!
- mattmaroon 19y ago
  "Count me skeptical. Good players eat such systems for snacks." You're thinking about it wrong. As he mentions, your bot doesn't play against good players. It plays .5/1 or 1/2 nl and makes a shit ton of money if it just breaks even. There aren't good players at those levels, because anyone who was good would win and quickly move up. You only need all of your if/then statements to add up to being better than the worst 5% of poker players, and that's not that hard to do. I know only because I've done it. Thats why extensible rules based systems are better than you think. They can get you to that with a good amount of work. Especially in short stack nl games. They're clearly not going to win at a $5/$10 nl table or a $215 SNG. But they don't need to to rake in millions per year.
xirium 19y ago
> none of the online poker sites have really stepped up to the plate and either a) made bots legal (similar to the way that they're legal on Internet Chess Club) or b) put effective prevention measures in place. That's a tough choice for a casino. Allow automation and you create a ghetto which discourages casual users. Disallow automation and you encourage an escalating "arms race". Regardless, the techniques discussed could be detected using anti-virus techniques. I'm surprised that native clients don't perform integrity checks or is this the state of play in the "arms race"? It would be fairly simple to disable integrity checks using methods to circumvent software DRM. Then it would be a matter of implementing distributed state to determine and ignore circumvented clients.
- ericb 19y ago
  Interestingly, the casino's financial interest, since it makes its money only on the rake, is to have as many players or bots as possible. In fact, from their perspective, an infinity of bots battling each other should be just fine--bots scale better than humans. From Ultimate bet's perspective, bots are great: It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until UltimateBet is rich.
  - xirium 19y ago
    I considered this possibility but a casino which is too open to bots could become marginalised. Real players would be scalped too easily and bot writers would be playing a zero-sum game. Both factors would reduce casino turnover and profit. So, it is in a casino's interest to leave the whole issue unresolved and with a level of tolerance which is in-step with competitors.
    - ericb 19y ago
      I agree to an extent, however, I would be most happy owning a bot battleground, personally. It would dispense with all that time wasted on "thinking" and reduce the speed to a processor/internet lag dependent game. Bots are no more zero sum than people, they are just faster. I love the idea of a money making machine that scales with processor cores/mhz.
ctkrohn 19y ago
It's interesting, but to be honest, if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures. The potential winnings are much higher, it's perfectly legal, and due to the depth of the markets such programs are capable of making much more money than poker bots. It's also doable, as funds like Renaissance Technologies (http://en.wikipedia.org/wiki/Renaissance_Technologies http://en.wikipedia.org/wiki/Renaissance_Technologies) and D. E. Shaw have demonstrated.
- hugh 19y ago
  if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures I'm not so sure about that. I think the main reason is: there's a lot more "stupid money" at the poker table than there is in the market. The bot as mentioned doesn't seem to do anything particularly fancy -- it just looks at the cards on the table and figures out your odds of winning based on that. This is information which is easily extracted mathematically, but which most players don't have direct access to. Good players will have a pretty good idea of these probabilities, whereas lousy players won't. If you can find yourself on a table full of lousy players, you're at a huge advantage. In the market, on the other hand, there's only one poker table, and all the best players in the world are already sitting at it. Any time statistical arbitrage opportunities arise in prices, you can bet that D.E. Shaw and Rennaisance Technologies have already noticed it and moved a few billion dollars that way. If you want to win, you don't just have to be smarter than six guys in their basements doing calculations in their heads, you have to be smarter than thousands of overpaid quants with supercomputers.
  - ctkrohn 19y ago
    Fair enough. You're probably right. I don't play online poker (and am a poor casual player) so I don't know what the level of skill is on those websites. I was just making an assumption based on how difficult I imagined it would be for a computer to play very good poker.

77 points by run4yourlives 19y ago

I was involved in a fairly elaborate pokerbot effort. Believe me, the mechanics of playing the game are the easy part. The hard part is writing the poker logic. Check this out if you want an intro to the "basic" concepts: Lokibot: http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html The bot described in that paper lost about 8c a hand at the $1 tables. After extensive modifications, we got it to the point where it was losing about 2c a hand. This when Bush made poker illegal in the US, and we killed the project.

goofygrin 19y ago

awesome article. I wonder mattmaroon's thoughts on this.

nanexcool 19y ago

Can't wait to read the rest of the series. Promises to have a bit of everything: OCR, screen-scraping, rules engine, WinAPI hooks, DLL injection, and that's just part one!

hollywoodcole 19y ago

Really Awesome, now I will apply to WoW.

henning 19y ago
Convincing others that your goldfarming bot is actually human seems like a highly challenging variant of the Turing test.
- Hexstream 19y ago
  Not sure it's so hard, first you introduce some randomness and inefficiencies in the gathering process so that it's more credible that it's a fallible human that's doing it. And as a second crucial step, you hook groups of bots into a "monitoring computer", at which an actual human sits and responds "No, I'm not a bot lol." when the supposedly fatal question arrives from an admin. You can even outsource this to the third world.
  - henning 19y ago
    You've never played WoW, have you? "Randomness" is a dead giveaway of a bot. Other players will grief you, report you, and you will be banned quickly. Just the computer vision aspect of it alone would be amazingly hard. Efficient goldfarming (e.g., with a frost mage that can take on 15-30 monsters at once) involves carefully coordinating a complicated series of actions where timing is critical and reacting quickly and intelligently when things go wrong is the difference between living and dying. It involves making complicated judgments about relative positioning, reasoning about surroundings, cause and effect, and other things that require human-level intelligence. If you can make a bot that can seek out and kill monsters in a non-trivial manner without dying all the time there's no reason you couldn't program a robot to replace a wide variety of menial real-world jobs. The consequences would be tremendous and using it to make $20/day off the Internet would be retarded because you could license the technology for billions.
    - dangoldin 19y ago
      You are taking this a bit too seriously, I think. Although I do agree that with that technology you are better off licensing it.
    - Harkins 19y ago
      The point of bots is that they don't have to be efficient, just marginally effective and able to run with limited or no supervision. The solution to inefficient bots is easy: create more instances.
      - TrevorJ 19y ago
        but therein lies the rub: it's easy for the administrators of the game to realize that no real person in the right mind would sit in the same location for 3 days and go after the same creatures over and over.
        Hexstream 19y ago
        Make the bot change locations and target creatures from time to time?... The point is that whatever heuristics are discovered to pinpoint bots won't be effective very long because of arms-race dynamics.
- TrevorJ 19y ago
  Not if you use a hunter class character. They have the reputation of being a bit slow :-)

ssharp 19y ago

His rules engine doesn't appear to trend other players at all. There are good tools out there that gather data on players and with enough data, you can pinpoint a player fairly well. It also shows his bot playing no-limit tables which seems like a horrible idea for a bot. There is a lot more to take into consideration in no-limit poker and I'm not sure a bot that only takes 3 or 4 things into consideration for every move could ever win at worthwhile stakes. If your bot can play 14 limit tables at a time with even the slightest % edge, you're going to make money. However, if you added more advanced player profiling algorithms into your bot, you're edge would likely increase and allow you to set it up at higher stakes.

mattmaroon 19y ago
I don't think I agree with you on no-limit vs. limit. There's a pretty simple strategy you can use to be a marginal winner at low stakes no-limit (or especially no-limit sit and goes). It' called short stacking, and essentially involves buying in for the minimum and playing mainly all-in or fold (with some preflop calling). You have to leave the table as soon as you get past 1.5x the minimum buy-in though. It was mainly done on Party Poker, which was the only site dumb enough to let you cash out and then return to the table for the min again. Any other site either made you bring back what you left with or wait an hour. On a site with enough tables in action, though, even that would not be a limitation. A bot could do fairly well at it. Also I can tell you from first hand experience that breaking even at low stakes NL sit and goes can be done with an AI consisting of nothing but if/then statments. I did this with WinHoldem back in the days before it was banned from everywhere. I wasn't able to replicate that success with limit. The main problem with WinHoldem was that it didn't join or leave tables on its own (also, it would often bug out when the tournament got heads up), which meant that the time spent managing it made it less profitable than just playing myself. That's why we tried to build a bot ourselves. Maybe I'll blog about that some time.
- sanswork 19y ago
  The reason limit is easier is just that there is less variables to account for much like it's easier to write a bot to play poker with fewer cards. While I will agree sit and gos are all strategy that strategy is dependent on putting your opponent on reasonable ranges which a bot like this won't do. Bots and winning bots are possible(stick most casual people up against the UofA bots for example), but I find it hard to believe this one is one of them.

gruseom 19y ago

A few years ago, I wrote a program that would screen-scrape to read all cards showing and then do a Monte Carlo simulation to randomly complete the hand and compute what % of times my cards would win (along with pot odds). It took a second or so to do the simulation, so I could use it to make decisions in real time. It was a fun project, but I soon stopped using it. Your odds against random cards are not at all your actual odds, because most players fold weaker cards. What matters most is who you're playing against. Anyway, the article, although it describes well the techniques for programmatically interacting with a poker client, is majorly misleading because this is not the hard part of making a bot. The hard part is figuring out whether, and how much, to bet. He alludes to this as "the million dollar question", but then omits to even mention it under "What skills will I need to write a bot?" Perhaps he'll address this in later articles? If you look at the last screen shot, it's clear that he built a UI for an end-user to specify the actual logic for poker play. ("Call if it has not been raised, I'm later than 7th", etc.) His "bot" doesn't actually make decisions. It collects rules from an end-user, calls those to decide what to do, and passes the result on to the poker client. In other words, the hard part of the job - writing a program to play poker - has been passed to somebody else. Maybe that somebody else is the author, creating the algorithm through his UI... but then why not just code it? There are serious researchers who have made credible bots (he links to them), bots which would (and, for all we know, do) make money on poker sites. But when I looked at it, the literature was not nearly detailed enough to code up their algorithms. (Also, they were working with simplified versions of poker, since full no-limit holdem is a hard nut to crack). What they didn't use were the "extensible rules-based systems" which this author claims "are a lot more powerful than you think". Count me skeptical. Good players eat such systems for snacks. Maybe they make money in low-stakes games against bad players, but I'd like to see evidence. In short: never mind the dll injection, show us your bankroll over 100,000 hands!

mattmaroon 19y ago
"Count me skeptical. Good players eat such systems for snacks." You're thinking about it wrong. As he mentions, your bot doesn't play against good players. It plays .5/1 or 1/2 nl and makes a shit ton of money if it just breaks even. There aren't good players at those levels, because anyone who was good would win and quickly move up. You only need all of your if/then statements to add up to being better than the worst 5% of poker players, and that's not that hard to do. I know only because I've done it. Thats why extensible rules based systems are better than you think. They can get you to that with a good amount of work. Especially in short stack nl games. They're clearly not going to win at a $5/$10 nl table or a $215 SNG. But they don't need to to rake in millions per year.

xirium 19y ago

> none of the online poker sites have really stepped up to the plate and either a) made bots legal (similar to the way that they're legal on Internet Chess Club) or b) put effective prevention measures in place. That's a tough choice for a casino. Allow automation and you create a ghetto which discourages casual users. Disallow automation and you encourage an escalating "arms race". Regardless, the techniques discussed could be detected using anti-virus techniques. I'm surprised that native clients don't perform integrity checks or is this the state of play in the "arms race"? It would be fairly simple to disable integrity checks using methods to circumvent software DRM. Then it would be a matter of implementing distributed state to determine and ignore circumvented clients.

ericb 19y ago
Interestingly, the casino's financial interest, since it makes its money only on the rake, is to have as many players or bots as possible. In fact, from their perspective, an infinity of bots battling each other should be just fine--bots scale better than humans. From Ultimate bet's perspective, bots are great: It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until UltimateBet is rich.
- xirium 19y ago
  I considered this possibility but a casino which is too open to bots could become marginalised. Real players would be scalped too easily and bot writers would be playing a zero-sum game. Both factors would reduce casino turnover and profit. So, it is in a casino's interest to leave the whole issue unresolved and with a level of tolerance which is in-step with competitors.
  - ericb 19y ago
    I agree to an extent, however, I would be most happy owning a bot battleground, personally. It would dispense with all that time wasted on "thinking" and reduce the speed to a processor/internet lag dependent game. Bots are no more zero sum than people, they are just faster. I love the idea of a money making machine that scales with processor cores/mhz.

ctkrohn 19y ago

It's interesting, but to be honest, if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures. The potential winnings are much higher, it's perfectly legal, and due to the depth of the markets such programs are capable of making much more money than poker bots. It's also doable, as funds like Renaissance Technologies (http://en.wikipedia.org/wiki/Renaissance_Technologies http://en.wikipedia.org/wiki/Renaissance_Technologies) and D. E. Shaw have demonstrated.

hugh 19y ago
if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures I'm not so sure about that. I think the main reason is: there's a lot more "stupid money" at the poker table than there is in the market. The bot as mentioned doesn't seem to do anything particularly fancy -- it just looks at the cards on the table and figures out your odds of winning based on that. This is information which is easily extracted mathematically, but which most players don't have direct access to. Good players will have a pretty good idea of these probabilities, whereas lousy players won't. If you can find yourself on a table full of lousy players, you're at a huge advantage. In the market, on the other hand, there's only one poker table, and all the best players in the world are already sitting at it. Any time statistical arbitrage opportunities arise in prices, you can bet that D.E. Shaw and Rennaisance Technologies have already noticed it and moved a few billion dollars that way. If you want to win, you don't just have to be smarter than six guys in their basements doing calculations in their heads, you have to be smarter than thousands of overpaid quants with supercomputers.
- ctkrohn 19y ago
  Fair enough. You're probably right. I don't play online poker (and am a poor casual player) so I don't know what the level of skill is on those websites. I was just making an assumption based on how difficult I imagined it would be for a computer to play very good poker.