10 ms·
How Primedice was exploited for $1M in Bitcoin
- mmosta 11y agoThe (costly) lesson learned here doesn't revolve around Primedice's RNG architecture but the strict importance of testing. We all face pressure to deliver but at the end of the day -like brakes on a car- testing is one thing you should never cut corners on. Although another 2 weeks of testing may not have explicitly exposed the vulnerability, it surely would have offered a better baseline from which to evolve better heuristics for the analysis of the exploit when it did occur.
- hga 11y agoI'd say the degree of testing depends on the domain. Facebook in its early days could probably afford to drop a some data on the floor. Boeing and its subcontractors can't simply can't screw up flight software (or this happens: https://en.wikipedia.org/wiki/2015_Seville_A400M_crash https://en.wikipedia.org/wiki/2015_Seville_A400M_crash). In-between, we're seeing a lot of Bitcoin and other sorts of handling money associated firms discover that "move fast and break things" can result rather quickly in a broken firm.
- rpcope1 11y agoYeah, this is definitely one of those places where if you didn't test and analyze the system relatively well before deployment, you probably shouldn't be surprised that someone came along and did this. Is it nice/fair, no, not really, but if you're going to be moving lots of money in and out, you really ought to be developing to the same standards as maybe a brokerage or bank (not that they do exceptionally well on this front either). Also doxxing this guy in this context seems rather juvenile, especially given that it's unlikely it will help your cause in any tangible way.
- andrewvc 11y agoTesting is important, but I don't think that's enough for this sort of problem. What this needs is in-depth code review. Tests are only as good as the tests you write, and if there was no test to account for this particular issue nothing would have been caught. For concurrency issues someone thinking real hard is as important as testing.
- PhantomGremlin 11y agosomeone thinking real hard is as important as testing With one caveat. The "thinking real hard" has to happen before and as the code is written. If you try to catch fundamental design problems and misconceptions during a code review, you're much too late. At that point there is a "status quo" i.e. the existing code, and a defensive resistance to change, even if you can demonstrate clear problems with said code. The end result is band-aids on top of band-aids, rather than a proper solution to the problems. Edit: the article itself alludes to band-aids: our developer had improperly patched the glitch i.e. they attempted a quick fix, instead of understanding at a fundamental level the overall mistakes in what they were doing.
- jstanley 11y agoI don't think he did much wrong here. It doesn't appear that he actually broke in to your computers, he just submitted lots of bets. If you're going to run a casino (whose entire business model is based on exploiting weaknesses of others for profit), don't be surprised if people try to exploit your weaknesses for profit. I'm not saying there's anything wrong with running a casino, I think it is fine. I just think that what he did is fine too. EDIT: Except the part where you apparently didn't pay out all of his winnings. I'm not fine with that part. Also doxing one of your customers just because he made a profit out of you.
- danneu 11y agoJust because you actually do have a moral problem with casinos doesn't mean a malicious user is a "customer".
- Lawtonfogle 11y agoJust because you have a moral problem with the customer doesn't mean they are a malicious user. Even if a user's actions are outright illegal, illegal is neither sufficient nor necessary for saying something is wrong in the moral views of anyone who uses a moral system divorced from the legal system. In short, deeply held moral views can differ so greatly that a common language to describe the situation can be impossible unless one sticks purely to the facts which include accepting the legal system does exist and make judgment calls but putting no authority in those calls.
- swashboon 11y agoThe indicated that they emailed the exploiter and basically told them to stop playing. They have the right to refuse service, it was circumvented by making new accounts. I don't see any reason they would need to pay after refusing services.
- genericuser 11y agoBasically would be the key word in your description of their actions. As they do not mention actually telling the user to stop playing. They only mention "demanding" their bitcoins back. Demanding something you are at that point probably not entitled to from a user of your service, and informing them that you have decided to refuse future service to them may appear basically the same, but they are in fact not. "We reached out to Hufflepuff via his bitcointalk forum account and demanded the return of the coins, however this backfired unbelievably hard. "
- pavel_lishin 11y ago> Sorry for the long read Didn't feel long at all; it was concise and had enough detail to satisfy a casual reader (me) with links to more detail if I wanted it.
- genericuser 11y ago"To understand how Hufflepuff beat our system, one must understand how our provably fair system (RNG) works." "Our database had seeds that were both inactive and in use at the same time all connected to Hufflepuff." To me it sounds like your company simply didn't understand how the system you created worked, it sounds like you merely understood how you wished it to work. Gambling has always favored people who actually understand the system in place, by not understanding the system you yourselves were implementing you created the opportunity for a customer to understand it better than you. What you describe with your database having seeds that were both in use and inactive at the same time, sounds to me like you did not do a good job managing concurrency on records which were crucial to your business. While I can not say whether the users did anything illegal, the post you have created to call attention to this does not make an adequate case for wrong doing on the users part. However your response does make an amazing case for you having responded to an implementation issue in your software very poorly, and I think would make future users who are aware of this less likely to use your service. That is just my opinion given the information you have presented, as you seem to focus more on maliciously providing information of the person you are accusing as opposed to proving your accusations that the person in question committed a crime while doing so.
- jballanc 11y agoI'd be willing to place a bet on what sort of database technology they weren't using. Hint: it has three letters, starts with an S, and ends with an L. ;-)
- jsprogrammer 11y agoSQL won't force you to respect flag columns in your application layer (unless your entire application is in SQL?).
- nyir 11y agoFrom the article it sounds like constraints could have been used to prevent having multiple seeds linked to the same account?
- imaginenore 11y ago> and time again to investigate and each time our developers could not find any wrong-doing. > ... > This was done by sending it more requests than it could handle in a small time period, think hundreds of requests in under a second Sounds like your developers really messed up. The server logs would be the #1 place I would look. Where else really? How do you not notice hundreds of requests per second (from the same IP I assume)?
- disordr 11y agoThis was my question as well. How do you not notice 100's of qps from the same IP, and then go on to say, he was betting $8k/second. server logs, and basic networking chomps seems like it would have found this much quicker.
- Lazare 11y agoI've never heard of Primedice before, but based on their blog post (and the general concept of a bitcoin casino)... ...I can't imagine the slightest reason why they would expect, would want, or would tolerate automated gambling. ANY evidence that someone is using a script or bot, making too many bets, making bets with an oddly regular pattern, making too many requests from a single IP, making requests from different IPs at the same time, etc. would seem to be a red flag, and should cause the associated account to be suspended. Right? Like, if there's a hole in your code, the attacker is very likely to automate the attack to try and maximise their gains. But if there's no hole in your code, your punters are very unlikely to automate their playing. Any hint of automation is a huge red flag; any hint of automation combined with a string of "lucky" wins should be an automatic account suspension, because you're obviously looking at successful attack. Saying that your developers couldn't find any "wrong-doing" just raises questions. Like, why do your developers need to look, shouldn't those checks be automated? And how did they miss it when it turns out it was obvious? And why were you even looking when the underlying activity was obviously illegitimate? Real casinos don't wait until they understand the scam before cutting someone off. So many questions. > How do you not notice hundreds of requests per second (from the same IP I assume) To be fair to them, I imagine a bitcoin casino is a magnet for weird traffic patterns. Giving them the benefit of the doubt, maybe they're getting bursts of crazy traffic all the time, and they had no way of knowing which ones were "potential timing attacks" and not just "bored script kiddy with a botnet". Maybe "Hufflepuff" was hitting them from an unrelated network constantly, and a small fraction of those times he also made a bet he already knew the outcome of. Maybe he was also making normal bets at the same time. (...or maybe not.) Still, even if the bad traffic was hidden by noise, I don't believe that his account activity could have looked normal the entire time he was building up his 1 million payout.
- lcswi 11y agoUh, that post includes some doxxing. From an observer's view, after fully reading it and seeing how hostile their response is, I can only applaud the player. To me it seems fair game. She exploited a weakness in the game, something that in meatspace is often hailed as genius. In bitcoin online gambling, race conditions are a part of the game that has to be expected to be attacked. With their doxxing they lost any bit of sympathy I had.
- davidcollantes 11y agoShe? How did you know?
- CUViper 11y agoMany people wouldn't even notice "he" used for a person of unknown gender. In the spirit of equality, some use "she" instead to subvert this male-default. In fact, the article uses male pronouns for the user -- did you think to ask how they know?
- lcswi 11y agoYeah, I am playing with switching he for she whenever the gender is unknown or irrelevant. It's weird how weird it is.
- ronald_dregan 11y agobuttcoin is stupid and if you have anything to do with them, you are stupid and deserve to be scammed
- obstinate 11y agoIt is really interesting to me how different Bitcoin users' understanding of morality and the typical person's are. To almost any joe or Jane on the street, exploiting a weakness in someone else's security -- be it an unlocked door or an unchecked error code -- to acquire money from them without their intent, is Wrong. On the other hand, taking retaliatory action, such as providing what detail you can about that person in the service of returning the ill-gotten gains, is right. Judging by the contents of this thread, the Bitcoin/hacker community feels just the opposite. How odd. I feel like this more than anything will impede the mass adoption of Bitcoin.
- danneu 11y agoThere's also a lot of moralizing around casinos, so when you combine the two, people can't really help themselves. "They deserve it!"
- Nadya 11y agoFew people see scamming a casino (a business designed around scamming people) as "wrong". They see it as "a taste of their own medicine". Go to any Joe or Jane on the street and ask them if swindling (conning) $50 from a lifetime "professional" con artist is wrong. Out-conning the con artist is typically seen as a thing of approval.
- mikeash 11y agoI'm not sure the difference is really there. The fact that this place is a casino matters a lot. My understanding of typical morality is that it's totally cool to exploit a weakness in a casino's games to shift the odds in your favor, and the casino's recourse is limited to trying to make this impossible, and banning people who do it (while still paying out whatever they won before they were banned). However, it's not cool to break the casino's games from within. To make an analogy, if you discover that e.g. a casino's blackjack decks have twice as many 7s in them as they should, and you use that knowledge to make a ton of money playing blackjack at the casino, that's legitimate. However, if you break into the casino and replace all their normal blackjack decks with ones which have twice as many 7s, then use that knowledge to make a ton of money at blackjack, that's bad. So the question is, which category does this particular attack fall into? Apparently they exploited a race condition, wherein outcomes became predictable if lots of games were played simultaneously. Is that exploiting the game within its rules, or is that breaking and entering? To me, this feels legitimate. An analogy to physical games here might be finding a roulette table where the dealer under some circumstances lets you place a bet after it becomes apparent where the ball is going to land.
- justonepost 11y agoLol. They (as any sane Casino operator would) should have booted this guy immediately on day 1, claiming they felt he was scamming the system. Sure, pay him out, but don't let him keep betting. Yeesh. I have to wonder if this was an inside job and they let him siphon those winnings. I am highly suspicious of most bitcoin companies, and I wonder if they're all just shells around vulnerabilities looking to siphon user deposits / investments.
- danneu 11y agoThat doesn't make much sense. If they have a legit exploit, then kicking them will just make them create a new account. Or perhaps distribute the +EV betting over a bunch of accounts so that it was much harder to detect. If they don't have an exploit, then you want them to keep betting and lose. And if you have no real basis for kicking them, then, what, your casino just doesn't pay out people that win big? The casino's only real option here was to discover the exploit.
- justonepost 11y agoNo, pay him out, but just don't let anyone bets that way continue betting. Betting that rapidly has to be a huge flag for even the dimmest of sys admins.
- scintill76 11y agoWhat is "bets that way"? It seems like initially all they knew was that some accounts were unusually active (betting every second, for hours) and implausibly lucky. They saw the flags, they just didn't know any useful way to react (which is also incompetence, but IMO different than what you said.) (I'm assuming a dumb rate-limiting solution would be off the table, as it would be ineffective at stopping the attack, and/or lose money as other players stop betting when they hit a limit.)
- Lazare 11y ago> What is "bets that way"? I dunno, maybe anyone who is clearly automated (ie, betting every second for hours) and probably cheating (ie, implausibly lucky)? > It seems like initially all they knew was that some accounts were unusually active (betting every second, for hours) and implausibly lucky. Exactly. > They saw the flags, they just didn't know any useful way to react The useful way to react would be to not let them continue betting.
- tiatia 11y ago"We heavily explored what we thought was every possibility, ran simulations and did the math and came to the conclusion that he was just incredibly lucky." ROTFL
- baakss 11y agoStrange choice to put a picture from the Ocean's Eleven series at the top of the article, seeing as how the thieves in those films are considered the protagonists/heroes. I think a bit more information included with the article to verify this was actually a 'hack' would be helpful. I mean, was this akin to counting cards or what? It's hard to tell from the article.
- danneu 11y agoSince the player was making hundreds of simultaneous requests, it sounds like a race condition in the casino's server-seed mechanism where the player was able to get the server-seed of future bets. Obviously, the server only wants to divulge the server-secret of closed-out bets so that you can verify them as provably fair. > Strange choice to put a picture from the Ocean's Eleven series at the top of the article, seeing as how the thieves in those films are considered the protagonists/heroes. It's not so strange after reading a number of these HN comments.
- baakss 11y agoI think the strangeness for me is that the article is written from the casino's perspective. They went so far as to dox the 'culprit' and are portraying him in a negative light in the article.
- danneu 11y agoIt looks like the attacker had some sort of celebrity status and fanfare throughout all of this: https://bitcointalk.org/index.php?topic=843892.0 https://bitcointalk.org/index.php?topic=843892.0 so maybe Ocean's Eleven isn't so far-fetched.
- csomar 11y agoAny information that leads to the return of the coins from this incident will be greatly rewarded. We invite you to analyze the above bitcoin addresses and find out where the bulk of the coins ended up if you have the skills. What does this mean? Are they trying to steal the thief (according to them). Shouldn't they inform the local authorities and let them handle the case?
- danneu 11y ago> Shouldn't they inform the local authorities and let them > handle the case? In which jurisdiction? The one containing the VPN IP addresses? Or the internet police? An open letter to the internet asking for help ends up sounding about 1000x more effective than phoning this in to the "authorities".
- junto 11y ago... did the math and came to the conclusion that he was just incredibly lucky Woah. Red flag. Just incredibly lucky doesn't exist. Odds are odds.
- csbrooks 11y agoI think they just mean, if you have 10,000 customers, it's not surprising if someone hits the 1 in 10,000 odds. You could say that person was "lucky", though they have no greater chance than anyone else of hitting those odds again.
- Lazare 11y agoThey don't give us enough info to say for sure, but something tells me Hufflepuffs winnings were a few orders of magnitude less likely than one in 10,000.
- Lazare 11y agoI'm reflexively sympathetic to the small, scrappy startup. But... ...man this is not a sympathetic story. You're running a gambling site (morally shady and an obvious magnet for abuse), using bitcoins (another huge abuse magnet), you had very supicious gambling patterns (a massive red flag that no real casino would accept), and you just let the guy keep playing? That's not how you're supposed to do that. And as for the whole "we had a timing bug in our code, and we couldn't find it even with hard proof that it existed, and then we finally thought we'd fixed it, and then we asked the guy for the million dollars back (??!?), and then it turned out we hadn't fixed it at all, and he hit us up for another few thousand"? Like, that just sounds screaming amateurish. If you can't be trusted to write secure code, or at least fix the bugs you find in it, maybe online bitcoin casinos aren't for you? And if you think asking people nicely to give you your money back works in casinos, maybe you don't understand the industry? Edit: I'm not trying to be a dick, but I feel like the proper blogpost to write would be a grovelling "hey guys, I know it's super obvious, but if you're doing an online gambling website, monitor your transactions for specific patterns. Know what a real punter looks like, and aggressively throttle anyone who doesn't behave like one. Otherwise you'll be stupid idiots who lose a million dollars over a stupid bug, and then have to write a magnanimous letter congratulating the guy who exploited you for winning so much money (man that was painful to write)." You made some huge mistakes, and it doesn't sound like you really learned from them, or even identified them. Hint: Your core mistake was not a timing bug that emerged when your system was under heavy load.
- curiously 11y agoSo bitcoin casinos are legal ?
- scintill76 11y agoSome pretty hilarious YouTube comments on the linked video[0]: "could you please share your strategy in that video", "Please, Send your method of play and programs for this need". Sorry, but I'm pretty sure nobody's going to share their bot for making 2000 BTC in an hour. [0] https://www.youtube.com/watch?v=lSLXv5Tz1ZY https://www.youtube.com/watch?v=lSLXv5Tz1ZY , linked from "$8000 worth of bitcoin every second for hours on end"