10 ms·
Race conditions on Facebook, DigitalOcean and others (fixed)
- unclesaamm 11y agoWow, it seems like there is room here for a 3rd party vendor to implement promo code handling as a service, and to do it right once and for all.
- yesmade 11y ago$3k for the facebook review bug. that's a little bit too much - update thanks for the downvotes guys. keep up the good work
- franjkovic 11y agoThe bounty actually surprised me, too. I expected between $1000-$2000. That is one of reasons I like reporting bugs to Facebook - they pay really good, critical bugs are fixed really fast (<1 day). One time they paid me $5000 for a bug I never could have found, but they did internally based on my low severity report. (http://josipfranjkovic.blogspot.com/2013/11/facebook-bug-bounty-secondary-damage.html http://josipfranjkovic.blogspot.com/2013/11/facebook-bug-bou...)
- mwsherman 11y agoIt’s impressive that they are able to fix them so quickly – one needs to imagine they get a non-trivial number of reports, and that some majority of them are junk. They have a good triage + repro + escalation system.
- franjkovic 11y agoFacebook puts out stats from their bug bounty program once a year. Most of bugs are invalid reports - in 2013 they had 14,763 reports, with 687 being valid. (https://www.fb.com/818902394790655 https://www.fb.com/818902394790655) They probably got a couple people working exclusively on bug bounty reports. I also have to say they did a great job changing communication channels from emails to tickets which show in /support/, it is way easier now. The downside is that you must have a Facebook account, not sure if it was needed before the change.
- yesmade 11y agocongratulations on both findings
- Gigablah 11y agoInstead of questioning why others are getting so much, question why you're getting so little.
- yesmade 11y agochill out man. you are turning this into something personal. it was only a comment at the amount he got for cheating the review system. even the OP said he wasn't expecting that much. stop jumping into the hate wagon everybody
- dang 11y agoPerhaps Gigablah was trying to be helpful? HN can be frustrating if you provoke it. The problem isn't so much what you said as how you put it: the combination of dismissive tone and superficial content puts readers here on edge, because too many comments are like that and we all find them annoying. As a result it's easy to have your good intentions misread. If you had explained the thought process behind your comment, I think it would have been received better.
- Gigablah 11y agoI wasn't judging you, haha. I'm just saying, it makes more strategic sense in general to bring yourself up to the level of others (however inflated) rather than bring others down to yours.
- totony 11y agoThis bug actually seems quite critical imo, defeats the purpose of a feature and permits abuse/cheating
- mikeash 11y agoWho are you to say that it's "too much," when it's their money than they can spend as they wish?
- andersonmvd 11y agoMore interesting than the bounty itself is to understand which defense works best at scale and the nitty gritty details of those kind of attacks. Intuitively I think that we just need to avoid inconsistencies between the Time of Check (TOC) and Time of Use (TOU), so veryfing the existence of a discount coupon while inserting it in one query should do the trick (INSERT INTO coupons (...) Values (...) WHERE NOT EXISTS (SELECT 1 FROM coupons WHERE (...)) instead of increasing the time between the TOC/TOU, e.g. one query to check if the coupon exists and a second one to insert the coupon. Besides it I am wondering if I am missing something, e.g. is this really a problem limited to the application layer or are the databases unable to prevent such attacks? I think I am right regarding the app protection, but let's see what people have to say :)
- zhoutong 11y agoOr just use a UNIQUE INDEX.
- underwater 11y agoNot every database is powered by SQL. Add to that sharding, caching, cross data center traffic and the problem becomes non trivial very quickly.
- hobarrera 11y agosharding is what confuses me the most. How would you avoid these race conditions with a distributed database?
- MichaelGG 11y agoPick a good shard key. In the case of a per-page-per-user ratings system, if you shard on the PageId, then you can locally check consistency to make sure there's no duplicate (PageId, UserId) keys. You can check the same if you shard on UserId, but then doing aggregates can be more difficult since you need to talk to every shard to find out a page's rating.
- mike_hearn 11y ago
- numair 11y agoI would be really interested to know how various forms of this bug are resolved. This seems like a problem that, on its surface, seems easy to fix, but isn't. Especially if you've designed your architecture for real-time-ness and global redundancy. Google's servers with atomic clocks come to mind...
- ekimekim 11y agocynical answer: I've seen alot of races get "fixed" by adding a sleep() or similar less cynical answer: Commonly you already have some kind of means to handle races - locking, transactions, some other variety of extra check - and the fix for newly discovered races is "oh, I didn't realise that could happen. add lock"
- hobarrera 11y agoIf you get three requests in at the same time, and sleep the tree for N (say, 400) miliseconds they'll all still run concurrently. Adding a random time to sleep might work, but some requests would run noticeably slower.
- MichaelGG 11y agoUnless the code is doing read-write-read. If you're using a system that doesn't reflect writes immediately (like Elasticsearch), waiting after the writes can give time for the system to flush and make the other writes visible then you can execute rollback logic. It'd be much better to make sure you're updating the same unique key and/or use the DB's conflict resolution system.
- ejcx 11y agoI actually fixed the issue that was reported to LastPass. I could be mistaken but I believe he reported the security issue through our regular support channel which is why it took three days to see (instead of our security channel). From the time I saw it, I fixed it with the patch going live within an hour or two. When I DID see it, tried it myself with a quick shell script that that curled and backgrounded the same request a bunch of times, I just kind of chuckled. It was a good bug. Josip is top notch.
- monksy 11y agoBTW: I just subscribed to LastPass a few days ago. I'm pretty happy with the service.
- jdubs 11y agoLastPass is awesome but I hate their website login process! It bothers me to no extreme that if I type in my email address with a wrong username, it pops back with, "Invalid password" while typing in a obviously random email, it pops back with a "Unknown email address. Would you like to create an account now?." I worry that a malicious attacker could finger the service for potential victims.
- ejcx 11y agoUsername enumeration is a valid concern. Requests on the login form (and some other places) are throtted. If you get too many emails wrong you will start only getting errors.
- innocenat 11y agoIt is already normally possible to test whether email address is registered by trying to register with that email address. Unless that process is secured too, it doesn't really make much sense to not pop up Unknown email address error.
- pwman 11y agoCorrect -- It's a pet peeve of mine when login processes obscure this saying invalid password when the sign up process doesn't -- if you're going to tell people usernames aren't available then you shouldn't be avoiding it on the login screen.
- inportb 11y agoSo the review bug was a security issue but the username bug wasn't? I wonder what else the review bug affected.
- franjkovic 11y agoI think they did not reward me because you cannot really hurt anyone by having multiple usernames.
- joshschreuder 11y agoWhat about squatting on valuable ones? But probably not a big deal unless it relates to Pages.
- emmab 11y agoIt would be cool if there was a browser addon that let you submit a form N times in parallel.
- ejcx 11y agoI do a lot of App Sec related things and I actually use mostly Chrome dev tools and command line instead of burp and other tools. The way I reproduced the bug when it was reported was by using the "Copy to curl" feature in Chrome, and then using it as follows for i in `seq 1 16`;do curl.*& #copied from chrome dev tools. & to background done
- bburky 11y agoAlso, curl gained a --next command line option somewhat recently. It lets you send off multiple requests in the same curl invocation. These requests will all be pipelined in the same HTTP connection, which might trigger slightly different behavior in the website. I have considered writing a program that will let me send of a bunch of HTTP requests at once, but wait to close all the connections at the exact same time. That would probably be the most effective way to trigger race conditions.
- SixSigma 11y agowhy would it ?
- odonnellryan 11y agoIf you go down to the "proof of concept" here it's not hard to test this: https://defuse.ca/race-conditions-in-web-applications.htm https://defuse.ca/race-conditions-in-web-applications.htm
- d_luaz 11y agoNo bounty for bug report? Should at least have a nominal fee of $100 (else no one would bother to report it).
- squiguy7 11y agoI agree. If I had my own company I would surely provide some incentive for bugs found in the product. Whether that incentive was monetary, a free membership, etc. I think it's important to acknowledge that all software systems are imperfect.
- reagan83 11y agoThe economics of bug bounty programs could lead to misaligned incentives. Because the overhead cost to validate and communicate around bug reports isn't zero, the % of non-bugs submitted could become imbalanced because it is free to submit. In most systems the reward is zero, so you can infer if a person has taken the time to submit a bug report it is because he/she is invested in seeing it fixed. Context: I work at a decent sized company in SV on this type of problem.
- d_luaz 11y agoSo the best solution is not to have a reward? Or not to have a publicized reward? Or don't depend on the public on bug hunting? Or just hope on goodwill?
- nmjohn 11y agoSo when I find a bug in say Paypal which allows complete account takeover and could sell it to an organized hacker group for say $100,000 or report it to Paypal "because I'm invested in seeing it fixed" and receive nothing - that is only an easy decision for the whitest of white hat hacker. Properly designed bug bounty programs are a cornerstone to any company who remotely cares about the security of their product, period. The idea of misaligned incentives due to poor bug reports being free to submit is ignorant - and worse toxic, because it sounds so true to an executive who has no actual understanding of the issue. A quality bug report should take no more than 1 minute for a reviewer to look at and know if it's really a bug or not. If it can't, it should be rejected saying provide more clear details. For example a dom based xss attack could be reported with just a target URL and it is quite clear what the problem is. That would take 10 seconds to analyze. Additionally, most bugs reported to most decent sized companies are reported by someone who has previously reported a bug to the company before. If someone is constantly reporting good bugs or the opposite, its quite easy to prioritize which of those individuals gets their emails read first.
- MichaelGG 11y agoWe should see lots more of these if people embrace eventual consistency instead of "slow" ACID transactions. And interestingly, the more larger scale a system, the more likely that globally consistent operations are too expensive to enable in general, and developers will overlook cases where they must implement some locking or double checking.
- pyvpx 11y agowhen did eventual consistency equate to race conditions, or even increased susceptibility to race conditions? I don't follow. could you explain your reasoning further?
- MichaelGG 11y agoIt's probably just an ease-of-use question. The more guarantees your database can deliver, the easier it is to reason about things and make sure you aren't being caught on a gotcha. It's not necessarily different than using a normal RDBMS, right - you could do a check in SQL outside a transaction and end up writing multiple times. But with an RDBMS, you can easily solve the situation by turning on a transaction and leaving no question about things. This is why things like VoltDB ("NewSQL") are pushing to keep SQL and ACID, and figure out a way to scale, instead of throwing it all aside and making the developer deal with consistency issues. It's not that you can't end up with the same functionality using eventual consistency, just that it's harder. Just look at Amazon's "apology based computing" (I think that was the name) and how they structure their systems to be resilient by being able to resolve multiple conflicting commands in a proper way (deciding, without communication, which command wins, figuring out rollbacks, etc.) It's fantastic, and perhaps it's the only feasible way to operate at their scale. But it's also a hell of a lot more complicated than "UseTransaction = true". (So my predictions/guesses: If developers that'd otherwise use a traditional ACID RDBMS switch to non-ACID (BASE?) systems, they'll end up introducing bugs due to the shifted responsibility of handling data consistency. And seeing how big servers are, and even how far sharding can take you with normal RDBMS, the scale at which people "need" to drop ACID is probably far higher than the point at which people are dropping it.)
- Kiro 11y agoI'm a novice but would like to know how these issues can arise. What kind of backend setup is needed for it to be a problem? What is happening when a race condition occurs in these examples?
- deleted 11y ago[deleted]
- spdy 11y agoIts actually quite simple as example for the promo code the code looks like this: 1. Code sent. 2. Check if valid. 3. Redeem code. 4. Invalid code. Now if i send 10 requests at the same time with the same code maybe 4-6 will hit the code part after 2. And your window of opportunity is the time it takes to go from 3 to 4. Sometimes certain tasks are put inside async queue, you have a slight delay to your database server or you need to wait for db replication to kick in. Because normally there is no code part to recheck how often this code was used.
- codenut 11y agoCan this issue be prevented if we use the promo code as the table primary key or document ID?
- jaysh 11y agoThat won't be enough because the promo codes are shared amongst many users. If the promo code became the primary key, then only one user would be able to redeem it. If you introduced some combination of a user ID and promo code, then it won't prevent a race of one user firing many queries with different promo codes and stacking them up. It would, however, fix the original problem.
- some1else 11y agoA simple Discount domain model with validations: Class Discount belongs_to :promo_code belongs_to :customer belongs_to :order validates_presence_of :promo_code, :customer, :order validates_associated :promo_code validates_uniqueness_of :promo_code_id, :scope => [:customer_id, :order_id] end Limiting down to a single Promo-code per order: Class Discount # ... validates_uniqueness_of :order_id, :scope => :customer_id end
- Rafert 11y agoI have reported the same issue with Digital Ocean (security) in November 2014, and they told me I was using the wrong address and that they forwarded it to the proper team. I triggered it by accident, using the same GitHub code twice, and I (or the DO staffer) didn't realize it was a race condition. I never heard back but they let me keep the balance :)
- janoelze 11y agoappreciating the joke (?) in the comments. https://i.imgur.com/zWE5ABQ.png https://i.imgur.com/zWE5ABQ.png
- jbkkd 11y agoNow that race condition bugs have been widely exposed, I have a feeling we'll start seeing more of these "attacks" in the near future. They are relatively easy to execute and don't raise a high suspicion.
- georgerobinson 11y agoCan anyone comment on how the author flooded HTTP requests to the endpoint URLs? Did he use developer tools in his browser and execute his own JavaScript, or use CURL in a tight loop with the cookie and CSRF token from his browser session?
- gislifb 11y agoWithout knowing exactly how he did it I assume this is possible by doing a POST with cURL inside a loop or with parallel. You can then get the exact request by using Chrome developer-tools. (Find the POST-request in the network-tab, right-click and select copy as cURL)
- tomcam 11y agoNow please fix race conditions everywhere else, like Baltimore.