6 ms·

Race conditions on the web

119 points by type0 10y ago

et1337 10y ago
I would love to hear how these were all patched. Wrap everything in a transaction? Use some kind of MQ? Make sure these critical DB calls are not cached in any way? I'm sure it's different for each case.
- EGreg 10y ago
  Um, simply generate a new code for every time an email is attached. Why not?
  - simplify 10y ago
    Instead of downvoting this comment, why not explain why they're wrong, both for the benefit of them and other readers?
    - sbuttgereit 10y ago
      Staying off topic for a moment... I bet there's an interesting study in this. Something along the lines of neurochemical or psychological rewards when one does something like "downvote" in self-regulating forum like this; seems like it would be narrowly focused enough an as compared to say, trolling, that you might be able discern something in from that sort of behavior. (I'm clearly no scientist :-) ) For the record I agree with you. I think when someone says something that is silly, naive, or somehow just wrong in a forum like this... it seems like it would be far more productive to both participants and observers to have someone just point out what's wrong (yes, that takes time, but you cared enough to down vote, right?) Some part of the time you'll just uncover a misunderstanding or actually find out the person that would otherwise be down voted may have had a point. Oh well... back to work.
- ejcx 10y ago
  Having patched one of this researchers race conditions, while I was at lastpass, I did exactly that. This only applies to coupon redemption
  - corobo 10y ago
    Which one?
Blaine0002 10y ago
It always amazes me when developers making crucial transaction systems don't know about or implement some form of locking. I personally use a cluster of consul servers for distributed locking ( http://consul.io http://consul.io by hashicorp ) although I don't use the rest of consuls features I'd sure like to learn how someday :)
- spriggan3 10y ago
  You can build what you think is the most robust and secure system in the world, someone somewhere will figure out how to break it. I don't think it's fair to insinuate that the people who wrote the code were "incompetent", especially given the size of Facebook's codebase. And given their audience they'll be more exposed to hackers than whatever thing you'r working on that isn't Facebook and doesn't have the same audience.
  - Noseshine 10y ago
    Exactly. When I write my code I am aware of a high number of things that could go wrong, and that I deliberately don't check. If it goes wrong I'll let it crash, or I make the deliberate decision too see if that condition ever actually happens in the real world. I'm not talking about security! It can be things like only checking if function parameters are what I expect them to be for some functions where I think it's important, or being aware that if some functions are called with other timings than I expect something could happen. The problem is I'm not writing the final app, I'm writing some sort of library (sort of), so I have no control over how it is going to be called in the end. I'll just add it to the documentation but I make few attempts at catching all or even a lot of such errors. If I would blow up my code at least tenfold if I tried to take care of all the possible conditions - creating a lot more of them in the process. Writing code feels amazingly fragile to me, and yet it works well. Note that that code has had several reviews from other developers, so I'm not talking about really bad code. After having delved deep into medical topics out of curiosity - hundreds of hours of anatomy, physiology, neuroscience, bio-chemistry, lots of statistics, I'm even less concerned. The ways things go wrong in a biological system are orders of magnitude more numerous, and the approach of nature is "fix it when it happens" (or start by creating a new instance). I think the more complex our own human-made systems become we'll have to use more and more of the nature method. We are already doing it everywhere, electronics or software. I see two competing forces: a) The human attempt to make systems more "provable", for example by formalization/"mathematization", b) Nature showing us that complex systems can only be done with a relaxed and laissez-faire attitude ("shit happens") after putting in a reasonable effort. The balance shifts towards b) for systems in rapidly changing environments, and to a) for systems in static conditions. So discussions about the subject should never be just about the system (piece of software) itself, it must include the environment it is to operate in.
paulmd 10y ago
I used to use a camera shopping website that had a race condition in their product listing/search functionality. Their site was quite slow, and if you tried to have more than one tab open to their site then you would get mixed-up results. The tabs might have the same content, mixed content (eg the breadcrumbs from the other tab), or sometimes nothing at all. This would occur even across a significant period of time, not just if you made the requests at the same time. I struggle to understand how that could even have happened. Were they storing the results of the DB lookup referenced against my IP somehow?
- mh- 10y ago
  probably collecting the search results into a cache of some sort, and the key was a session ID (likely stored in a cookie.)
- deleted 10y ago
  [deleted]
- JoshTriplett 10y ago
  Lenovo's website does this: if you try to browse and customize more than one ThinkPad at a time, the site will get completely confused and start replacing the older one with the newer one, losing the older one entirely.
- flukus 10y ago
  I'm guessing a lot of them stem from using session variables and from caching. I've seen everything from an nHibernate cache being stored in the session, serializing and deserializing a huge chunk of the database on every request, to developers not being aware that the session variable they get is user specific (They access it via Session[UserId]). Caching in particular seems to be something that's applied without much analysis, I've seen it slow down implementations more often than it sped them up. What I'd really like is for the browser to send a tab id on every request, so we can scope variable to a tab as well as user.
  - robocat 10y ago
    FYI: one place to store a tab id is in the window.name variable: * it is persistent across refreshes * it remains even if you leave page to another domain then return to page on same tab * it disappears if the tab is closed * it doesn't change as you navigate back/forward * would be most useful with xhr requests or a hidden field in form post requests. Downsides: * no origin security * no obvious nice way to use with page GETs for server to know which tab a page request is from.
spriggan3 10y ago
Interesting, it would be even more interesting to know how these problems were patched on Facebook's side.
matheweis 10y ago
I was once involved in helping diagnose a race condition in a website whereby a user could change another user's password if they changed their password at approximately the same time. When looking at the code it was not at all obvious that there was even potential for the issue until you knew it was going on... Some on this thread are acting like these are trivial problems to avoid. I think they are wrong...
- onion2k 10y ago
  The problems aren't trivial, and sometimes the solution isn't trivial, but that doesn't mean there aren't strategies that automatically mitigate some of the issues. Using database transactions, verifying things worked correctly, rolling back transactions with unexpected results, not making assumptions like "two users won't change passwords in the same millisecond", and so on can protect your product. PHP and MySQL are largely to blame as so many web developers started out building sites on a database that didn't support transactions if you used the default engine. Basic software quality assurance is missing from the majority of web development businesses.
- deleted 10y ago
  [deleted]
- homero 10y ago
  How's that work when you set passwords according to id?
  - lsaferite 10y ago
    Yeah, this is my question exactly. I've dealt with MANY race conditions over the years and the idea that two different user requests could update the same user account password seems... unlikely at best. Not saying it's impossible, but more detail would have been appreciated.
    - yardstick 10y ago
      I've seen this before in a Java system. A MVC Controller populated the HTTP params into a Bean, which it assigned to an instance field, and then used that subsequently to grab the value to update (don't think it was password, might have been email address iirc). This would work fine if the controlled was request scoped, but it turned out to be a singleton - so the field was shared across all requests!
breatheoften 10y ago
Bug bounty programs are interesting. I wonder if it wouldn't be a good use of tax payer dollars to pay people at the nsa or similar organization with computer security responsibilities to churn away at these programs. As in "spend one month a year doing big bounty programs, collect your normal salary, keep whatever you earn" ...
- deleted 10y ago
  [deleted]
vinchuco 10y ago
Off topic: FarmVille on a satellite connection could let you plant fields on top of each other even though they weren't supposed to intersect. Unintentional vertical farming.

119 points by type0 10y ago

I would love to hear how these were all patched. Wrap everything in a transaction? Use some kind of MQ? Make sure these critical DB calls are not cached in any way? I'm sure it's different for each case.

EGreg 10y ago
Um, simply generate a new code for every time an email is attached. Why not?
- simplify 10y ago
  Instead of downvoting this comment, why not explain why they're wrong, both for the benefit of them and other readers?
  - sbuttgereit 10y ago
    Staying off topic for a moment... I bet there's an interesting study in this. Something along the lines of neurochemical or psychological rewards when one does something like "downvote" in self-regulating forum like this; seems like it would be narrowly focused enough an as compared to say, trolling, that you might be able discern something in from that sort of behavior. (I'm clearly no scientist :-) ) For the record I agree with you. I think when someone says something that is silly, naive, or somehow just wrong in a forum like this... it seems like it would be far more productive to both participants and observers to have someone just point out what's wrong (yes, that takes time, but you cared enough to down vote, right?) Some part of the time you'll just uncover a misunderstanding or actually find out the person that would otherwise be down voted may have had a point. Oh well... back to work.
ejcx 10y ago
Having patched one of this researchers race conditions, while I was at lastpass, I did exactly that. This only applies to coupon redemption
- corobo 10y ago
  Which one?

Blaine0002 10y ago

It always amazes me when developers making crucial transaction systems don't know about or implement some form of locking. I personally use a cluster of consul servers for distributed locking ( http://consul.io http://consul.io by hashicorp ) although I don't use the rest of consuls features I'd sure like to learn how someday :)

spriggan3 10y ago
You can build what you think is the most robust and secure system in the world, someone somewhere will figure out how to break it. I don't think it's fair to insinuate that the people who wrote the code were "incompetent", especially given the size of Facebook's codebase. And given their audience they'll be more exposed to hackers than whatever thing you'r working on that isn't Facebook and doesn't have the same audience.
- Noseshine 10y ago
  Exactly. When I write my code I am aware of a high number of things that could go wrong, and that I deliberately don't check. If it goes wrong I'll let it crash, or I make the deliberate decision too see if that condition ever actually happens in the real world. I'm not talking about security! It can be things like only checking if function parameters are what I expect them to be for some functions where I think it's important, or being aware that if some functions are called with other timings than I expect something could happen. The problem is I'm not writing the final app, I'm writing some sort of library (sort of), so I have no control over how it is going to be called in the end. I'll just add it to the documentation but I make few attempts at catching all or even a lot of such errors. If I would blow up my code at least tenfold if I tried to take care of all the possible conditions - creating a lot more of them in the process. Writing code feels amazingly fragile to me, and yet it works well. Note that that code has had several reviews from other developers, so I'm not talking about really bad code. After having delved deep into medical topics out of curiosity - hundreds of hours of anatomy, physiology, neuroscience, bio-chemistry, lots of statistics, I'm even less concerned. The ways things go wrong in a biological system are orders of magnitude more numerous, and the approach of nature is "fix it when it happens" (or start by creating a new instance). I think the more complex our own human-made systems become we'll have to use more and more of the nature method. We are already doing it everywhere, electronics or software. I see two competing forces: a) The human attempt to make systems more "provable", for example by formalization/"mathematization", b) Nature showing us that complex systems can only be done with a relaxed and laissez-faire attitude ("shit happens") after putting in a reasonable effort. The balance shifts towards b) for systems in rapidly changing environments, and to a) for systems in static conditions. So discussions about the subject should never be just about the system (piece of software) itself, it must include the environment it is to operate in.

paulmd 10y ago

I used to use a camera shopping website that had a race condition in their product listing/search functionality. Their site was quite slow, and if you tried to have more than one tab open to their site then you would get mixed-up results. The tabs might have the same content, mixed content (eg the breadcrumbs from the other tab), or sometimes nothing at all. This would occur even across a significant period of time, not just if you made the requests at the same time. I struggle to understand how that could even have happened. Were they storing the results of the DB lookup referenced against my IP somehow?

mh- 10y ago
probably collecting the search results into a cache of some sort, and the key was a session ID (likely stored in a cookie.)
deleted 10y ago
[deleted]
JoshTriplett 10y ago
Lenovo's website does this: if you try to browse and customize more than one ThinkPad at a time, the site will get completely confused and start replacing the older one with the newer one, losing the older one entirely.
flukus 10y ago
I'm guessing a lot of them stem from using session variables and from caching. I've seen everything from an nHibernate cache being stored in the session, serializing and deserializing a huge chunk of the database on every request, to developers not being aware that the session variable they get is user specific (They access it via Session[UserId]). Caching in particular seems to be something that's applied without much analysis, I've seen it slow down implementations more often than it sped them up. What I'd really like is for the browser to send a tab id on every request, so we can scope variable to a tab as well as user.
- robocat 10y ago
  FYI: one place to store a tab id is in the window.name variable: * it is persistent across refreshes * it remains even if you leave page to another domain then return to page on same tab * it disappears if the tab is closed * it doesn't change as you navigate back/forward * would be most useful with xhr requests or a hidden field in form post requests. Downsides: * no origin security * no obvious nice way to use with page GETs for server to know which tab a page request is from.

spriggan3 10y ago

Interesting, it would be even more interesting to know how these problems were patched on Facebook's side.

matheweis 10y ago

I was once involved in helping diagnose a race condition in a website whereby a user could change another user's password if they changed their password at approximately the same time. When looking at the code it was not at all obvious that there was even potential for the issue until you knew it was going on... Some on this thread are acting like these are trivial problems to avoid. I think they are wrong...

onion2k 10y ago
The problems aren't trivial, and sometimes the solution isn't trivial, but that doesn't mean there aren't strategies that automatically mitigate some of the issues. Using database transactions, verifying things worked correctly, rolling back transactions with unexpected results, not making assumptions like "two users won't change passwords in the same millisecond", and so on can protect your product. PHP and MySQL are largely to blame as so many web developers started out building sites on a database that didn't support transactions if you used the default engine. Basic software quality assurance is missing from the majority of web development businesses.
deleted 10y ago
[deleted]
homero 10y ago
How's that work when you set passwords according to id?
- lsaferite 10y ago
  Yeah, this is my question exactly. I've dealt with MANY race conditions over the years and the idea that two different user requests could update the same user account password seems... unlikely at best. Not saying it's impossible, but more detail would have been appreciated.
  - yardstick 10y ago
    I've seen this before in a Java system. A MVC Controller populated the HTTP params into a Bean, which it assigned to an instance field, and then used that subsequently to grab the value to update (don't think it was password, might have been email address iirc). This would work fine if the controlled was request scoped, but it turned out to be a singleton - so the field was shared across all requests!

breatheoften 10y ago

Bug bounty programs are interesting. I wonder if it wouldn't be a good use of tax payer dollars to pay people at the nsa or similar organization with computer security responsibilities to churn away at these programs. As in "spend one month a year doing big bounty programs, collect your normal salary, keep whatever you earn" ...

deleted 10y ago
[deleted]

vinchuco 10y ago

Off topic: FarmVille on a satellite connection could let you plant fields on top of each other even though they weren't supposed to intersect. Unintentional vertical farming.