5 ms·

Smashing the state machine: the true potential of web race conditions

112 points by chatmasta 3y ago

rob-olmos 3y ago
Post on ACIDRain as an example of the ecomm time-gap issue back in the day: https://news.ycombinator.com/item?id=20027532 https://news.ycombinator.com/item?id=20027532
imetatroll 3y ago
What I love about this is that http/2 which is non-trivial compared to http/1.1 allows this to happen. The same http/2 that, in my opinion, takes away from the beauty of the web.
- albinowax_ 3y ago
  At least HTTP/2 pretty much kills request smuggling (assuming there's no downgrading behind the scenes)
aidos 3y ago
Hadn't thought about the possibility of using http/2 in order to generate race conditions. Nice. Interesting trick. Years ago, Stripe had a capture the flag related to distributed systems (I believe it was the second one they ran). One of the final levels involved attacking a system that would make calls to 3 other systems in turn that each held part of the key before calling back to your webhook. You could tell how many systems it had spoken to by watching for incremented port numbers between the requests. I got killed by the jitter, but I seem to recall someone saying there was a way to stream your requests at a lower lower level on the same socket. Caveat, 10 years ago, details hazy. Found the HN thread from the time though https://news.ycombinator.com/item?id=4453857 https://news.ycombinator.com/item?id=4453857
1vuio0pswjnm7 3y ago
HTTP/2 is nightmarish.
- josephg 3y ago
  The problem isn't HTTP/2. HTTP/1.1 is also vulnerable to these attacks; they're just harder to find, reproduce and ultimately fix.
- rewmie 3y ago
  > HTTP/2 is nightmarish. Why do you think that?
- deleted 3y ago
  [deleted]
- robocat 3y ago
  From article: Finally, use the single-packet attack (or last-byte sync if HTTP/2 isn't supported) to issue all the requests at once. You can do this in Turbo Intruder using the single-packet-attack template, or in Repeater using the 'Send group in parallel' option.
- 1vuio0pswjnm7 3y ago
  Does HTTP/2 use more resources per connection than HTTP/1.1.
josephg 3y ago
The core problem here is that web application developers don't understand or use database transactions. I've seen this myself at almost every web shop I've ever worked at - queries / updates would be made to the database in series - "fetch this, then update that, then update this other thing". Your code works fine locally, but when multiple sessions interact with that data all at the same time, who knows what will happen. I hadn't thought of using these race conditions to trigger security vulnerabilities. That makes all sorts of sense, and I'm going to use this work to justify my persnicketyness in future projects. It doesn't help that popular ORMs either don't expose transactions at all or they consider transaction handling to be an advanced feature. It looks like mongodb only (finally) added transactions in mongo 4.2. The general rule for how to avoid this problems isn't locking. Its using a database transaction per HTTP request, or "operation" - however you define that. Transactions are perfect because if any part of the request fails you can rollback the entire transaction. And if you really need to, you can emulate something similar yourself by having a "last modified" nonce on your record (a random number will do). If you ever do a read-write cycle - say from a web frontend - then the read should include reading the nonce, and the write should fail on the server if the nonce has changed since whatever value you previously read. (And you can quietly fix this by simply retrying the entire operation). From a security point of view, this is a fantastic find. I anticipate a massive amount of code out there is at best buggy, and at worst vulnerable to these sort of attacks.
- mickeyp 3y ago
  > The core problem here is that web application developers don't understand or use database transactions. I've seen this myself at almost every web shop I've ever worked at - queries / updates would be made to the database in series - "fetch this, then update that, then update this other thing". Your code works fine locally, but when multiple sessions interact with that data all at the same time, who knows what will happen. I mean, you're right, and it's a bugbear of mine, too. But let's not pretend that SELECT FOR UPDATE and out-of-DB manipulation of said data fixes everything. (Nor that a direct UPDATE statement would.) Ultimately, either your updates are linearisable or they are not. Changing someone's first name with or without this methodology does not alter the fact that "last one to change wins". Which change is the right one is in this case in the eye of the beholder and the domain you work in. But, yes, updates that build on the existing values to derive new ones are common data races because people, indeed, do not understand databases, transactions or isolation levels. There are easy fixes for this type of problem. But it's never quite as simple as making everything SERIALIZABLE and FOR UPDATE and expecting that to resolve version conflicts because two different things have different expectations of what a row value should be.
socketcluster 3y ago
This is an excellent article. >> Every pentester knows that multi-step sequences are a hotbed for vulnerabilities, but with race conditions, everything is multi-step. This is something that we spent a lot of time thinking about and why we decided to upgrade SocketCluster (an open source WebSocket RPC + pub/sub system https://socketcluster.io/ https://socketcluster.io/) to support async iterables (with for-await-of loops) as first-class citizens to consume messages/requests instead of callback listeners. Listener callbacks are inherently concurrent and, therefore, prone to vulnerabilities as adroitly described in this article. It's very difficult to enforce that certain actions are processed in-order using callbacks and the resulting code is typically anything but succinct... Some users have still not upgraded to the latest SC version because it's just so different from what they're used to but articles like this help to confirm our own observations and reinforce that it may be a good decision in the long term. For all of its benefits, though, one of the gotchas of a queue-based system to be aware of is the potential for backpressure to build up. In our case, we had to expose an API for backpressure monitoring/management.
lifeisstillgood 3y ago
Is there a good shortish course / book / site out there that would take a developer from "I get the ideas and happily use burp" to "so that's how pentesting works"? It seems a valuable way to spend the rapidly smaller training budgets
- yao420 3y ago
  The portswigger courses from the creators of burp and the employers of this researcher are the best. Free too.
  - lifeisstillgood 3y ago
    Makes me look a bit silly now ... thank you
deleted 3y ago
[deleted]
ishjoh 3y ago
Great article and I had no idea what to call 'time of check, time of use' (TOCTOU) which is a great name instead of me having to describe the situation. I'm building an app on top of django where I have to worry about this, if you're using django check out there support for select-for-update, and if you're database supports it nowait=True can be a great thing that will fail a read if a select for update is already run: https://docs.djangoproject.com/en/4.2/ref/models/querysets/#select-for-update https://docs.djangoproject.com/en/4.2/ref/models/querysets/#... Also worth mentioning optimistic locking if you're looking to solve the issue in a different way, there is more involved from the application side but it has some advantages as well. I tend to prefer select for update with nowait=True since it's simpler on the application side, but I have used optimistic locking in the past with great success and some systems support it OOTB. Here is a description from AWS for those curious: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
deleted 3y ago
[deleted]

112 points by chatmasta 3y ago

Post on ACIDRain as an example of the ecomm time-gap issue back in the day: https://news.ycombinator.com/item?id=20027532 https://news.ycombinator.com/item?id=20027532

imetatroll 3y ago

What I love about this is that http/2 which is non-trivial compared to http/1.1 allows this to happen. The same http/2 that, in my opinion, takes away from the beauty of the web.

albinowax_ 3y ago
At least HTTP/2 pretty much kills request smuggling (assuming there's no downgrading behind the scenes)

aidos 3y ago

Hadn't thought about the possibility of using http/2 in order to generate race conditions. Nice. Interesting trick. Years ago, Stripe had a capture the flag related to distributed systems (I believe it was the second one they ran). One of the final levels involved attacking a system that would make calls to 3 other systems in turn that each held part of the key before calling back to your webhook. You could tell how many systems it had spoken to by watching for incremented port numbers between the requests. I got killed by the jitter, but I seem to recall someone saying there was a way to stream your requests at a lower lower level on the same socket. Caveat, 10 years ago, details hazy. Found the HN thread from the time though https://news.ycombinator.com/item?id=4453857 https://news.ycombinator.com/item?id=4453857

1vuio0pswjnm7 3y ago

HTTP/2 is nightmarish.

josephg 3y ago
The problem isn't HTTP/2. HTTP/1.1 is also vulnerable to these attacks; they're just harder to find, reproduce and ultimately fix.
rewmie 3y ago
> HTTP/2 is nightmarish. Why do you think that?
deleted 3y ago
[deleted]
robocat 3y ago
From article: Finally, use the single-packet attack (or last-byte sync if HTTP/2 isn't supported) to issue all the requests at once. You can do this in Turbo Intruder using the single-packet-attack template, or in Repeater using the 'Send group in parallel' option.
1vuio0pswjnm7 3y ago
Does HTTP/2 use more resources per connection than HTTP/1.1.

josephg 3y ago

The core problem here is that web application developers don't understand or use database transactions. I've seen this myself at almost every web shop I've ever worked at - queries / updates would be made to the database in series - "fetch this, then update that, then update this other thing". Your code works fine locally, but when multiple sessions interact with that data all at the same time, who knows what will happen. I hadn't thought of using these race conditions to trigger security vulnerabilities. That makes all sorts of sense, and I'm going to use this work to justify my persnicketyness in future projects. It doesn't help that popular ORMs either don't expose transactions at all or they consider transaction handling to be an advanced feature. It looks like mongodb only (finally) added transactions in mongo 4.2. The general rule for how to avoid this problems isn't locking. Its using a database transaction per HTTP request, or "operation" - however you define that. Transactions are perfect because if any part of the request fails you can rollback the entire transaction. And if you really need to, you can emulate something similar yourself by having a "last modified" nonce on your record (a random number will do). If you ever do a read-write cycle - say from a web frontend - then the read should include reading the nonce, and the write should fail on the server if the nonce has changed since whatever value you previously read. (And you can quietly fix this by simply retrying the entire operation). From a security point of view, this is a fantastic find. I anticipate a massive amount of code out there is at best buggy, and at worst vulnerable to these sort of attacks.

mickeyp 3y ago
> The core problem here is that web application developers don't understand or use database transactions. I've seen this myself at almost every web shop I've ever worked at - queries / updates would be made to the database in series - "fetch this, then update that, then update this other thing". Your code works fine locally, but when multiple sessions interact with that data all at the same time, who knows what will happen. I mean, you're right, and it's a bugbear of mine, too. But let's not pretend that SELECT FOR UPDATE and out-of-DB manipulation of said data fixes everything. (Nor that a direct UPDATE statement would.) Ultimately, either your updates are linearisable or they are not. Changing someone's first name with or without this methodology does not alter the fact that "last one to change wins". Which change is the right one is in this case in the eye of the beholder and the domain you work in. But, yes, updates that build on the existing values to derive new ones are common data races because people, indeed, do not understand databases, transactions or isolation levels. There are easy fixes for this type of problem. But it's never quite as simple as making everything SERIALIZABLE and FOR UPDATE and expecting that to resolve version conflicts because two different things have different expectations of what a row value should be.

socketcluster 3y ago

This is an excellent article. >> Every pentester knows that multi-step sequences are a hotbed for vulnerabilities, but with race conditions, everything is multi-step. This is something that we spent a lot of time thinking about and why we decided to upgrade SocketCluster (an open source WebSocket RPC + pub/sub system https://socketcluster.io/ https://socketcluster.io/) to support async iterables (with for-await-of loops) as first-class citizens to consume messages/requests instead of callback listeners. Listener callbacks are inherently concurrent and, therefore, prone to vulnerabilities as adroitly described in this article. It's very difficult to enforce that certain actions are processed in-order using callbacks and the resulting code is typically anything but succinct... Some users have still not upgraded to the latest SC version because it's just so different from what they're used to but articles like this help to confirm our own observations and reinforce that it may be a good decision in the long term. For all of its benefits, though, one of the gotchas of a queue-based system to be aware of is the potential for backpressure to build up. In our case, we had to expose an API for backpressure monitoring/management.

lifeisstillgood 3y ago

Is there a good shortish course / book / site out there that would take a developer from "I get the ideas and happily use burp" to "so that's how pentesting works"? It seems a valuable way to spend the rapidly smaller training budgets

yao420 3y ago
The portswigger courses from the creators of burp and the employers of this researcher are the best. Free too.
- lifeisstillgood 3y ago
  Makes me look a bit silly now ... thank you

deleted 3y ago

[deleted]

ishjoh 3y ago

Great article and I had no idea what to call 'time of check, time of use' (TOCTOU) which is a great name instead of me having to describe the situation. I'm building an app on top of django where I have to worry about this, if you're using django check out there support for select-for-update, and if you're database supports it nowait=True can be a great thing that will fail a read if a select for update is already run: https://docs.djangoproject.com/en/4.2/ref/models/querysets/#select-for-update https://docs.djangoproject.com/en/4.2/ref/models/querysets/#... Also worth mentioning optimistic locking if you're looking to solve the issue in a different way, there is more involved from the application side but it has some advantages as well. I tend to prefer select for update with nowait=True since it's simpler on the application side, but I have used optimistic locking in the past with great success and some systems support it OOTB. Here is a description from AWS for those curious: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

deleted 3y ago

[deleted]