12 ms·

ACIDRain: Concurrency-Related Attacks on Database-Backed Web Applications (2017) [pdf]

139 points by zvrba 7y ago

founderling 7y ago
Somebody write a TlDr. You'll be my hero of the day!
- Skunkleton 7y ago
  ABSTRACT In theory, database transactions protect application data from cor- ruption and integrity violations. In practice, database transactions frequently execute under weak isolation that exposes programs to a range of concurrency anomalies, and programmers may fail to correctly employ transactions. While low transaction volumes mask many potential concurrency-related errors under normal operation, determined adversaries can exploit them programmatically for fun and profit. In this paper, we formalize a new kind of attack on database-backed applications called an ACIDRain attack, in which an adversary systematically exploits concurrency-related vulnerabil- ities via programmatically accessible APIs. These attacks are not theoretical: ACIDRain attacks have already occurred in a handful of applications in the wild, including one attack which bankrupted a popular Bitcoin exchange. To proactively detect the potential for ACIDRain attacks, we extend the theory of weak isolation to analyze latent potential for non-serializable behavior under concurrent web API calls. We introduce a language-agnostic method for detecting potential isolation anomalies in web applications, called Abstract Anomaly Detection (2AD), that uses dynamic traces of database accesses to efficiently reason about the space of possible concurrent interleavings. We apply a prototype 2AD analysis tool to 12 popular self-hosted eCommerce applications written in four languages and deployed on over 2M websites. We identify and verify 22 critical ACIDRain attacks that allow attackers to corrupt store inventory, over-spend gift cards, and steal inventory.
  - founderling 7y ago
    I mean a technical TlDr of how the actual attack works.
    - tadkar 7y ago
      Basically use the web APIs to issue two spend requests almost simultaneously, and the fact that it takes time for the database to synchronize means you can double spend. As the posters above are saying, it’s very similar to a race condition.
    - dc_gregory 7y ago
      The technical TLDR is "db level race condition", or the title of the article.
  - bouncycastle 7y ago
    So, umm... they re-discovered race conditions and gave it a fancy name?
    - homakov 7y ago
      Correct. “ACIDrain” is literally a race condition and made up name.
    - praptak 7y ago
      The value of tfa is the work they put into practical demonstration and analysis. Anybody can sit in their armchair and speculate about theoretical exploits. It's the actual practical exploits and mitigation techniques that matter.
- C4stor 7y ago
  TL;DR : Researchers found a way to analyze database logs to identify race conditions in databases, and demonstrated how once this knowledge acquired, it left websites open to be exploited through their public APIs.
homakov 7y ago
This is for anomaly detection based on reading logs. It may miss a vulnerability so you still need to test yourself. I have a Chrome extension which is perfect for race attacks. https://github.com/sakurity/racer https://github.com/sakurity/racer
- squanch 7y ago
  Thank you. This seems pretty neat!
sitkack 7y ago
This is awesome! This is another brick in the road to requiring formal methods. Hmm, typed multiple database-call transactions. Are there type safe database libraries for Idris or Rust?
- pyrale 7y ago
  Not what you ask, but squeal is a typed SQL wrapper for Haskell.
  - LunaSea 7y ago
    It's really just an ORM if I'm looking at the project.
    - deleted 7y ago
      [deleted]
    - pyrale 7y ago
      The ORM part is a thin wrapping that is there to explain Haskell enough to have type-checking. The goal is really to have Haskell type-check the requests you make.
      - LunaSea 7y ago
        Yup and I think it's a noble goal but it's still implemented in a way where you're not writing SQL but some kind of other syntax. Also the issue is that it's trying to be database agnostic which means that you can't use any of the cool features from your database of choice.
- namibj 7y ago
  There is something based on Bloom IIRC, but the whole concept should be well-described and discoverable from the Keeping CALM paper: https://arxiv.org/abs/1901.01930 https://arxiv.org/abs/1901.01930
bouncycastle 7y ago
On page 11… "Developer's response" section, quote: "In contrast, the developer of OpenCart responded to the inventory vulnerability by posting a comment—“use your brain! its [sic] not hard to come up with a solution that does not involve coding!”—then closed both the inventory and voucher vulnerability issues and blocked us from responding. "
- founderling 7y ago
  The response seems shockingly arrogant and irresponsible at first. But he is in fact right. A good reminder that people who are unfriendly and argue against public opinion are not necessarily wrong. I don't know opencart, but I know how it is handled in companies using other onlineshops: When the stock of an item goes negative, it is noticed by staff when they want to fulfill the order. ("Strange, this customer ordered red socks, but we don't have any"). Staff sets the item as "Cannot be delivered" in the order. Which ups the stock to zero again. Customer order gets fulfilled with a note "Sorry, the red socks could not be delivered.". It's not perfect, but also not a big drama. Same with coupons. You should have them being invalidated after the order process. And if you see the same coupon code used twice, you know it is fraud.
  - dunkelheit 7y ago
    The response is arrogant and irresponsible and not necessarily right either. This attitude is precisely the reason why most software is shit that requires constant manual babysitting. Bugs like this are no big deal until they are - even something insignificant as an item missing from the inventory can become a source of drama if you promised exactly this item to your child as a christmas present.
    - founderling 7y ago
      Items cannot be shipped sometimes. In real life, that happens way way way more often due to other circumstances then due to race conditions. Staff breaks the last item. Manufacturer sent a box with 100 items but it only contained 99. The color of the shelf stained the last item. Etc etc etc. Think about it: Depending on the shop software, two orders need to be placed within a few milliseconds. Containing the same item. And the item has to be very low on stock so the two orders cannot be fulfilled. That is a very very rare event. It's almost a theoretical event only. I have seen a bunch of companies grow an online business. There were tons of pain points. This race condition was none of them.
saurabhnanda 7y ago
Does this impact postgres backed applications?
- dunkelheit 7y ago
  Yes! The default isolation level in postgres is READ COMMITTED which is rather weak and allows for the types of concurrency bugs discussed in the article (unless you use SELECT FOR UPDATE or some other kind of locking). https://www.postgresql.org/docs/11/transaction-iso.html https://www.postgresql.org/docs/11/transaction-iso.html
timvisee 7y ago
Along with transactions, I also like to use 'Add' or 'Subtract' operations which usually provide better reliability instead of manually reading and setting, being a source for data races.
- varelaz 7y ago
  You may end up with 0xFF items on stock, add/subtract are cool, but not ideal, also negative number is not always more acceptable than exception. I recall FF bug with Civilization https://www.geek.com/games/why-gandhi-is-always-a-warmongering-jerk-in-civilization-1608515/ https://www.geek.com/games/why-gandhi-is-always-a-warmongeri...
sara7262 7y ago
See the video of most dangerous bird and how he killed owner https://howto105.blogspot.com/2019/04/most-dangerous-bird-killed-owner-in-usa.html https://howto105.blogspot.com/2019/04/most-dangerous-bird-ki... Funny dog driving own car See his video on link below http://bit.ly/2WdSKpA http://bit.ly/2WdSKpA Cute dog going for school like human baby see his cute video I am sure you can't stop laugh http://bit.ly/2Wq6Ynr http://bit.ly/2Wq6Ynr United nation recently showed the real face of india See what they said in report After knowing this I am sure you will hate them if you have soft heart https://youtu.be/oUskNtdWeIY https://youtu.be/oUskNtdWeIY Best way to get rid from face moles with the help of home remedies its easy and simple way You don't need to waste money for this https://zoomtips.blogspot.com/2019/05/easily-get-rid-from-face-moles-with.html https://zoomtips.blogspot.com/2019/05/easily-get-rid-from-fa... Girl made a strange world record even you can't believe this see her video on link below http://bit.ly/2ErUDEW http://bit.ly/2ErUDEW Trump is useful now lol See in video what China did with trump http://bit.ly/2VPVZis http://bit.ly/2VPVZis Lose weight without exercise Now weight losing is not hard job just follow this simple method https://rebrand.ly/loss-weight https://rebrand.ly/loss-weight Now don't worry about dry hair or dryness on scalp Just follow this simple methods which you can make with simple home remedies and get rid from hair dryness https://zoomtips.blogspot.com/2019/05/treatment-for-dry-hair-and-scalp.html https://zoomtips.blogspot.com/2019/05/treatment-for-dry-hair... This video is important for Facebook users must watch this otherwise you will loss your Facebook id https://rebrand.ly/Facebook-id-danger https://rebrand.ly/Facebook-id-danger
aneutron 7y ago
Very interesting. I wonder if they implemented the nested transactions aspect of 2AD (paper is dated at 2017, and they said they were working on it). That would make it extremely useful in pentesting.
varelaz 7y ago
While this is cool to harden your transactions, as more locks you add as more deadlocks and blocks appear, and it becomes harder and harder to keep transactions fast and isolated from each other. For example if you do select for update and record is not found, DB should lock key space to prevent adding such records, did you suppose to do this and block other inserts/updates? It's often easy to find problem with transactions in code, but very hard to fix all consequences. By the way because of this instead of select for update for single record I prefer update with no change. If there is no record, nothing will be locked.
- hobs 7y ago
  Blocking phantom reads/writes is usually only that of a serializeable transaction isolation level - and a range lock shouldnt block other writes, unless they are of the exact range specified, and then you are asking for a consistent view of reality. Generally I dont find adding more locks hurts things that are well designed - often you will find deadlocks turn into just waiting for your turn in line - and then you can tune your db code or your hardware or your whatever to maximize that throughput. Also fwiw if you are doing any sort of upsert and you are not locking for the update/insert combo, then you are definitely opening the door to a duplicate key insert - as long as you handle that though, nbd.
he0001 7y ago
Correct me if I’m wrong here, but wouldn’t these attacks be translatable to anywhere there are read/write discrepancies?
lgats 7y ago
Mobile Friendly View: http://pdf.cab/openaccess.thecvf.com/content_ECCV_2018/papers/Mikael_Persson_Lambda_Twist_An_ECCV_2018_paper.pdf http://pdf.cab/openaccess.thecvf.com/content_ECCV_2018/paper...

139 points by zvrba 7y ago

founderling 7y ago

Somebody write a TlDr. You'll be my hero of the day!

Skunkleton 7y ago
ABSTRACT In theory, database transactions protect application data from cor- ruption and integrity violations. In practice, database transactions frequently execute under weak isolation that exposes programs to a range of concurrency anomalies, and programmers may fail to correctly employ transactions. While low transaction volumes mask many potential concurrency-related errors under normal operation, determined adversaries can exploit them programmatically for fun and profit. In this paper, we formalize a new kind of attack on database-backed applications called an ACIDRain attack, in which an adversary systematically exploits concurrency-related vulnerabil- ities via programmatically accessible APIs. These attacks are not theoretical: ACIDRain attacks have already occurred in a handful of applications in the wild, including one attack which bankrupted a popular Bitcoin exchange. To proactively detect the potential for ACIDRain attacks, we extend the theory of weak isolation to analyze latent potential for non-serializable behavior under concurrent web API calls. We introduce a language-agnostic method for detecting potential isolation anomalies in web applications, called Abstract Anomaly Detection (2AD), that uses dynamic traces of database accesses to efficiently reason about the space of possible concurrent interleavings. We apply a prototype 2AD analysis tool to 12 popular self-hosted eCommerce applications written in four languages and deployed on over 2M websites. We identify and verify 22 critical ACIDRain attacks that allow attackers to corrupt store inventory, over-spend gift cards, and steal inventory.
- founderling 7y ago
  I mean a technical TlDr of how the actual attack works.
  - tadkar 7y ago
    Basically use the web APIs to issue two spend requests almost simultaneously, and the fact that it takes time for the database to synchronize means you can double spend. As the posters above are saying, it’s very similar to a race condition.
  - dc_gregory 7y ago
    The technical TLDR is "db level race condition", or the title of the article.
- bouncycastle 7y ago
  So, umm... they re-discovered race conditions and gave it a fancy name?
  - homakov 7y ago
    Correct. “ACIDrain” is literally a race condition and made up name.
  - praptak 7y ago
    The value of tfa is the work they put into practical demonstration and analysis. Anybody can sit in their armchair and speculate about theoretical exploits. It's the actual practical exploits and mitigation techniques that matter.
C4stor 7y ago
TL;DR : Researchers found a way to analyze database logs to identify race conditions in databases, and demonstrated how once this knowledge acquired, it left websites open to be exploited through their public APIs.

homakov 7y ago

This is for anomaly detection based on reading logs. It may miss a vulnerability so you still need to test yourself. I have a Chrome extension which is perfect for race attacks. https://github.com/sakurity/racer https://github.com/sakurity/racer

squanch 7y ago
Thank you. This seems pretty neat!

sitkack 7y ago

This is awesome! This is another brick in the road to requiring formal methods. Hmm, typed multiple database-call transactions. Are there type safe database libraries for Idris or Rust?

pyrale 7y ago
Not what you ask, but squeal is a typed SQL wrapper for Haskell.
- LunaSea 7y ago
  It's really just an ORM if I'm looking at the project.
  - deleted 7y ago
    [deleted]
  - pyrale 7y ago
    The ORM part is a thin wrapping that is there to explain Haskell enough to have type-checking. The goal is really to have Haskell type-check the requests you make.
    - LunaSea 7y ago
      Yup and I think it's a noble goal but it's still implemented in a way where you're not writing SQL but some kind of other syntax. Also the issue is that it's trying to be database agnostic which means that you can't use any of the cool features from your database of choice.
namibj 7y ago
There is something based on Bloom IIRC, but the whole concept should be well-described and discoverable from the Keeping CALM paper: https://arxiv.org/abs/1901.01930 https://arxiv.org/abs/1901.01930

bouncycastle 7y ago

On page 11… "Developer's response" section, quote: "In contrast, the developer of OpenCart responded to the inventory vulnerability by posting a comment—“use your brain! its [sic] not hard to come up with a solution that does not involve coding!”—then closed both the inventory and voucher vulnerability issues and blocked us from responding. "

founderling 7y ago
The response seems shockingly arrogant and irresponsible at first. But he is in fact right. A good reminder that people who are unfriendly and argue against public opinion are not necessarily wrong. I don't know opencart, but I know how it is handled in companies using other onlineshops: When the stock of an item goes negative, it is noticed by staff when they want to fulfill the order. ("Strange, this customer ordered red socks, but we don't have any"). Staff sets the item as "Cannot be delivered" in the order. Which ups the stock to zero again. Customer order gets fulfilled with a note "Sorry, the red socks could not be delivered.". It's not perfect, but also not a big drama. Same with coupons. You should have them being invalidated after the order process. And if you see the same coupon code used twice, you know it is fraud.
- dunkelheit 7y ago
  The response is arrogant and irresponsible and not necessarily right either. This attitude is precisely the reason why most software is shit that requires constant manual babysitting. Bugs like this are no big deal until they are - even something insignificant as an item missing from the inventory can become a source of drama if you promised exactly this item to your child as a christmas present.
  - founderling 7y ago
    Items cannot be shipped sometimes. In real life, that happens way way way more often due to other circumstances then due to race conditions. Staff breaks the last item. Manufacturer sent a box with 100 items but it only contained 99. The color of the shelf stained the last item. Etc etc etc. Think about it: Depending on the shop software, two orders need to be placed within a few milliseconds. Containing the same item. And the item has to be very low on stock so the two orders cannot be fulfilled. That is a very very rare event. It's almost a theoretical event only. I have seen a bunch of companies grow an online business. There were tons of pain points. This race condition was none of them.

saurabhnanda 7y ago

Does this impact postgres backed applications?

dunkelheit 7y ago
Yes! The default isolation level in postgres is READ COMMITTED which is rather weak and allows for the types of concurrency bugs discussed in the article (unless you use SELECT FOR UPDATE or some other kind of locking). https://www.postgresql.org/docs/11/transaction-iso.html https://www.postgresql.org/docs/11/transaction-iso.html

timvisee 7y ago

Along with transactions, I also like to use 'Add' or 'Subtract' operations which usually provide better reliability instead of manually reading and setting, being a source for data races.

varelaz 7y ago
You may end up with 0xFF items on stock, add/subtract are cool, but not ideal, also negative number is not always more acceptable than exception. I recall FF bug with Civilization https://www.geek.com/games/why-gandhi-is-always-a-warmongering-jerk-in-civilization-1608515/ https://www.geek.com/games/why-gandhi-is-always-a-warmongeri...

sara7262 7y ago

See the video of most dangerous bird and how he killed owner https://howto105.blogspot.com/2019/04/most-dangerous-bird-killed-owner-in-usa.html https://howto105.blogspot.com/2019/04/most-dangerous-bird-ki... Funny dog driving own car See his video on link below http://bit.ly/2WdSKpA http://bit.ly/2WdSKpA Cute dog going for school like human baby see his cute video I am sure you can't stop laugh http://bit.ly/2Wq6Ynr http://bit.ly/2Wq6Ynr United nation recently showed the real face of india See what they said in report After knowing this I am sure you will hate them if you have soft heart https://youtu.be/oUskNtdWeIY https://youtu.be/oUskNtdWeIY Best way to get rid from face moles with the help of home remedies its easy and simple way You don't need to waste money for this https://zoomtips.blogspot.com/2019/05/easily-get-rid-from-face-moles-with.html https://zoomtips.blogspot.com/2019/05/easily-get-rid-from-fa... Girl made a strange world record even you can't believe this see her video on link below http://bit.ly/2ErUDEW http://bit.ly/2ErUDEW Trump is useful now lol See in video what China did with trump http://bit.ly/2VPVZis http://bit.ly/2VPVZis Lose weight without exercise Now weight losing is not hard job just follow this simple method https://rebrand.ly/loss-weight https://rebrand.ly/loss-weight Now don't worry about dry hair or dryness on scalp Just follow this simple methods which you can make with simple home remedies and get rid from hair dryness https://zoomtips.blogspot.com/2019/05/treatment-for-dry-hair-and-scalp.html https://zoomtips.blogspot.com/2019/05/treatment-for-dry-hair... This video is important for Facebook users must watch this otherwise you will loss your Facebook id https://rebrand.ly/Facebook-id-danger https://rebrand.ly/Facebook-id-danger

aneutron 7y ago

Very interesting. I wonder if they implemented the nested transactions aspect of 2AD (paper is dated at 2017, and they said they were working on it). That would make it extremely useful in pentesting.

varelaz 7y ago

While this is cool to harden your transactions, as more locks you add as more deadlocks and blocks appear, and it becomes harder and harder to keep transactions fast and isolated from each other. For example if you do select for update and record is not found, DB should lock key space to prevent adding such records, did you suppose to do this and block other inserts/updates? It's often easy to find problem with transactions in code, but very hard to fix all consequences. By the way because of this instead of select for update for single record I prefer update with no change. If there is no record, nothing will be locked.

hobs 7y ago
Blocking phantom reads/writes is usually only that of a serializeable transaction isolation level - and a range lock shouldnt block other writes, unless they are of the exact range specified, and then you are asking for a consistent view of reality. Generally I dont find adding more locks hurts things that are well designed - often you will find deadlocks turn into just waiting for your turn in line - and then you can tune your db code or your hardware or your whatever to maximize that throughput. Also fwiw if you are doing any sort of upsert and you are not locking for the update/insert combo, then you are definitely opening the door to a duplicate key insert - as long as you handle that though, nbd.

he0001 7y ago

Correct me if I’m wrong here, but wouldn’t these attacks be translatable to anywhere there are read/write discrepancies?

lgats 7y ago

Mobile Friendly View: http://pdf.cab/openaccess.thecvf.com/content_ECCV_2018/papers/Mikael_Persson_Lambda_Twist_An_ECCV_2018_paper.pdf http://pdf.cab/openaccess.thecvf.com/content_ECCV_2018/paper...