12 ms·

OWASP Rabbit Hole

269 points by mdhb 4y ago

hislaziness 4y ago
Definitely Stripe should have done a better job here. However, also points to the risk of allowing service providers to handle your security. Maybe there should have been a back test and or a "maker" "checker" control for pushing new security rules.
- dncornholio 4y ago
  Cloudflare is blocking the shit out of legitimate requests, how is this Stripes fault? Putting the webhook callback behind CloudFlare was the mistake IMO.
  - bouke 4y ago
    Indeed, in the end Troy disabled a certain rule for incoming requests from Stripe but not all. Just before publishing the article he discovered more (webhook) requests being blocked by a different rule. This seems like a continuous source of issues where valid (webhook) requests are being (arbitrarily) blocked. Why wouldn't you just disable all rules for requests coming from Stripe?
  - michaelt 4y ago
    Yes - Stripe is even nice enough to provide a list of IP addresses their webhook requests could come from [1] And Cloudflare is well known for randomly throwing up "Checking your browser before accessing example.com" interstitials and captchas and suchlike - of course you don't want bot detection on your callback API. [1] https://stripe.com/docs/ips?locale=en-GB#webhook-notifications https://stripe.com/docs/ips?locale=en-GB#webhook-notificatio...
    - a2tech 4y ago
      A customer I work with has a large Drupal site hosted with a big time Drupal provider. Cloudflare has decided that image uploads from the browser via the JS uploader needs to be checked and injects itself into that...but the browser has no way to display it so the page just hangs. Its awful.
technion 4y ago
There are multiple places where he uses a feature only available to enterprise users (read: Not me) to resolve this. I guess the "normal" thing here would be to disable the WAF and be done with it.
- raverbashing 4y ago
  Yeah, while I would have tried to investigate this a bit, this seems to be an issue on Cloudflare Bypassing the rules is a workaround, but not a fix
- e1g 4y ago
  This jumped out at me too. Cloudflare keeps many diagnostic and debugging tools available only on the Enterprise plan (typically >$3k per month). When something like this happens to most small teams and startups, they are working in the dark even on the Business plan at $200/mo.
  - tills13 4y ago
    FWIW CloudFlare has worked with us time and time again to get pricing in line with our expectations. We have numerous "Business Plus" or "Free Plus" plans across various accounts and zones that include features that are, as listed in the marketing, only available to Enterprise. Just reach out to someone at Cloudflare. I'm sure they'd love your business.
- a2tech 4y ago
  And if you have to disable WAF, what exactly is Cloudflare doing for you?
  - mpalfrey 4y ago
    Would it still do DDOS even without the WAF?
    - mstrem 4y ago
      Our DDoS mitigation is separate yes and will still work.
yellow_lead 4y ago
WAF blocking things randomly isn't a good look for Cloudflare. I wonder how many outages this has caused for their customers.
- AlfeG 4y ago
  From my experience randomly blocking rules in WAF is essentialy a feature of the WAF. I have random issues with WAF all over the places from random providers on different projects. And yep. Disabling WAF were always a good enough solution untill we figure out which of the request parts are triggering rules
jamespo 4y ago
Running a WAF with a dynamic ruleset between you and your payment provider seems a bit risky to me.
- danuker 4y ago
  Using a third-party's WAF and trusting it will not log your data also shows some risk. By centralizing all data going through the web you paint a massive target on yourself (for black hats).
ricardo81 4y ago
Came across similar situations before [beyond previous issues with being a trusted bot by Cloudflare] The destination website user was wondering why our service didn't visit their website and why we were receiving 503s in return, with CF in the middle. Turns out in this case it was an additional bot blocking service they'd installed on their own server, which I'd presume is hard for them to debug if the request never reaches them (but in this case it did, just added confusion for them). Far from ideal to have a middle man arbitrate a web request or decide what is trusted. On the flip side, lots of rogue traffic that should be blocked. Think in some cases the WAFs are just a bit too aggressive with the grey area and UAs they don't know. It creates a barrier of entry to new players.
arpa 4y ago
It's like complex systems have complex failure modes, who'd thunk?
Liquidor 4y ago
I'm not expert and this was really nice and easy to read and understand. Thanks for sharing Troy. Question: Should we even use Cloudflare for server-server communication such as webhooks, API calls etc?
- jacquesm 4y ago
  If your API calls are so generic that you don't need to identify the caller using an API key or some other token then you might be able to get some mileage out of it. But personally I'd like all API calls to go to my servers and not someone else's. If only because I don't like 3rd parties having the ability to log the payloads of requests, and whoever terminates your SSL connections has the ability.
berkle4455 4y ago
Damn this is absurd. Is there not a way on Clouldflare to flag webhooks with a "if request contains this api key, let it through, block the rest"?
- a2tech 4y ago
  You can create custom rulesets, which is what the article's author did. This allowed the API calls to essentially bypass the WAF.
  - blibble 4y ago
    just remember sometimes cloudflare add new features which aren't controlled by existing rules "security off for this ip?" "I guess you don't mean our new shiny feature!"
- tyingq 4y ago
  It looks like "WAF - add exceptions" supports Wireshark style expressions, so there's things like: (http.request.uri.query contains "some-string") Or similar for checking headers, post body content, etc.
a2tech 4y ago
This boils down to 'Cloudflare did something' and without the Enterprise plan you'd never be able to pull the data required to diagnose the problem. Oh also, Cloudflare knows they did something but plays the blame game until you get to someone that openly acknowledges that they know something is broken. I've said it before and I'll say it again--Cloudflare is not making the Internet safer, its just making it less open. I know that is not the overwhelming sentiment of HN and me complaining about it isn't going to change anyones mind.
- cuuupid 4y ago
  As a bit of a meta point it’s a bit astonishing how good Cloudflare is at PR and community management. The sentiment I’ve seen here in the past is overwhelmingly pro-Cloudflare. Every now and then they’ll publish a hit piece on AWS here and there’ll be an AWS hate thread, but no one seems to question Cloudflare’s motives for promoting bad press about their competitor directly to their target market. Then again in most things, the underdog trying to upstage the incumbent is always a popular narrative. You’re right that talking about it likely isn’t going to change any sentiment.
  - throwaway2847 4y ago
    HN is like any other community and is thoroughly captured by some hype machines (Cloudflare, the language that shall not be named) and virulently opposed to others (cryptocurrency)
    - jcuenod 4y ago
      What are you talking about? Java doesn't have a hype machine...
  - groffee 4y ago
    > it’s a bit astonishing how good Cloudflare is at PR and community management It's really not, look at any cloudflare support thread here and how they crawl out of the woodwork for damage control because they failed in basic customer service it's a huge red flag, and everyone in the comments always sees it. cloudflare is cancer and needs to die.
tiffanyh 4y ago
> About Page: “I'm Troy Hunt, an Australian Microsoft Regional Director and Microsoft Most Valuable Professional for Developer Security. I don't work for Microsoft, but they're kind enough to recognise my community contributions by way of their award programs which I've been a part of since 2011.” Can someone explain what this Microsoft Regional Director role is because it sounds like he works for Microsoft but then says he does not.
- mpalfrey 4y ago
  Microsoft Regional Director is effectively someone who is an advisor to Microsoft - https://rd.microsoft.com/en-us/about/ https://rd.microsoft.com/en-us/about/ .
  - philipwhiuk 4y ago
    That's a really stupid choice of title for essentially a "Key Partner"
danpalmer 4y ago
See also: Cloudflare in front of Mastodon. A lot of naive Mastodon admins are putting default Cloudflare configurations in front of their instances. The problem is that inter-instance requests necessary for federation then get caught as bots (because they essentially are) and connections in the network degrade, causing eventual de-federation. Cloudflare is magic, in the good and bad ways. I'd only use it for very small personal sites or things that can afford downtime and don't need integrations, or for large businesses that can afford the enterprise plans and will actively manage the account and respond to business and tech needs with config changes.
- pbronez 4y ago
  I wonder if their Wildebeast ActivityPub implementation will have similar issues.
- worldofmatthew 4y ago
  "very small personal sites" don't need Cloudflare.
  - Dylan16807 4y ago
    Any site should be able to survive DDOS.
    - worldofmatthew 4y ago
      Most small websites will never get attacked.
    - BeFlatXIII 4y ago
      With which strategies would "any site" survive DDoS without paying Cloudflare or some other middleman?
      - Dylan16807 4y ago
        That was my point.
mpalfrey 4y ago
I wonder what is actually causing the error here. Looking at the 2 payloads diff on the site I can't spot anything. It's a weird one that's for sure.
- tyingq 4y ago
  Something real subtle probably, like enough of the redacted areas accidentally containing something like looks like SQL. Like WALTER contains ALTER, CASEY contains CASE, etc...enough to add up the score to fail. Hopefully more convoluted than simple substring matches like that, but I've seen WAFS do that. If it is adding up substring matches, there is stuff like "subscription_update", "null", "created", "discountable", "total_count", etc, that would set a high floor.
aaroncloud 4y ago
I have used a few different WAFs to date, AWS WAF, CloudFlare WAF and Akamai WAF. I have had issues with all but usually the frequent issue is an unexpected blocking of request. Would be interesting to benchmark a set of WAFs to see which ones block threats vs which block valid traffic.
- tyingq 4y ago
  A funny one that comes up is many WAFs either block, or only examine the first X bytes of large-sized post requests, for performance reasons. So your choice of unpredictable lost requests, or a loophole where the WAF is blind.
talkingtab 4y ago
I wonder what the take away from this is? The simple one is "bad cloudflare" or "bad stripe" or even "bad hibp". Or maybe all in conjunction. Or maybe none. But that seems simplistic to me. The smell of this is a system that is so poorly made that it has layer upon layer of obscure hacks to protect it. It appears that no one can understand why this happened and the best guess is that it had something legitimate that was misunderstood. Maybe the word "alter" and "table"? This is the equivalent of you walking into a bank, telling the person "Hi my name is Rob and I came to the bank today to ..." And then the bank goes into automatic shut down. This is broken. IMHO.
- jsnell 4y ago
  From the information given, bad Cloudflare. These kinds of content-matching rules should be triggering deterministically, and testable in a hermetic test environment. They also have sample payloads that get blocked vs. ones that gets through, despite being essentially identical. It should be about as easy to debug as it gets. That it's tricky to debug suggests there's something totally different just badly understood rules. Maybe a server with a hardware fault that's making it return bogus results (though that should be easy to find in monitoring), maybe some kind of race condition, or running of different rules in parallel + having global or request-scoped state such that the order in which the rules finish running matters.
perfecto_maduro 4y ago
We whitelisted the stripe IPs completely after getting burned once. If Stripe gets hacked so that the hackers jump off to our site, we have far bigger problems to worry about.
- danpalmer 4y ago
  It seems like Cloudflare should be doing this for you. It wouldn't be hard for them to keep a list of IPs from common known-good integrations. They could prompt on first hit to ask you if you want to allow-list those companies, or even just do it by default.
  - JohnFen 4y ago
    Can you imagine the firestorm that would happen if CF was found to be allowing traffic from certain other entities, no matter how trustworthy they're perceived to be, to bypass security controls by default? And the firestorm would be entirely warranted.
    - danpalmer 4y ago
      I see your point, but I think there is a version of this that would be fine. I imagine these rules would be shown in the product so users can see the configuration and override it, perhaps it's an explicit opt-in, perhaps there's a public application process for inclusion or some sort of stated conditions so that it's not seen as too political which services are chosen... there are lots of options.
      - JohnFen 4y ago
        I agree. I was really responding to having such a pass be enabled by default. The defaults for any security system should be toward maximal security. Users being able to loosen the rules to fit their situation is fine and desirable.
eis 4y ago
Ignoring the question of who is actually doing something wrong and the fact that most customers on lower tiers wouldn't even be able to get most of the presented information, it seems absurd to me that a customer on an Enterprise plan that costs thousands of dollars a month can't get a simple "this is why those requests trigger the firewall" answer from CloudFlare. The diff between the request that passed and the one that triggered the rule is simple enough and the WAF is not some opaque AI I hope. Why can't the triggering request be replayed through the WAF with output that shows the scores for each bit?
Perseids 4y ago
What I find bleak about the situation, is that it is a glaring design-fail, even though everyone involved should have the necessary expertise to do better. A callback from your payment provider should never go through a best-effort WAF. Instead, as you already have a strong business relationship, you could easily exchange/store/configure strong credentials with stripe [1]. When even a security professional doesn't do that, what does it say about the state of documentation of this feature? Looking at the documentation directly, what they advise you to do is kind of the worst idea they could come up with: https://stripe.com/docs/webhooks/signatures https://stripe.com/docs/webhooks/signatures – you need custom logic[2] to verify that their MAC ("signature" they call it incorrectly) is valid and you need to configure a different secret for each of your endpoints. And then you still need to handle replay attacks somehow, which is its own nightmare to do correctly. It's no wonder the WAF can't do that for you. From a few years old personal experience, I'm really irritated by stripes web-hook approach overall. Payment process information is such a vital business concern that "let's try to call them and if that fails... well we tried" is broken on principal alone. The obvious approach is to have an event list which you as customer long-poll or just poll every few seconds if your framework doesn't support async well. This is also trivial to do securely: You're HTTPS library already authenticates stripe during the TLS handshake and that is all that is necessary. [1] Best case scenario: Let stripe authenticate with mutual TLS, but I know this is quite a long way away from typical web server configurations. [2] Stripe's approach very much reminds of DPoP https://news.ycombinator.com/item?id=31266575 https://news.ycombinator.com/item?id=31266575 which shall in now way be construed as a compliment.
- davedx 4y ago
  > The obvious approach is to have an event list which you as customer long-poll or just poll every few seconds This is also much much easier for the payments integrations developers to test, compared to all the messing about and running dodgy proxies that testing webhooks involves. Webhooks for critical application paths just seem like a bad idea all around really
  - bmizerany 4y ago
    > Webhooks for critical application paths just seem like a bad idea all around really Well, put. I've been deep in the Stripe API for a while now, and the invoices API provides up-to-date account status information and what a customer is paying for. This can be referenced as needed and, if desired, cached with some TTL and referenced as the "truth." Then webhooks can be viewed as a convenient way to bust the cache quicker than waiting for the TTL to expire. It could be better, but it provides the necessary information without relying on webhooks. An example of how we use this API as a form of entitlement checks at Tier can be found here: https://github.com/tierrun/tier/blob/f7c32426d30ca314706ca7e64399340f306f1d0b/control/usage.go#L89-L116 https://github.com/tierrun/tier/blob/f7c32426d30ca314706ca7e...
davedx 4y ago
"I look at the managed WAF Cloudflare provides more favourably than I did before simply because I have a better understanding of how comprehensive it is. I want to write code and run apps on the web, that's my focus, and I want someone else to provide that additional layer on top that continuously adapts to block new and emerging threats. I want to understand it (and I now do, at least certainly better than before), but I don't want managing it day in and day out to be my job." My takeaway from this is actually that you can't just have a managed WAF that gets out of your way. This looks like it took easily a day or more of work. What's the advantage of using a CloudFlare "managed" WAF versus running your own within AWS? I guess the "infrastructure" is managed, but the operations isn't... I'm a pretty experienced engineer, and I honestly don't know if I'd have been able to solve this issue personally. Most likely I would have just whitelisted all of Stripe's IP's, like the initial hotfix did.
- pixl97 4y ago
  Working around WAFs for years in enterprise support this kind of crap happens all the time. In my current work we'll have a client try to access our app in some way that oddly explodes. In general a browser HAR file is very useful. Then we have to check our app (hosted on the customer's servers), then we'd have to look at the load balancer, and when that doesn't bear results it's quite often we find a WAF in network path. Most of the time it's near impossible to find the team that manages it and then get helpful information out of them about the issue.
NovemberWhiskey 4y ago
This is absolutely not the first (or second) time I've seen an outage triggered by a well-meaning security rules update on a WAF. To be honest, a lot of security-related deployment processes would be regarded as unacceptable, wild-west level shit if they occurred in the software lifecycle - like difficulty to identify that a change had even occurred, inability to see before/after for the change, release processes effected manually via consoles, change deployed directly to production without going through a lower environment, and big-banged as opposed to canaried etc. etc.
ufmace 4y ago
My first thought is - why is this traffic going through CloudFlare at all? There's no caching benefit and it's all going to be coming from a datacenter anyways. Maybe it's more trouble than it's worth to bother setting up a separate non-CDNed DNS for API routes for this particular site. But then how much time was spent trying to sort out why those requests were being blocked?
- wlonkly 4y ago
  Cloudflare blocks denial of service attacks, too, which is probably important for tools in the security space, and blocking denial of service attacks means making sure that attackers don't know how to bypass Cloudflare and hit your origin directly. If you let things go direct to the origin, then you're giving away information about where the origin is, even if the things are only Stripe.
  - ufmace 4y ago
    That's true, but it seems pretty low-risk to me, since only Stripe and maybe a few other services you're integrating with actually know the DNS name. It's possible someone could discover it and attack it, but I think I'd wait to see an attack actually happen that couldn't practically be mitigated otherwise before moving it behind Cloudflare etc.
dbetteridge 4y ago
We had a very similar issue recently but with s3 presigned url uploads, cloudflare WAF and OWASP rules. Worked fine one day on 'high', next day started blocking random file uploads with a 'blocked by cloudflare page'.
cromulent 4y ago
From someone else in the article's comments: > if we treated the customer's phone number as a hex representation of ASCII, it spelled something that was recognisable as a command. And the WAF team suggested they ask the customer to change their phone number. Goodness me.
billpg 4y ago
Suppose I use Stripe in a similar way to HIBP and a customer comes along just as my server has gone down, would Stripe try again later? I think it should wait a little while but I'd hope it'd automatically try again later as long as there's data that needs to be sync'd but isn't yet. I can maybe imagine 403 being a "Don't try again" signal in a way that 500 or timed-out isn't.
- philipwhiuk 4y ago
  If it doesn't you can manually re-trigger it. That does require you to monitor it of course.
- __float 4y ago
  Yes, it retries for up to three days according to documentation: https://stripe.com/docs/webhooks/best-practices#retry-logic https://stripe.com/docs/webhooks/best-practices#retry-logic
dt3ft 4y ago
I had the same issue with one of my sites. I activated CF owasp ruleset and all hell broke lose (it took a while to backtrack, aka down the rabbit hole I went). I deactivated it and the problems remained. It took several hours before things were back to normal. All in all - CF is too big now, they want to have ALL the features, but the only feature I want is DDoS protection…
ghoshbishakh 4y ago
I solved the stripe webhook issue for https://pinggy.io https://pinggy.io by adding a refresh button that calls the stripe API to double check all subscriptions for a user.
zX41ZdbW 4y ago
Case 1. It blocks my SQL playground due to SQL injections: https://play.clickhouse.com/play?user=play https://play.clickhouse.com/play?user=play - solution: disable WAF. Case 2: It damages my presentations by removing whitespaces in HTML elements styled as "white-space: pre" at https://presentations.clickhouse.com/ https://presentations.clickhouse.com/ - solution: disable auto minification. Case 3: It makes the debian packages repository inconsistent https://packages.clickhouse.com/ https://packages.clickhouse.com/ - solution: disable caching. In fact, Cloudflare is an amazing service - it is powerful and easy to use, you only have to take care when enabling and disabling its features.
amluto 4y ago
I’ve generally thought that POST we hooks are the wrong abstraction for this type of payment confirmation, and reading this makes me believe it even more: This whole process needs something like a message queue. Stripe should publish and event to the queue, and HIBP should receive that event. A message queue is still subject to network failures, but any sensible implementation will notice and recover missed events.
- wlonkly 4y ago
  The queue/DLQ in this case is on Stripe's end. Once the WAF problem was fixed, Stripe ran the webhooks correctly.
  - amluto 4y ago
    With a real queue, dropping one message on the underlying transport will either be detected quickly and obviously on both ends, or will be recovered automatically, or both. The implementation could be as simple as a queue id and sequence number in the POST along with a way to request replay (via GET to Stripe) of old messages indexed by the same id and sequence number and a way to periodically ask for the latest sequence number.
mac-chaffee 4y ago
I've been running the OWASP coreruleset in production for about a year now and it has been a big pain. The way I made it manageable was 1) training users that "if you see a 403 error, tell me ASAP" and 2) learning the ModSecurity rule syntax to be able to create rule exceptions for users very quickly. This is not possible to do at Cloudflare's scale. Even then, users who didn't know the intricate details of the Web Application Firewall (like Troy in this case) would waste hours hunting down the issue. Since less popular sites often have more illegitimate traffic than legitimate traffic, there was really no good way for me to proactively fix WAF issues. The conclusion I have drawn is that WAFs really only have a few very narrow use-cases. The main use-case is when you want to write your own rule to protect hosts from a specific zero day while they are being patched. Like a simple rule to detect Log4J [1] was an effective band-aid while we scrambled to implement real patches. But WAFs have an inherent weakness: clever attackers can pretty much always circumvent rules, or force to you write a rule that is so complex that it causes slowness or blocks legitimate traffic. Another use-case is when you have to deploy some untrusted code that is likely vulnerable to common (>1 year old) vulnerabilities. Like running an old/archived wordpress instance. This is the only time when the coreruleset makes sense IMO. As I see it, WAFs are a tool created in a simpler time when the number of possible attacks and applications was small. In the modern era where there is a constant deluge of zero-days, huge attack surfaces, tons of variability in applications, and lots of sites where RCE/SQLi is a feature (think CI job definitions, Juptyer notebooks, custom query languages), WAFs have lost their effectiveness. [1]: https://github.com/coreruleset/coreruleset/issues/2331 https://github.com/coreruleset/coreruleset/issues/2331
lobocinza 4y ago
Cloudflare is a white elephant in my current org. We are probably paying a lot to them and we are having issues that would be easy for me to fix if I was running my own deployment anywhere else but the platform team doesn't seem to know how to config things or don't care about it. So I end up with complaints from the business team that I can't action upon because the setup is obscure and the few parts that I can see and manage are poorly documented and I'm not even sure which feature is supported by the company's plan. My org is at fault but I doubt this is an exception.
AtNightWeCode 4y ago
I guess things like this is why CF added the traffic sequence section in the portal. Another thing on the same theme is that the DDOS rules are the first in line...

269 points by mdhb 4y ago

Definitely Stripe should have done a better job here. However, also points to the risk of allowing service providers to handle your security. Maybe there should have been a back test and or a "maker" "checker" control for pushing new security rules.

dncornholio 4y ago
Cloudflare is blocking the shit out of legitimate requests, how is this Stripes fault? Putting the webhook callback behind CloudFlare was the mistake IMO.
- bouke 4y ago
  Indeed, in the end Troy disabled a certain rule for incoming requests from Stripe but not all. Just before publishing the article he discovered more (webhook) requests being blocked by a different rule. This seems like a continuous source of issues where valid (webhook) requests are being (arbitrarily) blocked. Why wouldn't you just disable all rules for requests coming from Stripe?
- michaelt 4y ago
  Yes - Stripe is even nice enough to provide a list of IP addresses their webhook requests could come from [1] And Cloudflare is well known for randomly throwing up "Checking your browser before accessing example.com" interstitials and captchas and suchlike - of course you don't want bot detection on your callback API. [1] https://stripe.com/docs/ips?locale=en-GB#webhook-notifications https://stripe.com/docs/ips?locale=en-GB#webhook-notificatio...
  - a2tech 4y ago
    A customer I work with has a large Drupal site hosted with a big time Drupal provider. Cloudflare has decided that image uploads from the browser via the JS uploader needs to be checked and injects itself into that...but the browser has no way to display it so the page just hangs. Its awful.

technion 4y ago

There are multiple places where he uses a feature only available to enterprise users (read: Not me) to resolve this. I guess the "normal" thing here would be to disable the WAF and be done with it.

raverbashing 4y ago
Yeah, while I would have tried to investigate this a bit, this seems to be an issue on Cloudflare Bypassing the rules is a workaround, but not a fix
e1g 4y ago
This jumped out at me too. Cloudflare keeps many diagnostic and debugging tools available only on the Enterprise plan (typically >$3k per month). When something like this happens to most small teams and startups, they are working in the dark even on the Business plan at $200/mo.
- tills13 4y ago
  FWIW CloudFlare has worked with us time and time again to get pricing in line with our expectations. We have numerous "Business Plus" or "Free Plus" plans across various accounts and zones that include features that are, as listed in the marketing, only available to Enterprise. Just reach out to someone at Cloudflare. I'm sure they'd love your business.
a2tech 4y ago
And if you have to disable WAF, what exactly is Cloudflare doing for you?
- mpalfrey 4y ago
  Would it still do DDOS even without the WAF?
  - mstrem 4y ago
    Our DDoS mitigation is separate yes and will still work.

yellow_lead 4y ago

WAF blocking things randomly isn't a good look for Cloudflare. I wonder how many outages this has caused for their customers.

AlfeG 4y ago
From my experience randomly blocking rules in WAF is essentialy a feature of the WAF. I have random issues with WAF all over the places from random providers on different projects. And yep. Disabling WAF were always a good enough solution untill we figure out which of the request parts are triggering rules

jamespo 4y ago

Running a WAF with a dynamic ruleset between you and your payment provider seems a bit risky to me.

danuker 4y ago
Using a third-party's WAF and trusting it will not log your data also shows some risk. By centralizing all data going through the web you paint a massive target on yourself (for black hats).

ricardo81 4y ago

Came across similar situations before [beyond previous issues with being a trusted bot by Cloudflare] The destination website user was wondering why our service didn't visit their website and why we were receiving 503s in return, with CF in the middle. Turns out in this case it was an additional bot blocking service they'd installed on their own server, which I'd presume is hard for them to debug if the request never reaches them (but in this case it did, just added confusion for them). Far from ideal to have a middle man arbitrate a web request or decide what is trusted. On the flip side, lots of rogue traffic that should be blocked. Think in some cases the WAFs are just a bit too aggressive with the grey area and UAs they don't know. It creates a barrier of entry to new players.

arpa 4y ago

It's like complex systems have complex failure modes, who'd thunk?

Liquidor 4y ago

I'm not expert and this was really nice and easy to read and understand. Thanks for sharing Troy. Question: Should we even use Cloudflare for server-server communication such as webhooks, API calls etc?

jacquesm 4y ago
If your API calls are so generic that you don't need to identify the caller using an API key or some other token then you might be able to get some mileage out of it. But personally I'd like all API calls to go to my servers and not someone else's. If only because I don't like 3rd parties having the ability to log the payloads of requests, and whoever terminates your SSL connections has the ability.

berkle4455 4y ago

Damn this is absurd. Is there not a way on Clouldflare to flag webhooks with a "if request contains this api key, let it through, block the rest"?

a2tech 4y ago
You can create custom rulesets, which is what the article's author did. This allowed the API calls to essentially bypass the WAF.
- blibble 4y ago
  just remember sometimes cloudflare add new features which aren't controlled by existing rules "security off for this ip?" "I guess you don't mean our new shiny feature!"
tyingq 4y ago
It looks like "WAF - add exceptions" supports Wireshark style expressions, so there's things like: (http.request.uri.query contains "some-string") Or similar for checking headers, post body content, etc.

a2tech 4y ago

This boils down to 'Cloudflare did something' and without the Enterprise plan you'd never be able to pull the data required to diagnose the problem. Oh also, Cloudflare knows they did something but plays the blame game until you get to someone that openly acknowledges that they know something is broken. I've said it before and I'll say it again--Cloudflare is not making the Internet safer, its just making it less open. I know that is not the overwhelming sentiment of HN and me complaining about it isn't going to change anyones mind.

cuuupid 4y ago
As a bit of a meta point it’s a bit astonishing how good Cloudflare is at PR and community management. The sentiment I’ve seen here in the past is overwhelmingly pro-Cloudflare. Every now and then they’ll publish a hit piece on AWS here and there’ll be an AWS hate thread, but no one seems to question Cloudflare’s motives for promoting bad press about their competitor directly to their target market. Then again in most things, the underdog trying to upstage the incumbent is always a popular narrative. You’re right that talking about it likely isn’t going to change any sentiment.
- throwaway2847 4y ago
  HN is like any other community and is thoroughly captured by some hype machines (Cloudflare, the language that shall not be named) and virulently opposed to others (cryptocurrency)
  - jcuenod 4y ago
    What are you talking about? Java doesn't have a hype machine...
- groffee 4y ago
  > it’s a bit astonishing how good Cloudflare is at PR and community management It's really not, look at any cloudflare support thread here and how they crawl out of the woodwork for damage control because they failed in basic customer service it's a huge red flag, and everyone in the comments always sees it. cloudflare is cancer and needs to die.

tiffanyh 4y ago

> About Page: “I'm Troy Hunt, an Australian Microsoft Regional Director and Microsoft Most Valuable Professional for Developer Security. I don't work for Microsoft, but they're kind enough to recognise my community contributions by way of their award programs which I've been a part of since 2011.” Can someone explain what this Microsoft Regional Director role is because it sounds like he works for Microsoft but then says he does not.

mpalfrey 4y ago
Microsoft Regional Director is effectively someone who is an advisor to Microsoft - https://rd.microsoft.com/en-us/about/ https://rd.microsoft.com/en-us/about/ .
- philipwhiuk 4y ago
  That's a really stupid choice of title for essentially a "Key Partner"

danpalmer 4y ago

See also: Cloudflare in front of Mastodon. A lot of naive Mastodon admins are putting default Cloudflare configurations in front of their instances. The problem is that inter-instance requests necessary for federation then get caught as bots (because they essentially are) and connections in the network degrade, causing eventual de-federation. Cloudflare is magic, in the good and bad ways. I'd only use it for very small personal sites or things that can afford downtime and don't need integrations, or for large businesses that can afford the enterprise plans and will actively manage the account and respond to business and tech needs with config changes.

pbronez 4y ago
I wonder if their Wildebeast ActivityPub implementation will have similar issues.
worldofmatthew 4y ago
"very small personal sites" don't need Cloudflare.
- Dylan16807 4y ago
  Any site should be able to survive DDOS.
  - worldofmatthew 4y ago
    Most small websites will never get attacked.
  - BeFlatXIII 4y ago
    With which strategies would "any site" survive DDoS without paying Cloudflare or some other middleman?
    - Dylan16807 4y ago
      That was my point.

mpalfrey 4y ago

I wonder what is actually causing the error here. Looking at the 2 payloads diff on the site I can't spot anything. It's a weird one that's for sure.

tyingq 4y ago
Something real subtle probably, like enough of the redacted areas accidentally containing something like looks like SQL. Like WALTER contains ALTER, CASEY contains CASE, etc...enough to add up the score to fail. Hopefully more convoluted than simple substring matches like that, but I've seen WAFS do that. If it is adding up substring matches, there is stuff like "subscription_update", "null", "created", "discountable", "total_count", etc, that would set a high floor.

aaroncloud 4y ago

I have used a few different WAFs to date, AWS WAF, CloudFlare WAF and Akamai WAF. I have had issues with all but usually the frequent issue is an unexpected blocking of request. Would be interesting to benchmark a set of WAFs to see which ones block threats vs which block valid traffic.

tyingq 4y ago
A funny one that comes up is many WAFs either block, or only examine the first X bytes of large-sized post requests, for performance reasons. So your choice of unpredictable lost requests, or a loophole where the WAF is blind.

talkingtab 4y ago

I wonder what the take away from this is? The simple one is "bad cloudflare" or "bad stripe" or even "bad hibp". Or maybe all in conjunction. Or maybe none. But that seems simplistic to me. The smell of this is a system that is so poorly made that it has layer upon layer of obscure hacks to protect it. It appears that no one can understand why this happened and the best guess is that it had something legitimate that was misunderstood. Maybe the word "alter" and "table"? This is the equivalent of you walking into a bank, telling the person "Hi my name is Rob and I came to the bank today to ..." And then the bank goes into automatic shut down. This is broken. IMHO.

jsnell 4y ago
From the information given, bad Cloudflare. These kinds of content-matching rules should be triggering deterministically, and testable in a hermetic test environment. They also have sample payloads that get blocked vs. ones that gets through, despite being essentially identical. It should be about as easy to debug as it gets. That it's tricky to debug suggests there's something totally different just badly understood rules. Maybe a server with a hardware fault that's making it return bogus results (though that should be easy to find in monitoring), maybe some kind of race condition, or running of different rules in parallel + having global or request-scoped state such that the order in which the rules finish running matters.

perfecto_maduro 4y ago

We whitelisted the stripe IPs completely after getting burned once. If Stripe gets hacked so that the hackers jump off to our site, we have far bigger problems to worry about.

danpalmer 4y ago
It seems like Cloudflare should be doing this for you. It wouldn't be hard for them to keep a list of IPs from common known-good integrations. They could prompt on first hit to ask you if you want to allow-list those companies, or even just do it by default.
- JohnFen 4y ago
  Can you imagine the firestorm that would happen if CF was found to be allowing traffic from certain other entities, no matter how trustworthy they're perceived to be, to bypass security controls by default? And the firestorm would be entirely warranted.
  - danpalmer 4y ago
    I see your point, but I think there is a version of this that would be fine. I imagine these rules would be shown in the product so users can see the configuration and override it, perhaps it's an explicit opt-in, perhaps there's a public application process for inclusion or some sort of stated conditions so that it's not seen as too political which services are chosen... there are lots of options.
    - JohnFen 4y ago
      I agree. I was really responding to having such a pass be enabled by default. The defaults for any security system should be toward maximal security. Users being able to loosen the rules to fit their situation is fine and desirable.

eis 4y ago

Ignoring the question of who is actually doing something wrong and the fact that most customers on lower tiers wouldn't even be able to get most of the presented information, it seems absurd to me that a customer on an Enterprise plan that costs thousands of dollars a month can't get a simple "this is why those requests trigger the firewall" answer from CloudFlare. The diff between the request that passed and the one that triggered the rule is simple enough and the WAF is not some opaque AI I hope. Why can't the triggering request be replayed through the WAF with output that shows the scores for each bit?

Perseids 4y ago

What I find bleak about the situation, is that it is a glaring design-fail, even though everyone involved should have the necessary expertise to do better. A callback from your payment provider should never go through a best-effort WAF. Instead, as you already have a strong business relationship, you could easily exchange/store/configure strong credentials with stripe [1]. When even a security professional doesn't do that, what does it say about the state of documentation of this feature? Looking at the documentation directly, what they advise you to do is kind of the worst idea they could come up with: https://stripe.com/docs/webhooks/signatures https://stripe.com/docs/webhooks/signatures – you need custom logic[2] to verify that their MAC ("signature" they call it incorrectly) is valid and you need to configure a different secret for each of your endpoints. And then you still need to handle replay attacks somehow, which is its own nightmare to do correctly. It's no wonder the WAF can't do that for you. From a few years old personal experience, I'm really irritated by stripes web-hook approach overall. Payment process information is such a vital business concern that "let's try to call them and if that fails... well we tried" is broken on principal alone. The obvious approach is to have an event list which you as customer long-poll or just poll every few seconds if your framework doesn't support async well. This is also trivial to do securely: You're HTTPS library already authenticates stripe during the TLS handshake and that is all that is necessary. [1] Best case scenario: Let stripe authenticate with mutual TLS, but I know this is quite a long way away from typical web server configurations. [2] Stripe's approach very much reminds of DPoP https://news.ycombinator.com/item?id=31266575 https://news.ycombinator.com/item?id=31266575 which shall in now way be construed as a compliment.

davedx 4y ago
> The obvious approach is to have an event list which you as customer long-poll or just poll every few seconds This is also much much easier for the payments integrations developers to test, compared to all the messing about and running dodgy proxies that testing webhooks involves. Webhooks for critical application paths just seem like a bad idea all around really
- bmizerany 4y ago
  > Webhooks for critical application paths just seem like a bad idea all around really Well, put. I've been deep in the Stripe API for a while now, and the invoices API provides up-to-date account status information and what a customer is paying for. This can be referenced as needed and, if desired, cached with some TTL and referenced as the "truth." Then webhooks can be viewed as a convenient way to bust the cache quicker than waiting for the TTL to expire. It could be better, but it provides the necessary information without relying on webhooks. An example of how we use this API as a form of entitlement checks at Tier can be found here: https://github.com/tierrun/tier/blob/f7c32426d30ca314706ca7e64399340f306f1d0b/control/usage.go#L89-L116 https://github.com/tierrun/tier/blob/f7c32426d30ca314706ca7e...

davedx 4y ago

"I look at the managed WAF Cloudflare provides more favourably than I did before simply because I have a better understanding of how comprehensive it is. I want to write code and run apps on the web, that's my focus, and I want someone else to provide that additional layer on top that continuously adapts to block new and emerging threats. I want to understand it (and I now do, at least certainly better than before), but I don't want managing it day in and day out to be my job." My takeaway from this is actually that you can't just have a managed WAF that gets out of your way. This looks like it took easily a day or more of work. What's the advantage of using a CloudFlare "managed" WAF versus running your own within AWS? I guess the "infrastructure" is managed, but the operations isn't... I'm a pretty experienced engineer, and I honestly don't know if I'd have been able to solve this issue personally. Most likely I would have just whitelisted all of Stripe's IP's, like the initial hotfix did.

pixl97 4y ago
Working around WAFs for years in enterprise support this kind of crap happens all the time. In my current work we'll have a client try to access our app in some way that oddly explodes. In general a browser HAR file is very useful. Then we have to check our app (hosted on the customer's servers), then we'd have to look at the load balancer, and when that doesn't bear results it's quite often we find a WAF in network path. Most of the time it's near impossible to find the team that manages it and then get helpful information out of them about the issue.

NovemberWhiskey 4y ago

This is absolutely not the first (or second) time I've seen an outage triggered by a well-meaning security rules update on a WAF. To be honest, a lot of security-related deployment processes would be regarded as unacceptable, wild-west level shit if they occurred in the software lifecycle - like difficulty to identify that a change had even occurred, inability to see before/after for the change, release processes effected manually via consoles, change deployed directly to production without going through a lower environment, and big-banged as opposed to canaried etc. etc.

ufmace 4y ago

My first thought is - why is this traffic going through CloudFlare at all? There's no caching benefit and it's all going to be coming from a datacenter anyways. Maybe it's more trouble than it's worth to bother setting up a separate non-CDNed DNS for API routes for this particular site. But then how much time was spent trying to sort out why those requests were being blocked?

wlonkly 4y ago
Cloudflare blocks denial of service attacks, too, which is probably important for tools in the security space, and blocking denial of service attacks means making sure that attackers don't know how to bypass Cloudflare and hit your origin directly. If you let things go direct to the origin, then you're giving away information about where the origin is, even if the things are only Stripe.
- ufmace 4y ago
  That's true, but it seems pretty low-risk to me, since only Stripe and maybe a few other services you're integrating with actually know the DNS name. It's possible someone could discover it and attack it, but I think I'd wait to see an attack actually happen that couldn't practically be mitigated otherwise before moving it behind Cloudflare etc.

dbetteridge 4y ago

We had a very similar issue recently but with s3 presigned url uploads, cloudflare WAF and OWASP rules. Worked fine one day on 'high', next day started blocking random file uploads with a 'blocked by cloudflare page'.

cromulent 4y ago

From someone else in the article's comments: > if we treated the customer's phone number as a hex representation of ASCII, it spelled something that was recognisable as a command. And the WAF team suggested they ask the customer to change their phone number. Goodness me.

billpg 4y ago

Suppose I use Stripe in a similar way to HIBP and a customer comes along just as my server has gone down, would Stripe try again later? I think it should wait a little while but I'd hope it'd automatically try again later as long as there's data that needs to be sync'd but isn't yet. I can maybe imagine 403 being a "Don't try again" signal in a way that 500 or timed-out isn't.

philipwhiuk 4y ago
If it doesn't you can manually re-trigger it. That does require you to monitor it of course.
__float 4y ago
Yes, it retries for up to three days according to documentation: https://stripe.com/docs/webhooks/best-practices#retry-logic https://stripe.com/docs/webhooks/best-practices#retry-logic

dt3ft 4y ago

I had the same issue with one of my sites. I activated CF owasp ruleset and all hell broke lose (it took a while to backtrack, aka down the rabbit hole I went). I deactivated it and the problems remained. It took several hours before things were back to normal. All in all - CF is too big now, they want to have ALL the features, but the only feature I want is DDoS protection…

ghoshbishakh 4y ago

I solved the stripe webhook issue for https://pinggy.io https://pinggy.io by adding a refresh button that calls the stripe API to double check all subscriptions for a user.

zX41ZdbW 4y ago

Case 1. It blocks my SQL playground due to SQL injections: https://play.clickhouse.com/play?user=play https://play.clickhouse.com/play?user=play - solution: disable WAF. Case 2: It damages my presentations by removing whitespaces in HTML elements styled as "white-space: pre" at https://presentations.clickhouse.com/ https://presentations.clickhouse.com/ - solution: disable auto minification. Case 3: It makes the debian packages repository inconsistent https://packages.clickhouse.com/ https://packages.clickhouse.com/ - solution: disable caching. In fact, Cloudflare is an amazing service - it is powerful and easy to use, you only have to take care when enabling and disabling its features.

amluto 4y ago

I’ve generally thought that POST we hooks are the wrong abstraction for this type of payment confirmation, and reading this makes me believe it even more: This whole process needs something like a message queue. Stripe should publish and event to the queue, and HIBP should receive that event. A message queue is still subject to network failures, but any sensible implementation will notice and recover missed events.

wlonkly 4y ago
The queue/DLQ in this case is on Stripe's end. Once the WAF problem was fixed, Stripe ran the webhooks correctly.
- amluto 4y ago
  With a real queue, dropping one message on the underlying transport will either be detected quickly and obviously on both ends, or will be recovered automatically, or both. The implementation could be as simple as a queue id and sequence number in the POST along with a way to request replay (via GET to Stripe) of old messages indexed by the same id and sequence number and a way to periodically ask for the latest sequence number.

mac-chaffee 4y ago

I've been running the OWASP coreruleset in production for about a year now and it has been a big pain. The way I made it manageable was 1) training users that "if you see a 403 error, tell me ASAP" and 2) learning the ModSecurity rule syntax to be able to create rule exceptions for users very quickly. This is not possible to do at Cloudflare's scale. Even then, users who didn't know the intricate details of the Web Application Firewall (like Troy in this case) would waste hours hunting down the issue. Since less popular sites often have more illegitimate traffic than legitimate traffic, there was really no good way for me to proactively fix WAF issues. The conclusion I have drawn is that WAFs really only have a few very narrow use-cases. The main use-case is when you want to write your own rule to protect hosts from a specific zero day while they are being patched. Like a simple rule to detect Log4J [1] was an effective band-aid while we scrambled to implement real patches. But WAFs have an inherent weakness: clever attackers can pretty much always circumvent rules, or force to you write a rule that is so complex that it causes slowness or blocks legitimate traffic. Another use-case is when you have to deploy some untrusted code that is likely vulnerable to common (>1 year old) vulnerabilities. Like running an old/archived wordpress instance. This is the only time when the coreruleset makes sense IMO. As I see it, WAFs are a tool created in a simpler time when the number of possible attacks and applications was small. In the modern era where there is a constant deluge of zero-days, huge attack surfaces, tons of variability in applications, and lots of sites where RCE/SQLi is a feature (think CI job definitions, Juptyer notebooks, custom query languages), WAFs have lost their effectiveness. [1]: https://github.com/coreruleset/coreruleset/issues/2331 https://github.com/coreruleset/coreruleset/issues/2331

lobocinza 4y ago

Cloudflare is a white elephant in my current org. We are probably paying a lot to them and we are having issues that would be easy for me to fix if I was running my own deployment anywhere else but the platform team doesn't seem to know how to config things or don't care about it. So I end up with complaints from the business team that I can't action upon because the setup is obscure and the few parts that I can see and manage are poorly documented and I'm not even sure which feature is supported by the company's plan. My org is at fault but I doubt this is an exception.

AtNightWeCode 4y ago

I guess things like this is why CF added the traffic sequence section in the portal. Another thing on the same theme is that the DDOS rules are the first in line...