19 ms·
The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai
by matt_heimer 1y ago
The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem.
If your site discusses databases then turning on the default SQL injection attack prevention rules will break your site. And there is another ruleset for file inclusion where things like /etc/hosts and /etc/passwd get blocked.
I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts.
Fine tuning the rules is time consuming. You often have to just completely turn off the ruleset because when you try to keep the ruleset on and allow the use-case there are a ton of changes you need to get implemented (if its even possible). Page won't load because /etc/hosts was in a query param? Okay, now that you've fixed that, all the XHR included resources won't load because /etc/hosts is included in the referrer. Now that that's fixed things still won't work because some random JS analytics lib put the URL visited in a cookie, etc, etc... There is a temptation to just turn the rules off.
- _blk 1y ago100! [good] security just doesn't work as a mixing pattern... I'm not saying it's necessarily bad to use those additional protections but they come with severe limitations so the total value (as in cost/benefit) is hard to gauge.
- gfiorav 1y agoI agree. From a product perspective, I would also support the decision. Should we make the rules more complex by default, potentially overlooking SQL injection vulnerabilities? Or should we blanket prohibit anything that even remotely resembles SQL, allowing those edge cases to figure it out? I favor the latter approach. That group of Cloudflare users will understand the complexity of their use case accepting SQL in payloads and will be well-positioned to modify the default rules. They will know exactly where they want to allow SQL usage. From Cloudflare’s perspective, it is virtually impossible to reliably cover every conceivable valid use of SQL, and it is likely 99% of websites won’t host SQL content.
- Y_Y 1y agoWhy not just whitelist the thousand most common words? That should be good enough for 99% of approriate content, and the smelly nerds who make websites or talk about them can take their tiny market segment and get bent.
- krferriter 1y agoIf your web application is relying on Cloudflare filtration of input values to prevent SQL injection, your web application is vulnerable to SQL injection.
- p_ing 1y agoDefense in-depth. I would hope few would want a vulnerable web app and simply protect it via a WAF. But just because your web app is 'invulnerable' doesn't mean you should forgo the WAF.
- krferriter 1y agoBut what is being defended against? This is blocking legitimate user behavior. Would it be defense in depth to also prohibit semicolons or two consecutive hyphen characters in all content? If your app is constructing paths to read from the server's filesystem based on substrings contained within client-provided field values, throwing an error if `"/etc/hosts"` appears in any input is not going to save you.
- p_ing 1y agoUnknown or unforeseen attacks. The WAF ruleset can be updated much faster than code. WAFs also provide flexibility in how requests are responded to, or even disallow access from IP ranges, certain browsers, etc. WAFs do throw false positives and do require adjustments OOTB for most sites, but you’re missing the forest by focusing on this single case.
- RKFADU_UOFCCLEL 1y agoWhat? If I construct my queries the right way (e.g., not concatenating strings together like it's the year 1990), then I never will want a WAF "helping" me by blocking my users because they have an apostrophe in their name.
- ryandrake 1y agoThis looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively, aggressively, and in this case, to the wrong content altogether. Applying the filter to "other stuff" sent to and among the servers might make sense, but there doesn't seem to be any security benefit to filtering actual text payload that's only going to be displayed as blog content. This seems like a pretty cut and dried bug to me. 1: https://en.wikipedia.org/wiki/Scunthorpe_problem https://en.wikipedia.org/wiki/Scunthorpe_problem
- rurp 1y agoThis is exactly what I was thinking as well, it's a great Scunthorpe example. Nothing from the body of a user article should ever be executed in any way. If blocking a list of strings is providing any security at all you're already in trouble because attackers will find a way around that specific block list.
- pmarreck 1y agoCorrect. And a great example of it.
- chrisjj 1y ago> This looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively No. > aggressively No. >, and in this case, to the wrong content altogether. Yes - making it not a Scunthorpe problem.
- paxys 1y ago"You never know..." is the worst form of security, and makes systems less secure overall. Passwords must be changed every month, just to be safe. They must be 20 alphanumeric characters (with 5 symbols of course), just to be safe. We must pass every 3-letter compliance standard with hundreds of pages of checklists for each. The server must have WAF enabled, because one of the checklists says so. Ask the CIO what actual threat all this is preventing, and you'll get blank stares. As an engineer what incentive is there to put effort into knowing where each form input goes and how to sanitize it in a way that makes sense? You are getting paid to check the box and move on, and every new hire quickly realizes that. Organizations like these aren't focused on improving security, they are focused on covering their ass after the breach happens.
- chii 1y ago> Ask the CIO what actual threat all this is preventing the CIO is securing his job.
- reaperducer 1y agothe CIO is securing his job. Every CIO I have worked for (where n=3) has gotten where they are because they're a good manager, even though they have near-zero current technical knowledge. The fetishizing of "business," in part through MBAs, has been detrimental to actually getting things done. A century ago, if someone asked you what you do and you replied, "I'm a businessman. I have a degree in business," you'd get a response somewhere between "Yeah, but what to you actually do" and outright laughter.
- alabastervlog 1y agoIt's a relatively recent change, too. Transition from "the executives and managers mostly came up through 10-25 years of doing 'lower' jobs in the company, and very much know how the business actually works" to "we hire MBAs to those roles directly" was throughout the '70s-'90s. Finance and business grads have really taken over the economy, not just through technocratic "here's how to do stuff" advice but by personally taking all the reigns of power. They're even hard at work taking over medicine and pushing doctors out of the work-social upper-middle-class. Already did it with professors. Lawyers seem safe, so far.
- mjr00 1y ago> I disagree with other posts here, it is partially a balance between security and usability. And economics. Many people here are blaming incompetent security teams and app developers, but a lot of seemingly dumb security policies are due to insurers. If an insurer says "we're going to jack up premiums by 20% unless you force employees to change their password once every 90 days", you can argue till you're blue in the face that it's bad practice, NIST changed its policy to recommend not regularly rotating passwords over a decade ago, etc., and be totally correct... but they're still going to jack up premiums if you don't do it. So you dejectedly sigh, implement a password expiration policy, and listen to grumbling employees who call you incompetent. It's been a while since I've been through a process like this, but given how infamous log4shell became, it wouldn't surprise me if insurers are now also making it mandatory that common "hacking strings" like /etc/hosts, /etc/passwd, jndi:, and friends must be rejected by servers.
- Wowfunhappy 1y agoMaybe it wouldn't make a difference, but if I was the IT person telling users they have to change their passwords every 90 days, I would 100% include a line in the email blaming the insurance company.
- foobarchu 1y agoI'm not in an IT dept (developer instead), but I'd bet money that would get you a thorough dressing down by an executive involved with the insurance. That sort of blaming goes over well with those at the bottom of the hierarchy, and poorly with those at the top.
- Wowfunhappy 1y agoThe insurance people are not a part of the company, so I'm not sure who would be offended. I wouldn't be mean about it. I'm imagining adding a line to the email such as: > (Yes, I know this is annoying, but it's required by our insurance company.) What is the insurance company going to do, jack up our rates because we accurately stated what their policy was?
- coldpie 1y ago> There is a temptation to just turn the rules off Definitely, though I have seen other solutions, like inserting non-printable characters in the problematic strings (e.g. "/etc/ho<b></b>sts" or whatever, you get the idea). And honestly that seems like a reasonable, if somewhat annoying, workaround to me that still retains the protections.
- krferriter 1y agoI don't get why you'd have SQL injection filtering of input fields at the CDN level. Or any validation of input fields aside from length or maybe some simple type validation (number, date, etc). Your backend should be able to handle arbitrary byte content in input fields. Your backend shouldn't be vulnerable to SQL injection if not for a CDN layer that's doing pre-filtering.
- ordersofmag 1y agoA simple reason would be if you're just using it as a proxy signal for bad bots and you want to reduce the load on your real servers and let them get rejected at the CDN level. Obvious SQL injection attempt = must be malicious bot = I don't want my servers wasting their time
- chrisjj 1y ago> A simple reason would be if you're just using it as a proxy signal for bad bots Who would be that stupid?
- immibis 1y agoBecause someone said "we need security" and someone else said "what is security" and someone else said "SQL injection is security" and someone looked up SQL injections and saw the word "select" and "insert". WAFs are always a bad idea (possible exception: in allow-but-audit mode). If you knew the vulnerabilities you'd protect against them in your application. If you don't know the vulnerabilities all you get is a fuzzy feeling that Someone Else is Taking Care of it, meanwhile the vulnerabilities are still there. Maybe that's what companies pay for? The feeling?
- pxc 1y agoWAFs can be a useful site of intervention during incidents or when high-severity vulns are first made public. It's not a replacement for fixing the vuln, that still has to happen, but it gives you a place to mitigate it that may be faster or simpler than deploying code changes.
- 1y ago
- stingraycharles 1y agoYup. Were a database company that needs to be compliant with SOC2, and I’ve had extremely long and tiring arguments with our auditor why we couldn’t adhere to some of these standard WAF rulesets because it broke our site (we allow people to spin up a demo env and trigger queries). We changed auditors after that.
- spydum 1y agosounds like your security policy is wrong (or doesnt have a provision for exceptions managed by someone with authority to grant them), or your auditor was swerving out of his lane. As far as I've seen: SOC2 doesn't describe any hard security controls - it just asks to evaluate your policy versus your implemented controls.
- stingraycharles 1y agoYou are absolutely correct, which is why we switched auditors. We use a third party to verify compliance of all our cloud resources (SecureFrame), and one of their checks is that specific AWS WAF rulesets are enabled on e.g. CloudFront endpoints. These are managed rulesets by AWS. We disabled this check, auditor swerved out of his lane, I spent more several hours explaining things he didn’t understand, and things resolved after our CEO had a call with him (you can imagine how the discussion went). All in all, if the auditor would have been more reasonable it wouldn’t have been an issue, but I’ve always been wary of managed firewall rulesets because of this reason.
- RKFADU_UOFCCLEL 1y agoThere's no "trade-off" here. Blocking IPs that send "1337 h4x0r buzzword /etc/passwd" in it is completely naive and obtrusive, which is the modus operandi of the CDN being discussed here. There are plenty of other ways of hosting a website.
- kiitos 1y ago> I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts. I might be out of the loop here, but it seems to me that any WAF that's triggered when the string "/etc/hosts" is literally anywhere in the content of a requested resource, is pretty obviously broken.
- schnable 1y agoI don't think so. This rule for example probably block attacks on a dozen old WordPress vulnerabilities.
- kiitos 1y agoAnd a rule that denies everything blocks all vulnerabilities entirely. A false positive from a conservative evaluation of a query parameter or header value is one thing, conceivably understandable. A false positive due to the content of a blog post is something else altogether.
- afiori 1y agoThis is a strawman, especially if like the parent claims this was improving security for one of the most popular website backends ever. Rules like this might very well have had incredible positive impact on ten of thousands of websites at the cost of some weird debugging sessions for dozens of programmers (made up numbers obviously).
- kiitos 1y agoLook, any WAF that blocks a document like <!DOCTYPE html> <html lang="en"> <body> <p>/etc/hosts is a file on Unix hosts</p> is pretty clearly broken. And you can't meaningfully measure product metrics like impact for fundamentally broken products.
- julik 1y agoThis is what surprises me in this story. I could not, at first glance, assume that either Substack people or Cloudflare people were incompetent. Oh: I resisted tooth and nail about turning on a WAF at one of my gigs (there was no strict requirement for it, just cargo cult). Turns out - I was right.
- oakwhiz 1y agoI've had the issue where filling out form fields for some company website triggers a WAF and then nobody in the company is able to connect me to the responsible party who can fix the WAF rules. So I'm just stuck.
- lsofzz 1y ago> The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem. I agree. There is a business opportunity here. Right in the middle of your sentences. Hint: Context-Aware WAF. Many platforms have emerged in the last decade - some called it smart WAF, some called it nextgen WAF.. All vaporware garbage that consumes tons and tons of system resource and still manages to do a shit job of _actually_ WAF'ing web requests. To be truly context-aware, you need to compute a priori about the situation - the user, the page, the interactions etc.
- matt-p 1y agoIn my experience the pain of false positives required to outweigh the "WAF is best practice" is just very very heigh. Most big businesses would rather lose/frustrate a small percentage of customers, to be "safe".
- chrisjj 1y ago> The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content They shouldn't be doing that job at all. The content of user data is none of their business.