16 ms·

Writing "/etc/hosts" breaks the Substack editor

• selfselfgo 1y ago
  [dead]
• deleted 1y ago
  [deleted]
• paxys 1y ago
  This isn't a "security vs usability" trade-off as the author implies. This has nothing to do with security at all. /etc/hosts See, HN didn't complain. Does this mean I have hacked into the site? No, Substack (or Cloudflare, wherever the problem is) is run by people who have no idea how text input works.
  • macspoofing 1y ago
    My thought exactly - this isn't an example of balance between "security vs usability" - this is just wrong behaviour.
  • orlp 1y ago
    This is like banning quotes from your website to 'solve' SQL injection...
    • betenoire 1y ago
      "magic quotes" have entered the chat :D
  • eli 1y ago
    It's a text string that is frequently associated with attacks and vulnerabilities. In general you want your WAF to block those things. This is indeed the point of a WAF. Except you also don't want it to get in the way of normal functionality (too much). That is what the security vs usability trade off is. This particular rule is obviously off. I suspect it wasn't intended to apply to the POST payload of user content. Perhaps just URL parameters. On a big enough website, users are doing weird stuff all the time and it can be tricky to write rules that stop the traffic you don't want while allowing every oddball legitimate request.
    • SonOfLilit 1y ago
      Your auditor wants your WAF to block those things. _You_, at least I, never ever want to have a WAF at all, as they cause much more harm than good and, as a product category, deserve to die.
  • mystifyingpoi 1y ago
    > is run by people who have no idea how text input works That's a very uncharitable view. It's far more likely that they are simply using some WAF with sane defaults and never caught this. They'll fix it and move on.
    • immibis 1y ago
      with insane defaults FTFY
  • gav 1y ago
    It's more so that Cloudflare has a WAF product that checks a box for security and makes people who's job it is to care about boxes being checked happy. For example, I worked with a client that had a test suite of about 7000 or so strings that should return a 500 error, including /etc/hosts and other ones such as: ../../apache/logs/error.log AND%20(SELECT%208203%20FROM%20(SELECT(SLEEP(5)))xGId) /../..//../..//../..//../winnt/system32/netstat.exe?-a We "failed" and were not in compliance as you could make a request containing one of those strings--ignoring that neither Apache, SQL, or Windows were in use. We ended up deploying a WAF to block all these requests, even though it didn't improve security in any meaningful way.
    • krferriter 1y ago
      > For example, I worked with a client that had a test suite of about 7000 or so strings that should return a 500 error > We "failed" and were not in compliance as you could make a request containing one of those strings--ignoring that neither Apache, SQL, or Windows were in use. this causes me pain
    • WesolyKubeczek 1y ago
      Why in the world should those be 500 even? Those all are "40x client fuckup". I guess someone was told, when compiling those strings, that they should observe this known-good implementation (that actually crashed upon receiving such things) and record whatever it returns, and then mandate it of everyone else from now on.
    • ta1243 1y ago
      What is their desired behaviour if not a 404? A 500? a FIN? a RST?
      • gav 1y ago
        The desired result is a 500 so it's possible to audit. As much as this is a pain, the alternative can be more painful. I used to have a client that would forward me an email from their security team every six weeks saying "we found a SQL injection issue with your site, can you look into this and confirm that it's fixed?" and I'd reply back saying "that not possible" and they'd go "ok, we've marked this as a false positive". Eventually I got bored of having the same conversation over and over, so I asked them to show what they were finding. It turned out their scan would do the following: html1 = request("https://example.com/search?query=test") html2 = request("https://example.com/search?query=test' or 1=1--") if (html1 != html2) sql_injection_vulnerable = true Which of course is total nonsense, just because it returns different content doesn't mean anything. This is a perfect use case for a WAF, I can stick one in front and then have it return 500s for all these requests and not worry about it any more. In our case, we didn't have a WAF, but they had a obvious User-Agent, and it turns out that blocking all of their requests passed the scan too :)
• petercooper 1y ago
  I ran into a similar issue with OpenRouter last night. OpenRouter is a “switchboard” style service that provides a single endpoint from which you can use many different LLMs. It’s great, but last night I started to try using it to see what models are good at processing raw HTML in various ways. It turns out OpenRouter’s API is protected by Cloudflare and something about specific raw chunks of HTML and JavaScript in the POST request body cause it to block many, though not all, requests. Going direct to OpenAI or Anthropic with the same prompts is fine. I wouldn’t mind but these are billable requests to commercial models and not OpenRouter’s free models (which I expect to be heavily protected from abuse).
  • esafak 1y ago
    Did you report it?
    • petercooper 1y ago
      Not yet, other than on X, because the project's comms is oriented around Discord which involves some hoop jumping. (Update: On the way to doing that, I decided to run my tests again and they now work without Cloudflare being touchy, so I'll keep an eye on it!) (Update 2: They just replied to me on X and said they had fixed their Cloudflare config - happy days!)
• Y_Y 1y ago
  Does it block `/etc//hosts` or `/etc/./hosts`? This is a ridiculous kind of whack-a-mole that's doomed to failure. The people who wrote these should realize that hackers are smarter and more determined than they are and you should only rely on proven security, like not executing untrusted input.
  • mystifyingpoi 1y ago
    No one expects any WAF to be a 100% solution that catches all exfiltration attempts ever, and it should not be treated this way. But having it is generally better than not having it.
    • paxys 1y ago
      > But having it is generally better than not having it. So is HN and every other site in the world insecure because it allows users to post "/etc/hosts" ?
      • mystifyingpoi 1y ago
        Maybe? I don't know nor care. Assuming that HN has a vuln with path traversal, a sanely configured WAF would block the traversal attempt.
        • smallnix 1y ago
          *some traversal attempts
        • latexr 1y ago
          I propose someone who doesn’t know or care how a system works shouldn’t be prescribing what to do to make it secure. Otherwise this is like suggesting every gate must have a lock to be secure, even those which aren’t connected to any walls. https://i.imgur.com/ntYUQB1.jpeg https://i.imgur.com/ntYUQB1.jpeg
          • MatthiasPortzel 1y ago
• untill 1y ago
  [flagged]
  • netsharc 1y ago
    Huh, a bit like "adult-content" filters that would censor Scunthorpe or Wikipedia articles about genitals, maybe Cloudflare saw a market to sell protection for donkeys who can't protect their webapps from getting request-injected.
  • tester756 1y ago
    Who knows how many attacks such a "stupid" thing blocks every month?
  • eli 1y ago
    You figured all that out just because the headers indicate the site passed through Cloudflare at one point? That's quite a leap! If Cloudflare had a default rule that made it impossible to write that string on any site with their WAF, wouldn't this be a lot more widespread? Much more likely someone entered a bad rule into Cloudflare, or Cloudflare isn't involved in that rule at all.
  • deleted 1y ago
    [deleted]
  • rainforest 1y ago
    I think Cloudflare WAF is a good product compared to other WAFs - by definition a WAF is intended to layer on validation that properly built applications should be doing, so it's sort of expected that it would reject valid potentially harmful content. I think you can fairly criticise WAF products and the people who advocate for them (and created the need for them) but I don't think the CF team responsible can really be singled out.
  • kccqzy 1y ago
    Unfortunately this is probably a case where the market demands stupidity. The quality engineers don't have a say over market forces.
  • arnaudsm 1y ago
• aidog 1y ago
  It's something I ran into quite a few times in my career. It's a weird call to get if the client can't save their cms site, due to typing something harmless. I think worst was when there was a dropdown that I defined which had a value in the mod rules that was not allowed.
  • sfoley 1y ago
    I cannot reproduce this.
• badgersnake 1y ago
  Seems like a case of somebody installing something they couldn’t be bothered to understand to tick a box marked security. The outcome is the usual one, stuff breaks and there is no additional security.
• netsharc 1y ago
  Reminds me of an anecdote about an e-commerce platform: someone coded a leaky webshop, so their workaround was to watch if the string "OutOfMemoryException" shows up in the logs, and then restart the app. Another developer in the team decided they wanted to log what customers searched for, so if someone typed in "OutOfMemoryException" in the search bar...
  • skipants 1y ago
    I've actually gone through this a few times with our WAF. A user got IP-banned because the WAF thought a note with the string "system(..." was PHP injection.
  • PhilipRoman 1y ago
    Careless analysis of free-form text logs is an underrated way to exploit systems. It's scary how much software blindly logs data without out of band escaping or sanitizing.
    • ycombinatrix 1y ago
      Why would someone "sanitize" OutOfMemoryException out of their logs? That is a silly point to make.
      • owebmaster 1y ago
        An OutOfMemoryException log should not be the same as a search log Error: OutOfMemoryException And Search: OutOfMemoryException Should not be related in any way
        • dusanz 1y ago
          Until someone searches for "Error: OutOfMemoryException"
          • PhilipRoman 1y ago
            If structured logging is too much, unique prefixes solve this issue. Basically you need some token that user provided data is unable to output to the log. If you rigorously escape all newlines, you can then use start-of-line and end-of-line as unforgeable tokens. The possibilities are endless and it all comes down to understanding whether the intersection of two grammars is empty.
• pimanrules 1y ago
  We faced a similar issue in our application. Our internal Red Team was publishing data with XSS and other injection attack attempts. The attacks themselves didn't work, but the presence of these entries caused our internal admin page to stop loading because our corporate firewall was blocking the network requests with those payloads in them. So an unsuccessful XSS attack became an effective DoS attack instead.
  • darkwater 1y ago
    This is funny and sad at the same time.
• 0xbadcafebee 1y ago
  Worth noting that people here are assuming that the author's assumption is correct, that his writing /etc/hosts is causing the 403, and that this is either a consequence of security filtering, or that this combination of characters at all that's causing the failure. The only evidence he has, is he gets back a 403 forbidden to an API request when he writes certain content. There's a thousand different things that could be triggering that 403. It's not likely to be a WAF or content scanner, because the HTTP request is using PUT (which browser forms don't use) and it's uploading the content as a JSON content-type in a JSON document. The WAF would have to specifically look for PUTs, open up the JSON document, parse it, find the sub-string in a valid string, and reject it. OR it would have to filter raw characters regardless of the HTTP operation. Neither of those seem likely. WAFs are designed to filter on specific kinds of requests, content, and methods. A valid string in a valid JSON document uploaded by JavaScript using a JSON content-type is not an attack vector. And this problem is definitely not path traversal protection, because that is only triggered when the string is in the URL, not some random part of the content body.
  • apetresc 1y ago
    It sure looks like the author did his due diligence; he has a chart of all the different phrases in the payload which triggered the 403 and they all corresponded to paths to common UNIX system configuration files. Nobody could prove that's exactly what's happening without seeing Cloudflare's internal WAF rules, but can you think of any other reasonable explanation? The endpoint is rejecting a PUT who's payload contains exactly /etc/hosts, /etc/passwd, or /etc/ssh/sshd_config, but NOT /etc/password, /etc/ssh, or /etc/h0sts. What else could it be?
    • simonw 1y ago
      Yeah, the author clearly put the work in to demonstrate what's happening here.
  • ryandrake 1y ago
    If you change a single string in the HTTP payload and it works, what other explanation makes sense besides a text scanner somewhere along the path to deploying the content?
• arp242 1y ago
  Few years ago I had an application that allowed me to set any password, but then gave mysterious errors when I tried to use that password to login. Took me a bit to figure out what was going on, but their WAF blocked my "hacking attempt" of using a ' in the password. The same application also stored my full password in localStorage and a cookie (without httponly or secure). Because reasons. Sigh. I'm going to do a hot take and say that WAFs are bollocks mainly used by garbage software. I'm not saying a good developer can't make a mistake and write a path traversal, but if you're really worried about that then there are better ways to prevent that than this approach which obviously is going to negatively impact users in weird and mysterious ways. It's like the naïve /(fuck|shit|...)/g-type "bad word filter". It shows a fundamental lack of care and/or competency. Aside: is anyone still storing passwords in /etc/passwd? Storing the password in a different root-only file (/etc/shadow, /etc/master.passwd, etc.) has been a thing on every major system since the 90s AFAIK?
  • tlb 1y ago
    It's more that /etc/hosts and /etc/passwd are good for testing because they always exist with predictable contents on almost every system. If you inject "cat /etc/passwd" to various URLs you can grep for "root:" to see if it worked. So it's really blocking doorknob-twisting scripts.
    • arp242 1y ago
      Oh yeah, I've used it for that purpose. Seems rather silly to block that outright though since you can use many commonly distributed files.
  • reverendsteveii 1y ago
    my bank requires non-alphanumeric characters in their passwords but will reject a password if it has alphanumeric characters it associates with command injection attacks. as far as WAFs being garbage, they absolutely are, but this is a great time for a POSIWID analysis. A WAF says its purpose is to secure web apps. It doesn't do that, but people keep buying them. Now we're faced with a crossroads: we either have to assume that everyone is stupid or that the actual purpose of a WAF is something other than its stated purpose. I personally only assume stupidity as a last resort. I find it lazy and cynical, and it's often used to dismiss things as hopeless when they're not actually hopeless. To just say "Oh well, people are dumb" is a thought-terminating cliche that ignores potential opportunities. So we do the other thing and actually take some time to think about who decides to put a WAF in-place and what value it adds for them. Once you do that, you see myriad benefits because a WAF is a cheap, quick solution that allows non-technical people to say they're doing something. You're the manager of a finance OU that has a development group in it whose responsibility is some small web app. Your boss just read an article about cyber security and wants to know what this group two levels below you is doing about cyber security. Would you rather come back with "We're gonna need a year, $1 million and every other dev priority to be pushed back in order to develop a custom solution" or "We can have one fired up tomorrow for $300/mo, it's developed and supported by Microsoft and it's basically industry standard." The negative impact of these things is obvious to us because this is what we do, but we're not always the decision-makers for stuff like that. Often the decision-makers are actually that naive and/or they're motivated less by the ostensible goal of better web app security and more by the goal of better job security. As far as etc/passwd you're right that passwords don't live there anymore but user IDs often do and those can indicate which services are running as daemons on a given system. This is vital because if you can figure out what services are running you can start version fingerprinting them and then cross-referencing those versions with the CVE database.
• deleted 1y ago
  [deleted]
• mrgoldenbrown 1y ago
  Everything old is new again :) We used to call this the Scunthorpe problem. https://en.m.wikipedia.org/wiki/Scunthorpe_problem https://en.m.wikipedia.org/wiki/Scunthorpe_problem
  • greghendershott 1y ago
    See also: Recent scrubbing US government web sites for words like "diversity", "equity", and "inclusion". Writing about biology, finance, or geology? Shrug. Dumb filtering is bad enough when used by smart people with good intent.
    • netsharc 1y ago
      Huh, quick tell one Musk's DOGE l33t h4ck3ers about reverse proxies, and put all government sites behind one, that looks for those words and returns an error... Error 451 would be the most appropriate! For bonus, the reverse proxy will run on a system infiltrated by Russian (why not Chinese as well) hackers.
  • reverendsteveii 1y ago
    "I wonder why it's called Scunthorpe....?" sits quietly for a second "Oh nnnnnnnooooooooooooooo lol!"
  • kreddor 1y ago
    I remember back in the old days on the Eve Online forums when the word cockpit would always turn up as "c***pit". I was quite amused by that.
    • Too 1y ago
      This is actually a better solution, replacing dangerous words with placeholders, instead of blocking the whole payload. That at least gives the user some indication of what is going on. Not that I'm for any such WAF filters in the first place, just if having to choose between the lesser of two evils I'd choose the more informative.
• righthand 1y ago
  Similar: Writing `find` as the first word in your search will prevent Firefox from accepting the “return” key is pressed. Pretty annoying.
  • apetresc 1y ago
    I can't reproduce this; is it still the case, or some ancient thing?
    • kmoser 1y ago
      I can't reproduce it, either, using FF 137.0.2 (64-bit) on Windows.
  • jandrese 1y ago
    Are you sure you don't have a custom search rule configured in Firefox? I just tried this on my local instance and there was no problem.
  • righthand 1y ago
    EDIT: Apparently this is caused by the “findplus” extension. Removed!
• matt_heimer 1y ago
  The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem. If your site discusses databases then turning on the default SQL injection attack prevention rules will break your site. And there is another ruleset for file inclusion where things like /etc/hosts and /etc/passwd get blocked. I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts. Fine tuning the rules is time consuming. You often have to just completely turn off the ruleset because when you try to keep the ruleset on and allow the use-case there are a ton of changes you need to get implemented (if its even possible). Page won't load because /etc/hosts was in a query param? Okay, now that you've fixed that, all the XHR included resources won't load because /etc/hosts is included in the referrer. Now that that's fixed things still won't work because some random JS analytics lib put the URL visited in a cookie, etc, etc... There is a temptation to just turn the rules off.
  • _blk 1y ago
    100! [good] security just doesn't work as a mixing pattern... I'm not saying it's necessarily bad to use those additional protections but they come with severe limitations so the total value (as in cost/benefit) is hard to gauge.
  • gfiorav 1y ago
    I agree. From a product perspective, I would also support the decision. Should we make the rules more complex by default, potentially overlooking SQL injection vulnerabilities? Or should we blanket prohibit anything that even remotely resembles SQL, allowing those edge cases to figure it out? I favor the latter approach. That group of Cloudflare users will understand the complexity of their use case accepting SQL in payloads and will be well-positioned to modify the default rules. They will know exactly where they want to allow SQL usage. From Cloudflare’s perspective, it is virtually impossible to reliably cover every conceivable valid use of SQL, and it is likely 99% of websites won’t host SQL content.
• eniac111 1y ago
  https://en.wikipedia.org/wiki/Bush_hid_the_facts https://en.wikipedia.org/wiki/Bush_hid_the_facts
• jmmv 1y ago
  I encountered this a while ago and it was incredibly frustrating. The "Network error" prevented me from updating a post I had written for months because I couldn't figure out why my edits (which extended the length and which I assumed was the problem) couldn't get through. Trying to contact support was difficult too due to AI chatbots, but when I finally did reach a human, their "tech support" obviously didn't bother to look at this in any reasonable timeframe. It wasn't until some random person on Twitter suggested the possibility of some magic string tripping over some stupid security logic that I found the problem and could finally edit my post.
• SonOfLilit 1y ago
  After having been bitten once (was teaching a competitive programming team, half the class got a blank page when submitting solutions, after an hour of debugging I narrowed it down to a few C++ types and keywords that cause 403 if they appear in the code, all of which happen to have meaning in Javascript), and again (working for a bank, we had an API that you're supposed to submit a python file to, and most python files would result in 403 but short ones wouldn't... a few hours of debugging and I narrowed it down to a keyword that sometimes appears in the code) and then again a few months later (same thing, new cloud environment, few hours burned on debugging[1]), I had the solution to his problem in mind _immediately_ when I saw the words "network error". [1] the second time it happened, a colleague added "if we got 403, print "HAHAHA YOU'VE BEEN WAFFED" to our deployment script, and for that I am forever thankful because I saw that error more times than I expected
  • simonw 1y ago
    Do you remember if that was Cloudflare or some other likely WAF?
    • SonOfLilit 1y ago
      First time something on-prem, maybe F5. Second time AWS. Oh, I just remembered I had another encounter with the AWS WAF. I had a Jenkins instance in our cloud account that I was trying to integrate with VSTS (imagine github except developed by Microsoft, and still maintained, nevermind that they own github and it's undoubtedly a better product). Whenever I tried to trigger a build, it worked, but when VSTS did, it failed. Using a REST monitor service I was able to record the exact requests VSTS was making and prove that they work with curl from my machine... after a few nights of experimenting and diffing I noticed a difference between the request VSTS made to the REST monitor and my reproduction with curl: VSTS didn't send a "User-Agent" header, so curl supplied one by default unless I added I think -H "User-Agent:", and therefore did not trigger the first default rule in the AWS WAF, "if your request doesn't list a user agent you're a hacker". HAHAHA I'VE BEEN WAFFED AGAIN.
  • netsharc 1y ago
    +++ATH
• simonw 1y ago
  "How could Substack improve this situation for technical writers?" How about this: don't run a dumb as rocks Web Application Firewall on an endpoint where people are editing articles that could be about any topic, including discussing the kind of strings that might trigger a dumb as rocks WAF. This is like when forums about web development implement XSS filters that prevent their members from talking about XSS! Learn to escape content properly instead.
  • serial_dev 1y ago
    Surprisingly simple solution
  • ZeroTalent 1y ago
    hire a cybersec person. I don't think they one.
  • awoimbee 1y ago
    I'm in the position where I have to run a WAF to pass security certifications. The only open source WAFs are modsecurity and it's beta successor, coraza. These things are dumb, they just use OWASP's coreruleset which is a big pile of unreadable garbage.
• teddyh 1y ago
  > For now, I'll continue using workarounds like "/etc/h*sts" (with quotes) or alternative spellings when discussing system paths in my Substack posts. Ahh, the modern trend of ”unalived”¹ etc. comes to every corner of society eventually. 1. <https://knowyourmeme.com/memes/unalive https://knowyourmeme.com/memes/unalive>
• nottorp 1y ago
  So everyone should start looking for vulnerabilities in the substack site? If that's their idea of security...
• Null-Set 1y ago
  This looks like it was caused by this update https://developers.cloudflare.com/waf/change-log/2025-04-22/ https://developers.cloudflare.com/waf/change-log/2025-04-22/ rule 100741. It references this CVE https://github.com/tuo4n8/CVE-2023-22047 https://github.com/tuo4n8/CVE-2023-22047 which allows the reading of system files. The example given shows them reading /etc/passwd
  • mrspuratic 1y ago
    AFAICT it's also (though I'm very rusty) in ModSecurity, if XML content processing is enabled then rules like these will trip: SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile lfi-os-files.data" SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile unix-shell.data" ... where the referenced files contain the usual list of *nix suspects including the offending filename (lfi-os-files.data, "local file inclusion" attacks) The advantage (whack-a-mole notwithstanding) of a WAF is it orders of magnitude easier to tweak WAF rules than upgrade say, Weblogic, or other teetering piles of middleware.
    • julik 1y ago
      So that's why immediately when I hear "WAF" I read "...and the site will break in weird and exciting ways due to arbitrary, badly developed heuristics outside of your control, every odd day of every even week" - I remember the glory days of shared hosting and mod_security. Turns out the hunches were right all along.
• blenderob 1y ago
  > This case highlights an interesting tension in web security: the balance between protection and usability. But it doesn't. This case highlights a bug, a stupid bug. This case highlights that people who should know better, don't! The tension between security and usability is real but this is not it. Tension between security and usability is usually a tradeoff. When you implement good security that inconveniences the user. From simple things like 2FA to locking out the user after 3 failed attempts. Rate limiting to prevent DoS. It's a tradeoff. You increase security to degrade user experience. Or you decrease security to increase user experience. This is neither. This is both bad security and bad user experience. What's the tension?
  • _1tem 1y ago
    I would say it’s a useful security practice in general to apply WAF as a blanket rule to all endpoints and then remove it selectively when issues like this occur. It’s much, much, harder to evaluate every single public facing endpoint especially when hosting third party software like Wordpress with plugins.
  • crabbone 1y ago
    Precisely. This also reminded me, I think in the PHP 3 era, PHP used to "sanitize" the contents of URL requests to blanket combat SQL injections, or perhaps, it was a configuration setting that would be frequently turned on in shared hosting services. This, of course, would've been very soon discovered by the authors of the PHP site and various techniques were employed to circumvent this restriction, overall giving probably even worse outcomes than if the "sanitation" wasn't there to begin with.
    • TRiG_Ireland 1y ago
      The days of addslashes() and stripslashes()!
• josephcsible 1y ago
  WAFs were created by people who read https://thedailywtf.com/articles/Injection_Rejection https://thedailywtf.com/articles/Injection_Rejection and didn't realize that TDWTF isn't a collection of best practices.
• deleted 1y ago
  [deleted]
• skybrian 1y ago
  Did anyone try reporting this to Substack?
• wglb 1y ago
  The problem with WAF is discussed in https://users.ece.cmu.edu/~adrian/731-sp04/readings/Ptacek-Newsham-ids98.pdf https://users.ece.cmu.edu/~adrian/731-sp04/readings/Ptacek-N.... One of the authors of the paper has said "WAFs are just speed bump to a determined attacker."
  • p_ing 1y ago
    Locks are a speedbump for a lockpick. Doors are a speedbump for a car. Well yeah, sure, doesn't mean I'm going to have an open doorframe or a door without a lock.
    • wat10000 1y ago
      The difference is that a door tends to be the only thing between you and an attacker. A speedbump is better than nothing. This isn't like having a lock on your door, this is like having a cheap, easily pickable padlock on your bank vault. If the vault has a proper lock then the padlock serves no purpose, and if it doesn't then you're screwed regardless.
      • Macha 1y ago
        And to extend the metaphor to cover the false positives these systems produce, sometimes the padlock seizes shut if the air temperature is in a certain range, and the team that put it there refuses to take responsibility for the fact they've locked your customers from accessing their assets with the valid key.
      • 9x39 1y ago
        I think a WAF is closer to a component of an entry control point, like on a military base. It's a tool for manned security to interact with and inspect traffic. Unmanned, they're just an obstacle to route around, but manned, they're an effective way to provide asymmetries to the defender. WAFs can have thousands of rules ranging from basic to the sophisticated, not unlike mechanisms you can deploy at a checkpoint. Security devices like IDSes or WAFs allow deploying filtering logic without touching an app directly, which can be hard/slow across team boundaries. They can allow retroactive analysis and flagging to a central log analysis team. Being able to investigate whether an adversary came through your door after the fact is powerful, you might even be able to detect a breach if you can filter through enough alerts. People are more likely to get dismissed for not installing an IDS or WAF than having one. Its effectiveness is orthogonal to the politics of its existence, most of the time.
• Osiris 1y ago
  I understand applying path filters in URLS and search strings, but I find it odd that they would apply the same rules to request body content, especially content encoded as valid JSON, and especially for a BLOG platform where the content would be anything.
• johnklos 1y ago
  Content filtering should be highly context dependent. If the WAF is detached from what it's supposed to filter, this happens. If the WAF doesn't have the ability to discern between command and content contexts, then the filtering shouldn't be done via WAF. This is like spam filtering. I'm an anti-spam advocate, so the idea that most people can't discuss spam because even the discussion will set off filters is quite old to me. People who apologize for email content filtering usually say that spam would be out of control if they didn't have that in place, in spite of no personal experience on their end testing different kinds of filtering. My email servers filter based on the sending server's configuration: does the EHLO / HELO string resolve in DNS? Does it resolve back to the connecting IP? Does the reverse DNS name resolve to the same IP? Does the delivery have proper SPF / DKIM? Et cetera. My delivery-based filtering works worlds better than content-based filtering, plus I don't have to constantly update it. Each kind has advantages, but I'd rather occasional spam with no false positives than the chance I'm blocking email because someone used the wrong words. With web sites and WAF, I think the same applies, and I can understand when people have a small site and don't know or don't have the resources to fix things at the actual content level, but the people running a site like Substack really should know better.
  • _1tem 1y ago
    SPF and DKIM are now more commonly implemented correctly by spammers than by major email providers. https://news.ycombinator.com/item?id=43468995 https://news.ycombinator.com/item?id=43468995
    • johnklos 1y ago
      Yes, but they are still effective at preventing spam from spammers who are pretending to be others.
  • Anamon 1y ago
    Yes to smart filtering at the right layer. The whole reverse DNS checks et al. are so effective. I recently moved my personal mailbox from a host who didn't do these kinds of checks to one that does. My received spam volume instantly went from about 20 a day (across all my aliases) to less than 1 a week.
• dvorack101 1y ago
  Indeed a severe case of paranoia? 1. Create a new post. 2. Include an Image, set filter to All File types and select "/etc/hosts". 3. You get served with an weird error message box displacing a weird error message. 4. After this the Substack posts editor is broken. Heck, every time i access the Dashboard, it waits forever to build the page. Did find this text while browsing the source for an error (see original ascii art: https://pastebin.com/iBDsuer7 https://pastebin.com/iBDsuer7): SUBSTACK WANTS YOU TO BUILD A BETTER BUSINESS MODEL FOR WRITING https://substack.com/jobs
  • julik 1y ago
    "Who signed off on your WAF rules" would be a great reverse interview question then.
• jkrems 1y ago
  Could this be trivially solved client-side by the editor if it just encoded the slashes, assuming it's HTML or markdown that's stored? Replacing `/etc/hosts` with `/etc/hosts` for storage seems like an okay workaround. Potentially even doing so for anything that's added to the WAF rules automatically by syncing the rules to the editor code.
• driverdan 1y ago
  This is a common problem with WAFs and, more specifically, Cloudflare's default rulesets. If your platform has content that is remotely technical you'll end up triggering some rules. You end up needing a test suite to confirm your real content doesn't trigger the rules and if it does you need to disable them.
• nicoledevillers 1y ago
  it was a cf managed waf rule for a vulnerability that doesn't apply to us. we've disabled it.
  • SonOfLilit 1y ago
    This comment deserves to be much higher, assuming this user speaks for Substack (no previous submissions or comments, but the comment implies it).
    • nicoledevillers 1y ago
      i don't speak for substack, but i do work there and changed the WAF rule :)
  • silverwind 1y ago
    Why not review rules before applying them?
• robertlagrant 1y ago
  > This case highlights an interesting tension in web security: the balance between protection and usability. This isn't a tension. This rule should not be applied at the WAF level. It doesn't know that this field is safe from $whatever injection attacks. But the substack backend does. Remove the rule from the WAF (and add it to the backend, where it belongs) and you are just as secure and much more usable. No tension.
  • worewood 1y ago
    There is a tension, but it's between paying enough to developers to actually produce decent code or pay a 3rd-party to firewall the application.
    • marcosdumay 1y ago
      Again, there is no tension. People will manage to circumvent the firewall if they want to attack your site. But you will still pay, and get both the DoS vulnerabilities created by the firewall and the new attack vectors in the firewall itself.
  • _1tem 1y ago
    I would say it’s a decent security practice to apply WAF as a blanket rule to all endpoints and then remove it selectively when issues like this occur. It’s much, much, harder to evaluate every single public facing endpoint especially when hosting third party software like Wordpress with plugins.
    • SonOfLilit 1y ago
      I don't agree. WAFs usually add more attack surface than they remove. https://www.macchaffee.com/blog/2023/wafs/ https://www.macchaffee.com/blog/2023/wafs/ Of course, Wordpress is basically undefendable, so I'd never ever host it on a machine that has anything else of value (including e.g. db credentials that give access to much more than the public content on the WP installation).
• chaitrack 1y ago
  [dead]
• nickagliano 1y ago
  As a card carrying Substack hater, I’m not suprised. > "How could Substack improve this situation for technical writers?" They don’t care about (technical) writers. All they care about is building a TikTok clone to “drive discoverability” and make the attention-metrics go up. Chris Best is memeing about it on his own platform. Very gross.
  • paulpauper 1y ago
    Reminds me of Slashdot and breaking the page by widening it with certain characters
• mifydev 1y ago
  It's /con/con all over again
• curtisszmania 1y ago
  [dead]
• godelski 1y ago
  I don't get it. Why aren't those files just protected so they have no read or write permissions? Isn't this like the standard way to do things? Put the blog in a private user space with minimal permissions. Why would random text be parsed? I read the article but this doesn't make sense to me. They suggested directory transversal but your text shouldn't have anything to do with that and transversal is solved by permission settings
  • tryauuum 1y ago
    this is the usual approach with web application firewalls, block all the 100500 known attacks. Doesn't matter if they are not applicable to your website. Some of them are obviously OS-depended (having .exe in the URLs) but it doesn't matter, it's blocked just in case I do understand this appoach. From the defence point of view it makes sense, if you have to create a solution to protect millions of websites it doesn't make sense to tailor it to specifics of a single one
    • godelski 1y ago
      Can you help me understand the exploit here? I'd really like to understand. (What's below is written not as "this is how it should be done" but instead "what I understand should be done". To provide context to what I do and do not understand so that my misunderstandings can be more directly addressed) I understand being over zealous, an abundance of caution. But what I'm confused about is why normal text could lead to an exploit in the first place. I can only understand this being a problem if arbitrary text is being executed. Which that would appear to be a HUGE security hole ripe for exploitation. But path access is handled by privileges. So even if arbitrary text is being executed, how can that lead to exploitation without already having a major security hole? Maybe I'm not understanding substack? I've only been a reader. But why is the writer not in a container or chroot? If you want you be overly zealous why not use two vms? Put them in a vm to write, once they've edited then run tests and then use that as the image for the userspace in the ephemeral vm that is viewed by readers. Would this still be exploitable? I don't mean image for the whole vm, I really do mean a subimage so you can lock that up.
• swyx 1y ago
  substack also does wonderful things like preserve weird bullet points, lack code block displays, and make it impossible to customize the landing page of your site beyond the 2 formats they give you. generally think that Substack has done a good thing for its core audience of longform newsletter writer creators who want to be Ben Thompson. however its experience for technical people, for podcasters, for people who want to start multi-channel media brands, and for people who write for reach over revenue (but with optional revenue) has been really poor. (all 4 of these are us with Latent.Space). I've aired all these complaints with them and theyve done nothing, which is their prerogative. i'd love for "new Substack" to emerge. or "Substack for developers".
  • rmccue 1y ago
    Ben Thompson is working on Passport, which seems to be a self-hosted (WordPress-based) Substack: https://stratechery.com/2021/passport/ https://stratechery.com/2021/passport/ He gave a talk on it at WordCamp Asia at the start of last year, although I haven’t heard of any progress recently on it.
    • ZeroTalent 1y ago
      You're missing the point of Substack: 1. It's a social media platform with a network that is still easy to extract organic growth from. 2. 99% email deliverability without configuring anything. It's whitelisted everywhere.
• t1234s 1y ago
  writing "bcc: someone@email.com" sometimes triggers WAF rules
• thayne 1y ago
  As soon as I saw the headline, I knew this was due to a WAF. I worked on a project where we had to use a WAF for compliance reasons. It was a game of wack-a-mole to fix all the places where standard rules broke the application or blocked legitimate requests. One notable, and related example is any request with the string "../" was blocked, because it might be a path traversal attack. Of course, it is more common that someone just put a relative path in their document.
• HenryBemis 1y ago
  Aaaahh they are trying to prevent a Little Bobby Tables story..
• ChrisArchitect 1y ago
  Just tried to post a tweet with this article title and link and got a similar error (on desktop twitter.com). Lovely.
• julik 1y ago
  Ok so: there is a blogging/content publishing engine, which is somewhat of a darling of the startup scene. There is a cloud hosting company with a variety of products, which is an even dearer darling of the startup scene. Something is posted on the blobbing/content publishing engine that clearly reveals that * The product provided for blogging/content publishing did a shitty job of configuring WAF rules for its use cases (the utility of a "magic WAF that will just solve all your problems" being out of the picture for now) * The WAF product provided by the cloud platform clearly has shitty, overreaching rules doing arbitrary filtering on arbitrary strings. That filtering absolutely can (and will) break unrelated content if the application behind the WAF is developed with a modicum of security-mindedness. You don't `fopen()` a string input (no, I will not be surprised - yes, sometimes you do `fopen()` a string input - when you are using software that is badly written). So I am wondering: 1. Was this sent to Substack as a bug - they charge money for their platform, and the inability to store $arbitrary_string on a page you pay for, as a user, is actually a malfunction and disfunction"? It might not be the case "it got once enshittified by a CIO who mandated a WAF of some description to tick a box", it might be the case "we grabbed a WAF from our cloud vendor and haven't reviewed the rules because we had no time". I don't think it would be very difficult for me, as an owner/manager at the blogging platform, to realise that enabling a rule filtering "anything that resembles a Unix system file path or a SQL query" is absolutely stupid for a blogging platform - and go and turn it the hell off at the first user complaint. 2. Similarly - does the cloud vendor know that their WAF refuses requests with such strings in them, and do they have a checkbox for "Kill requests which have any character an Average Joe does not type more frequently than once a week"? There should be a setting for that, and - thinking about the cloud vendor in question - I can't imagine the skill level there would be so low as to not have a config option to turn it off. So - yes, that's a case of "we enabled a WAF for some compliance/external reasons/big customer who wants a 'my vendor uses a WAF' on their checklist", but also the case of "we enabled a WAF but it's either buggy or we haven't bothered to configure it properly". To me it feels like this would be 2 emails first ("look, your thing <X> that I pay you money for clearly and blatantly does <shitty thing>, either let me turn it off or turn it off yourself or review it please") - and a blog post about it second.
• iefbr14 1y ago
  So "/etc/h*sts" is not stopped by the filters? Nice to know for the hackers :)
• deleted 1y ago
  [deleted]
• donatj 1y ago
  We briefly had a WAF forced upon us and it caused so many problems like this we were able to turn it off, for now. I'm sure it'll be back.
• Sharo2025 1y ago
  [flagged]
• halffullbrain 1y ago
  At least, in this case, the WAF in question had the decency to return 403. I've worked with a WAF installation (totally different product), where the "WAF fail" tell was HTTP status 200 (!) and "location: /" (and some garbage cookies), possibly to get browsers to redirect using said cookies. This was part of the CSRF protection. Other problems were with "command injection"-patterns (like in the article, expect with specific Windows commands, too - they clash with everyday words which the users submit), and obviously SQL injections which cover some relevant words, too. The bottom line is that WAFs in their "hardened/insurance friendly" standard configs are set up to protect the company from amateurs exposing buggy, unsupported software or architectures. WAF's are useful for that, but you still gave all the other issues with buggy, unsupported software. As others have written, WAFs can be useful to protect against emerging threats, like we saw with the log4j exploit which CloudFlare rolled out protection for quite fast. Unless you want compliance more than customers, you MUST at least have a process to add exceptions to "all the rules"-circus they put in front of the buggy apps. Whack-a-mole security filtering is bad, but whack-a-mole relaxation rule creation against an unknown filter is really tiring.
  • Too 1y ago
    Almost equally fun are the ones that simply drop the connection and leave you waiting for a timeout.
• lofaszvanitt 1y ago
  Using a WAF is the strongest indicator that someone doesn't know what's happening and where or something underneath is smelly and leaking profusely.
• 0xDEAFBEAD 1y ago
  Weird idea: What if user content was stored and transmitted encrypted by default? Then an attacker would have to either (a) identify a plaintext which encrypts to an attack ciphertext (annoying, and also you could keep your WAF rules operational for the ciphertext, with minimal inconvenience to users) or (b) attack the system when plaintext is being handled (could still dramatically reduce attack surface).
• vintermann 1y ago
  That reminds me of issues I once had with Microsoft's boneheaded WAF. We had base64 encoded data in a cookie, and whenever certain particular characters were produced next to each other in the data - I think the most common was "--" - the WAF would tilt and stop the "attempted SQL injection attack". So every so often someone would get an illegal login cookie and just get locked out of the system until they deleted it or it expired. Took a while to find out what went wrong, and even longer to figure out how to remove the more boneheaded rules from the WAF.
• stefs 1y ago
  this feels like blocking terms like "null" or "select" just because you failed to properly parameterize your SQL queries.
• mike-cardwell 1y ago
  Just rot13 any request data using javascript before posting, and rot13 it again on the server side. Problem solved. (jk)
• sudb 1y ago
  I had a problem recently trying to send LLM-generated text between two web servers under my control, from AWS to Render - I was getting 403s for command injection from Render's Cloudflare protection which is opaque and unconfigurable to users. The hacky workaround which has been stably working for a while now was to encode the offending request body and decode it on the destination server.
• gitroom 1y ago
  The amount of headaches I've had from WAFs blocking legit stuff is unreal. I just wish the folks turning those rules on had to use them for a week themselves.
• toogan 1y ago
  The title would be improved with "Writing the string ...". I first read it as "Writing the file" which was pretty weird.
  • TRiG_Ireland 1y ago
    It's in quotation marks, which I'd say makes it clear enough for most people.
• Habgdnv 1y ago
  I have a lifetime Pastebin account that I hadn't used for some years. Last year I enrolled in a "linux administration" class and tried to use that pastebin (famous for sharing code) to share some code/configurations with other students. When I tried to paste my homework I kept getting a Cloudflare error page. I don't even remember what I was pasting, but it was normal linux stuff. I contacted pastebin support - of course I got ghosted. I am sharing this in relation to the WAF comments and how much the companies implementing WAF care about your case.
• mattrighetti 1y ago
  This reminds me of that time I was discussing with friends about something we did in our computer science class that day and I realised writing toString in the Whatsapp client for macOS would crash the application. At the time I didn’t have the skills to understand why so I recorded the bug on my phone to share with friends :)
• bastawhiz 1y ago
  I once helped maintain some PHP software that was effectively a CMS. You'd drop a little PHP snippet into any page (e.g., that you make with Dreamweaver) and it would automatically integrate it with the CMS functionality. We had unending trouble with mod_security. The worst issue I can remember was that any POST request whose body contained the word "delete" was automatically rejected. That was the full rule. To this day I still can't imagine what the developers were thinking.
• chrisjj 1y ago
  > Substack's filter is well-intentioned - protecting their platform from potential attacks. There is sadly no evidence in this article that the supposed filter does protect the platform from potential attacks.
• rpigab 1y ago
  Between around 2005 and 2011 in France, if a child was born and parents Mr Bar and Mrs Baz wanted to transmit both of their last names, he or she had to be named "Foo Bar--Baz". No, that's not a typo, that's two hyphens. Check out "Circulaire du 6 décembre 2004 relative au nom de famille" if you don't believe me. Yes, the people in charge probably didn't think or know of SQL comments. However, it worked well as long as input is sanitized and not concatenated, which is often the case using modern frameworks or common sense. However, nowadays, we just put a WAF in front of everything, it's cheaper that way because common sense is hard to come by. People like Foo Bar--Baz still exist, and unless they've had their name changed, they're sometimes running into extremely wierd issues in the web software they're using.