24 ms·

Don’t try to sanitize input – escape output

• vxNsr 7y ago
  It's interesting how so much of programming is knowing about the existence of libraries and not trying to rebuild things that already exist. I'm sure there are thousands of people out there who sanitize inputs and outputs naively and don't know about great libs like DomPurify. I know I'm guilty of trying to build something from first principles only to google it after banging my head against edge cases and finding a ready-made library or util that with a tiny bit of finesse or modification does the job.
  • CiPHPerCoder 7y ago
    > I'm sure there are thousands of people out there who sanitize inputs and outputs naively and don't know about great libs like DomPurify. Also HTMLPurifier for PHP, and probably several others in other languages.
  • inapis 7y ago
    TBF, most of these libraries are not easily discovered. You have to luck on the chance that the library author used similar words as your search query. Barring major stuff (authentication, databases etc), you will rarely know about a library in existence. Search engines are still limited by the language of humans. Unless and until, you note down every possible library you come across for future reference, this problem is here to stay. Maybe a language having a vast standard library won’t suffer from this problem but it will definitely have other problems.
    • vxNsr 7y ago
      yeah, creating a personal search db is time consuming and kinda impossible... a while ago everyone was coming up with bookmark managers that kinda sorta could function as a personal search db, but it still required a lot of customization. Also they were all cloud based, and didn't really function offline.
  • simonw 7y ago
    One of the most useful skills I've developed over the years is a very strong instinct for "someone else will have solved this already in an open source library", combined with intuition on exactly how they would have described it so I can find it with search.
  • SahAssar 7y ago
    DomPurify by default allows style tags ( https://github.com/cure53/DOMPurify/blob/master/src/tags.js#L99 https://github.com/cure53/DOMPurify/blob/master/src/tags.js#... ) which can be used for tracking visits and some types of password extraction from certain types of frameworks (see https://www.mike-gualtieri.com/posts/stealing-data-with-css-attack-and-defense https://www.mike-gualtieri.com/posts/stealing-data-with-css-... and https://github.com/maxchehab/CSS-Keylogging https://github.com/maxchehab/CSS-Keylogging ). Also regarding these types of mitigations see https://www.youtube.com/watch?v=lG7U3fuNw3A https://www.youtube.com/watch?v=lG7U3fuNw3A which explains how some good mitigations might not catch all edge cases. I don't think there is a golden rule for this stuff besides having a very strict CSP that only allows same-site resources and no eval/inline with reporting to see when someone has tried to do something bad.
• justinator 7y ago
  Why is Billy the Kid, a Wild West outlaw, talking like a pirate?
  • benhoyt 7y ago
    Good call! Guess I was thinking of Captain Kidd. Fixed, kind of. :-) Then again, I'm not sure what Billy the Kid's doing surfing the web in 2020 either...
• skybrian 7y ago
  It seems like this depends on your types? If you are storing user input to a database using a text field, it's best to assume the field can contain arbitrary text, since the database allows that. But if you're storing the input into a number field then it must be parsed as a number and you can assume it's a number. If you constrained the number field to a certain range then you can assume the number is in that range. Storing arbitrary text is common, so we usually need to know how to render it correctly. There are fancier types and constraints, though.
  • choeger 7y ago
    You should still sanitize the output, IMO. The code in question has no ties to the DB schema (usually) and if that changes for some reason you have no means (usually) to figure out where you have to change the code.
  • dTal 7y ago
    Each piece of code should enforce its own contracts explicitly across communication boundaries. If your backend relies on x input being parseable as a number, it shouldn't assume that it is just because you know that it ultimately came from a number picker in an HTML form - it should check for itself. This is defensive not just for security, but so that when you change something later and screw it up you get actual error messages instead of silent breakage.
• deleted 7y ago
  [deleted]
• ehsankia 7y ago
  So then it's fine for an input that deletes your entire database, as long as none of that data makes it back out?
  • jfkebwjsbx 7y ago
    If you are treating inputs as code, you have bigger issues to worry about.
    • madsbuch 7y ago
      What is the exact (philosophical?) distinction between data and code? When you extract the host name from email to be able to send a email, you are interpreting a string. We could call this process executing a program that outputs the host name.
      • jeen02 7y ago
        That's the dumbest question I've ever heard
      • jfkebwjsbx 7y ago
        No, that is data, not code. You don't seem to understand the distinction between running algorithms on data and taking programs as input, which is what the GP talked about.
        • dikei 7y ago
          No, he made a good point. It's a matter of perspective, what is considered data in one situation, can be consider code in other situations. Hell, they even created the NX-bit to prevent malicious memory space becoming executable.
          • throwaway373438 7y ago
            Mixing code and data is an inherent attribute of von Newmann architecture computer systems.
• throwawayjava 7y ago
  This feels like a distinction without a difference. Escaping outputs is just one way of sanitizing inputs. Sometimes it works. Sometimes it doesn't. The author of this post even realizes that their prognostication is not general and then offers the advice to "be sure to get security review"... At the end of the day, you need to make sure that any untrusted source is treated in a safe way by every sink and does not otherwise interfere with system specs (e.g., mangling user output). Whether that happens at line 5 (where the input is read) or line 155 (where the command is generated) doesn't really matter. Or to be more precise, is determined by whatever design patterns the framework developer chose. What matters at the end of the day is that command injection isn't possible and the system's specs (including UI/UX specs) are respected. Crucially, both input and output constraints are informed by the nature of both the source and the sink. Hence the existence of libraries like DomPurify and HTMLPurifier, which consider one very particular type of sink. Sometimes you will write code in domains where others haven't written excellent libraries but where sanitization (of either input or output) is needed. E.g., embedded systems. I'd replace the author's advice with "carefully specify the semantics of your sources and sinks", which is ultimately what the author's actual advice (basically, "use trusted libraries and, when not, be sure to get security review") boils down to.
  • tptacek 7y ago
    Not really, no. Output filtering is done in the context of a specific output domain. Input sanitization isn't; the developer who builds sanitization has to guess at all the possible output domains. "Filter outputs not inputs" is a very old appsec truism.
    • eandre 7y ago
      I think the confusion comes from that not everybody thinks of "passing data to the database layer" as an output, but only an input to the next layer. If you think of this input as an output from the previous layer, then your advice makes perfect sense. But I don't think everyone thinks that way so it might help to clarify what "output" means in this context.
      • 7y ago
• jancsika 7y ago
  In the example given, why can't the programmer simply set the textContent for the given element to the arbitrary user input?
• jedberg 7y ago
  You should do both. Sanitize your inputs so that it can be safely stored in your data store, and then sanitize your output so it can be safely displayed. We did this at reddit. We had basic SQL sanitization on the way in, and then a full pass on the way back to the user. The advantage this gave us is that when someone discovered a new way to hack our sanitization, all we had to do was update the output filter and everything was magically safe. We didn't have to do full database scans to find all the bad data and change it. Edit: Apparently I shouldn't have simplified "paramaterazation of SQL" as "sanitize your input". I used the more generic term since I was talking about any kind of data store. But yes, it was of course parameterized.
  • iainmerrick 7y ago
    What does the “basic SQL sanitization on the way in” consist of and what does it do for you? The article makes sense to me; if I’m just storing strings in a database, I don’t see why they would need to be sanitized at rest, even if they contain malicious SQL code. Only when I actually come to use those strings for some purpose.
    • bardan 7y ago
      But what if at some point somewhere down the line someone forgets to sanitize the output? Surely better procedure to sanitize at both ends. Nobody is perfect.
      • laumars 7y ago
        You might need to escape strings differently depending on where you’re outputting eg HTML or JSON.
      • cjfd 7y ago
        This is completely the wrong attitude. Code should be correct, not fail-safe. All of the safety hatches that people tend to introduce make the code less predictable and eventual problems tend to arise far away from where they originated, making them difficult to diagnose. What is the result of this sanitization? Now we have some undefined, changing internal string format running around in our application, and possibly multiple undefined and changing internal string formats, where it is also unclear at what point a string is supposed to be in what format. If something is an arbitrary string it should be allowed to be an arbitrary string and the things handling that should escape it the appropriate way. The article is correct.
• 0xff00ffee 7y ago
  These two issues have been relevant for over 20 years, older than today's college grads. I find it fascinating the new blog posts are being written on a regular basis explaining these two pitfalls. And there are probably a million blogs with these same two examples going back decades. This is an epic re-post in spirit. I'm tempted to ask, "Why hasn't this been fixed yet?" Where "fixed" means, "Something new programmers just starting off their careers don't have to jam into their brains?" I know the expected answer will be: it's an abstraction of a more complex problem of understanding data and how it is used, and we can talk about how JS and PHP have added native functions to construct custom code to address this problem (hack cough cough hack). But these two cases in particular stick in my craw because there is no fundamental solution presenting yet. So I ask again (and I've been asking this since 2004-ish): Why do these two examples still persist? Why do the frameworks not eliminate them by construction? This is such a repeated pattern, why is it even there?
  • iainmerrick 7y ago
    I haven’t used Elm, but I’ve read that it uses strong typing to distinguish between escaped and non-escaped data. That sounds like a good general solution to the problem, as the compiler will prevent you from using unescaped data in a dangerous context.
    • chopin 7y ago
      I've heard that some Haskell frameworks do this as well. I am heavily using Java's Servlet framework and the blatant spraying of Strings everywhere is astounding in this age. I understand that backwards compatibility is an issue, but one could have set another API beside for optional use and deprecate the current one.
      • squiggleblaz 7y ago
        I imagine all Haskell frameworks do; the ones I've tried surely do. Haskellers are accustomed to mixing string types, since the default String type is inefficient (a linked list of Char) and most import a library to provide immutable ~Pascal strings. And since this is such a common occurrence, the syntax has support form multiple string literal types. An application developer can literally create their own. It's also trivial to create a new type in Haskell that has minimal runtime cost, so it's pretty harmless. I think if you don't have an easy way of creating string literals in the type you want, the developers will at some point reach for the deprecated api, and at that point you're just requiring good hygiene. Which is exactly what you're trying to stop. Language support is critical in being able to get away with this.
• firefoxd 7y ago
  It's definitely an interesting article but he didn't go deep enough with subject of escaping the output. I always recommend this excellent article from Joel Spolsky, Making wrong code look wrong. https://www.joelonsoftware.com/2005/05/11/making-wrong-code-look-wrong/ https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...
  • nothrabannosir 7y ago
    Agreed, it seems what this article (along with most of the comments) is dancing around is the idea of a stronger type system. String != string, e.g. UserText vs SqlStatement. Using explicit conversion methods between those types helps clarify the actual “boundaries” of your system, or rather the independent parts and their individual boundaries. Joel Spolsky’s article illustrates the problem well. The problem stems from our simplistic type systems, which reflect a value’s storage class (“array of bytes”) rather than its semantic type (“raw input from user”, “sql safe TEXT literal”). Once your type system can differentiate between those two, it can help you identify where conversion between the two (aka “escaping” or “encoding”) is necessary. Then the problem of “dangerous string” disappears, because there is no more string: if it’s a UserText, it can’t be concatenated with a SqlStatement without conversion. Just like an “int” for example, or an array. Anyway I’m just rehashing Spolsky’s article, poorly. Don’t let my inaccurate summary reflect negatively on his point :p
• josh_fyi 7y ago
  I think he is wrong on handling code as input and visible output (for sites like StackOverflow). No need to filter such input. Escaping your strings will handle that as well. The code <tag/> , for example, will be escaped to <tag/> , appearing in the rendered page as <tag/> (but not _interpreted_ as <tag/> ).
• pwdisswordfish2 7y ago
  > Incidentally, the mother in the xkcd comic says, “I hope you’ve learned to sanitize your database inputs.” Which is somewhat confusing, but I’ll give Randall the benefit of the doubt and assume he meant “escape your database parameters”. What a strangely roundabout way of saying that programming advice found in a webcomic may be actually wrong. (To be fair, that Xkcd comic was a product of its time, when ‘sanitisation’ was all the rage.)
  • zarmin 7y ago
    colloquial sanitization?
• deleted 7y ago
  [deleted]
• emayljames 7y ago
  The examples are very outdated. In PHP you would convert the offending characters to html entities >at input, yes..input<. It then only needs filtered >one time<.
  • function_seven 7y ago
    So the SMS message I send from my service would look like this? Alert: Bob &amp; Jane O&apos;Brien just commented on your recent post If you store HTML in your database, now you have to convert from that to whatever you're outputting to later. And you haven't covered the dangerous characters for other contexts. Just store whatever the user sent you. When outputting to various formats, convert accordingly.
    • emayljames 7y ago
      This whole article is based on a DB row being used for everything. That is just not reality. From a cost point of view, if you're gonna just use that data a couple of ways, it is pointless to constantly convert it coming out. Even then, if that was the case, convert from entities, no more costly than what you are saying.
• winrid 7y ago
  Generally I try to sanitize the input and store the original raw value in case we find sanitation bugs. I'd prefer fixing the old data when needed to having overhead on every read, at least for FastComments...
  • winrid 7y ago
    downvotes?
    • zAy0LfpBZLC8mAC 7y ago
      Yes. Sanitizing inputs is a bad idea, still.
      • winrid 7y ago
        Maybe Sanitizing is the wrong word for what I'm doing. For example, I need to strip marketing/tracking information from URLs before saving them or else someone coming from Google will have a different URL than someone coming from FB and then the comments won't load. I guess I meant normalization.
• skyzyx 7y ago
  This is bad advice. Definitely sanitize on input, then escape on output. You should never be knowingly storing unsafe input.
  • e12e 7y ago
    Actually, rather than sanitize input, I would recommend whitelist and reject in most cases. ID should be an integer, but you get a string with spaces around an integer? - error out. There's html in your text input? Error out. In the vast majority of cases you control the client (web form) - anything "surprising" will then be an error - or worse; malicious. In the case of a json service, if the client doesn't submit valid json for your schema / api - error out.
    • e12e 7y ago
      TFA touches on this with validation - validation is ux improvement - help the user submit only valid data. At that point there should be no proper use-case for invalid data to be submitted - hence reject, not sanitize and insert.
    • duncans 7y ago
      > There's html in your text input? - error out How do you detect HTML? Less-than/greater-than signs? Users are now banned from entering Less-than/greater-than signs?
      • hombre_fatal 7y ago
        I once registered for a forum to ask a question, but they had it configured so new users couldn't submit URLs, probably to deal with spam. Their solution was to reject posts that contained what looked like URLs which means any time you don't put a space after a period, it's probably a URL. Like "That is fine.Pizza is good." -> http://fine.pizza http://fine.pizza detected, post rejected But it gets worse. I had a code block in my post and it was also detecting URLs in my code. `console.log` -> URL detected!
      • e12e 7y ago
• varelaz 7y ago
  As for XSS I think browsers could have done more to fix this. For example add some tag like <unsafe> or <sandbox> for part of HTML that cannot have access to cookies and javascript on the page and disables any active components, like iframes and objects. Developers can use them to renderer rich content provided by users. Right now you can do this with iframes and CORS only, but that's too heavy to implement. These tags could have their own CORS limits for example. Why I think that it's a browser problem. Security of the output needs to be reviewed with browser features added and only on browser level it can be up to date with all new features.
  • brlewis 7y ago
    Do you think CSP doesn't do the job? https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
    • deleted 7y ago
      [deleted]
    • varelaz 7y ago
      It does but only on whole document. Problem that by default it allows everything, people are lazy to find how it works and setup it right. Also you need to open every third party separatelly which could work bad for ads. If there will be a sandbox, you can allow only what you need for a particular part. I understand that this conception looks complex and more like iframe. Basically right now a lot of ads content rendered in iframes without src which are kind of sandboxes in this case.
  • megous 7y ago
    Some people would still come up with: echo "<unsafe>$comment_content</unsafe>"; and feel falsely safe.
• AstralStorm 7y ago
  The sane way is to escape input. Be 8-bit safe by using hex escapes and if necessary, use specialized Unicode hex collation rules. Every text processing is supposed to have clear escape rules. Albeit there were bugs where accidental unescaping does happen.
  • varelaz 7y ago
    Problem with escaping that you need to know what you escape for, escaping is needed for almost everywhere (SQL, URL encoding, HTML, JSON, YAML) and double escaping could break the content.
• peterwwillis 7y ago
  You don't "sanitize" input, you "sane-itize" input. The whole point of checking the input is to make sure it's valid, not to try to scrape away cruft you don't think is valid. Example: phone numbers. Is the input you accepted a phone number? Well, if you were just "sanitizing input", you might pass it through some generic "sanitizing function" that just checks if it had "malicious characters" or something and strip those out. But what you should actually be checking is is this a phone number? By making sure the input is what it's supposed to be, you not only gain a better security posture, but you improve your program's reliability by making it operate in the way you expect it to. Some input fields like "give me a random block of text and I'll store it" are very hard to validate, so for those fields you can encode them as Base64 for storage, and at output time decide how to format them safely. Also consider wrappers for any functions which pass input from a user. Perl's Taint Mode (https://en.wikipedia.org/wiki/Taint_checking https://en.wikipedia.org/wiki/Taint_checking) is a global way to enforce this, but for languages that don't have a Taint Mode, you'll have to implement it yourself.
  • michael-ax 7y ago
    ++ for "validate" and "sane-itize"! its is about input validation after all
  • jessaustin 7y ago
    Some people really go overboard with this sort of validation, however. I've had to argue with vendors who didn't believe email addresses can end in something other than ".com" or ".org".
• hamilyon2 7y ago
  Escaping output does not always work when, e.g. you have thousand of integrated systems and don't control any of them nor their upgrades. If you don't filter malicious inputs, they will forever live in your database and it takes one bad release of some reporting tool somewhere for your users to become vulnerable.
  • wccrawford 7y ago
    If you rely on sanitizing before storing, you can end up with data in your database that somehow missed being sanitized, or is maliciously entered in your database. You must escape the outputs, no matter how hard you try to sanitize the inputs. Losing "control" of any integrated systems means your system is vulnerable, even if only to someone at a terminal typing things into the DB manually.
    • hamilyon2 7y ago
      Of course you must escape your outputs, get in-depth layers you don't even need now, track how data flows in your application, patch every known vulnerability, sandbox every single thing, implement capability - based security, and many more. I didn't mean otherwise. Filtering input is not sufficient. But it is not optional.
  • strictnein 7y ago
    Not sure why you're being downvoted. Probably because a lot of the people doing so have only worked in small startups. In a larger org the data you have may end up in a lot of places you didn't expect. Integrated into systems both new and decades old.
    • hombre_fatal 7y ago
      I've worked on systems so shitty that some input was triple-escaped so that it would trickle down three unescape layers in old buggy clients deployed to the field. The thing is, there's no wisdom to take away from this monstrosity. It's just old shitty legacy ad-hoc code. The grandparent post thinks there is. That you might need to break every rule in the book to integrate shit-tier software doesn't need to be said when talking about software best practices.
• keymone 7y ago
  Bad advice. Input is the boundary of your system. Always protect the boundary of your system orders of magnitude more than it’s internals. That’s like programming basics.
  • zAy0LfpBZLC8mAC 7y ago
    There is nothing to protect there, the idea that some data is inherently dangerous is nonsense.
    • keymone 7y ago
      RCE due to buffer overflows, google it.
      • megous 7y ago
        You don't need to change the data to make it less dangerous. Just fix the program processing it. It's the program that's dangerous, not the data.
        • strictnein 7y ago
          Because you have full control over all programs in an enterprise?
      • zAy0LfpBZLC8mAC 7y ago
        And what about buffer overflows makes particular data inherently(!) dangerous? As in, what would be an example of data where it is impossible to make it non-dangerous by fixing the buffer overflow that it exploits?
        • keymone 7y ago
          not interested in debating semantics. data is turned from input streams of bytes into coherent structures in your system. the process of turning data into coherent structures is input sanitization. sanitization or validation or conformation - you can argue how to call it and where to use it, but if you don't - your internal bugs will be exploitable externally, good luck.
• lenkite 7y ago
  Always use prepared statements to set query parameters. This will handle 99.9% of all query use-cases. Constructing dynamic SQL from user input is a fool's game.
• rawfan 7y ago
  It's actually quite easy: 1. Sanitize input when you actually need/want to do that but at least to a degree that it's save to put in a database (e.g. through prepared statements) 2. Always validate input 3. Always escape output (unless you have a reason not to).
• Hitton 7y ago
  Only escaping output has one significant disadvantage. Say that you are escaping &. You'll get &amp;. Your user then wants to edit the text. You save the edited text. Now when you escape and output it again, you get &amp;amp;. Rinse and repeat.
  • ptx 7y ago
    No, that is escaping the input. The article recommendeds storing the text as-is and escaping (that is, converting it to HTML) when it's displayed.
  • zAy0LfpBZLC8mAC 7y ago
    No, the browser will decode &amp; in the HTML into the '&' character and will submit that character back to you if it is part of some form field.
    • Hitton 7y ago
      Wow, I feel really stupid now. That means I have misunderstood how does it work and have evaded it needlessly till now (not that I do much web app development).
• NohatCoder 7y ago
  A method I have generally found useful is to make a whitelist of safe characters (something like alphanumeric, comma, dot and space), and escape everything else. You might escape a bunch of stuff that technically didn't need escaping, but the method is simple, rock solid, and doesn't mangle anyone's names.
  • DuckyC 7y ago
    My name contains Ø, and im guessing i would not be able to enter that with your method. I would consider that mangling my name if i had to write o or oe.
    • NohatCoder 7y ago
      No, escape means keep, in HTML for instance Ø would become Ø escaped, but it is still there visible, same as every other character.
      • hombre_fatal 7y ago
        This kind of thinking is how your users end up getting emails from your buggy service like "Hello Østein &amp; friends, ..." and your JSON API consumers encounter the same silly output. Don't escape input. Escape based on output. Escaping doesn't mean anything until you've also specified an output format. It's not always HTML.
        • deleted 7y ago
          [deleted]
        • NohatCoder 7y ago
          You are grossly misrepresenting my post, I have said nothing about whether the escaping should be applied to input or output, please edit or delete your comment.
• ethhics 7y ago
  Postel’s law—be conservative in what you do, but liberal in what you accept from others
  • SigmundA 7y ago
    Exactly what came to mind: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle Having done plenty of interfaces between systems these are words to live by.
    • gregw2 7y ago
      It's good wisdom but beware the counter-argument. Being liberal in what you accept causes your implementation to have to maintain a wide range of implicitly defined odd inputs in perpetuity and leads to a form of lock-in based on handling of edge-cases. HTML and browser implementations that allowed all sorts of garbage behaviors are a good example of this. The robustness principle works, but it does come with a cost, like all things.
    • deleted 7y ago
      [deleted]
    • zAy0LfpBZLC8mAC 7y ago
      No, it's terrible advice as it only causes unnecessary interoperability problems and vulnerabilities. There is no reason why anyone should need to generate invalid input to your program and it is never a better idea to make every consumer more complex to deal with broken input than to make one producer create non-broken output. The only robustness to invalid input you should have is that you should not fall over when you encounter broken input, but simply reject it.
      • SigmundA 7y ago
        Both TCP which Postel wrote the spec and HTML follow this principle so it it seems have its merits. You know what followed your principle? XHTML...and the arguments were the same, its not well formed just reject it, why would you ever accept broken input. Sure it makes parsing faster and simpler and yet what actually works and is robust in the real world, HTML...
• dangerface 7y ago
  Why not both? I don't filter my input I just make sure its sane.
• billpg 7y ago
  I wrote a very similar piece six or seven years ago. People responded here that I was just making sematic arguments or otherwise fell over themselves to ignore the point I was trying to make. It is good to see from the responses here that we've learnt absolutely nothing. Shameless plug: https://blog.hackensplat.com/2013/09/never-sanitize-your-inputs.html https://blog.hackensplat.com/2013/09/never-sanitize-your-inp...
  • palant 7y ago
    I've been saying the same for at least a decade, e.g. in https://palant.de/2016/03/02/why-you-should-go-with-secure-by-default-for-your-web-application/ https://palant.de/2016/03/02/why-you-should-go-with-secure-b.... It's ridiculous that somebody still has to explain it.
• Izmaki 7y ago
  What is horrible advice is to tell people to never sanitise input, but then forget to switch the focus to what should be done instead. There's too much time spend justifying the headline vs. explaining what should be done instead and why this is more effective.
  • z3t4 7y ago
    Instead prevent "injections" by using innerText instead of innerHTML and parameterize SQL queries instead of concatenating strings. But you always want to sanitize user input! Ever wondered how the average age of your user base was so high, only to discover that some users claim they are several million years old. You don't want to sanitize, you want to sanitYize. People writing their e-mail as street address and vice versa.
• rossdavidh 7y ago
  So, essentially he is saying: 1) go ahead and accept the risky thing from your users, and store it right there in your database, but 2) make sure that you remember, in every single place in your code where you read that out of the database, to treat it properly, and 3) make sure that every other programmer, now or at any time in the future, remembers to do this also, in any code they write which reads user input out of the database and puts it on the screen. What a bad idea. Don't leave landmines there for other maintainers of the code to step on. Especially because the other maintainer may actually be you, six months or a year from now. Sanitize your inputs. Also, escape your outputs.
  • BurningFrog 7y ago
    One of my most important rules: Whenever you say, think, or imply something starting with "So from now on we just have to remember...", you're really saying "Let's decide this part of the system will always keep breaking!".
  • swiley 7y ago
    Or maybe just treat strings and user input as opaque objects and leave it to the libraries to escape them.
  • inimino 7y ago
    "The risky thing" here is a text string. So yes, you should be able to accept an arbitrary string and store it in your database without shooting yourself in the foot, and the same when you read it out of your database and put it in HTML, or JSON, or CSV, or XML, or ASN.1, or protobufs, or in another database. Handling escaping in HTML or REST API is web dev 101, this shouldn't be controversial.
  • coding123 7y ago
    I like the way react does it: dangerouslySetInnerHTML={{__html: variable}} Not only can you easily see every place in the code that is using this construct, but, the framework provides a warning. I definitely don't think it's bad to sanitize HTML content. but in general MOST text in a web app should be just text, and rendered with whatever HTML it contains. In very few places should the web application give user rich-text (aka HTML) access. In any place that the application does that, sanitization should be used.
• wurp 7y ago
  The article misstates what 'sanitizing inputs' means. I agree with posters who recommend passing data as parameters to methods that don't require sanitized input (e.g. stored procedures or KeyValue APIs). Also, sanitizing input means transforming input so you retain the original content, but without escape or control characters. Sanitizing input does not mean throwing part of the input away (except when you know it is meaningless in your context, e.g. spaces at the end of a name).
• dana321 7y ago
  In the example given there, use json_encode($name) That will encode any data structure properly for output into JavaScript.
• ch 7y ago
  One's output is another's input.
• _nalply 7y ago
  It depends. If you take arguments to some sub-system (an example are database keys like the id of an entity instance), then you need to sanitize input. Anyway, today I learnt something. If you have free-form data like text it makes sense not to sanitize it because in this case sanitizing depends on the output domain. For example < is dangerous for HTML and ' is dangerous for SQL, and so on.
• water42 7y ago
  Don't filter input. Instead, prevent certain characters from being input in text elements. This is a user experience problem, not a software problem. The software can validate that a "name" is rejected if it does not follow the front end validations, but it doesn't need to do any more than that. Of course, this argument does not extend beyond a "name" field to more complex fields. But more complex fields are less susceptible to introducing UX problems if certain characters are sanitized.
• kazinator 7y ago
  Problem with this kind of description is that input and output are often two names for the same thing. One processing element's output is the next one's input. Basically, don't substitute HTML cruft onto text, except at that stage in processing when that text is just about to be inserted into HTML. Don't do HTML-ization prematurely. You wouldn't encode and store data in Base64 just in case it might be needed that way in some future processing step.
• patrec 7y ago
  Arrrghh, it's 2020 and developers are still too dumb to understand the difference between a string (list of characters) and a tree (sql, html, ...). No, the solution is not to "sanitize" input, output or both! The solution is to not use the same type for text and trees! We should rid the world of brain-damaged templating solutions like jinja, go template and similar garbage that pretends your SQL or HTML is some flat string that just needs a bit of extra contextual "escaping" magic [1]. If you interpolate into a proper AST there is no problem and you need no "escaping". For efficiency reasons you may not want to directly interpolate into an actual ast and then de-serialize all of it to a string again just to output it down a socket in the next line, but that's just an optimization. Bourne shell fucked this up as well (for an easier case of interpolation, since there is basically no nesting) and it remains a constant source of severe bugs and security holes in shell scripting as well. By contrast lisp has been doing this right for literally decades: `(html (div ,some-string-to-interpolate ,@a-list-of-inline-elements-to-splice)) [1] Yes, I get it's "convenient" to have a "universal" solution that works for any type of file. But html and sql in particular are easily important enough to have a correct solution and it's not hard, it's not inherently slower and it's way more convenient in any real sense than puzzling about several rube-goldberg sanitization schemes, running some stupid security linter and paying $$$ to pen-testers for hunting down this crap.
  • matheusmoreira 7y ago
    Parser support is non-existent in most if not all languages. Every language I know is able to parse regular languages at best. Parsing HTML and SQL and manipulating the resulting tree is not the first solution developers think of. We should be able to look up some RFC, give the EBNF grammar to a library and get a parser out of it. In order to do that today, we need to use ancient parser generator tools. Why? A parse(grammar, input) -> tree function would be easier to use. The Earley algorithm can receive a grammar as input. Related: http://langsec.org/ http://langsec.org/
    • patrec 7y ago
      Well, I'm not some prefix-fanatic, but much of the problem would not exist in the first place if we had just used some sexp style syntax for HTML, it would be more pleasant to edit, and faster and much easier to parse for both humans and machines to boot. Another billion dollar mistake. So I feel a bit ambivalent about attempts to lower the costs of pushing out more over complex grammars into the world. When have you last used in earnest a non-sexp/internal DSL for something like build systems that didn't engender in you an occasional urge to visit physical violence on its creators? But what I'd unambiguously like to see is easier parsing of "sane" languages and the death of perl-style regexps. Still my guess would be: 95% of trouble comes from two: html (including js, css, svg etc. unfortunately) and sql. And most of the remaining 5% from bash :) So just dealing with those three would make a big dent. Also things are not quite so bleak as you make them out to be: jsx is much saner and any established mainstream language has a conforming html5 parser these days (sadly, to do it properly you also want something that deals with the various other languages that get munged into html: css, javascript, gimped xml and here the situation is less good). SQL is thornier (and has many wildly different dialects), but unless you need dynamic queries, parameterized queries are available everywhere. In fact a 1% effort/80% of the benefits approach is to not bother with parsing at all and to just use different types for e.g. html and text (interpolate html into html as string-interpolation as text (i.e. plain strings) into html as escaped string).
• dwheeler 7y ago
  I think the title here is misleading. The title is, "Don’t try to sanitize input. Escape output." The article itself is only talking about sanitizing user input to "prevent cross-site scripting attacks". He later on does require input checking: "Input sanitization is usually a bad idea, but input validation is a good thing... by all means validate it and return an error if it’s invalid." It's vital for secure programs to check their inputs and minimize what they will accept. I do think it's a good idea to reject "&" and "<" when you can. But I also agree that in most cases you can't completely forbid all inputs that have HTML metacharacters. In the case of cross-site scripting, the best countermeasure is output escaping. Many modern frameworks do output escaping by default; Rails (for example) has done it for years (in Rails, any "normal" string is automatically escaped when sent back out as HTML). A good reason to prefer one framework over another is because it has secure defaults; if your framework doesn't escape by default, you should consider using a better framework. You can't depend on just one thing to suddenly make your software secure. You need to validate inputs so that only valid inputs are accepted into your program. You need to escape output, because there are often legitimate input characters that must be escaped. You need to prefer tools that have safe defaults. Use tools to scan your results, to find what you missed. It's not rocket science, but it does require a set of approaches; there is no silver bullet for making secure software.
• thdrdt 7y ago
  Ofcourse you can validate input! Validation != Sanitation In this thread some people seem to confuse those. You can validate if all input characters are utf-8. But the moment you start to 'sanitize' non-utf-8 input into utf-8 you are in trouble. It's best to notify the user that the input validation failed and that you don't accept the input.
• Jugurtha 7y ago
  "Don't tell me what the fuck to do - tell me what you did and what has worked for you in a given context."
• tobyhinloopen 7y ago
  This is a terrible idea.
• HelloNurse 7y ago
  A good point, but it is obscured by a terrible choice of vocabulary: in this article "sanitizing input" actually means specifically trying to sanitize input of the whole system as soon as it is received (quite impossible, given the open-ended and conflicting nature of sanitization needs) and "escaping output" actually means sanitizing the input of a specific subsystem for a specific purpose. Given a faithfully persisted and assumed unsafe original text, SQL query builders can turn everything into SQL strings or die trying, XML parsers can check entity expansion size and other traps, HTML generation templates can introduce fancy markup to surround arbitrary text, XML generation templates can escape input wholesale in CDATA sections, and so on. It's the traditional principle of separation of concerns.
• jiveturkey 7y ago
  amazing that in 2020 this is still so poorly understood. not as evidenced the TFA (which I didn't bother to read), but by the strong, uh "opinions" here on HN.