8 ms·
Don’t try to sanitize input, escape output (2020)
- lsb 2y agoOf the “six famous bad ideas in computer security”, the first and second are “default permit” and “enumerating badness”. http://www.ranum.com/security/computer_security/editorials/dumb/ http://www.ranum.com/security/computer_security/editorials/d...
- josephg 2y agoThis isn't “default permit” or “enumerating badness”. Its kind of the opposite. The idea is that you don't want to store text in your database in a form that is safe when rendered as HTML, JS, JSON, SQL, etc. That would be "enumerating badness". Instead, at the moment you render the text as HTML, you encode the text into an HTML-friendly form (via escape characters). If you want to embed the text into a SQL query, have your SQL library add sql-specific escape characters where needed in the text. Same for your JSON library, and so on. Its the responsibility of an encoding library to encode and decode text in the appropriate way. A JSON or SQL library should be able to encode then decode any arbitrary unicode string, even one which contains quote characters. Just like how any arbitrary unicode string should be able to be used on a webpage, in a text field without being able to interact with the rest of the page in any way. Most libraries already do this if used properly. SQL libraries (using parameters) will escape text where needed. React will embed text in an html-safe way. JSON libraries escape quotes in strings. And so on.
- simonw 2y agoThis is such an important lesson, but it's a difficult one to convince people of - telling people NOT to sanitize their input goes against so much existing thinking and teaching about web application security. It's worth emphasizing that there's still plenty of scope for sensible input validation. If a field is a number, or one of a known list of items (US States for example) then obviously you should reject invalid data. But... most web apps end up with some level of free-form text. A comment on Hacker News. A user's bio field. A feedback form. Filtering those is where things go wrong. You don't want to accidentally create a web development discussion forum where people can't talk about HTML because it gets stripped out of their comments!
- deleted 2y ago[deleted]
- toolslive 2y agoAn anecdote: I haven't checked recently, but at least until 2015 there was a Belgian govt web site for the register of companies that would yield 0 companies in the city of Aalter. The reason is not hard to find. In fear of SQL injection they would filter out all SQL keywords; in this case "alter".
- Dalewyn 2y agoClassic Scunthorpe.
- endofreach 2y agoYou beat me to it. I couldn't remember the exact term. I just knew the "pattern" – and that it wasn't something that would have made the issue as obvious as eg. the pee-ass-word.
- shalmanese 2y agoDon't you mean Clbuttic Scunthorpe?
- xxs 2y agoI have never sanitized any input, full respect to whatever text there is. It's simple - html/xml/javascript/json/url is not text. You render it with whatever tools you have to - and that tools happen not to be concat. You render xml - use DOM, xslt, etc. html - same story, use whatever templating engine you wish. json - use your own model and render it to json. SQL - prepared statements.
- zelphirkalt 2y agoI think this misses important aspects. One does not often want to render HTML in place of a user name on a website as HTML, for example.
- marticode 2y agoWhy not not both? Escaping output should be a requirement but doesn't hurt to remove obvious garbage in the input (including harmless stuff like pointless spaces)
- sfvisser 2y agoThe article clearly states why not: first you probably screw up your data (example is names, but there are many more examples) and second what is garbage depends on your output usage. Html/sql/JSON/etc all require a different sanitization.
- cjfd 2y agoThe article tell us that input validation is okay but argues against input sanitization. I.e., if there is obvious garbage in the input tell the user that the input is wrong and don't store it and also do not try to correct it. That seems good advice in most cases. Your example of point less spaces is good though. I think it is generally a good idea to remove those.
- dotancohen 2y agoRegarding harmless stuff like pointless spaces. I know a person who uses two spaces between his first and last name, because his culture users a second given name yet he has none. So, one space between the first given name and the second (nonexistent) given name, then another space between the second (nonexistent) given name and the family name. You might think it is weird or unnecessary, but that is his identity. One could counter with far weirder or seemingly unnecessary things we accept regarding peoples' identity today.
- WillAdams 2y agoObvious solution is for this person to adopt the "Zero Width Non Joiner" character as their middle name --- that would ensure that this doubled-up-space would be preserved.
- dotancohen 2y ago
- KingOfCoders 2y agoI think the challenge is, you share data with other systems. If you don't treat "sharing" as "output" you're in trouble.
- cowboylowrez 2y agoyeah I think this is the key takeaway for me, because the sanitation needed depends on the consumer of the output. i was never a successful developer but when i was at least trying, i would constantly run into other devs that would happily code forms that allowed sql injection, even in relatively normal winforms apps, not to mention poor understanding of regular ol concurrency / transactions. since i had a poor reputation (which i take full responsibility for), my concerns would always be dismissed by a combination of "elitist ivory tower thinking", "toxic interactions" and rebuffed with comebacks like the "database server just handles this" etc etc. if your comment is anything but solid black for the duration of folks reading it, its just more evidence that the vast majority of developers are just shit at their jobs haha
- KingOfCoders 2y ago"my concerns" Sorry, decades ago I was one of those people.
- cowboylowrez 2y agobuilding software is just a wierd activity for humans i think lol
- ungamedplayer 2y agoThe reason you sanitise input is because the data can attack the host and the client. This post has a narrow view on attackers.
- sfvisser 2y agoIt’s really not. The point is that sanitizing output is a better way to protect the host and the client.
- cowboylowrez 2y agoyes, and the article clearly agrees with you, different destinations of the data have different escaping requirements.
- worthless-trash 2y agoSure, keep writing exploitable code then, have fun!
- ww520 2y agoStill looking for a way to safely parse HTML string into DOM while avoiding XSS attacks. Most solutions end up with sanitizing input.
- danielheath 2y agoOur approach at work: parse it as HTML, define a short list of known-acceptable tags & attributes, and strip everything else. Limiting attributes to ["href", "src"] and tags to ["p", "br", "h1", "ul", "ol", "li", "span", "div", "img"] gets you remarkably close to rendering the safe bits of HTML - add to that list upon request. If you want to take it further, use an `iframe srcdoc=""` with sandbox attributes set.
- masklinn 2y ago> Limiting attributes to ["href", "src"] You need to clean that up as well to avoid e.g. javascript: links, and then there are more issues with SVG if you allow media uploads. Then you need to be very sure you’re using a proper html5 parser and your rendering is completely canonicalized or you open yourself up to filter evasions (https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Evasion_Cheat_Sheet.html https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Ev...) And of course I assume that’s what you meant but you should not add upon request, you should evaluate the addition.
- danielheath 2y agoYes - just double checked those, thankfully the framework builtins are correct (staying up to date with a well maintained framework does wonders for your security posture).
- furstenheim 2y agoWasn't there this case of a security issue coming from abusing different parsers, in different places? Server, client, or different browsers
- 2y ago
- TheChaplain 2y agoDisagree. Escaping/sanitizing on output takes extras cycles/energy that can be spared if the same process is done once upon submission. Think more sustainable.
- oops 2y agoYes I love seeing < in my database. Every time I see it I think oh boy how many cycles will I save when I display this in HTML!
- sameoldtune 2y agoIt’s a trade off for sure. But if you know at the time of collection all the ways in which your data is going to be used in the future, you work at a more well-run organization than me!
- TheChaplain 2y agoObviously one thing does not exclude the other, but the more processed data is before, the less has to be done later for every page retrieval. It surprises me that this seem unfamiliar these days?
- kolinko 2y agoThat is true only for simple systems that don't change often and that have a single view of the data. For most real world use cases your approach is break even at best, and often way worse.
- BoredPositron 2y agoYou mean it's not suitable for the deploy every hour crowd it has nothing to do with complexity.
- vouwfietsman 2y agoNot sure the escaping/sanitizing proposition can hold a candle to the overwhelming performance dumpster fire that is modern web dev.
- zzo38computer 2y agoIt is a reasonable idea, but there are other things that can be done too. However, in the stuff about SQL, you could use SQL host parameters (usually denoted by question marks) if the database system you use supports it, which can avoid SQL injection problems. If you deliberately allow the user to enter SQL queries, there are some better ways to handle this. If you use a database system that allows restricting SQL queries (like the authorizer callback and several other functions in SQLite which can be used for this purpose), then you might use that; I think it is better than trying to write a parser for the SQL code which is independent of the database, and expecting it to work. Another alternative is to allow the database (in CSV or SQLite format) to be downloaded (and if the MIME type is set correctly, then it is possible that a browser or browser extension will allow the user to do so using their own user interface if they wish to do so; otherwise, an external program can be used). Some of the other problems mentioned, and the complexity involved, are due to problems with the messy complexity of HTML and WWW, in general. For validation, you should of course validate on the back end, and you may do so in the front end too (especially if the data needed for validation is small and is intended to be publicly known). However, if JavaScripts are disabled, then it should still send the form and the server will reply with an error message if the validation fails; if JavaScripts are enabled then it can check for the error before sending it to the server; therefore it will work either way.
- foota 2y agoIt's buried a bit in the article, but if you have to sanitize input to allow only some kinds of inputs (e.g., specific tags), you should really be parsing it fully to an AST and then acting on that (or using a library doing the same) since otherwise you're going to be subject to all sorts of pain.
- sameoldtune 2y agoThis doesn’t have to be over engineered either! After all an enum is a perfectly valid AST.
- antonvs 2y agoWhich raises another pithy phrase in this area: parse, don't validate. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
- dudeinjapan 2y agoPorque no los dos?
- zer0c00ler 2y ago[dead]
- buro9 2y agoI store the raw input in my database, but run it through bluemonday before rendering it. Simples. https://github.com/microcosm-cc/bluemonday https://github.com/microcosm-cc/bluemonday
- Doxin 2y agoThat seems like it's doing a LOT more than what you'd want a sanitizer to do. For HTML you really just want to escape a couple characters with special meaning. You probably can get away with just replacing < and > with their substitution sequences. That way people can still discuss XSS exploits without your sanitizer deleting a bunch of the text they entered on purpose.
- terraexpert 2y agoWe were contacted by a bug hunter once stating he has access to our database and asking for a bounty for his finding, he even provided a sample of first 100 users from the users table in the database. After some investigating, I figured out how did he obtain the data. He was one of the first 100 users, he set one of his fields to an xss hunter payload, and slept on it. After two years, a developer had a dump of data to test some things on, and he loaded the data into an sql development software on his mac, and using his vscode muscle memory, he did a command+shift+p to show the vscode command bar, but on the sql editor it opened "Print Preview", and the software rendered the current table view into a webview to ease the printing, where the xss payload got executed and page content was sent to the researcher. Escape input, you never know where will it be rendered.
- orangepanda 2y agoIf you dont know where it will be rendered, how do you know what to sanitize for?
- KronisLV 2y agoSanitize it for all XSS. Or better yet, avoid something like HTML or anything that can contain executable instructions, when all you need instead is a regular markup language. I’ve seen HTML be used for user rich text input and it was an absolute mess, with old data that wasn’t properly sanitized, the sanitization library itself getting outdated, someone putting potentially unsafe content in from another system and so on, whereas people would sometimes bikeshead and worry about breaking old style classes or display of the data across multiple systems instead of addressing just how serious the potential risks are. Not all of the details here might be accurate, but honestly just use Markdown or something like that for user input, disallow HTML altogether and never use the raw input.
- jefftk 2y agoThat sounds like a pretty serious bug in the sql development software!
- HHad3 2y ago
- WillAdams 2y agoI still wish that the Unicode folks had set up a bunch of duplicate code points which could have been used exclusively for processing marked-up text and that the folks making markup systems/languages had followed through. Say one was updating TeX to take advantage of this --- all the normal Unicode character points would then have catcodes set to make them appropriate to process as text (or a matching special character), while "processing-marked-up" characters would then be set up so that for example: - \ (processing-marked-up variant) would work to begin TeX commands - # (processing-marked-up variant) would work to enumerate macro command arguments - & (processing-marked-up variant) would work to delineate table columns &c. and the matching "normal" characters when encountered would simply be set.
- Udo 2y agoThey're not even related. Sanitizing input is at best a formatting/style issue. Escaping output is a security issue.
- ecjhdnc2025 2y agoEhhh!? I don't get this at all. You obviously do both. 1) you get your input data into the form that is meaningful in the database by validating, sanitising and transforming it. Because you know what form that data should be in, and that's the only form that belongs in your database. Data isn't just output, sometimes it is processed, queried, joined upon. 2) you correctly format/transform it for output formats. Now you know what the normalised form is in the database, you likely have a simpler job to transform it for output. It's not just lazy to suggest there's a choice here, it's wrong.
- growse 2y agoValidating data on input is just checking for "known badness", and is pretty ineffective. If you've got specific structure requirements for the data you store, parse it into that structure.
- ecjhdnc2025 2y agoRight -- I would include that in "transform" in my example above. I've seen too many forum developers spend far too much time after the fact dealing with their decision to "just use TinyMCE" ==> Oh hey, you're a server-side HTML parsing expert now anyway; wasn't that what you were trying to avoid?
- chx 2y agoThis has been the way for Drupal since ... 2005 at least. My memory becomes fuzzy before that. Since 2015 it's highly automated too thanks to Twig autoescape.
- kazinator 2y agoOf course you should sanitize input, and escape everything properly in the context-specific way. Defining what is valid for an input field and rejecting everything else helps the user catch mistakes. It's not just for security. Some kinds of information are tricky to sanitize. Names, addresses and such. Especially in an application or site that has global users. Do the wrong thing and you end up aggravating users, who are not able to input something legitimate. But maybe don't allow, say, a date field to be "la la la" or even "December 47, 2023".
- deleted 2y ago[deleted]
- atmanactive 2y agoAbsolutely the worst advice ever!
- hinkley 2y agoThis is another place where 80% of the time one way works but 20% of the time you need to go the other way. Of course once the product is in production you can swim one direction but not fight the current going in the other. You can always move to escaping output, but retroactively sanitizing input is a giant pain in the ass. But the problem comes in with your architecture, and whether you can discern data you generated from data the customers generated. Choose the wrong metaphors and you end up with partially formatted data existing halfway up your call stack instead of only at the view layer. And now you really are fucked. Rails has a cheat for this. It sets a single boolean value on the strings which is meant to indicate the provenance of the string content. If it has already been escaped, it is not escaped again. If you are combining escaped and unescaped data, you have to write your own templating function that is responsible for escaping the unescaped data (or it can lie and create security vulnerabilities. "It's fine! This data will always be clean!" Oh foolish man.) The better solution is to push the formatting down the stack. But this is a rule that Expediency is particularly fond of breaking.
- shaftway 2y agoI've always been a big fan of structuring data on input, escaping it on output. I think the big problem with just escaping output is that you can accidentally change what the output will actually be in ways that your users can't predict. If I am explaining some HTML in a field and drop `<i>...</i>` in there today, your escaper may escape this properly. But next month when you decide to change your output to actually allow an `<i>` tag, then all of a sudden my comment looks like some italicized dots, which broke it. Instead if you structure it, and store it in your datastore as a tree of nodes and tags, then next month when you want to support `<i>` you update the input reader to generate the new structure, and the output writer to handle the new tags. You preserve old values while sanitizing or escaping things properly for each platform.