10 ms·
You should do both. Sanitize your inputs so that it can be safely stored in your data store, and then sanitize your output so it can be safely displayed. We d
by jedberg 7y ago
You should do both. Sanitize your inputs so that it can be safely stored in your data store, and then sanitize your output so it can be safely displayed.
We did this at reddit. We had basic SQL sanitization on the way in, and then a full pass on the way back to the user. The advantage this gave us is that when someone discovered a new way to hack our sanitization, all we had to do was update the output filter and everything was magically safe.
We didn't have to do full database scans to find all the bad data and change it.
Edit: Apparently I shouldn't have simplified "paramaterazation of SQL" as "sanitize your input". I used the more generic term since I was talking about any kind of data store. But yes, it was of course parameterized.
- iainmerrick 7y agoWhat does the “basic SQL sanitization on the way in” consist of and what does it do for you? The article makes sense to me; if I’m just storing strings in a database, I don’t see why they would need to be sanitized at rest, even if they contain malicious SQL code. Only when I actually come to use those strings for some purpose.
- bardan 7y agoBut what if at some point somewhere down the line someone forgets to sanitize the output? Surely better procedure to sanitize at both ends. Nobody is perfect.
- laumars 7y agoYou might need to escape strings differently depending on where you’re outputting eg HTML or JSON.
- cjfd 7y agoThis is completely the wrong attitude. Code should be correct, not fail-safe. All of the safety hatches that people tend to introduce make the code less predictable and eventual problems tend to arise far away from where they originated, making them difficult to diagnose. What is the result of this sanitization? Now we have some undefined, changing internal string format running around in our application, and possibly multiple undefined and changing internal string formats, where it is also unclear at what point a string is supposed to be in what format. If something is an arbitrary string it should be allowed to be an arbitrary string and the things handling that should escape it the appropriate way. The article is correct.
- Scarblac 7y agoCode should ideally be correct and fail-safe; deal gracefully with bad input, ensure correct output, at all levels. Ideally. That still doesn't mean you should care about output sanitization on data entry, as you don't know how it will be output yet.
- zAy0LfpBZLC8mAC 7y ago> Code should ideally be correct and fail-safe; deal gracefully with bad input, ensure correct output, at all levels. Ideally. NO! If you get bad input, you should fail, loudly. Anything else is a recipe for disaster. (Not as in "crash the whole system", of course, but as in "reject the request".)
- eythian 7y agoThis tends to leads to systems that don't work and die on real-world edge cases.
- zAy0LfpBZLC8mAC 7y agoNo, it doesn't. That is what "accept anything" programming leads to. The idea that "accepting all the inputs" somehow gives you an advantage is an illusion: If the semantics of some input are not well-defined, then the only thing you gain by accepting it anyway are hard to debug interoperability problems and vulnerabilities. When some input is not well-defined accoridng to the spec, then your interpretation is just a random guess, and the next developer will make a different random guess as to what that input means, and so an interoperability problem and potential vulnerability is born. If you reject the invalid input, you will notice the error and thus fix the source of the invalid input to produce input for which the semantics are actually well-defined.
- ashearer 7y agoThis sounds good in theory, but I'll give a counterexample. Requirement: Name input box. Implementation: We'll sanitize the input by rejecting any characters likely to be dangerous if mishandled, like single quotes, or anything else we don't immediately imagine to be useful. If a character turns out to be needed later, that's no problem. We'll just change the list. Security audit: Passes Later customer complaint: I can't sign up! — J. O'Brien Dev team: Sorry, too bad. We'd have to re-audit everything and possibly modify code to allow your last name, because there might be code somewhere that relies on the original sanitization for security. That was the point of sanitizing on input, after all. If you want to sign up, it would be easiest for us if you would just change your name.
- XMPPwocky 7y agoDouble-escaping is silly & it\'s just plain incorrect.
- Scarblac 7y agoYou can't sanitize for output at input time, as the sanitization that needs to be applied is different for HTML, JS and JSON. You don't know that at input time.
- jve 7y agoWell, use libraries/frameworks that ENFORCE you to sanitize and makes that exceptional case to output raw content. Examples. PHP: Using mysql_escape_string is a no-no - you will forget to add it one day. Using parametrized queries you won't write unsafe SQL. .NET Core - Outputting to HTML by default only outputs those chars to HTML which are in predefined UTF range. All other chars will be converted to HTML entities. If you want to output raw, you must explicitly use @Html.Raw https://docs.microsoft.com/en-us/aspnet/core/mvc/views/razor?view=aspnetcore-3.1#expression-encoding https://docs.microsoft.com/en-us/aspnet/core/mvc/views/razor...
- JimDabell 7y agoYou're thinking about this as if data can be in one of two states – untrusted or sanitised. This is not the case. When you output arbitrary data, you need to encode it in a way that is suitable for that context. These contexts might be: - Generating a web page. - Including in a JSON response from an API. - Sending an email. - Storing in an SQL database. These all use different formats / protocols that use different syntax to encode data. How you correctly encode data for one of them is different to how you correctly encode data for another of them. There is no method of taking untrusted data and "sanitising" it so that it is correct for all of them. What works for one will break for the rest. If you want to handle arbitrary data correctly and safely, store it as-is and when the time comes to use it, encode it appropriately for the context you are using it in. Where possible, use tools and systems that get it right by default instead of requiring developers to remember to encode correctly, e.g. generate HTML with templating engines that encode data as HTML by default, and use parameterised queries with SQL.
- TeMPOraL 7y ago> generate HTML with templating engines that encode data as HTML by default Don't, unless you're sure the templating engine actually parses the HTML into a tree of nodes before interpolating and re-emitting it. Otherwise it's likely someone will interpolate something in an improper context, e.g. inside <script> or <style> block.
- thephyber 7y agoThis doesn't pass the sniff test. If someone can forget to sanitize the output, someone can also forget to sanitize the input. The most important things are to understand where the content is used, use the appropriate output encoding/escaping, have rigorous tests to ensure your expectation correctly escapes nasty strings, and that you keep the output escaping code up to date to protect against novel attacks and new browser/app features. I worked at a social media company with one of the largest text-based user-content-stores in the world at the time. Some of the features had input-side encoding and some had output-side encoding. I was there ~10 years after the bad practice of input-side encoding started and it very quickly became too cumbersome to know exactly which fields were encoded with what encoding (and I mean both character encoding and htmlentities / specialchars / specific character stripping / etc). We started getting ridiculous bugs like passwords could not contain '&' characters or logins would fail matching what we had in the DB. It's not about being perfect. That will never happen. It's about storing exactly what the user submitted (if it is accepted by the POST submission logic) and to correctly encode the output for the correct security context (HTML, XML, JSON, html entities, html attribute, script tag, styles/stylesheet, urls, uploaded filename / file contents, filesystem injection, command injection, etc). These all have different rules. You can unintentionally open yourself to a vulnerability in one if you only expect the output to be displayed in HTML.
- benhoyt 7y agoThere's value in doing both in certain cases (as long as you're definitely escaping output), but I'd be careful about the motivation. You say: "Sanitize your inputs so that it can be safely stored in your data store" -- but it's safe to store any string in your database as long as you escape/encode it correctly (i.e., use parameterized queries). For example, what if someone posts a legitimate comment on reddit helping someone with SQL syntax for deleting tables and includes "DROP TABLE users" -- did you sanitize that away?
- jedberg 7y ago> did you sanitize that away? No, see my edit above. It was just parameterized.
- iamaelephant 7y agoAre you implying that Reddit is concatenating SQL queries instead of parameterizing properly? Because based on the reliability of the website I'm not surprised by this, but it's still hilarious.
- jedberg 7y agoNo it is not just concatenated.
- laumars 7y agoAre you sure that’s what is happening at Reddit? You shouldn’t need to sanitise your inputs for SQL. Paramatised SQL has been a thing in some languages for two decades now. This really is a long solved problem by now. Output is a different matter though but that’s because of rendering content safely down to HTML, JavaScript or JSON (to name a few examples). SQL shouldn’t come into the equation by this point.
- tie_ 7y agoThis. I'm tired of people implying or right out stating that SQL injection is an input validation problem. Why couldn't you have foo' OR 1=1; as a title of your post? It is all good characters as far as text entry is concerned. SQL injection is really a problem of how you pass parameters to your SQL layer. Parametrized queries are the (easy and widely available) solution. If you are concatenating input to your SQL queries, you're doing it wrong.
- squiggleblaz 7y agoHow many people named O'Brien are told they can't sign up, or passwords get rejected because they contain special characters? It's crazy. Even if you're using 1990s technology without parameterised queries, it's not like it's impossible to say `insert into users (name, motto) values ('O\'Brien', 'foo\' OR 1=1;')`.
- fwsgonzo 7y agoYep. I've stopped counting the amount of websites that force me to use weak passwords. It's crazy that this is still a thing in 2020. I wish the controls on browsers came with a green V that implements best practices (8+ symbols, no filter) so that people who made websites understand that this is what they should conform to. Not their own misconceptions about password security.
- lodovic 7y agoI wish websites would stop enforcing a "password policy". An insecure password should be a choice. If you are so sure you cannot secure your site, leave authentication to a third party provider. All this leads to is zillions of user accounts that are used only once.
- zAy0LfpBZLC8mAC 7y agoNow, you have ammended your comment to really say something very different (how is parameterized SQL 'basic' anything when it's actually the correct complete solution to the problem?). But in any case, this still suggests a complete misunderstanding of the point of that blog post. As far as that blog post's point is concerned, the SQL database is an output of your program. And the whole point is that you need to escape/encode all outputs correctly, but you should not ever sanitize anything.
- jedberg 7y agoBecause as others have said, parameterized SQL is a long standing solution, so I consider it pretty basic, but I still consider it a form of sanitization that not everyone does, even though it's a long standing solution. I think it is totally fair to interpret the article as saying that the database is a program output. And if you interpret it that way, what I said doesn't make sense. That's just not how I interpreted it.
- IshKebab 7y agoIf it's even possible for someone to "find a way to hack your sanitisation" then you're doing it wrong.
- bloak 7y ago> Sanitize your inputs so that it can be safely stored in your data store What kind of data store can't hold all possible strings of characters? Except perhaps '\0'. I'll make an exception for '\0'. (EDIT: There is of course the question of what you mean by a character. Sometimes it's an octet. Sometimes it's a Unicode code point.)
- jedberg 7y agoIt’s more a question of how you get them in there. For example if your data store is JSON based, you’ll need to escape some strings.
- TeMPOraL 7y agoThat sounds like a broken datastore API. In a properly designed API, you don't need to escape anything, because the API implementation ensures your data doesn't get read as code.
- anonsivalley652 7y agoYeah, I "LOL" at these type of One True Way™ proclamatory headlines. What I don't understand is the lack of using proper escaping functions when generating SQL. Templating SQL without escaping is the surest way to a SQL injection. --- Bad "SELECT * FROM USERS WHERE name = '%'" % (name) --- Good "SELECT * FROM USERS WHERE name = '%'" % sq_esc(name) # where sq_esc() doesn't add outer ', but escapes anything that needs it And to defensive coding: 0. Sanitize input. Always, always. 1. Assert pre-condition invariants.* 2. Process. 3. Assert post-condition invariants.* 4. Generate correct output by understanding the output domain. * Unit tests, smoke tests, integration tests and code-coverage alone are insufficient to cover complex code paths. Fuzzing with asserted invariants is a good way to shake the dust out of hairy code.
- DelightOne 7y agoDefensive coding makes sense. What is sql templating used for in the face of parameterized queries?
- lucideer 7y agoReplying to the edit for further clarification: > Apparently I shouldn't have simplified "paramaterazation of SQL" as "sanitize your input" It's not a simplification, it's incorrect. An SQL query is output. You are sending data out of your application code, via the SQL library driver. It may seem like I'm splitting hairs here, but I've seen this distinction misunderstood in this way often enough in situations that severely compromise security. Some people think of "output" as purely the end product of application flow, and all I/O that happens in between is somehow lumped together as indistinct "input". There's a reason I/O has two separate letters, and the distinction is crucial for securing your application. TL;DR: values passed to an I/O function (like stdout, file.write(), response.write(), db.query(), etc.) are all output, those returned from an I/O function (like stdin, file read, db query results or requestObject.getQueryParam()) are input. Sanitize the former, NOT the latter. Validate the latter (ideally. Though I'd say this is more about stability than security).
- dropmann 7y agoIt depends on the perspective, in case of SQL I would argue that sanitizing the input is the same as escaping the output, because the query you are sending to the database is the output. Escaping the output however as a term implies you are doing it right, while sanitizing the input could also mean you just replace("DROP", "") etc. (My last name is Dropmann, I know what I am talking about)
- ashearer 7y agoThe difference is where it's done. "Sanitizing the input" implies that it happens when the value is read, so that all uses of the value are stuck with a single result. "Escaping the output", in your example, would happen in the database or its driver, for parameterized queries. HTML output of the same value in the same request would be escaped differently within a function that builds HTML output.
- TeMPOraL 7y ago> Sanitize your inputs so that it can be safely stored in your data store, That's not sanitizing input, that's escaping output - if you subdivide concerns appropriately. The data you're saving into database is the output of your program. Really, the problem is of language translation. User input is an unstructured blob. SQL, or HTML, are structured languages, with their own semantics. Whenever you cross the language barrier, you need to translate data from one language to the other. Parametrized queries are the usual API to SQL drivers, and they do this for you under the hood, producing a valid SQL query string[0]. When going to HTML+JS, you need to invoke some library (or do translation yourself). (I really don't like the term "escaping". Translating between languages is more than just sticking slashes in front of double quotes.) This is why "sanitizing inputs" is a nonsense concept. The problem is of language translation, and you can't translate if you don't know the destination language. A blob correctly sanitized for SQL will not be correctly sanitized for HTML, and an input correctly sanitized for both will look bad in either. SQL injection and XSS are the same bug. Failure to translate between languages. Usually caused by a pretty stupid but somehow very popular idea - building target language expressions by gluing plain strings together[1]. -- [0] - Sometimes. I remember this is how it worked in the past, but not sure if server RDBMS APIs haven't changed since. For comparison, with SQLite, you're passing the query with placeholders to the SQLite functions, and the parameter values are passed as arguments. The SQLite internals turn this into an executable query, but I'm pretty sure this does not involve a query string with actual parameter values in it ever existing in memory. [1] - A related source of footguns is using templating engines for web pages. HTML is a tree of nodes, not a plain string. Using a template system is a recipe for XSS problems.
- gmueckl 7y agoAny actually decent RDBMS isn't stupid enough to first escape parameters, then parsing the query string to find placeholders, then do a bunch of string concatenation and then run the concatenated string through a second parser. It is really simpler and more robust to parse the query string once and grab the actual data value from the parameter array once a placeholder is found. However, a lot of client side libraries are cheating and embed the parameters into the query string before passing it to the server instead of implementing the proper parts of the protocol. This is about as safe as doing plain string concatenation in the first place. I don't trust the library authors to actually get this right.
- the8472 7y agoMost input by humans is unicode. Most people do not understand unicode. Don't try to sanitize it other than checking it's valid utf8 - which hopefully your programming language's string type/http parameter deserializer/DB engine already do for you. For example some people think that stripping ZWNJ, ZWJ or other kinds of spaces is a thing they ought to do because it confuses their markup parsers or can be used to encode hidden information in posts or stuff like that. Guess what, it breaks emoji, arabic, some asian languages and a bunch of other things. If you only sanitize your output and realize you made a mistake you can easily fix it by changing your output sanitzing algorithm. If you santizied your input you threw away data and can't fix your problem.
- nine_k 7y agoSee also: wtf-8 encoding, used by Mozilla for potentially broken Unicode input.
- klodolph 7y agoWTF-8 is for a very specific application—to encode in an 8-bit sequence a 16-bit sequence which is nominally UTF-16 but may contain unpaired surrogates. It’s not used for weird user inputs, instead, it’s used for e.g. interop with the Win32 API.
- deleted 7y ago[deleted]
- pirate_dev 7y agoCan you explain to me why reddit feels like it is held together with duct tape? IMHO, it has the most problems with site uptime and basic functionality of any major site on the net. I am always getting search problems, site unavailable, or some other such glitch with it. I can't believe you guys just don't know what you're doing, so what does the present setup offer that is worth this shitty performance?
- cerberusss 7y ago> always getting search problems, site unavailable, or some other such glitch This sounds like my spouse saying "you always...". It's obviously not "always"; I think you'll get a better reply when you quantify it. For example, in the last month, how many times did you get a search problem (which was it), a site unavailable or something else (what was it).
- pirate_dev 7y agoIt's about once per session for me. Seriously, compared to every other major site I know of, it is in a class of its own for flaky UX. At least one of the things I describe above per afternoon, lets say. Often many more than one if it is having serious problems. As for the issues with you and your spouse, I will just say it sounds like there is a opportunity for improved communication there. Best of luck.
- robotnikman 7y agoIIRC I do not think he is part of Reddit's administration team anymore.
- Alex3917 7y ago> You should do both. You should really do all three: 1) Sanitize input on the server. 2) Use a CSP to prevent your browser from rendering any asset unless specifically whitelisted. 3) Use your front end framework to sanitize text being displayed.
- fetbaffe 7y agoI'm no fan of sanitizing inputs, transform unsafe input to safe input and the storing it, because as you say, someone will find a way to circumvent that. What I always do is to exactly specify what is allowed in any input by parsing, schema validation. If it is HTML I run a HTML parser to validate accepted tags & attributes. If it is plain text i validate that there is no HTML in it, etc. If the input fails the filter then you deny the request. This has the advantage that you always know what data structures you are storing in the database and that will make future data migrations much easier. Drawback is that if your filter is too strict then you deny a valid request, however it is easier too loosen a filter later than migrate unwanted/unknown data that you accidentally accepted. Stored input is also part of your database schema. And of course, always escape output even if you know the data is "safe".