5 ms·

One point in and its already dead wrong, you never filter input, only output. Edit: Everyone talking about databases: paramaterized queries, check them out.

codeguy 14y ago
I'd definitely filter input that goes into a database. But if you think the wording can be made better, definitely send a pull request. Thanks!
xd 14y ago
SQL injection attacks alone are almost always a result of not filtering input...
- FuzzyDunlop 14y ago
  It's sometimes known as sanitising input, so I'm guessing that's what they were trying to refer to.
- kellishaver 14y ago
  With parametrized queries, that becomes a non-issue, but I still see no point in cluttering the database with data that's just going to be filtered out at some point - might as well filter it before it goes into the DB to begin with. The exception, of course, being those rare cases when some users need to see the filtered data and others need to see the raw data, but even then, you likely won't want to allow everything.
  - silentworks 14y ago
    I am not sure if you are referring to the same sort of filtering mentioned in the document. The data is filtered before it hits the database layer, this saves resource being used.
    - kellishaver 14y ago
      That's what I meant. I just kind of articulated it poorly, my apologies.
  - xd 14y ago
    Parametized queries are just another way of filtering input ..
    - kellishaver 14y ago
      Yes, but on the site, they make a clear distinction between parametrized queries with PDO and other types of input filtering. I misunderstood and thought you were referring specifically to the latter.
    - mtts 14y ago
      No they're not. They're a method for ensuring that what is meant to go into some field in some database actually ends up there and does not start doing things it should not be doing.
- jeltz 14y ago
  In my anecdotal experience SQL injection attacks are almost as often caused by relying on filtering data and then forgetting to filter it for some fields. You should instead rely on parametrized queries and only use string operations to build queries where you must (and then do it very carefully). SQL injections are very rare to non-existent in code where the programmers generally rely on parametrized queries.
- unconed 14y ago
  This the wrong way to look at it, and yes, the PHP world always does this wrong because they're too focused on HTML. SQL injection occurs when you're not escaping data while producing output, namely, an SQL query sent to the DB. XSS attacks occur when you're not escaping data while producing HTML, but you don't need angle brackets to do it. <a title="$string"> allows for XSS injection with just a quote character. Header injection attacks occur when you're not escaping data while producing HTTP/MIME headers, and all you need is a line-break character. The escaping always depends on the output context, and the data in the database cannot be made safe for all these contexts. You will still need to allow people named "O'Brian", let people post "<_<" smileys, use Unicode in their names, etc.
  - jasonlotito 14y ago
    > the PHP world always does this wrong because they're too focused on HTML. No. The PHP world recommends, time and time again, using PDO and binding variables to queries. I've yet to meet an individual who does it the other way, other than people who are relying on extremely outdated tutorials (7+ years ago). Hell, even this document does. This document, unfortunately, uses the word filter in the wrong way, but the intention is still fine.
    - sopooneo 14y ago
      One thing I've run into is that PDO does not let you parameterize the field name in "ORDER BY" clauses. And there are a few other little things that I've wanted to do that it doesn't allow. It may be that me even running into these problems means I'm doing something wrong at a structural level. Regardless, in those statements where I want something to be dynamic, but it's not allowed by PDO, I do an exact equality check against a white list.
      - jasonlotito 14y ago
        Well, that's not what binding is supposed to do. After all, field names aren't something you should be exposing at all. > I do an exact equality check against a white list. Pretty much the way you are supposed to do it (except, I'd actually use the value in the white list rather than the matched input value, as the two should not be the same).
- gcb 14y ago
  Output is not what goes to the user/page, in this context it is what your code sends anywhere, such as to mysql_execute or something
wvenable 14y ago
You've been downvoted but I completely agree. You cannot sanitize text for all possible situations; you can only escape it depending on the output format (database, html, etc). These functions should be used to "validate" input rather than "filter" it.
- LeonidasXIV 14y ago
  I absolutely hate how for example phpBB creates garbled bbcode in the database because they thought that they should filter it or something, instead of doing it on output now every consumer of the data has to unfuck the content, because they messed up.
jeltz 14y ago
Agreed, most input should not be filtered. There are some exceptions though like stripping whitespace and dashes from credit card numbers and phone numbers. The filtering of input has nothing to do with security (that should be handled by output escaping and paramterized queries) and should only be used for improving the quality of the saved data.
birdcircle 14y ago
Input is almost always filtered. Without mincing the semantics of that, let's say you receive free-form NL query as input. You don't filter that to reduce it to core terms you send to the database? You don't remove prepositions? You don't tokenize at all? Input is ambiguous and reduced/filtered into pseudo-meaningful terms to return relevant output.
- unconed 14y ago
  This is true for all structured data, but the problem always comes from freeform text fields, which you cannot filter for one context without mangling or destroying the data for all other purposes—except perhaps trivial things like trimming leading/trailing whitespace.
- prodigal_erik 14y ago
  If I want to tweet I <3 O'Reilly books you could pre-encode that as safe to paste into raw SQL, or as well-formed (X)HTML, but you can't do both simultaneously. Either encoding would end up distorting the content in the other context. You have to encode during output (and writing to a database counts) using the rules of the system consuming that output. Lots of crappy web forums visibly mangle punctuation in a futile effort to avoid this.
  - MartinCron 14y ago
    What a great example. I am going to remember that one.
smsm42 14y ago
What's wrong with filtering input? If you expect a number, what's wrong with telling the user "strawberry pie" is not a valid value?
- _ikke_ 14y ago
  There's a fine line between filtering and validating. What's meant with filtering is things like add_slashes and mysql_real_escape_string, a uniform transformation on the input, regardless of it's destination. Validating on the other hand is checking if the values satisfy the expected requirements. You only want to apply certain filters if you know they apply the current output. (htmlentities if output as html, mysql_real_escape_string if embedded in a query).
  - smsm42 14y ago
    Filtering can't be done "regardless of destination", of course, since kind of filtering depends on the required result. Otherwise, difference between filtering and validation is largely in application semantics - some may prefer one, some another. Output is different things, of course filters can be applied to the output too.
- wilmoore 14y ago
  What everyone is missing with the filtering argument is: (1) normalization (things like trim, etc. so you don't do stupid things like invalidate an email address just because it was input with trailing spaces). The result is passed to a validator. (2) validation (if a number is expected, it's only valid if given a sequence of digits within the correct range). Valid values are passed on to the next phase (likely persisted). Sanitization is generally not needed and in most cases likely to cause data corruption. I'm sure there are use cases for it but it should be the exception, not the rule.

codeguy 14y ago

I'd definitely filter input that goes into a database. But if you think the wording can be made better, definitely send a pull request. Thanks!

xd 14y ago

SQL injection attacks alone are almost always a result of not filtering input...

FuzzyDunlop 14y ago
It's sometimes known as sanitising input, so I'm guessing that's what they were trying to refer to.
kellishaver 14y ago
With parametrized queries, that becomes a non-issue, but I still see no point in cluttering the database with data that's just going to be filtered out at some point - might as well filter it before it goes into the DB to begin with. The exception, of course, being those rare cases when some users need to see the filtered data and others need to see the raw data, but even then, you likely won't want to allow everything.
- silentworks 14y ago
  I am not sure if you are referring to the same sort of filtering mentioned in the document. The data is filtered before it hits the database layer, this saves resource being used.
  - kellishaver 14y ago
    That's what I meant. I just kind of articulated it poorly, my apologies.
- xd 14y ago
  Parametized queries are just another way of filtering input ..
  - kellishaver 14y ago
    Yes, but on the site, they make a clear distinction between parametrized queries with PDO and other types of input filtering. I misunderstood and thought you were referring specifically to the latter.
  - mtts 14y ago
    No they're not. They're a method for ensuring that what is meant to go into some field in some database actually ends up there and does not start doing things it should not be doing.
jeltz 14y ago
In my anecdotal experience SQL injection attacks are almost as often caused by relying on filtering data and then forgetting to filter it for some fields. You should instead rely on parametrized queries and only use string operations to build queries where you must (and then do it very carefully). SQL injections are very rare to non-existent in code where the programmers generally rely on parametrized queries.
unconed 14y ago
This the wrong way to look at it, and yes, the PHP world always does this wrong because they're too focused on HTML. SQL injection occurs when you're not escaping data while producing output, namely, an SQL query sent to the DB. XSS attacks occur when you're not escaping data while producing HTML, but you don't need angle brackets to do it. <a title="$string"> allows for XSS injection with just a quote character. Header injection attacks occur when you're not escaping data while producing HTTP/MIME headers, and all you need is a line-break character. The escaping always depends on the output context, and the data in the database cannot be made safe for all these contexts. You will still need to allow people named "O'Brian", let people post "<_<" smileys, use Unicode in their names, etc.
- jasonlotito 14y ago
  > the PHP world always does this wrong because they're too focused on HTML. No. The PHP world recommends, time and time again, using PDO and binding variables to queries. I've yet to meet an individual who does it the other way, other than people who are relying on extremely outdated tutorials (7+ years ago). Hell, even this document does. This document, unfortunately, uses the word filter in the wrong way, but the intention is still fine.
  - sopooneo 14y ago
    One thing I've run into is that PDO does not let you parameterize the field name in "ORDER BY" clauses. And there are a few other little things that I've wanted to do that it doesn't allow. It may be that me even running into these problems means I'm doing something wrong at a structural level. Regardless, in those statements where I want something to be dynamic, but it's not allowed by PDO, I do an exact equality check against a white list.
    - jasonlotito 14y ago
      Well, that's not what binding is supposed to do. After all, field names aren't something you should be exposing at all. > I do an exact equality check against a white list. Pretty much the way you are supposed to do it (except, I'd actually use the value in the white list rather than the matched input value, as the two should not be the same).
gcb 14y ago
Output is not what goes to the user/page, in this context it is what your code sends anywhere, such as to mysql_execute or something

wvenable 14y ago

You've been downvoted but I completely agree. You cannot sanitize text for all possible situations; you can only escape it depending on the output format (database, html, etc). These functions should be used to "validate" input rather than "filter" it.

LeonidasXIV 14y ago
I absolutely hate how for example phpBB creates garbled bbcode in the database because they thought that they should filter it or something, instead of doing it on output now every consumer of the data has to unfuck the content, because they messed up.

jeltz 14y ago

Agreed, most input should not be filtered. There are some exceptions though like stripping whitespace and dashes from credit card numbers and phone numbers. The filtering of input has nothing to do with security (that should be handled by output escaping and paramterized queries) and should only be used for improving the quality of the saved data.

birdcircle 14y ago

Input is almost always filtered. Without mincing the semantics of that, let's say you receive free-form NL query as input. You don't filter that to reduce it to core terms you send to the database? You don't remove prepositions? You don't tokenize at all? Input is ambiguous and reduced/filtered into pseudo-meaningful terms to return relevant output.

unconed 14y ago
This is true for all structured data, but the problem always comes from freeform text fields, which you cannot filter for one context without mangling or destroying the data for all other purposes—except perhaps trivial things like trimming leading/trailing whitespace.
prodigal_erik 14y ago
If I want to tweet I <3 O'Reilly books you could pre-encode that as safe to paste into raw SQL, or as well-formed (X)HTML, but you can't do both simultaneously. Either encoding would end up distorting the content in the other context. You have to encode during output (and writing to a database counts) using the rules of the system consuming that output. Lots of crappy web forums visibly mangle punctuation in a futile effort to avoid this.
- MartinCron 14y ago
  What a great example. I am going to remember that one.

smsm42 14y ago

What's wrong with filtering input? If you expect a number, what's wrong with telling the user "strawberry pie" is not a valid value?

_ikke_ 14y ago
There's a fine line between filtering and validating. What's meant with filtering is things like add_slashes and mysql_real_escape_string, a uniform transformation on the input, regardless of it's destination. Validating on the other hand is checking if the values satisfy the expected requirements. You only want to apply certain filters if you know they apply the current output. (htmlentities if output as html, mysql_real_escape_string if embedded in a query).
- smsm42 14y ago
  Filtering can't be done "regardless of destination", of course, since kind of filtering depends on the required result. Otherwise, difference between filtering and validation is largely in application semantics - some may prefer one, some another. Output is different things, of course filters can be applied to the output too.
wilmoore 14y ago
What everyone is missing with the filtering argument is: (1) normalization (things like trim, etc. so you don't do stupid things like invalidate an email address just because it was input with trailing spaces). The result is passed to a validator. (2) validation (if a number is expected, it's only valid if given a sequence of digits within the correct range). Valid values are passed on to the next phase (likely persisted). Sanitization is generally not needed and in most cases likely to cause data corruption. I'm sure there are use cases for it but it should be the exception, not the rule.