5 ms·

I think a better way to think of this may be in terms of canonicalization. Inside your application, you should decide on a single canonical way to represent da

I think a better way to think of this may be in terms of canonicalization. Inside your application, you should decide on a single canonical way to represent data, one which fits the type of processing and expected use of the application. For example, you might decide that all strings should be UTF8, and should be interpreted (and stored) as whatever the user initially wrote. You might decide that any structured data should be parsed and then stored as protobufs in a BigTable. Or you might decide that an RDBMS is your native datastore and use whatever the native string encoding is for it, as well as parse & normalize data into tables upon input. Then, whenever you take input, your job is to validate and encode it. If you get a Windows-1252 string, you should re-encode it to utf8 for further storage. If it has data that are invalid UTF-8 codepoints, you should either strip, replace with a replacement character, or notify the user with a validation failure. Same with structured data that fails your normalization rules - you should usually notify the user. And when you send output, you should escape based on the intended output device. If you're putting it in an HTML page, HTML-escape it. If it's a URL, url-encode it. If it's a database query, SQL escape it. If it's a CSV, quote it. Thinking in these terms keeps the internal logic of your application simple (there are no format conversions except at system boundaries), and it also gives you a lot of flexibility to preserve the user's intent and add new output formats later.

platz 5y ago
so you would prevent stored XSS attacks by escaping on the output step instead of the canonicalization step
- simonw 5y ago
  Right - the way to avoid XSS is to escape on output. Most good template languages these days implement auto-escaping of variables that are interpolated into HTML. You still have to be careful embedding content into non-HTML contexts. One classic example there is outputting a blob of JSON inside a <script> tag - you need to make sure that you handle the case where a string could contain "</script><script>evil_code_here()</script>".
  - zelphirkalt 5y ago
    Also, in languages, which do not treat HTML as a simple string (looking at PHP and many others) or have libraries for doing exactly that, using any kind of data inside any HTML element, where it is put as text, will automatically make it escaped as text, with no overhead for the developer.
  - colejohnson66 5y ago
    React (technically JSX) has a nice feature where all output is escaped. So this doesn't work: const evil = "<script>alert('')</script>"; .. <div>{evil}</div> That'll output: <div><script>alert('')</script></div> If you must output a raw HTML string, they make you acknowledge that you're aware of what you're doing: <div dangerouslySetInnerHTML={{__html: evil}} />
Sohcahtoa82 5y ago
> If it's a database query, SQL escape it. No, you parameterize it.
- magicalhippo 5y ago
  At least our db doesn't support parameterized lists for use with IN operator. Not often I need this but when I do I'm glad our db layer has macros for SQL with automatic string escaping.
  - rileymat2 5y ago
    Which db is that? Just curious, the few I have used take (?,?)
    - tehbeard 5y ago
      I assume they're on about an unbounded, pass array in situation (?) rather than having to dynamically build that part of the query (?,?,?...). Done both, manually doing it looks messy, and higher chance of typos.
      - chopin 5y ago
        I've done the latter, in JDBC and I built the (?,?,...) part programmatically. However you have to be aware how many ?s are elsewhere in the PreparedStatement such that filling the query with the array contents works correctly. It'd be a much better design decision if JDBC actually supported named parameters (as JPQL does).
    - magicalhippo 5y ago
      Sure but it also has a limitation of ~2000 parameters. And it's kind of a PITA to dynamically generate and set the parameters compared to just setting one parameter. Sybase SQLAnywhere btw.
      - rileymat2 5y ago
omegalulw 5y ago
> Thinking in these terms keeps the internal logic of your application simple (there are no format conversions except at system boundaries), This is an excellent point but are there any devs who don't do this? This seems like such an obvious thing to do. I mean I guess if you dealing with tech debt and you want to upgrade from ASCII/bytes to UTF-8 you will not have this invariant for a shirt while but why would you not maintain the invariant in new code?

platz 5y ago

so you would prevent stored XSS attacks by escaping on the output step instead of the canonicalization step

simonw 5y ago
Right - the way to avoid XSS is to escape on output. Most good template languages these days implement auto-escaping of variables that are interpolated into HTML. You still have to be careful embedding content into non-HTML contexts. One classic example there is outputting a blob of JSON inside a <script> tag - you need to make sure that you handle the case where a string could contain "</script><script>evil_code_here()</script>".
- zelphirkalt 5y ago
  Also, in languages, which do not treat HTML as a simple string (looking at PHP and many others) or have libraries for doing exactly that, using any kind of data inside any HTML element, where it is put as text, will automatically make it escaped as text, with no overhead for the developer.
- colejohnson66 5y ago
  React (technically JSX) has a nice feature where all output is escaped. So this doesn't work: const evil = "<script>alert('')</script>"; .. <div>{evil}</div> That'll output: <div><script>alert('')</script></div> If you must output a raw HTML string, they make you acknowledge that you're aware of what you're doing: <div dangerouslySetInnerHTML={{__html: evil}} />

Sohcahtoa82 5y ago

> If it's a database query, SQL escape it. No, you parameterize it.

magicalhippo 5y ago
At least our db doesn't support parameterized lists for use with IN operator. Not often I need this but when I do I'm glad our db layer has macros for SQL with automatic string escaping.
- rileymat2 5y ago
  Which db is that? Just curious, the few I have used take (?,?)
  - tehbeard 5y ago
    I assume they're on about an unbounded, pass array in situation (?) rather than having to dynamically build that part of the query (?,?,?...). Done both, manually doing it looks messy, and higher chance of typos.
    - chopin 5y ago
      I've done the latter, in JDBC and I built the (?,?,...) part programmatically. However you have to be aware how many ?s are elsewhere in the PreparedStatement such that filling the query with the array contents works correctly. It'd be a much better design decision if JDBC actually supported named parameters (as JPQL does).
  - magicalhippo 5y ago
    Sure but it also has a limitation of ~2000 parameters. And it's kind of a PITA to dynamically generate and set the parameters compared to just setting one parameter. Sybase SQLAnywhere btw.
    - rileymat2 5y ago

omegalulw 5y ago

> Thinking in these terms keeps the internal logic of your application simple (there are no format conversions except at system boundaries), This is an excellent point but are there any devs who don't do this? This seems like such an obvious thing to do. I mean I guess if you dealing with tech debt and you want to upgrade from ASCII/bytes to UTF-8 you will not have this invariant for a shirt while but why would you not maintain the invariant in new code?