5 ms·

It's not, though. It's the easiest thing in the world: Just use a library that never emits unescaped content by default, or if you make a single-character typo.

It's not, though. It's the easiest thing in the world: Just use a library that never emits unescaped content by default, or if you make a single-character typo. The problem is that most of the libraries aren't that.

mrkeen 4y ago
Escaping isn't always a yes/no question. If someone enters "foo bar" into your frontend, should the backend only see "foo%20bar" ?
- pavlov 4y ago
  The back-end should see a 7-byte buffer with values [102 111 111 032 098 097 114], assume it's UTF-8 and convert that to its internal string representation?
- ezfe 4y ago
  no, the backend has no reason to see `foo%20bar` - you escape when you're combining that string with other strings (ie into HTML, into a SQL query, etc.)
  - evilDagmar 4y ago
    There's just a whole library of CVEs for people who attempt to escape things being sent to SQL. Use parameterized queries already.
    - 8n4vidtmkvmk 4y ago
      every one says "just use parameterized queries" but they don't handle arrays which makes the idea rather useless.
      - jiggawatts 4y ago
        Many database engines can handle arrays, or table-valued variables which are basically the same thing. Most ORMs will also abstract away arrays for you, so you as the developer never need to deal with escaping of data in arrays.
        8n4vidtmkvmk 4y ago
        Which relational DB supports this natively? ORMs don't count. They're just editing the SQL.
      - orf 4y ago
        Use one that does? Or build it yourself? ARRAY[?, ?, ?, ?]
        8n4vidtmkvmk 4y ago
        I do, but I feel like it defeats the purpose. In order to insert those ?s you have to parse the query, which is exactly what we're trying to avoid.
        orf 4y ago
        Sorry, I’m not clear on why you need to parse the query? Imagine you want to INSERT (int, array[], int) into a table. Why can’t you just generate the following SQL? INSERT INTO foo VALUES (?, ARRAY[?,?,?], ?) Where param 1-3 are indexes 0-2 in the array? This is basically what everything else does when handing arrays, or other composite types like coordinates. If the language or libraries are not suitable for doing this then the language or libraries are the problem, not the approach.
        8n4vidtmkvmk 4y ago
        If you're using an ORM/SQL builder, sure. If you prefer writing raw SQL because ORMs are often quite limiting, then you have to write something like `INSERT INTO foo VALUES ?` and then you pass it `[1, [2,3,4], 5]` as params, which is mostly fine but you still have to parse out that "?" from the query. Why parse? Because what if you wrote 'VALUES "?"' now it's a string and shouldn't be replaced. It would be much nicer if you could just send the query to the SQL server and figure out what to do with the params.
        orf 4y ago
        I think your argument is “if I try hard enough to make what I want to do really hard, it will be really hard”. I agree. I’m not sure I buy your argument about ORMs being limiting, given your current… limitations. Of course you could just do CAST(? AS …) or use something like “string_to_array(?)”. But that’s just working around not being able to compose queries.
Joker_vD 4y ago
Different contexts require different escaping schemes, you know?
masklinn 4y ago
> It's the easiest thing in the world: Just use a library that never emits unescaped content by default That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with. And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML. More broadly, I think one of the core issues is this: > Escape user input User input is a broad and complicated category, and it's easy for user input to be "laundered" as it moves through an application. And then escaping is an explicit action, which means it can be missed or forgotten, which is also a problem. This means the solution is really that APIs should default to escaping most everything. Rather than having to mark "untrusted" content, it's trusted content which should be marked thus. "Escaping" is the wrong default. But of course that doesn't solve all the issues. Like markdown, where you want the output of the Markdown converter to be trusted (otherwise the output won't be properly formatted on display), what you don't want trusted is the input, and that means you don't want the input to be laundered through the Markdown converter. Which is an issue in most Markdown libraries, as they inherit the "trusted input" model from Gruber's original Markdown, where HTML passthrough was a feature. In that sense one design I did enjoy is Jinja and Markupsafe in the Python ecosystem: - Like most modern template libraries, Jinja escapes content by default. - Also (though somewhat sadly) like most template libraries Jinja allows marking a value as safe at point-of-use, however that's dangerous as content can be mixed and it's easy for safe content to suddenly be swapped out for user input and become unsafe through seemingly unrelated changes. - So a better method is to use `markupsafe.Markup` at the source, it's a string subclass which the library considers safe (because Jinja uses `markupsafe.escape` internally), the neat thing is any combination between a Markup instance and a non-Markup string will implicitly escape the non-Markup parameter(s). This means you can mark safe content as safe at the source (where it's easy to prove it's safe because e.g. it's a literal), then most transformations will maintain the safety invariants. Though obviously it only works with content you know will ultimately be markup-injected. And non-method APIs can't be overridden (e.g. re, or HTML/XML libraries) so they're not Markup-aware, they'll treat Markup objects as regular strings which that complicates processing pipelines if you want to conserve safety invariants. At the same time, those are laundering opportunities so care is useful.
- 4y ago

mrkeen 4y ago

Escaping isn't always a yes/no question. If someone enters "foo bar" into your frontend, should the backend only see "foo%20bar" ?

pavlov 4y ago
The back-end should see a 7-byte buffer with values [102 111 111 032 098 097 114], assume it's UTF-8 and convert that to its internal string representation?
ezfe 4y ago
no, the backend has no reason to see `foo%20bar` - you escape when you're combining that string with other strings (ie into HTML, into a SQL query, etc.)
- evilDagmar 4y ago
  There's just a whole library of CVEs for people who attempt to escape things being sent to SQL. Use parameterized queries already.
  - 8n4vidtmkvmk 4y ago
    every one says "just use parameterized queries" but they don't handle arrays which makes the idea rather useless.
    - jiggawatts 4y ago
      Many database engines can handle arrays, or table-valued variables which are basically the same thing. Most ORMs will also abstract away arrays for you, so you as the developer never need to deal with escaping of data in arrays.
      - 8n4vidtmkvmk 4y ago
        Which relational DB supports this natively? ORMs don't count. They're just editing the SQL.
    - orf 4y ago
      Use one that does? Or build it yourself? ARRAY[?, ?, ?, ?]
      - 8n4vidtmkvmk 4y ago
        I do, but I feel like it defeats the purpose. In order to insert those ?s you have to parse the query, which is exactly what we're trying to avoid.
        orf 4y ago
        Sorry, I’m not clear on why you need to parse the query? Imagine you want to INSERT (int, array[], int) into a table. Why can’t you just generate the following SQL? INSERT INTO foo VALUES (?, ARRAY[?,?,?], ?) Where param 1-3 are indexes 0-2 in the array? This is basically what everything else does when handing arrays, or other composite types like coordinates. If the language or libraries are not suitable for doing this then the language or libraries are the problem, not the approach.
        8n4vidtmkvmk 4y ago
        If you're using an ORM/SQL builder, sure. If you prefer writing raw SQL because ORMs are often quite limiting, then you have to write something like `INSERT INTO foo VALUES ?` and then you pass it `[1, [2,3,4], 5]` as params, which is mostly fine but you still have to parse out that "?" from the query. Why parse? Because what if you wrote 'VALUES "?"' now it's a string and shouldn't be replaced. It would be much nicer if you could just send the query to the SQL server and figure out what to do with the params.
        orf 4y ago
        I think your argument is “if I try hard enough to make what I want to do really hard, it will be really hard”. I agree. I’m not sure I buy your argument about ORMs being limiting, given your current… limitations. Of course you could just do CAST(? AS …) or use something like “string_to_array(?)”. But that’s just working around not being able to compose queries.

Joker_vD 4y ago

Different contexts require different escaping schemes, you know?

masklinn 4y ago

> It's the easiest thing in the world: Just use a library that never emits unescaped content by default That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with. And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML. More broadly, I think one of the core issues is this: > Escape user input User input is a broad and complicated category, and it's easy for user input to be "laundered" as it moves through an application. And then escaping is an explicit action, which means it can be missed or forgotten, which is also a problem. This means the solution is really that APIs should default to escaping most everything. Rather than having to mark "untrusted" content, it's trusted content which should be marked thus. "Escaping" is the wrong default. But of course that doesn't solve all the issues. Like markdown, where you want the output of the Markdown converter to be trusted (otherwise the output won't be properly formatted on display), what you don't want trusted is the input, and that means you don't want the input to be laundered through the Markdown converter. Which is an issue in most Markdown libraries, as they inherit the "trusted input" model from Gruber's original Markdown, where HTML passthrough was a feature. In that sense one design I did enjoy is Jinja and Markupsafe in the Python ecosystem: - Like most modern template libraries, Jinja escapes content by default. - Also (though somewhat sadly) like most template libraries Jinja allows marking a value as safe at point-of-use, however that's dangerous as content can be mixed and it's easy for safe content to suddenly be swapped out for user input and become unsafe through seemingly unrelated changes. - So a better method is to use `markupsafe.Markup` at the source, it's a string subclass which the library considers safe (because Jinja uses `markupsafe.escape` internally), the neat thing is any combination between a Markup instance and a non-Markup string will implicitly escape the non-Markup parameter(s). This means you can mark safe content as safe at the source (where it's easy to prove it's safe because e.g. it's a literal), then most transformations will maintain the safety invariants. Though obviously it only works with content you know will ultimately be markup-injected. And non-method APIs can't be overridden (e.g. re, or HTML/XML libraries) so they're not Markup-aware, they'll treat Markup objects as regular strings which that complicates processing pipelines if you want to conserve safety invariants. At the same time, those are laundering opportunities so care is useful.

4y ago