4 ms·

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable. A type that says (say) "this

Sure, you need a different type for each different form of escaping you want to track, but that doesn't make the idea unworkable. A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.

simiones 4y ago
The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely used in HTML - so you will go from user_input_string to html_pcdata_escaped_ user_input_string. Then you compose a bit of HTML that contains the button and this part; let's say you store it in some HTML DOM object. Then you want to send this HTML object as JSON, so you have to know to convert html_pcdata_escaped_ user_input_string into json_string_escaped_user_input_string - but that loses type information which may hurt us later, so maybe we want to actually store it as json_string_escaped_html_pcdata_escaped_user_input_string. Then, if we want to use this as part of an SQL query string, by the same considerations, we want to put it in a mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string - which is getting really ugly, and easy to mess up. Of course, the order of escaping matters, so an mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string and a json_string_escaped_mysql_like_filter_escaped_html_pcdata_escaped_user_input_string are different things that need to be decoded differently (of course, for SQL in particular we could use prepared queries instead). Also, we can't ever concatenate this with any other string-like type until perhaps the final use point (such as sending a query string to the DB), since we need to remember which part of the string is escaped in which way, and for what types of uses it is safe (an HTML-escaped string may still contain SQLi or JSON injection). The point is that even with proper types, this is not easy to manage or fix. It also requires quite advanced type systems to be able to use these in normal contexts - say, you want to store several such strings with different provenances in a Map or Set or even List, without "forgetting" the provenance.
- mjw1007 4y ago
  As far as I'm concerned, "the point is" that what we're describing here -- keeping track of the type of each piece of data -- is the best way to think about this class of problems (as opposed to talking about "sanitising" or "safe data", for example). It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with this well as a requirement when we're choosing a language). For me, effects like "we can't ever concatenate this with any other string-like type" are desirable features, not problems with this approach: either it's possible to convert both strings to a common form, or I shouldn't be trying to combine them.
  - simiones 4y ago
    Sure - I'm just saying that the article is right that this problem is difficult, not easy, and that it doesn't get significantly easier if we accurately keep track.
- jiggawatts 4y ago
  > The point is that even with proper types, this is not easy to manage or fix. In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge". You're thinking of a type like: HtmlString<JsonString<Utf8String>>> In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as HtmlString. When writing to databases, there isn't even an escape step at all, because you use parametrised queries, right? Right!? The database stores "string", not "DatabaseEscapedString". This is similar to how instants in time ought to be handled. You store them as UTC and convert to the user's time zone at the last moment. You don't pass around some monstrosity that somehow keeps track of +10-5+3 in order to arrive at +7. That would be absurd. Instead you pass around the "Z" UTC timestamp and add +7 when needed.
  - simiones 4y ago
    > In practice the type that is "passed around" is almost always just "string" That's what happens in practice, of course. The GP was proposing something else, and I was explaining how complicated that gets. > and this is converted at the last moment to a single destination format, such as HtmlString. I explained before why this doesn't work unless we're talking about the final destination of this string. Otherwise, if that string is being taken through various encodings (say user input to JSON to sprintf format string to HTTP body), and if you need to combine safe and unsafe input, then what you're saying doesn't work anymore. Here is a sketch of an example: userInput := read() jsonFormat := "{\"context\": \"%s\", \"input\": \"" + json.escape(userInput) + "\"}" //easy and safe finalJson := "" sprintf(finalJson, jsonFormat, "some context") // oops - unsafe if original input was "%s" sprintf(finalJson, printf.escape(jsonFormat), "some context") // oops - does the wrong thing - it will output "{\"context\": \"%s\", \"input\": \"%s"\"} //let's try the other way around? userInput := read() formatStr := "{\"context\": \"%s\", \"input\": \"" + printf.escape(userInput) + "\"}" //easy and safe finalJsonStr := "" sprintf(finalJson, formatStr, "some context") // oops - unsafe if userInput was "safe-looking\", \"bypassAuth\": \"true\"}" sprintf(finalJson, json.escape(formatStr), "some context") // oops - does the wrong thing - it will output // "\"{\\\"context\\\":\\\"some context\\\", \\\"input\\\": \\\"safe-looking\\\", \\\"bypassAuth\\\": \\\"true\\\"}\"" - that is, a JSON string instead of a JSON object The only solution to get this to work is to keep the user input string entirely separate from any other string, and apply escaping to it individually at every level where it is used. Additionally, you will need to remember what escaping has been applied to it, and in what order, so that it can be un-escaped back to the original value when needed.

simiones 4y ago

The biggest problem appears once you start wanting to combine such strings. Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen. You have to first escape the raw text from the user so that it can be safely used in HTML - so you will go from user_input_string to html_pcdata_escaped_ user_input_string. Then you compose a bit of HTML that contains the button and this part; let's say you store it in some HTML DOM object. Then you want to send this HTML object as JSON, so you have to know to convert html_pcdata_escaped_ user_input_string into json_string_escaped_user_input_string - but that loses type information which may hurt us later, so maybe we want to actually store it as json_string_escaped_html_pcdata_escaped_user_input_string. Then, if we want to use this as part of an SQL query string, by the same considerations, we want to put it in a mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string - which is getting really ugly, and easy to mess up. Of course, the order of escaping matters, so an mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string and a json_string_escaped_mysql_like_filter_escaped_html_pcdata_escaped_user_input_string are different things that need to be decoded differently (of course, for SQL in particular we could use prepared queries instead). Also, we can't ever concatenate this with any other string-like type until perhaps the final use point (such as sending a query string to the DB), since we need to remember which part of the string is escaped in which way, and for what types of uses it is safe (an HTML-escaped string may still contain SQLi or JSON injection). The point is that even with proper types, this is not easy to manage or fix. It also requires quite advanced type systems to be able to use these in normal contexts - say, you want to store several such strings with different provenances in a Map or Set or even List, without "forgetting" the provenance.

mjw1007 4y ago
As far as I'm concerned, "the point is" that what we're describing here -- keeping track of the type of each piece of data -- is the best way to think about this class of problems (as opposed to talking about "sanitising" or "safe data", for example). It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with this well as a requirement when we're choosing a language). For me, effects like "we can't ever concatenate this with any other string-like type" are desirable features, not problems with this approach: either it's possible to convert both strings to a common form, or I shouldn't be trying to combine them.
- simiones 4y ago
  Sure - I'm just saying that the article is right that this problem is difficult, not easy, and that it doesn't get significantly easier if we accurately keep track.
jiggawatts 4y ago
> The point is that even with proper types, this is not easy to manage or fix. In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge". You're thinking of a type like: HtmlString<JsonString<Utf8String>>> In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as HtmlString. When writing to databases, there isn't even an escape step at all, because you use parametrised queries, right? Right!? The database stores "string", not "DatabaseEscapedString". This is similar to how instants in time ought to be handled. You store them as UTC and convert to the user's time zone at the last moment. You don't pass around some monstrosity that somehow keeps track of +10-5+3 in order to arrive at +7. That would be absurd. Instead you pass around the "Z" UTC timestamp and add +7 when needed.
- simiones 4y ago
  > In practice the type that is "passed around" is almost always just "string" That's what happens in practice, of course. The GP was proposing something else, and I was explaining how complicated that gets. > and this is converted at the last moment to a single destination format, such as HtmlString. I explained before why this doesn't work unless we're talking about the final destination of this string. Otherwise, if that string is being taken through various encodings (say user input to JSON to sprintf format string to HTTP body), and if you need to combine safe and unsafe input, then what you're saying doesn't work anymore. Here is a sketch of an example: userInput := read() jsonFormat := "{\"context\": \"%s\", \"input\": \"" + json.escape(userInput) + "\"}" //easy and safe finalJson := "" sprintf(finalJson, jsonFormat, "some context") // oops - unsafe if original input was "%s" sprintf(finalJson, printf.escape(jsonFormat), "some context") // oops - does the wrong thing - it will output "{\"context\": \"%s\", \"input\": \"%s"\"} //let's try the other way around? userInput := read() formatStr := "{\"context\": \"%s\", \"input\": \"" + printf.escape(userInput) + "\"}" //easy and safe finalJsonStr := "" sprintf(finalJson, formatStr, "some context") // oops - unsafe if userInput was "safe-looking\", \"bypassAuth\": \"true\"}" sprintf(finalJson, json.escape(formatStr), "some context") // oops - does the wrong thing - it will output // "\"{\\\"context\\\":\\\"some context\\\", \\\"input\\\": \\\"safe-looking\\\", \\\"bypassAuth\\\": \\\"true\\\"}\"" - that is, a JSON string instead of a JSON object The only solution to get this to work is to keep the user input string entirely separate from any other string, and apply escaping to it individually at every level where it is used. Additionally, you will need to remember what escaping has been applied to it, and in what order, so that it can be un-escaped back to the original value when needed.