4 ms·

When I see anything that touches web coding practices for Go I always look up an area that I know best: sanitization. I wrote https://github.com/microcosm-cc/b

by buro9 9y ago

When I see anything that touches web coding practices for Go I always look up an area that I know best: sanitization. I wrote https://github.com/microcosm-cc/bluemonday https://github.com/microcosm-cc/bluemonday which is a pure Go HTML sanitizer inspired by https://github.com/owasp/java-html-sanitizer https://github.com/owasp/java-html-sanitizer . The key things to understand about HTML sanitizers: * They must be whitelist based * They must be aware of context * You must sanitize ALL user input even if you don't think you're going to render it on a web page. The book linked to in the article does not seem to understand any of the above. The section on sanitization has the equivalent of "string replace" as the primary recommendation. Elsewhere in the XSS section a focus is on escaping content before it is rendered. Sanitization needs to know not to run on <pre> blocks, and to escape HTML entities automatically, and to understand what links are safe and which are not. XSS can be really interesting and quite targeted. It can be that a user-agent contains the XSS, because the target may not be the person reading the page but the admin looking at a web page of their web server logs through an analytics program on the same domain. The bluemonday package I wrote can deal with all of these things but that isn't the point, the point is that this is an area I know and the book falls way short of a decent standard for creating a secure and safe web application. And if it falls short in this area (the first 2 chapters) then I would assume that it falls short in all areas.

coldtea 9y ago
>You must sanitize ALL user input even if you don't think you're going to render it on a web page Not really.
- MajesticHobo 9y ago
  Yes, really. Otherwise, you must take extra care not to reflect any input data back in any response to the user, whether it's in the HTML body or not. See: HTTP response splitting.
  - coldtea 9y ago
    No, not really. In fact, you have it totally backwards: you're not supposed to sanitize all user input before storing it. Instead you're supposed to sanitize any user input before you output it back to your webpage. Even more so: it's the output that dictates what sanitization you should perform, not the input. You don't do input sanitization for HTML (for XSS etc) when you store your data in your DB. Instead you should sanitize the input for SQL Injection issues. And similarly for whatever other output -- if you take user input and run a shell command, you should sanitize for shell safety, not run html sanitization.
    - MajesticHobo 9y ago
      You've said nothing that contradicts my post. As long as the data is sanitized before it can affect the storage/transport mechanism for its content type, you're good.
      - lomnakkus 9y ago
        > As long as the data is sanitized before it can affect the storage/transport mechanism for its content type, you're good. No, not really. Storing the user's data as is is almost always of paramount importance. The fact that it may be output as HTML/XML/MarkDown/whatever means that it really is at output-time that you must sanitize/escape/quote. That's why the moral of the Bobby Tables story isn't: "Oh, just remove all semicolons". It's "use prepared queries".
        MajesticHobo 9y ago
        I don't disagree with sanitizing data at output time when it's clear that A) the input won't affect anything else and B) output is going to happen. But realize not all input winds up in a SQL database, not all input will be considered valid in all contexts, and not all input eventually becomes output. Sometimes, data really does need to be sanitized at the point of submission. If you disagree, that's more of a point about application design than appsec.
        9y ago
osteele 9y ago
> You must sanitize ALL user input even if you don't think you're going to render it on a web page. I'm not able to make sense of this. Sanitize it for what context? SQL? JSON? HTML? Inclusion as a command-line argument? All of these, and hope that sanitizing it for one context doesn't un-sanitize it for others?
- adamdoupe 9y ago
  Context here means the context of the output page. Usually this means the HTML context. Different sanitization is needed depending on _where_ in the HTML document the input is used. For instance, if the input is used in between HTML tags (let's say $foo is user input in this PHP example): ... <body><?php echo $foo ?></body> Here, the input that you need to transition to JavaScript execution is a < character (among other things): <script>alert(1)</script>. Therefore, to correctly sanitize this, you would call the PHP `htmlentities` function: ... <body><?php echo htmlentities($foo) ?></body> Now, this XSS vulnerability is fixed. What if foo is used in a different context? ... <body><a href='<?php echo htmlentities($foo) ?>'>... Here, what we need to transition the HTML parser to executing JavaScript is a ' character, and this can be exploited by the following input (in between the double quotes): "' onclick='alert(1)" The key problem is that `htmlentities` is not valid sanitization in the context of an HTML attribute value. In this example, you need to use `urlencode` ... <body><a href='<?php echo urlencode($foo) ?>'>... The general idea also applies to CSS, JSON, and JavaScript. SQL is a different vulnerability class (SQL injection). I highly recommend the following research paper from 2011 that discusses the context-sensitivity of JavaScript in depth: http://www.comp.nus.edu.sg/~prateeks/papers/scriptgard-ccs11.pdf http://www.comp.nus.edu.sg/~prateeks/papers/scriptgard-ccs11... In my mind, the context-sensitivity of XSS is one of the key reasons why it is so prevalent.
  - deleted 9y ago
    [deleted]
  - deleted 9y ago
    [deleted]
- taeric 9y ago
astockwell 9y ago
Honest and earnest question for you (as bluemonday's author): Where should gophers draw the line between Go's standard html/template library for rendering "safe" HTML, vs. needing to use bluemonday? The question assumes that any and all user inputs are checked/validated at the point of receipt. Is bluemonday just the "on steriods" version (e.g. provides more granular control, whitelist of specific tags, etc)? Perhaps it is more of a use-case question - I haven't worked on comment or wiki systems but that seems like it would be much harder to validate user inputs, and thus maybe that is where the line should be drawn? When user inputs are expressly expected to have markup?
- buro9 9y ago
  If you're using the Go (text|html)/template and you are not rendering any user or untrusted input... i.e. this is 100% a little blog where you are provably the only one providing content... you're gold. No need to do much else. If you accept any user input and this will be displayed or processed anywhere (be it in HTML fragments, input to SQL, etc)... then sanitize it and use bluemonday or any other sanitizer applicable to your content. For user generated content, like comments on Hacker News, there is a pretty rich policy provided by bluemonday so you don't have to think about it and are going to be alright. That looks like this: p := bluemonday.UGCPolicy() htmlOut := p.Sanitize(htmlIn) If you have custom needs beyond "I accept UGC"... then I'm afraid you're going to have to think about your needs and how to express your own policy, but even then, you can use bluemonday's built in policies as a starting point.
pram 9y ago
Nice. I use bluemonday, and I also took a lot of inspiration from how microcosm was written. Just wanted to say thanks for the effort!

by buro9 9y ago

coldtea 9y ago

>You must sanitize ALL user input even if you don't think you're going to render it on a web page Not really.

MajesticHobo 9y ago
Yes, really. Otherwise, you must take extra care not to reflect any input data back in any response to the user, whether it's in the HTML body or not. See: HTTP response splitting.
- coldtea 9y ago
  No, not really. In fact, you have it totally backwards: you're not supposed to sanitize all user input before storing it. Instead you're supposed to sanitize any user input before you output it back to your webpage. Even more so: it's the output that dictates what sanitization you should perform, not the input. You don't do input sanitization for HTML (for XSS etc) when you store your data in your DB. Instead you should sanitize the input for SQL Injection issues. And similarly for whatever other output -- if you take user input and run a shell command, you should sanitize for shell safety, not run html sanitization.
  - MajesticHobo 9y ago
    You've said nothing that contradicts my post. As long as the data is sanitized before it can affect the storage/transport mechanism for its content type, you're good.
    - lomnakkus 9y ago
      > As long as the data is sanitized before it can affect the storage/transport mechanism for its content type, you're good. No, not really. Storing the user's data as is is almost always of paramount importance. The fact that it may be output as HTML/XML/MarkDown/whatever means that it really is at output-time that you must sanitize/escape/quote. That's why the moral of the Bobby Tables story isn't: "Oh, just remove all semicolons". It's "use prepared queries".
      - MajesticHobo 9y ago
        I don't disagree with sanitizing data at output time when it's clear that A) the input won't affect anything else and B) output is going to happen. But realize not all input winds up in a SQL database, not all input will be considered valid in all contexts, and not all input eventually becomes output. Sometimes, data really does need to be sanitized at the point of submission. If you disagree, that's more of a point about application design than appsec.
        9y ago

osteele 9y ago

> You must sanitize ALL user input even if you don't think you're going to render it on a web page. I'm not able to make sense of this. Sanitize it for what context? SQL? JSON? HTML? Inclusion as a command-line argument? All of these, and hope that sanitizing it for one context doesn't un-sanitize it for others?

adamdoupe 9y ago
Context here means the context of the output page. Usually this means the HTML context. Different sanitization is needed depending on _where_ in the HTML document the input is used. For instance, if the input is used in between HTML tags (let's say $foo is user input in this PHP example): ... <body><?php echo $foo ?></body> Here, the input that you need to transition to JavaScript execution is a < character (among other things): <script>alert(1)</script>. Therefore, to correctly sanitize this, you would call the PHP `htmlentities` function: ... <body><?php echo htmlentities($foo) ?></body> Now, this XSS vulnerability is fixed. What if foo is used in a different context? ... <body><a href='<?php echo htmlentities($foo) ?>'>... Here, what we need to transition the HTML parser to executing JavaScript is a ' character, and this can be exploited by the following input (in between the double quotes): "' onclick='alert(1)" The key problem is that `htmlentities` is not valid sanitization in the context of an HTML attribute value. In this example, you need to use `urlencode` ... <body><a href='<?php echo urlencode($foo) ?>'>... The general idea also applies to CSS, JSON, and JavaScript. SQL is a different vulnerability class (SQL injection). I highly recommend the following research paper from 2011 that discusses the context-sensitivity of JavaScript in depth: http://www.comp.nus.edu.sg/~prateeks/papers/scriptgard-ccs11.pdf http://www.comp.nus.edu.sg/~prateeks/papers/scriptgard-ccs11... In my mind, the context-sensitivity of XSS is one of the key reasons why it is so prevalent.
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
taeric 9y ago

astockwell 9y ago

Honest and earnest question for you (as bluemonday's author): Where should gophers draw the line between Go's standard html/template library for rendering "safe" HTML, vs. needing to use bluemonday? The question assumes that any and all user inputs are checked/validated at the point of receipt. Is bluemonday just the "on steriods" version (e.g. provides more granular control, whitelist of specific tags, etc)? Perhaps it is more of a use-case question - I haven't worked on comment or wiki systems but that seems like it would be much harder to validate user inputs, and thus maybe that is where the line should be drawn? When user inputs are expressly expected to have markup?

buro9 9y ago
If you're using the Go (text|html)/template and you are not rendering any user or untrusted input... i.e. this is 100% a little blog where you are provably the only one providing content... you're gold. No need to do much else. If you accept any user input and this will be displayed or processed anywhere (be it in HTML fragments, input to SQL, etc)... then sanitize it and use bluemonday or any other sanitizer applicable to your content. For user generated content, like comments on Hacker News, there is a pretty rich policy provided by bluemonday so you don't have to think about it and are going to be alright. That looks like this: p := bluemonday.UGCPolicy() htmlOut := p.Sanitize(htmlIn) If you have custom needs beyond "I accept UGC"... then I'm afraid you're going to have to think about your needs and how to express your own policy, but even then, you can use bluemonday's built in policies as a starting point.

pram 9y ago

Nice. I use bluemonday, and I also took a lot of inspiration from how microcosm was written. Just wanted to say thanks for the effort!