7 ms·

How to safely escape JSON inside HTML SCRIPT elements

86 points by dmsnell 1y ago

dmsnell 1y ago
Discussing why parsing HTML SCRIPT elements is so complicated, the history of why it became the way it is, and how to safely and securely embed JSON content inside of a SCRIPT element today.
- dmsnell 1y ago
  This was my first submission, and the above comment was what I added to the text box. It wasn’t clear to me what the purpose was, but it seemed like it would want an excerpt. I only discovered after submitting that it created this comment. I guess people just generally don’t add those? Still, to help me out, could someone clarify why this was down-voted? I don’t want to mess up again if I did, but I don't understand what that was.
  - flomo 1y ago
    I don't know, but I see early posts which look like AI bot summaries (presumably to collect karma). Probably not necessary for a link.
  - shakna 1y ago
    > Leave url blank to submit a question for discussion. If there is no url, text will appear at the top of the thread. If there is a url, text is optional. Most people will opt for text to be optional with a link - unless they're showing their own product (Show HN). Because there is an expectation that you will attempt to read an article, before conversing about it.
  - bawolff 1y ago
    I think its just because as a comment it looks pretty random and somewhat off topic since its a summary of the article instead of an opinion on it. I think most of the time people dont add a comment to submissions, but if they do its more of the form: I found X interesting because of [insert non obvious reason why X is interesting] or some additional non-obvious context needed. In any case, i don't think there is any reason to worry too much. There was no ill intent and at the end of the day its all just fake internet points.
comex 1y ago
If you're evaluating JSON as JavaScript, you also need to make sure none of the objects have a key named "__proto__", or else you can end up with some strange results. (This is related to the 'prototype pollution' attack, although searching that phrase will mostly give you information about the more-dangerous variant where two objects are being merged together with some JS library. If __proto__ is just part of a literal, the behavior is not as dangerous, but still surprising.)
- o11c 1y ago
  But note that there's also `<script type="application/json">` these days (usually only useful with `id=`) ... and `importmap` I guess.
  - themafia 1y ago
    It's even more general: type This attribute indicates the type of script represented. The value of this attribute will be one of the following: [...] Any other value The embedded content is treated as a data block, and won't be processed by the browser. Developers must use a valid MIME type that is not a JavaScript MIME type to denote data blocks. All of the other attributes will be ignored, including the src attribute. Although 'importmap' has specific functionality, as does 'speculationrules', although they operate similarly. My favorite is type="module" which competes with the higher level attribute nomodule="true". Anyways it looks like <script> has taken a lot of abuse over the years: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
    - masklinn 1y ago
      > My favorite is type="module" which competes with the higher level attribute nomodule="true". Anyways it looks like <script> has taken a lot of abuse over the years: It "conflicts" in the same way noscript[1] and script "conflict" no? They're basically related features, but can't really be made exclusive because the mere act of trying to do so wouldn't work: as the link indicates, executing code in a !module browser reserves the type (requires a specific set of types) so you can't use that as a way to opt in !module browsers. [1] an other fun element with wonky parsing rules besides
dullcrisp 1y ago
Wait can someone explain why a script tag inside a comment inside a script tag needs to be closed, while a script tag inside a script tag without a comment does not? They explained why comments inside script tags are a thing, but nothing further than that.
- AdieuToLogic 1y ago
  From the post: Everything until the tag closer </script> is inside the script element. And: In fact, script tags can contain any language (not necessarily JavaScript) or even arbitrary data. In order to support this behavior, script tags have special parsing rules. For the most part, the browser accepts whatever is inside the script tag until it finds the script close tag </script>. Note the sentence fragment "even arbitrary data." This explains the second part of your question as to why nested script tags without HTML comments do not require matching closing tags. Similar compatibility hacks exist for other closing tags (search for Chrome closing tags being optional for a fun ride down a rabbit hole). As to: why a script tag inside a comment inside a script tag needs to be closed ... Well, this again is due to maximizing backward compatibility in order to support broken browsers (thanks IE4, you bastard!). As the article states: When JavaScript was first introduced, many browsers did not support it. So they would render the content of the script tag – the JavaScript code itself. The normal way to get around that was to put the script into a comment ... HTH
  - dullcrisp 1y ago
    So did these older browsers also check for the presence of a comment before turning on double-escaping mode? Or did they always have two levels of script tag escaping but that behavior only got preserved when inside an HTML comment? No other JavaScript behavior is different inside an HTML comment, and I’m still missing the connection between the HTML comment and the embedded </script> not closing the tag besides that they were two things that older browsers might have done.
- dmsnell 1y ago
  The other comment explains this, but I think it can also be viewed differently. It’s helpful to recognize that the inner script tags are not actual script tags. Yes, once entering a script element, the browser switches parsers and wants to skip everything until a closing script tag appears. The STYLE element, TITLE, TEXTAREA, and a few others do this. Once they chop up the HTML like this they send the contents to the separate inner parser (in this case, the JS engine). SCRIPT is unique due to the legacy behavior^1. HTML5 specifies these “inner” tags as transitions into escape modes. The entire goal is to allow JavaScript to contain the string “</script>” without it leaking to the outer parser. The early pattern of hiding inside an HTML comment is what determined the escaping mechanism rather than making some special syntax (which today does exist as noted in the post). The opening script tag inside the comment is actually what triggers the escaping mode, and so it’s less an HTML tag and more some kind of pseudo JS syntax. The inner closing tag is therefore the escaped string value and simultaneously closes the escaped mode. Consider the use of double quotes inside a string. We have to close the outer quote, but if the inner quote is escaped like `\”` then we don’t have to close it — it’s merely data and not syntax. There is only one level of nesting, and eight opening tags would still be “closed” by the single closing tag. ^1: (edit) This is one reason HTML and XML (XHTML) are incompatible. The content of SCRIPT and STYLE elements are essentially just bytes. In XML they must be well-formed markup. XML parsers cannot parse HTML.
TOGoS 1y ago
> Not so fast, things are about to get messy That ship sailed several paragraphs ago, when <script> got special treatment by the HTML parser. Too bad we couldn't all agree to parse <![CDATA[...]]> consistently, or, you know, just &-escape the text like we do /everywhere else/ in HTML.
- forty 1y ago
  What's wrong with CDATA? Do you have concrete examples when that would not work?
  - TOGoS 1y ago
    As per the 'special parsing rules for script tags', browsers don't actually treat it as what you'd expect it means. <script>console.log("<![CDATA[Hello, this string content in a CDATA section!]]>");</script> Results in this being output to the console: <![CDATA[Hello, this string content in a CDATA section!]]> Browsers don't do what you intend if you wrap the whole script in CDATA, either. They treat the "<![CDATA[" sequence as literally part of the script! Which of course throws a syntax error. I tend to use them anyway, as sort of a HTML/XHTML polyglot thing, because deep in my heart I still think HTML should be valid XML: <script>/* <![CDATA[ */ // my script here, and you *still* need to be careful not // to include close-script or close-cdata sequences /* ]]> */</script> In summary, the 'special parsing rules for script tags' add a great amount of complexity not just to the parsing code, but for anybody who has to emit markup, especially if different parsers disagree on what kind of escaping rules are active within a given section. Yes, the HTML5 spec codified the neurotypical "I would rather make you guess what I mean than just use the proper words to say it clearly" behavior, so at least browsers agree on it, but it's a mess and a pain to deal with because now you have to remember 1000 exceptions to what would have been simple rules.
    - pwdisswordfishz 1y ago
      > deep in my heart I still think HTML should be valid XML Never was, never will be. Just write XHTML instead.
pastureofplenty 1y ago
This reminded me of how in the early 2000s I was taught to enclose the content of SCRIPT tags in HTML comments, e.g. <script language="JavaScript"> </script>
maxbond 1y ago
I would say avoid trying to understand arcane nuances better than the adversary. Assume they've simultaneously got more time on their hands and sat on the relevant standards committees. Adopt a strategy that's robust to having missed a small nuance in the standard or in the particular implementation by this or that browser. (That doesn't mean there isn't value in a blog post enumerating the edge cases, of course.) Kaminsky described a very simple and nearly-universal technique to deal with escaping/injection issues. Encode the embedded data as base64 and decode it on the client side. This projects arbitrary data into a fixed, known domain (generally `[a-zA-Z0-9+/]*`) which you can ensure is free from control characters. (You may need to use a particular variant to achieve this, eg for URLs the last characters used are generally `-_` because both + and / are significant in that context.) After decoding, you can pass it to JSON.parse().
- Dylan16807 1y ago
  To me, escaping < for web stuff is just as non-arcane and non-nuanced as base64. And yeah use URL-safe base64 when you do use it. -_ with no padding.
  - maxbond 1y ago
    Yeah, that's fair, and I did forget about `=`/padding when I discussed base64. This instance is a solved problem with a simple solution, blessed by the standards body. The advantage of the base64 technique is that it provides fewer degrees of freedom, and so is more robust to unforseen vectors of attack. It's defensive programming. But it comes at a cost of memory/bandwidth.
wdiamond 1y ago
"<"+"/script>", its a matter of parsing not value
asddubs 1y ago
Is there any specific reason to use JSON_UNESCAPED_SLASHES, or is it just because it becomes unnecessary? The article mentions it several times, but never explains why to use it.
chrismorgan 1y ago
> Imagine if script tags required HTML escaping: There are two situations in which it does. ① XML syntax, which is absolutely still a thing: data:application/xhtml+xml,<html xmlns="http://www.w3.org/1999/xhtml"><script>console.log( 1 > 0 && 0 < 1 )</script></html> ② Inside an SVG <script> element in HTML syntax: data:text/html,<svg><script>console.log( 1 > 0 && 0 < 1 )</script></svg>
westurner 1y ago
What about CDATA; which XML and XHTML support? HTML5 does not support CDATA. CDATA: https://en.wikipedia.org/wiki/CDATA https://en.wikipedia.org/wiki/CDATA <![CDATA[ ]]> This would work for XHTML but not HTML5 IIUC: <script> <![CDATA[ x = {"<!--":""}; ]]> <![CDATA[ {{json.dumps(["<!--"])}} ]]> </script>

86 points by dmsnell 1y ago

Discussing why parsing HTML SCRIPT elements is so complicated, the history of why it became the way it is, and how to safely and securely embed JSON content inside of a SCRIPT element today.

dmsnell 1y ago
This was my first submission, and the above comment was what I added to the text box. It wasn’t clear to me what the purpose was, but it seemed like it would want an excerpt. I only discovered after submitting that it created this comment. I guess people just generally don’t add those? Still, to help me out, could someone clarify why this was down-voted? I don’t want to mess up again if I did, but I don't understand what that was.
- flomo 1y ago
  I don't know, but I see early posts which look like AI bot summaries (presumably to collect karma). Probably not necessary for a link.
- shakna 1y ago
  > Leave url blank to submit a question for discussion. If there is no url, text will appear at the top of the thread. If there is a url, text is optional. Most people will opt for text to be optional with a link - unless they're showing their own product (Show HN). Because there is an expectation that you will attempt to read an article, before conversing about it.
- bawolff 1y ago
  I think its just because as a comment it looks pretty random and somewhat off topic since its a summary of the article instead of an opinion on it. I think most of the time people dont add a comment to submissions, but if they do its more of the form: I found X interesting because of [insert non obvious reason why X is interesting] or some additional non-obvious context needed. In any case, i don't think there is any reason to worry too much. There was no ill intent and at the end of the day its all just fake internet points.

comex 1y ago

If you're evaluating JSON as JavaScript, you also need to make sure none of the objects have a key named "__proto__", or else you can end up with some strange results. (This is related to the 'prototype pollution' attack, although searching that phrase will mostly give you information about the more-dangerous variant where two objects are being merged together with some JS library. If __proto__ is just part of a literal, the behavior is not as dangerous, but still surprising.)

o11c 1y ago
But note that there's also `<script type="application/json">` these days (usually only useful with `id=`) ... and `importmap` I guess.
- themafia 1y ago
  It's even more general: type This attribute indicates the type of script represented. The value of this attribute will be one of the following: [...] Any other value The embedded content is treated as a data block, and won't be processed by the browser. Developers must use a valid MIME type that is not a JavaScript MIME type to denote data blocks. All of the other attributes will be ignored, including the src attribute. Although 'importmap' has specific functionality, as does 'speculationrules', although they operate similarly. My favorite is type="module" which competes with the higher level attribute nomodule="true". Anyways it looks like <script> has taken a lot of abuse over the years: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/script https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...
  - masklinn 1y ago
    > My favorite is type="module" which competes with the higher level attribute nomodule="true". Anyways it looks like <script> has taken a lot of abuse over the years: It "conflicts" in the same way noscript[1] and script "conflict" no? They're basically related features, but can't really be made exclusive because the mere act of trying to do so wouldn't work: as the link indicates, executing code in a !module browser reserves the type (requires a specific set of types) so you can't use that as a way to opt in !module browsers. [1] an other fun element with wonky parsing rules besides

dullcrisp 1y ago

Wait can someone explain why a script tag inside a comment inside a script tag needs to be closed, while a script tag inside a script tag without a comment does not? They explained why comments inside script tags are a thing, but nothing further than that.

AdieuToLogic 1y ago
From the post: Everything until the tag closer </script> is inside the script element. And: In fact, script tags can contain any language (not necessarily JavaScript) or even arbitrary data. In order to support this behavior, script tags have special parsing rules. For the most part, the browser accepts whatever is inside the script tag until it finds the script close tag </script>. Note the sentence fragment "even arbitrary data." This explains the second part of your question as to why nested script tags without HTML comments do not require matching closing tags. Similar compatibility hacks exist for other closing tags (search for Chrome closing tags being optional for a fun ride down a rabbit hole). As to: why a script tag inside a comment inside a script tag needs to be closed ... Well, this again is due to maximizing backward compatibility in order to support broken browsers (thanks IE4, you bastard!). As the article states: When JavaScript was first introduced, many browsers did not support it. So they would render the content of the script tag – the JavaScript code itself. The normal way to get around that was to put the script into a comment ... HTH
- dullcrisp 1y ago
  So did these older browsers also check for the presence of a comment before turning on double-escaping mode? Or did they always have two levels of script tag escaping but that behavior only got preserved when inside an HTML comment? No other JavaScript behavior is different inside an HTML comment, and I’m still missing the connection between the HTML comment and the embedded </script> not closing the tag besides that they were two things that older browsers might have done.
dmsnell 1y ago
The other comment explains this, but I think it can also be viewed differently. It’s helpful to recognize that the inner script tags are not actual script tags. Yes, once entering a script element, the browser switches parsers and wants to skip everything until a closing script tag appears. The STYLE element, TITLE, TEXTAREA, and a few others do this. Once they chop up the HTML like this they send the contents to the separate inner parser (in this case, the JS engine). SCRIPT is unique due to the legacy behavior^1. HTML5 specifies these “inner” tags as transitions into escape modes. The entire goal is to allow JavaScript to contain the string “</script>” without it leaking to the outer parser. The early pattern of hiding inside an HTML comment is what determined the escaping mechanism rather than making some special syntax (which today does exist as noted in the post). The opening script tag inside the comment is actually what triggers the escaping mode, and so it’s less an HTML tag and more some kind of pseudo JS syntax. The inner closing tag is therefore the escaped string value and simultaneously closes the escaped mode. Consider the use of double quotes inside a string. We have to close the outer quote, but if the inner quote is escaped like `\”` then we don’t have to close it — it’s merely data and not syntax. There is only one level of nesting, and eight opening tags would still be “closed” by the single closing tag. ^1: (edit) This is one reason HTML and XML (XHTML) are incompatible. The content of SCRIPT and STYLE elements are essentially just bytes. In XML they must be well-formed markup. XML parsers cannot parse HTML.

TOGoS 1y ago

> Not so fast, things are about to get messy That ship sailed several paragraphs ago, when <script> got special treatment by the HTML parser. Too bad we couldn't all agree to parse <![CDATA[...]]> consistently, or, you know, just &-escape the text like we do /everywhere else/ in HTML.

forty 1y ago
What's wrong with CDATA? Do you have concrete examples when that would not work?
- TOGoS 1y ago
  As per the 'special parsing rules for script tags', browsers don't actually treat it as what you'd expect it means. <script>console.log("<![CDATA[Hello, this string content in a CDATA section!]]>");</script> Results in this being output to the console: <![CDATA[Hello, this string content in a CDATA section!]]> Browsers don't do what you intend if you wrap the whole script in CDATA, either. They treat the "<![CDATA[" sequence as literally part of the script! Which of course throws a syntax error. I tend to use them anyway, as sort of a HTML/XHTML polyglot thing, because deep in my heart I still think HTML should be valid XML: <script>/* <![CDATA[ */ // my script here, and you *still* need to be careful not // to include close-script or close-cdata sequences /* ]]> */</script> In summary, the 'special parsing rules for script tags' add a great amount of complexity not just to the parsing code, but for anybody who has to emit markup, especially if different parsers disagree on what kind of escaping rules are active within a given section. Yes, the HTML5 spec codified the neurotypical "I would rather make you guess what I mean than just use the proper words to say it clearly" behavior, so at least browsers agree on it, but it's a mess and a pain to deal with because now you have to remember 1000 exceptions to what would have been simple rules.
  - pwdisswordfishz 1y ago
    > deep in my heart I still think HTML should be valid XML Never was, never will be. Just write XHTML instead.

pastureofplenty 1y ago

This reminded me of how in the early 2000s I was taught to enclose the content of SCRIPT tags in HTML comments, e.g. <script language="JavaScript"> </script>

maxbond 1y ago

I would say avoid trying to understand arcane nuances better than the adversary. Assume they've simultaneously got more time on their hands and sat on the relevant standards committees. Adopt a strategy that's robust to having missed a small nuance in the standard or in the particular implementation by this or that browser. (That doesn't mean there isn't value in a blog post enumerating the edge cases, of course.) Kaminsky described a very simple and nearly-universal technique to deal with escaping/injection issues. Encode the embedded data as base64 and decode it on the client side. This projects arbitrary data into a fixed, known domain (generally `[a-zA-Z0-9+/]*`) which you can ensure is free from control characters. (You may need to use a particular variant to achieve this, eg for URLs the last characters used are generally `-_` because both + and / are significant in that context.) After decoding, you can pass it to JSON.parse().

Dylan16807 1y ago
To me, escaping < for web stuff is just as non-arcane and non-nuanced as base64. And yeah use URL-safe base64 when you do use it. -_ with no padding.
- maxbond 1y ago
  Yeah, that's fair, and I did forget about `=`/padding when I discussed base64. This instance is a solved problem with a simple solution, blessed by the standards body. The advantage of the base64 technique is that it provides fewer degrees of freedom, and so is more robust to unforseen vectors of attack. It's defensive programming. But it comes at a cost of memory/bandwidth.

wdiamond 1y ago

"<"+"/script>", its a matter of parsing not value

asddubs 1y ago

Is there any specific reason to use JSON_UNESCAPED_SLASHES, or is it just because it becomes unnecessary? The article mentions it several times, but never explains why to use it.

chrismorgan 1y ago

> Imagine if script tags required HTML escaping: There are two situations in which it does. ① XML syntax, which is absolutely still a thing: data:application/xhtml+xml,<html xmlns="http://www.w3.org/1999/xhtml"><script>console.log( 1 > 0 && 0 < 1 )</script></html> ② Inside an SVG <script> element in HTML syntax: data:text/html,<svg><script>console.log( 1 > 0 && 0 < 1 )</script></svg>

westurner 1y ago

What about CDATA; which XML and XHTML support? HTML5 does not support CDATA. CDATA: https://en.wikipedia.org/wiki/CDATA https://en.wikipedia.org/wiki/CDATA <![CDATA[ ]]> This would work for XHTML but not HTML5 IIUC: <script> <![CDATA[ x = {"<!--":""}; ]]> <![CDATA[ {{json.dumps(["<!--"])}} ]]> </script>