4 ms·
[This comment is wrong; see masklinn’s response for my misunderstanding.] > Now the function would rather insert an escape character before the character code
by chrismorgan 7y ago
[This comment is wrong; see masklinn’s response for my misunderstanding.]
> Now the function would rather insert an escape character before the character code so that the result is still readable and valid UTF-8/UTF-16 code:
> JSON.stringify('\uD83D');
> // '"\\ud83d"'
I’m inclined to consider this a misfeature, unbreaking something that I’m glad was broken and should have remained broken.
Unpaired surrogates are (to simplify terminology a little) basically invalid Unicode. On the web platform, the only situation where unpaired surrogates should be encountered is as a transient state on user input, on platforms that send non-BMP characters through in two pieces (which is most browsers on Windows). Beyond that, nothing should ever deal in unpaired surrogates, because they will make things blow up and your life miserable.
Unpaired surrogates cannot be represented in UTF-8, or in well-formed UTF-16 (and that’s where JavaScript did the annoyingly bad thing that we’re paying for decades later, going with UTF-16 and not requiring well-formedness). Hence I’d quibble with the description of the result as “valid UTF-8/UTF-16 code”. (Sure, the actual JSON byte stream will be valid, but it contains an escaped string that is not valid.) U+FFFD REPLACEMENT CHARACTER was at least as valid, arguably more valid.
Various JSON parsers will choke on such strings, as observed in what I’d consider the canonical JSON spec: https://tools.ietf.org/html/rfc7159#section-8.2 https://tools.ietf.org/html/rfc7159#section-8.2.
In the I-JSON restricted subset, which is what I’d say everything should actually limit itself to, such strings are disallowed and will cause parse failure: https://tools.ietf.org/html/rfc7493#section-2.1 https://tools.ietf.org/html/rfc7493#section-2.1.
- masklinn 7y agoYou're misreading the article section, although the article is also partially wrong: the old behaviour was to output the lone surrogate in the JSON stream[0] (the "�" here is confusing: it is a display artefact of the specific font, not U+FFFD), the new behaviour is to replace the unpaired surrogate by the literal 6-character long escaped representation of the unpaired surrogate. That is, formerly U+D83D would pass through straight as U+D83D (or possibly be replaced by U+FFFD in some implementations), whereas it is now "encoded" as the sequence U+005C U+0075 U+0044 U+0038 U+0033 U+0044. That is, the entire point of the "well-formed stringify" proposal[1] was to stop generating broken JSON: > Rather than return unpaired surrogate code points as single UTF-16 code units, represent them with JSON escape sequences. [0] it's possible that some implementations would replace the lone surrogate by U+FFFD but according to MDN: > Before this change JSON.stringify would output lone surrogates if the input contained any lone surrogates; such strings could not be encoded in valid UTF-8 or UTF-16 this is also the behaviour I observe on an old safari, and the one which is quoted in the tc39 proposal[1]: > JSON.stringify can return strings including code points that have no representation in UTF-8 (specifically, surrogate code points U+D800 through U+DFFF). And contrary to the description of JSON.stringify, such strings are not "in UTF-16" because "isolated UTF-16 code units in the range D800₁₆..DFFF₁₆ are ill-formed" per The Unicode Standard, Version 10.0.0, Section 3.4 at definition D91 and excluded from being "in UTF-16" per definition D89. [1] https://github.com/tc39/proposal-well-formed-stringify https://github.com/tc39/proposal-well-formed-stringify
- deleted 7y ago[deleted]
- chrismorgan 7y agoAh, you’re right; thanks for the correction. The article does contain an error here, for the code point on the page is actually U+FFFD, not U+D83D (which of course is unrepresentable there). I trusted that (being a little surprised that it’d do the replacement, which is more what I’d expect of Python, but accepting it all the same) and didn’t pull out an old browser to confirm what actually happens, which is, as you say, an unpaired surrogate code point. The article should probably say '"\ud83d"' instead of '"�"' (even though the number of backslashes still then requires thought), because yielding U+FFFD there would be a completely valid (and preferable, in my opinion) solution.
- uncletaco 7y agoI love when the reaction to being wrong is to explain how the thing is responsible for their error and so they waste a few sentences explaining how it could have avoided leading then astray.
- masklinn 7y ago> The article does contain an error here, for the code point on the page is actually U+FFFD, not U+D83D (which of course is unrepresentable there). The MDN page also does that, that's a bit misleading but TFA commits way worse a sin: it actually states the unpaired codepoint is swapped for the replacement character: > In earlier versions, these would be replaced with a special character: which is just wrong. > yielding U+FFFD there would be a completely valid (and preferable, in my opinion) solution. I agree. The proposal does not explain whether U+FFFD was considered, I expect they picked the escaped version to limit or avoid data loss: if you get an U+FFFD you have no idea what it used to be.
- burpsnard 7y agoI'm suddenly reminded of the hours i spent noodling around with sgml.xml entity catalogues (™ © etc)