4 ms·

How to Create a UTF-16 Surrogate Pair by Hand, with Python

23 points by chubot 3y ago

poorlyknit 3y ago
The biggest takeaway/shock for me in this article is the fact that JSON string literals can't contain escaped characters outside the Basic Multilingual Plane (i.e. whose code points are greater than U+FFFF). Quoting RFC 8259: To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a 12-character sequence, encoding the UTF-16 surrogate pair. So in order to encode U+1F914 THINKING FACE you can either do {"text": "<thinking face>"} (Emoji omitted bc of HN) or {"text": "\ud83e\udd14"} but not {"text": "\u1f914"}. This seems to be a relic from ECMAScript which (iirc, only skimmed it) assumes UCS-2/UTF-16. From that perspective it makes a lot of sense but it steel feels a little icky to me having surrogates referenced in a standard that is supposed to be UTF-8 only :)
- chubot 3y ago
  Right exactly. The conventional ways of writing it would be \U0001F914 -- must be exactly 8 digits \u{1f914} -- 1 to 6 digits Almost all languages have these now, but JavaScript didn't when JSON was invented. The "J8 strings" extension mentioned technically only needs \a syntax for bytes, which is \yff since \xff is (oddly) a synonym for \u00ff and thus unsuitable. But I also want to add \u{123456} because it allows people to move away from the weird UTF-16 legacy in a UTF-8 format.
  - poorlyknit 3y ago
    Yea, it seems weird to me that they decided to do that in ECMAScript in the first place. The user/programmer should never have to interface with surrogates unless they are doing something specific to UTF-16.
    - masklinn 3y ago
      Javascript was released about 6 months before unicode 2.0, and furthermore was in various ways designed to be close to Java, whose strings are sequences of 16 bits code units. Because this is part of the string interface it’s not really fixable. Not only that but while Unicode 2.0 was released in June 1996, it wasn’t necessarily super sought after for a while, lots of people were wary of utf8 and while 16 bit code units was fine 32 was a bit much. While it was singularly late to the party, it took MySQL until 2010 to support non-BMP content, unless you were willing to store content as blobs. Java and Javascript similarly took a while just to give access to actual codepoints, in Java5 (2004, String#codePointAt, it took until Java 8 for an iterator to be added with CharSequence#codePoints) and ES6 (2015, String.prototype.codePointAt and String.prototype[@@iterator])
- TRiG_Ireland 3y ago
  As JSON must be in Unicode (I think the latest RFC restricts it to UTF-8), there's no actual _need_ to escape emoji. But if you do need to (perhaps you're in an ASCII-only environment), you have to do it that way, yes. Weird. (I asked once, on Stack Overflow, why this was the case, and was merely told that it was historical reasons, which matches what you surmised: it's a relic.)
  - poorlyknit 3y ago
    Right! In practice you'd just put the symbols directly. Funnily enough {"text": "\ud83e\udd14"} is the only way to encode the JSON from my post on HN :D
- dataflow 3y ago
  Wouldn't \u1f914 have been interpreted as \u1f19 followed by '4'? How were you imagining this might work?
  - deleted 3y ago
    [deleted]
- deleted 3y ago
  [deleted]
js2 3y ago
> As another data point, the node.js interpreter chooses to print the � replacement char instead of raising an exception. You can do the same in Python: >>> print(s.encode("utf-8","replace").decode("utf-8")) ? Or do this: import sys sys.stdout.reconfigure(errors="replace") Alternately, start python with: PYTHONIOENCODING=utf8:replace You can also use "ignore" to drop the illegal character entirely.
Aardwolf 3y ago
Surrogate pairs are such a useless thing to exist. A unicode character is just a number (of up to around 21 bits). The encoding of this number, and the unicode value itself, can (and should) be made completely independent. Note how UTF-8 did not introduce any useless values in unicode itself. That's because this code is sane (if a bit inefficient due to over-focus on error correction) and correctly makes its encoded byte values independent of the numeric value being encoded. For UTF-16, initially they thought 16 bits would be enough, and then they had to hack something on top of it to make it variable length. Fair enough, but even that hack, they could have done without introducing useless unicode values: they could have defined the variable length 16-bit code to use the unused codes to indicate variable length, and then add/subtract values while decoding it, that is only use the unused UTF-16 values to tell how to use the remaining UTF-16 bits to take existing consecutive values from unicode. This would have been not a bigger mess than surrogates are, be transparent to users (since the decoder deals with the math), and left the mess inside of UTF-16 only (which at this point they should have known was already game over since the planned 16 bits didn't fit) instead of infecting the rest with its mess. So not only is UTF-16 the ugliest encoding ever by being the worst of both worlds (variable length and yet bigger than a byte and not ascii compatible) and damages both windows (ruined filename loading in multiplatform C libraries, horrible "wstrings" in windows code) and java/javascript (UTF-16 in the language while the internet they interface with are UTF-8, exposing programmers to "surrogates"), but in addition in the making of this ugly code they managed to also uglify unicode itself, and UTF-8 (which has to deal with the fact it can refer to surrogates), and any other future encoding of unicode too!

23 points by chubot 3y ago

The biggest takeaway/shock for me in this article is the fact that JSON string literals can't contain escaped characters outside the Basic Multilingual Plane (i.e. whose code points are greater than U+FFFF). Quoting RFC 8259: To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a 12-character sequence, encoding the UTF-16 surrogate pair. So in order to encode U+1F914 THINKING FACE you can either do {"text": "<thinking face>"} (Emoji omitted bc of HN) or {"text": "\ud83e\udd14"} but not {"text": "\u1f914"}. This seems to be a relic from ECMAScript which (iirc, only skimmed it) assumes UCS-2/UTF-16. From that perspective it makes a lot of sense but it steel feels a little icky to me having surrogates referenced in a standard that is supposed to be UTF-8 only :)

chubot 3y ago
Right exactly. The conventional ways of writing it would be \U0001F914 -- must be exactly 8 digits \u{1f914} -- 1 to 6 digits Almost all languages have these now, but JavaScript didn't when JSON was invented. The "J8 strings" extension mentioned technically only needs \a syntax for bytes, which is \yff since \xff is (oddly) a synonym for \u00ff and thus unsuitable. But I also want to add \u{123456} because it allows people to move away from the weird UTF-16 legacy in a UTF-8 format.
- poorlyknit 3y ago
  Yea, it seems weird to me that they decided to do that in ECMAScript in the first place. The user/programmer should never have to interface with surrogates unless they are doing something specific to UTF-16.
  - masklinn 3y ago
    Javascript was released about 6 months before unicode 2.0, and furthermore was in various ways designed to be close to Java, whose strings are sequences of 16 bits code units. Because this is part of the string interface it’s not really fixable. Not only that but while Unicode 2.0 was released in June 1996, it wasn’t necessarily super sought after for a while, lots of people were wary of utf8 and while 16 bit code units was fine 32 was a bit much. While it was singularly late to the party, it took MySQL until 2010 to support non-BMP content, unless you were willing to store content as blobs. Java and Javascript similarly took a while just to give access to actual codepoints, in Java5 (2004, String#codePointAt, it took until Java 8 for an iterator to be added with CharSequence#codePoints) and ES6 (2015, String.prototype.codePointAt and String.prototype[@@iterator])
TRiG_Ireland 3y ago
As JSON must be in Unicode (I think the latest RFC restricts it to UTF-8), there's no actual _need_ to escape emoji. But if you do need to (perhaps you're in an ASCII-only environment), you have to do it that way, yes. Weird. (I asked once, on Stack Overflow, why this was the case, and was merely told that it was historical reasons, which matches what you surmised: it's a relic.)
- poorlyknit 3y ago
  Right! In practice you'd just put the symbols directly. Funnily enough {"text": "\ud83e\udd14"} is the only way to encode the JSON from my post on HN :D
dataflow 3y ago
Wouldn't \u1f914 have been interpreted as \u1f19 followed by '4'? How were you imagining this might work?
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]

js2 3y ago

> As another data point, the node.js interpreter chooses to print the � replacement char instead of raising an exception. You can do the same in Python: >>> print(s.encode("utf-8","replace").decode("utf-8")) ? Or do this: import sys sys.stdout.reconfigure(errors="replace") Alternately, start python with: PYTHONIOENCODING=utf8:replace You can also use "ignore" to drop the illegal character entirely.

Aardwolf 3y ago

Surrogate pairs are such a useless thing to exist. A unicode character is just a number (of up to around 21 bits). The encoding of this number, and the unicode value itself, can (and should) be made completely independent. Note how UTF-8 did not introduce any useless values in unicode itself. That's because this code is sane (if a bit inefficient due to over-focus on error correction) and correctly makes its encoded byte values independent of the numeric value being encoded. For UTF-16, initially they thought 16 bits would be enough, and then they had to hack something on top of it to make it variable length. Fair enough, but even that hack, they could have done without introducing useless unicode values: they could have defined the variable length 16-bit code to use the unused codes to indicate variable length, and then add/subtract values while decoding it, that is only use the unused UTF-16 values to tell how to use the remaining UTF-16 bits to take existing consecutive values from unicode. This would have been not a bigger mess than surrogates are, be transparent to users (since the decoder deals with the math), and left the mess inside of UTF-16 only (which at this point they should have known was already game over since the planned 16 bits didn't fit) instead of infecting the rest with its mess. So not only is UTF-16 the ugliest encoding ever by being the worst of both worlds (variable length and yet bigger than a byte and not ascii compatible) and damages both windows (ruined filename loading in multiplatform C libraries, horrible "wstrings" in windows code) and java/javascript (UTF-16 in the language while the internet they interface with are UTF-8, exposing programmers to "surrogates"), but in addition in the making of this ugly code they managed to also uglify unicode itself, and UTF-8 (which has to deal with the fact it can refer to surrogates), and any other future encoding of unicode too!