10 ms·

JavaScript string encoding

103 points by iamwil 9y ago

userbinator 9y ago
Unless of course you specify hex or base64, in which case it does refer to the encoding of the output string I've seen this... questionable design decision in another interpreted/scripting language too. Those are encodings, but clearly not encodings at the same "layer of abstraction" as e.g. UTF-8 or Shift-JIS or UTF-32 or UTF-16, because you could have a UTF-16 string containing "base64" or "hex".
andreasgonewild 9y ago
As much as I enjoy making fun of JavaScript, it seems more likely to me that the reason JavaScript uses UTF16 internally is the same as for most other languages that support Unicode; it's more efficient and convenient to process. UTF8 has variable character boundaries, which means that indexing/counting requires decoding char by char; but it works wonders as an exchange format since it's compact and mostly any language can deal with it.
- mikeash 9y ago
  All Unicode encodings require intelligent indexing. JavaScript uses UTF-16 because that (or rather its predecessor UCS-2) was the standard when it was being created. Same reason Java and Apple's Objective-C frameworks use it.
  - andreasgonewild 9y ago
    Not to the same extent as UTF8. It's not like UTF8 invalidated all other encodings; they still fill a purpose and UTF16 seems to still be a popular choice for internal processing, despite the misguided push to use UTF8 for everything.
    - mikeash 9y ago
      What's the difference? Both UTF-8 and UTF-16 are variable-length encodings where careless mutation with integer indexes can produce invalid results. UTF-8 is 1-4 bytes per code point, whereas UTF-16 is only 1-2 code units per code point, but that doesn't really make it easier. And proper handling really requires detecting grapheme cluster boundaries, which is the same difficulty regardless of whether you use UTF-8, UTF-16, or UTF-32.
      - andreasgonewild 9y ago
        So you're looking at up to four times as many chars per code point to take into account, but you still claim it's mostly the same thing. Good luck with the lobbying then, I think we're going to have to agree to disagree on this one.
        mikeash 9y ago
        Can you describe an operation (other than "count the number of UTF-16 code units") which is easier to code for UTF-16 than UTF-8?
        aurelian15 9y ago
        Well, even counting the number of code units is straight forward for UTF-8: while (*c) count += ((*(c++) & 0xC0) == 0x80) ? 0 : 1; See https://stackoverflow.com/questions/9356169/utf-8-continuation-bytes https://stackoverflow.com/questions/9356169/utf-8-continuati... for more details.
        mikeash 9y ago
        That's counting code points, not code units. A code point is the Unicode "character" number. A code unit is the smallest unit used by an encoding, such as a byte in UTF-8 or two bytes in UTF-16. Counting the number of UTF-8 code units in a UTF-8 string is of course trivial. Counting the number of UTF-16 code units in a UTF-8 strings would take more work. But there's probably no reason you'd want to compute that anyway.
    - YSFEJ4SWJUVU6 9y ago
      I don't know what angle you are coming from, but by far and large most uses of UTF-16 are entirely due to legacy reasons – APIs and languages made before Unicode extended beyond what fits in 16 bits. Back then, the trade-off made sense, and was popular too, because variable-sized encodings were much more rare. These days UTF-16 really is the worst of both worlds, but we are stuck with what we have.
  - aurelian15 9y ago
    This. Even if you would represent Unicode strings as 32-bit codewords, it would still require careful processing, e.g. to extract single characters from the string. For example, the single character "Ï" can both be represented as the single codeword "u+CF" and the codeword sequence "u+308 u+49". Due to these complications I favour the old method of just treating strings as byte sequences (and possibly enforcing UTF-8 as encoding, since it is a strict superset of ASCII) and to use specialized functions for Unicode processing.
    - andreasgonewild 9y ago
      But processing UTF8 is still without a doubt more complex; you're not going to weasel your way out of that fact, no matter how many cases you can think of where they are comparable. Why can't several alternatives be allowed to coexist and complement each other? Why does everything have to be UTF8, or JavaScript, or Rust, or Go or whatever?
      - aurelian15 9y ago
        Can you elaborate? I didn't say that you should use UTF-8 (that's just what I prefer personally), but my point was that you should never make any assumption about a Unicode string without consulting the corresponding Unicode tables and essentially have to treat strings as "opaque sequence of something" anyways. May as well be a byte sequence. Regarding your last point, I'm totally with you (if I understood you correctly). Of course applications should support multiple input/output encodings, but as a programmer you have to decide on some internal representation. That being said, I really don't see how processing UTF-8 is significantly more complex than processing, say, UTF-16. In both cases you need to handle continuation units for the extraction of Unicode code points.
        userbinator 9y ago
        That being said, I really don't see how processing UTF-8 is significantly more complex than processing, say, UTF-16. In both cases you need to handle continuation units for the extraction of Unicode code points. UTF-8 has 4 valid cases, one for each length, and many more invalid cases for each length (2-byte sequence missing trail byte, 3-byte sequence missing 1 trail byte, 3-byte sequence missing 2 trail bytes, 4-byte sequence missing 3 trail bytes, 4-byte sequence missing 2 trail bytes, ..., overlongs, UTF-8'd surrogates, overflow, etc.) Differences between implementations' treatment of error cases have lead to some security concerns; see https://hsivonen.fi/broken-utf-8/ https://hsivonen.fi/broken-utf-8/ and discussion at https://news.ycombinator.com/item?id=14451822 https://news.ycombinator.com/item?id=14451822 for an example. UTF-16 has two valid cases (one or two code units) and two error cases (lead surrogate not followed by trail surrogate, lone trail surrogate). It's more like a DBCS, except each code unit is 2 instead of 1 byte.
  - pjc50 9y ago
    And Windows ("#define _UNICODE", which actually makes everything use UCS-2)
- pornel 9y ago
  UTF-16 is a variable-width encoding. Thanks to surrogate pairs some code points take 2 bytes, and some take 4. Even if you use UCS-2 (the 2-byte "UTF-16" infamous for mangling emoji) constant-time indexing of code points still doesn't give you constant time access to what humans would call characters ("grapheme clusters" in Unicode), because of decomposed characters with modifiers and joiners. Languages that use UTF-16 mostly use it only because they're older than UTF-8 (or are tied to a platform older than UTF-8).
  - millstone 9y ago
    UTF-16 is a variable width encoding that is also more convenient to process compared to UTF-8. For example you don't have to deal with invalid code units, non-shortest forms, etc. I think you're right that older languages use UTF-16 and newer ones use UTF-8. But it also seems empirically true that UTF-16 languages do better at grappling with Unicode's subtleties, compared to UTF-8. The temptation of UTF8-is-bsically-C-strings is hard to ignore.
    - duskwuff 9y ago
      > For example you don't have to deal with invalid code units, non-shortest forms, etc. You have to deal with unpaired surrogates, though. And just because that wasn't annoying enough, any UTF-8 sequence which contains an encoded surrogate (which is technically invalid, but not prohibited by most implementations) is impossible to encode as UTF-16.
- duskwuff 9y ago
  Javascript doesn't specify anything about what character encoding is used "internally". Indeed, as the first comment on the article points out, some implementations (like V8) internally use ASCII to store strings that contain only ASCII characters. What Javascript does is use UTF-16 semantics for Unicode strings. The reason why it does this is simple: when those methods were implemented in the mid-1990s, UTF-16 was largely synonymous with Unicode. No characters beyond U+FFFF were defined until the release of Unicode 3.1 in March 2001.
josteink 9y ago
> A string is a series of bytes. Incorrect or wildly inaccurate. A string is conceptually text which may be (is) represented internally as bytes, through some means of encoding that text. And thus the concept of encodings are introduced. That (and how) text is represented should be an implementation-detail though: Strings represent text, not bytes. I think most people miss this distinction and that's the main source of confusion for encoding-related problems among programmers.
- tyingq 9y ago
  Depends on the language. Some co-opt the string as a generic buffer too, and have no other suitable type to use.
- dgreensp 9y ago
  Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, but that's irrelevant to how to think of the string type in JavaScript, or in an ideal world. One level of abstraction that works is to think of a string as a series of integers (which may be much larger than 65535) representing code points. Whether your VM uses a fixed-length or variable-length encoding to encode this series of integers should be irrelevant, aside from performance and memory consumption concerns. JavaScript's "charAt" and numeric indexing of strings violate this abstraction, making them not useful in Unicode-aware code. Of course, the "series of code points" abstraction has limits too, since what we think of as a single "character" can be composed of multiple code points, which calls into the question the whole concept of a "character," which is fuzzy in the first place. This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary. There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. This line of thinking does not make sense to me, and I'm not sure what scheme is being proposed. First of all, "most strings" are representable as UTF-8? All strings are representable as UTF-8. And if you are reading data from a UTF-8 source and you don't want to pay the time and memory cost of decoding it and re-encoding it in a less space-efficient code, leave it as a Buffer of bytes! Don't decode it. Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8. Node does make you specify which encoding you want when converting between strings and Buffers, but it's always embraced UTF-8 as its primary encoding, as far as I'm aware. Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16. This is kind of silly. Java made the same decision and was certainly not "invented" in ten man-days!
huhlig 9y ago
This just leads me to believe that Node and it's followers never learned the lessons from their stint with php. Also, "Encoding is the process of squashing the graphics you see on screen, say, 世 - into actual bytes." No it's a way of representing one value in a way a system can more easily handle in a hopefully lossless fashion. Encoding has nothing to do with what's on screen other than that being one representation of the data.
benjaminjackman 9y ago
I wonder if there is ever going to be an encoding that replaces UTF-8? Or have we hit on some sort of permanent local maxima (not a global one in the sense that UTF-8 carries the baggage of being backwards compatible with ASCII ... though maybe you could argue that's more of a unicode problem than a UTF8 encoding format one). At this point UTF-8 seems pretty permanent, what would come along to replace it? And if it is likely to be permanent shouldn't node / javascript in general be moving towards deprecating UCS-2 / UTF-16 and giving first class support to UTF-8? I saw all this because a couple of years ago I had to write a UTF-8 converter before ScalaJS natively supported for a serialization library I had written. I was kind of surprised that the javascript support was so lacking, luckily writing a UTF-8 encoder/decoder isn't that hard of an endeavor.
- pwdisswordfish 9y ago
  Functions like String.prototype.charCodeAt, String.prototype.indexOf, String.prototype.substr and such (which operate on indices into 16-bit-wide strings) will have to be supported somehow, so I doubt it.
  - gumby 9y ago
    Only matters to the Windows platform, though. EDIT: early morning brain fart on my part, js of course too. Grr.
    - pwdisswordfish 9y ago
      What? These are cross-platform features, defined in the ECMAScript spec. Nearly all string-processing code written in JavaScript uses them.
  - carussell 9y ago
    String.fromCodePoint and String#codePointAt were introduced for this reason. Or rather, the time to bring the UTF-8 Everywhere initiative to JS would have been when these two methods were introduced for ES6. Unfortunately, that didn't happen and the committee went with UTF-16 (which is not the same thing as UCS-2—the fact that they differ is why these methods were introduced in the first place).
gumby 9y ago
> You could easily represent all of the characters in the Unicode set with an encoding that says simply "assign one number, 4 bytes (or 32 bits) long, for each character in the Unicode set." Actually you can't, at least not in a standard way. There are combinations of code points that don't have a single code point equivalent. Flag emjois are an example, but there are many letterlike versions as well. There are enough unallocated points in the 32-bit space that probably you could manage to make single-point equivalents on your own.
- whipoodle 9y ago
  Hmm, I don't understand. AFAIK flag emojis are ligatures across two code points representing the two-letter international code for the country.
  - mahkoh 9y ago
    gumby is deliberately interpreting the word character to mean grapheme cluster (whereas the author meant code point) to make a point.
    - mikeash 9y ago
      We really should avoid that word when discussing Unicode. It just leads to confusion.
- johncolanduoni 9y ago
  > There are enough unallocated points in the 32-bit space that probably you could manage to make single-point equivalents on your own. Unicode doesn't prevent you from drowning letters in many combining marks (e.g. an accent, an umlaut, and an overhead tilde), or doing this with letters where they make no sense and for which no font will have useful positioning information (e.g. over a copyright sign). There's over 1700 combining marks in Unicode, which gives you way over 2^32 ways to use the combining marks with the letter a, let alone the rest of the letters in Unicode.
cantstop99 9y ago
Lol what a fuckup of a language. So glad the entire internet depends on it.
fuckhn88 9y ago
Lol what a fuckup of a language. So glad the entire internet depends on it.
benhacker127001 9y ago
I am a mobile phone and software encryption specialist. Contact me if you need a phone hack or any other related services email: eightspyders@gmail.com
zengid 9y ago
> There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. I'm guessing you'd have to write a C++ module for this, but any suggestions on how one might do this successfully?
- kevinburke 9y ago
  You could use JavaScript, bit shifting and codePointAt, I think, but it would be a huge pain.
- pwdisswordfish 9y ago
  That's completely bogus. Forget unpaired surrogates; this scheme cannot even represent odd-length ASCII strings.
0x0 9y ago
I don't think this article is very good. It seems to make the "newbie unicode error" of assuming that strings "have" an encoding (or that strings "are" UTF-8) and thinking of bytes in string. For example in the JSON paragraph, it makes the cardinal sin of referring to "... creating UTF-8 strings". No such thing! Strings are an array of integer unicode code points. Stop thinking about bytes at this level. The internal representation of strings and chars does not matter, because you as a programmer ever only see integer code points. Encoding only enters the picture the moment you want to convert your string to or from a byte array (for example to write to disk or send over the network). The encoding, such as "UTF-8", then specifies how to map between the array of abstract code points to an array of 8bit bytes.
- alangpierce 9y ago
  Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a language like that, you should probably have a basic understanding of how UTF-16 works. I'd say a JavaScript programmer should think of things this way: * JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points. * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. * Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.
  - 0x0 9y ago
    It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)
twotwotwo 9y ago
Commented this on the blog; cross-posting here: V8 turns out to have a ton of internal string representations. They don’t affect semantics (the general point that JS string functions think in UTF-16 is valid) but they’re interesting. V8 apparently stores all-ASCII strings as ASCII, so stuff like HTML tag names or base64 blobs doesn’t double in size. Like Go, V8 lets you take a substring as a pointer into the larger string; the internal class is called SlicedString, but from JS-land you don’t see anything different from a string literal. As in Go, keeping a short substring of a long parent string keeps the whole parent ‘alive’ across GCs so sometimes folks will be surprised all those bytes are still allocated. Unlike Go, V8 has a ConsString type, so concatenating strings sometimes doesn’t really immediately copy the underlying bytes anywhere. Building a string with a loop that runs str += newPiece probably goes faster than expected because of this. [It turns out it flattens the string the next time you index into it, or at least used to, which has some perf implications of its own: https://gist.github.com/mraleph/3397008 https://gist.github.com/mraleph/3397008] That’s mostly from this post by a member of the Dart team http://mrale.ph/blog/2016/11/23/making-less-dart-faster.html http://mrale.ph/blog/2016/11/23/making-less-dart-faster.html ; his blog has a lot of interesting stuff about how these fine-tuned language implementations really work. Kind of amazing the lengths V8 (and other JS engine) teams went to to make the code they saw in the wild work well.
- evilpie 9y ago
  Here is a comment describing strings in SpiderMonkey: http://searchfox.org/mozilla-central/rev/999385a5e8c2d360cc37286882508075fc2078bd/js/src/vm/String.h#44 http://searchfox.org/mozilla-central/rev/999385a5e8c2d360cc3.... If you scroll down there is ASCII art showing the different string sub-classes.
- ubernostrum 9y ago
  V8 apparently stores all-ASCII strings as ASCII, so stuff like HTML tag names or base64 blobs doesn’t double in size. Python does this too. In Python 2, and in Python 3 until 3.3, a compile-time flag determined the internal Unicode storage of the Python interpreter; a "narrow" build of the interpreter used 2-byte Unicode storage with surrogate pairs, while a "wide" build used 4-byte Unicode storage. As of Python 3.3, the internal storage of Unicode is dynamic. Python 3 source code is always parsed as UTF-8, but then as string objects are created by the interpreter their memory representation is chosen on a per-string basis, to be able to accommodate the widest code point present in the string. So Python will choose either a one-byte, two-byte, or four-byte encoding to store the string in memory, depending on what code points are present in it. This is very nice because it means iteration over a Python string is always iteration over its code points, the length of a string is always the number of code points in it, and indexing always yields the code point at index, since the internal storage of the string is fixed width and never has to include surrogates (in pre-3.3 Python, "narrow" builds would actually yield up things for which ord() gave a value in the surrogate range, and code points requiring surrogates added 2 to the length of a string rather than 1).

103 points by iamwil 9y ago

Unless of course you specify hex or base64, in which case it does refer to the encoding of the output string I've seen this... questionable design decision in another interpreted/scripting language too. Those are encodings, but clearly not encodings at the same "layer of abstraction" as e.g. UTF-8 or Shift-JIS or UTF-32 or UTF-16, because you could have a UTF-16 string containing "base64" or "hex".

andreasgonewild 9y ago

As much as I enjoy making fun of JavaScript, it seems more likely to me that the reason JavaScript uses UTF16 internally is the same as for most other languages that support Unicode; it's more efficient and convenient to process. UTF8 has variable character boundaries, which means that indexing/counting requires decoding char by char; but it works wonders as an exchange format since it's compact and mostly any language can deal with it.

mikeash 9y ago
All Unicode encodings require intelligent indexing. JavaScript uses UTF-16 because that (or rather its predecessor UCS-2) was the standard when it was being created. Same reason Java and Apple's Objective-C frameworks use it.
- andreasgonewild 9y ago
  Not to the same extent as UTF8. It's not like UTF8 invalidated all other encodings; they still fill a purpose and UTF16 seems to still be a popular choice for internal processing, despite the misguided push to use UTF8 for everything.
  - mikeash 9y ago
    What's the difference? Both UTF-8 and UTF-16 are variable-length encodings where careless mutation with integer indexes can produce invalid results. UTF-8 is 1-4 bytes per code point, whereas UTF-16 is only 1-2 code units per code point, but that doesn't really make it easier. And proper handling really requires detecting grapheme cluster boundaries, which is the same difficulty regardless of whether you use UTF-8, UTF-16, or UTF-32.
    - andreasgonewild 9y ago
      So you're looking at up to four times as many chars per code point to take into account, but you still claim it's mostly the same thing. Good luck with the lobbying then, I think we're going to have to agree to disagree on this one.
      - mikeash 9y ago
        Can you describe an operation (other than "count the number of UTF-16 code units") which is easier to code for UTF-16 than UTF-8?
        aurelian15 9y ago
        Well, even counting the number of code units is straight forward for UTF-8: while (*c) count += ((*(c++) & 0xC0) == 0x80) ? 0 : 1; See https://stackoverflow.com/questions/9356169/utf-8-continuation-bytes https://stackoverflow.com/questions/9356169/utf-8-continuati... for more details.
        mikeash 9y ago
        That's counting code points, not code units. A code point is the Unicode "character" number. A code unit is the smallest unit used by an encoding, such as a byte in UTF-8 or two bytes in UTF-16. Counting the number of UTF-8 code units in a UTF-8 string is of course trivial. Counting the number of UTF-16 code units in a UTF-8 strings would take more work. But there's probably no reason you'd want to compute that anyway.
  - YSFEJ4SWJUVU6 9y ago
    I don't know what angle you are coming from, but by far and large most uses of UTF-16 are entirely due to legacy reasons – APIs and languages made before Unicode extended beyond what fits in 16 bits. Back then, the trade-off made sense, and was popular too, because variable-sized encodings were much more rare. These days UTF-16 really is the worst of both worlds, but we are stuck with what we have.
- aurelian15 9y ago
  This. Even if you would represent Unicode strings as 32-bit codewords, it would still require careful processing, e.g. to extract single characters from the string. For example, the single character "Ï" can both be represented as the single codeword "u+CF" and the codeword sequence "u+308 u+49". Due to these complications I favour the old method of just treating strings as byte sequences (and possibly enforcing UTF-8 as encoding, since it is a strict superset of ASCII) and to use specialized functions for Unicode processing.
  - andreasgonewild 9y ago
    But processing UTF8 is still without a doubt more complex; you're not going to weasel your way out of that fact, no matter how many cases you can think of where they are comparable. Why can't several alternatives be allowed to coexist and complement each other? Why does everything have to be UTF8, or JavaScript, or Rust, or Go or whatever?
    - aurelian15 9y ago
      Can you elaborate? I didn't say that you should use UTF-8 (that's just what I prefer personally), but my point was that you should never make any assumption about a Unicode string without consulting the corresponding Unicode tables and essentially have to treat strings as "opaque sequence of something" anyways. May as well be a byte sequence. Regarding your last point, I'm totally with you (if I understood you correctly). Of course applications should support multiple input/output encodings, but as a programmer you have to decide on some internal representation. That being said, I really don't see how processing UTF-8 is significantly more complex than processing, say, UTF-16. In both cases you need to handle continuation units for the extraction of Unicode code points.
      - userbinator 9y ago
        That being said, I really don't see how processing UTF-8 is significantly more complex than processing, say, UTF-16. In both cases you need to handle continuation units for the extraction of Unicode code points. UTF-8 has 4 valid cases, one for each length, and many more invalid cases for each length (2-byte sequence missing trail byte, 3-byte sequence missing 1 trail byte, 3-byte sequence missing 2 trail bytes, 4-byte sequence missing 3 trail bytes, 4-byte sequence missing 2 trail bytes, ..., overlongs, UTF-8'd surrogates, overflow, etc.) Differences between implementations' treatment of error cases have lead to some security concerns; see https://hsivonen.fi/broken-utf-8/ https://hsivonen.fi/broken-utf-8/ and discussion at https://news.ycombinator.com/item?id=14451822 https://news.ycombinator.com/item?id=14451822 for an example. UTF-16 has two valid cases (one or two code units) and two error cases (lead surrogate not followed by trail surrogate, lone trail surrogate). It's more like a DBCS, except each code unit is 2 instead of 1 byte.
- pjc50 9y ago
  And Windows ("#define _UNICODE", which actually makes everything use UCS-2)
pornel 9y ago
UTF-16 is a variable-width encoding. Thanks to surrogate pairs some code points take 2 bytes, and some take 4. Even if you use UCS-2 (the 2-byte "UTF-16" infamous for mangling emoji) constant-time indexing of code points still doesn't give you constant time access to what humans would call characters ("grapheme clusters" in Unicode), because of decomposed characters with modifiers and joiners. Languages that use UTF-16 mostly use it only because they're older than UTF-8 (or are tied to a platform older than UTF-8).
- millstone 9y ago
  UTF-16 is a variable width encoding that is also more convenient to process compared to UTF-8. For example you don't have to deal with invalid code units, non-shortest forms, etc. I think you're right that older languages use UTF-16 and newer ones use UTF-8. But it also seems empirically true that UTF-16 languages do better at grappling with Unicode's subtleties, compared to UTF-8. The temptation of UTF8-is-bsically-C-strings is hard to ignore.
  - duskwuff 9y ago
    > For example you don't have to deal with invalid code units, non-shortest forms, etc. You have to deal with unpaired surrogates, though. And just because that wasn't annoying enough, any UTF-8 sequence which contains an encoded surrogate (which is technically invalid, but not prohibited by most implementations) is impossible to encode as UTF-16.
duskwuff 9y ago
Javascript doesn't specify anything about what character encoding is used "internally". Indeed, as the first comment on the article points out, some implementations (like V8) internally use ASCII to store strings that contain only ASCII characters. What Javascript does is use UTF-16 semantics for Unicode strings. The reason why it does this is simple: when those methods were implemented in the mid-1990s, UTF-16 was largely synonymous with Unicode. No characters beyond U+FFFF were defined until the release of Unicode 3.1 in March 2001.

josteink 9y ago

> A string is a series of bytes. Incorrect or wildly inaccurate. A string is conceptually text which may be (is) represented internally as bytes, through some means of encoding that text. And thus the concept of encodings are introduced. That (and how) text is represented should be an implementation-detail though: Strings represent text, not bytes. I think most people miss this distinction and that's the main source of confusion for encoding-related problems among programmers.

tyingq 9y ago
Depends on the language. Some co-opt the string as a generic buffer too, and have no other suitable type to use.
dgreensp 9y ago
Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, but that's irrelevant to how to think of the string type in JavaScript, or in an ideal world. One level of abstraction that works is to think of a string as a series of integers (which may be much larger than 65535) representing code points. Whether your VM uses a fixed-length or variable-length encoding to encode this series of integers should be irrelevant, aside from performance and memory consumption concerns. JavaScript's "charAt" and numeric indexing of strings violate this abstraction, making them not useful in Unicode-aware code. Of course, the "series of code points" abstraction has limits too, since what we think of as a single "character" can be composed of multiple code points, which calls into the question the whole concept of a "character," which is fuzzy in the first place. This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary. There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. This line of thinking does not make sense to me, and I'm not sure what scheme is being proposed. First of all, "most strings" are representable as UTF-8? All strings are representable as UTF-8. And if you are reading data from a UTF-8 source and you don't want to pay the time and memory cost of decoding it and re-encoding it in a less space-efficient code, leave it as a Buffer of bytes! Don't decode it. Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8. Node does make you specify which encoding you want when converting between strings and Buffers, but it's always embraced UTF-8 as its primary encoding, as far as I'm aware. Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16. This is kind of silly. Java made the same decision and was certainly not "invented" in ten man-days!

huhlig 9y ago

This just leads me to believe that Node and it's followers never learned the lessons from their stint with php. Also, "Encoding is the process of squashing the graphics you see on screen, say, 世 - into actual bytes." No it's a way of representing one value in a way a system can more easily handle in a hopefully lossless fashion. Encoding has nothing to do with what's on screen other than that being one representation of the data.

benjaminjackman 9y ago

I wonder if there is ever going to be an encoding that replaces UTF-8? Or have we hit on some sort of permanent local maxima (not a global one in the sense that UTF-8 carries the baggage of being backwards compatible with ASCII ... though maybe you could argue that's more of a unicode problem than a UTF8 encoding format one). At this point UTF-8 seems pretty permanent, what would come along to replace it? And if it is likely to be permanent shouldn't node / javascript in general be moving towards deprecating UCS-2 / UTF-16 and giving first class support to UTF-8? I saw all this because a couple of years ago I had to write a UTF-8 converter before ScalaJS natively supported for a serialization library I had written. I was kind of surprised that the javascript support was so lacking, luckily writing a UTF-8 encoder/decoder isn't that hard of an endeavor.

pwdisswordfish 9y ago
Functions like String.prototype.charCodeAt, String.prototype.indexOf, String.prototype.substr and such (which operate on indices into 16-bit-wide strings) will have to be supported somehow, so I doubt it.
- gumby 9y ago
  Only matters to the Windows platform, though. EDIT: early morning brain fart on my part, js of course too. Grr.
  - pwdisswordfish 9y ago
    What? These are cross-platform features, defined in the ECMAScript spec. Nearly all string-processing code written in JavaScript uses them.
- carussell 9y ago
  String.fromCodePoint and String#codePointAt were introduced for this reason. Or rather, the time to bring the UTF-8 Everywhere initiative to JS would have been when these two methods were introduced for ES6. Unfortunately, that didn't happen and the committee went with UTF-16 (which is not the same thing as UCS-2—the fact that they differ is why these methods were introduced in the first place).

gumby 9y ago

> You could easily represent all of the characters in the Unicode set with an encoding that says simply "assign one number, 4 bytes (or 32 bits) long, for each character in the Unicode set." Actually you can't, at least not in a standard way. There are combinations of code points that don't have a single code point equivalent. Flag emjois are an example, but there are many letterlike versions as well. There are enough unallocated points in the 32-bit space that probably you could manage to make single-point equivalents on your own.

whipoodle 9y ago
Hmm, I don't understand. AFAIK flag emojis are ligatures across two code points representing the two-letter international code for the country.
- mahkoh 9y ago
  gumby is deliberately interpreting the word character to mean grapheme cluster (whereas the author meant code point) to make a point.
  - mikeash 9y ago
    We really should avoid that word when discussing Unicode. It just leads to confusion.
johncolanduoni 9y ago
> There are enough unallocated points in the 32-bit space that probably you could manage to make single-point equivalents on your own. Unicode doesn't prevent you from drowning letters in many combining marks (e.g. an accent, an umlaut, and an overhead tilde), or doing this with letters where they make no sense and for which no font will have useful positioning information (e.g. over a copyright sign). There's over 1700 combining marks in Unicode, which gives you way over 2^32 ways to use the combining marks with the letter a, let alone the rest of the letters in Unicode.

cantstop99 9y ago

Lol what a fuckup of a language. So glad the entire internet depends on it.

fuckhn88 9y ago

Lol what a fuckup of a language. So glad the entire internet depends on it.

benhacker127001 9y ago

I am a mobile phone and software encryption specialist. Contact me if you need a phone hack or any other related services email: eightspyders@gmail.com

zengid 9y ago

> There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. I'm guessing you'd have to write a C++ module for this, but any suggestions on how one might do this successfully?

kevinburke 9y ago
You could use JavaScript, bit shifting and codePointAt, I think, but it would be a huge pain.
pwdisswordfish 9y ago
That's completely bogus. Forget unpaired surrogates; this scheme cannot even represent odd-length ASCII strings.

0x0 9y ago

I don't think this article is very good. It seems to make the "newbie unicode error" of assuming that strings "have" an encoding (or that strings "are" UTF-8) and thinking of bytes in string. For example in the JSON paragraph, it makes the cardinal sin of referring to "... creating UTF-8 strings". No such thing! Strings are an array of integer unicode code points. Stop thinking about bytes at this level. The internal representation of strings and chars does not matter, because you as a programmer ever only see integer code points. Encoding only enters the picture the moment you want to convert your string to or from a byte array (for example to write to disk or send over the network). The encoding, such as "UTF-8", then specifies how to map between the array of abstract code points to an array of 8bit bytes.

alangpierce 9y ago
Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a language like that, you should probably have a basic understanding of how UTF-16 works. I'd say a JavaScript programmer should think of things this way: * JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points. * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. * Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.
- 0x0 9y ago
  It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)

twotwotwo 9y ago

Commented this on the blog; cross-posting here: V8 turns out to have a ton of internal string representations. They don’t affect semantics (the general point that JS string functions think in UTF-16 is valid) but they’re interesting. V8 apparently stores all-ASCII strings as ASCII, so stuff like HTML tag names or base64 blobs doesn’t double in size. Like Go, V8 lets you take a substring as a pointer into the larger string; the internal class is called SlicedString, but from JS-land you don’t see anything different from a string literal. As in Go, keeping a short substring of a long parent string keeps the whole parent ‘alive’ across GCs so sometimes folks will be surprised all those bytes are still allocated. Unlike Go, V8 has a ConsString type, so concatenating strings sometimes doesn’t really immediately copy the underlying bytes anywhere. Building a string with a loop that runs str += newPiece probably goes faster than expected because of this. [It turns out it flattens the string the next time you index into it, or at least used to, which has some perf implications of its own: https://gist.github.com/mraleph/3397008 https://gist.github.com/mraleph/3397008] That’s mostly from this post by a member of the Dart team http://mrale.ph/blog/2016/11/23/making-less-dart-faster.html http://mrale.ph/blog/2016/11/23/making-less-dart-faster.html ; his blog has a lot of interesting stuff about how these fine-tuned language implementations really work. Kind of amazing the lengths V8 (and other JS engine) teams went to to make the code they saw in the wild work well.

evilpie 9y ago
Here is a comment describing strings in SpiderMonkey: http://searchfox.org/mozilla-central/rev/999385a5e8c2d360cc37286882508075fc2078bd/js/src/vm/String.h#44 http://searchfox.org/mozilla-central/rev/999385a5e8c2d360cc3.... If you scroll down there is ASCII art showing the different string sub-classes.
ubernostrum 9y ago
V8 apparently stores all-ASCII strings as ASCII, so stuff like HTML tag names or base64 blobs doesn’t double in size. Python does this too. In Python 2, and in Python 3 until 3.3, a compile-time flag determined the internal Unicode storage of the Python interpreter; a "narrow" build of the interpreter used 2-byte Unicode storage with surrogate pairs, while a "wide" build used 4-byte Unicode storage. As of Python 3.3, the internal storage of Unicode is dynamic. Python 3 source code is always parsed as UTF-8, but then as string objects are created by the interpreter their memory representation is chosen on a per-string basis, to be able to accommodate the widest code point present in the string. So Python will choose either a one-byte, two-byte, or four-byte encoding to store the string in memory, depending on what code points are present in it. This is very nice because it means iteration over a Python string is always iteration over its code points, the length of a string is always the number of code points in it, and indexing always yields the code point at index, since the internal storage of the string is fixed width and never has to include surrogates (in pre-3.3 Python, "narrow" builds would actually yield up things for which ord() gave a value in the surrogate range, and code points requiring surrogates added 2 to the length of a string rather than 1).