4 ms·

Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a

Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a language like that, you should probably have a basic understanding of how UTF-16 works. I'd say a JavaScript programmer should think of things this way: * JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points. * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. * Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.

0x0 9y ago
It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)
- burntsushi 9y ago
  > was fairly counter-productive with all its talk about "UTF-8 strings" Eh. I agree and disagree. I agree in the sense that the phrase "UTF-8 string" is generally a misnomer and is a good signal that there's some confusion somewhere, but I don't find think I find it as damning as you do. In particular, not all languages represent strings as sequences of codepoints, and instead make their internal UTF-8 byte representation a first class part of their string API. Two languages that come to mind are Go and Rust, where Go's conventionally uses UTF-8 and Rust uses enforced UTF-8. But in both cases, accessing the raw bytes is not only a standard operation, but is necessary whenever you want to do high performance string processing. That is, if someone said Go/Rust had "UTF-8 strings," that wouldn't be altogether wrong. UTF-8 is a first class aspect of both string APIs, while both provide standard functions you'd expect from Unicode strings.
chrismorgan 9y ago
An important subtlety: JavaScript doesn’t use UTF-16 for its strings; it uses UCS-2 code units. But internally, JavaScript engines can use whatever encoding they like, though most use UCS-2 most of the time. (Servo, for example, represents documents in WTF-8 rather than the conventional UCS-2. This is good for memory efficiency and certain other things, though bad for arbitrary indexing which is rare.) Python 2 is in the same boat concerning internal representation (it will use UTF-16 much of the time), but it doesn’t expose any UTF-16ness in the API.
dom0 9y ago
> * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. Don't do that (the array part). Guaranteeing O(1) indexing isn't very useful but precludes efficient storage.

0x0 9y ago

It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)

burntsushi 9y ago
> was fairly counter-productive with all its talk about "UTF-8 strings" Eh. I agree and disagree. I agree in the sense that the phrase "UTF-8 string" is generally a misnomer and is a good signal that there's some confusion somewhere, but I don't find think I find it as damning as you do. In particular, not all languages represent strings as sequences of codepoints, and instead make their internal UTF-8 byte representation a first class part of their string API. Two languages that come to mind are Go and Rust, where Go's conventionally uses UTF-8 and Rust uses enforced UTF-8. But in both cases, accessing the raw bytes is not only a standard operation, but is necessary whenever you want to do high performance string processing. That is, if someone said Go/Rust had "UTF-8 strings," that wouldn't be altogether wrong. UTF-8 is a first class aspect of both string APIs, while both provide standard functions you'd expect from Unicode strings.

chrismorgan 9y ago

An important subtlety: JavaScript doesn’t use UTF-16 for its strings; it uses UCS-2 code units. But internally, JavaScript engines can use whatever encoding they like, though most use UCS-2 most of the time. (Servo, for example, represents documents in WTF-8 rather than the conventional UCS-2. This is good for memory efficiency and certain other things, though bad for arbitrary indexing which is rare.) Python 2 is in the same boat concerning internal representation (it will use UTF-16 much of the time), but it doesn’t expose any UTF-16ness in the API.

dom0 9y ago

> * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. Don't do that (the array part). Guaranteeing O(1) indexing isn't very useful but precludes efficient storage.