3 ms·
Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, b
by dgreensp 9y ago
Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, but that's irrelevant to how to think of the string type in JavaScript, or in an ideal world.
One level of abstraction that works is to think of a string as a series of integers (which may be much larger than 65535) representing code points. Whether your VM uses a fixed-length or variable-length encoding to encode this series of integers should be irrelevant, aside from performance and memory consumption concerns. JavaScript's "charAt" and numeric indexing of strings violate this abstraction, making them not useful in Unicode-aware code. Of course, the "series of code points" abstraction has limits too, since what we think of as a single "character" can be composed of multiple code points, which calls into the question the whole concept of a "character," which is fuzzy in the first place.
This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary.
There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary.
This line of thinking does not make sense to me, and I'm not sure what scheme is being proposed. First of all, "most strings" are representable as UTF-8? All strings are representable as UTF-8. And if you are reading data from a UTF-8 source and you don't want to pay the time and memory cost of decoding it and re-encoding it in a less space-efficient code, leave it as a Buffer of bytes! Don't decode it.
Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8.
Node does make you specify which encoding you want when converting between strings and Buffers, but it's always embraced UTF-8 as its primary encoding, as far as I'm aware.
Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16.
This is kind of silly. Java made the same decision and was certainly not "invented" in ten man-days!
- wrs 9y agoRight, that was one of the obviously-correct representations at the time. When the pioneering major Unicode-based systems (later Mac OSes, Windows NT, Java, etc.) were being built, the Unicode consortium thought code points were just 16 bits, so two-byte characters were a fine solution, trading off some memory but retaining "normal" string characteristics like O(1) indexing and easy size calculation. Pretty much everybody went with that, except Plan 9, which nobody was paying attention to as they invented UTF-8, which is now saving our bacon--thanks! Then after all those runtimes and APIs were established, the Unicode consortium realized 65,000 characters wasn't actually enough and blew up all those assumptions. Since then, things have been very uncomfortable in those systems, string-wise. Next-generation systems are incorporating the semantics of modern Unicode, but it'll take a long while to fix this--you can't just redefine a primitive type like "string" in an existing giant codebase.
- dgreensp 9y agoIndeed! I didn't know Plan 9 was behind UTF-8. :) We've come a long, long way since the 90s and even the 00s. It's shocking and gratifying how well emojis work online. A great motivator for Unicode support, too. :)