4 ms·

> A string is a series of bytes. Incorrect or wildly inaccurate. A string is conceptually text which may be (is) represented internally as bytes, through some

> A string is a series of bytes. Incorrect or wildly inaccurate. A string is conceptually text which may be (is) represented internally as bytes, through some means of encoding that text. And thus the concept of encodings are introduced. That (and how) text is represented should be an implementation-detail though: Strings represent text, not bytes. I think most people miss this distinction and that's the main source of confusion for encoding-related problems among programmers.

tyingq 9y ago
Depends on the language. Some co-opt the string as a generic buffer too, and have no other suitable type to use.
dgreensp 9y ago
Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, but that's irrelevant to how to think of the string type in JavaScript, or in an ideal world. One level of abstraction that works is to think of a string as a series of integers (which may be much larger than 65535) representing code points. Whether your VM uses a fixed-length or variable-length encoding to encode this series of integers should be irrelevant, aside from performance and memory consumption concerns. JavaScript's "charAt" and numeric indexing of strings violate this abstraction, making them not useful in Unicode-aware code. Of course, the "series of code points" abstraction has limits too, since what we think of as a single "character" can be composed of multiple code points, which calls into the question the whole concept of a "character," which is fuzzy in the first place. This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary. There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. This line of thinking does not make sense to me, and I'm not sure what scheme is being proposed. First of all, "most strings" are representable as UTF-8? All strings are representable as UTF-8. And if you are reading data from a UTF-8 source and you don't want to pay the time and memory cost of decoding it and re-encoding it in a less space-efficient code, leave it as a Buffer of bytes! Don't decode it. Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8. Node does make you specify which encoding you want when converting between strings and Buffers, but it's always embraced UTF-8 as its primary encoding, as far as I'm aware. Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16. This is kind of silly. Java made the same decision and was certainly not "invented" in ten man-days!
- wrs 9y ago
  Right, that was one of the obviously-correct representations at the time. When the pioneering major Unicode-based systems (later Mac OSes, Windows NT, Java, etc.) were being built, the Unicode consortium thought code points were just 16 bits, so two-byte characters were a fine solution, trading off some memory but retaining "normal" string characteristics like O(1) indexing and easy size calculation. Pretty much everybody went with that, except Plan 9, which nobody was paying attention to as they invented UTF-8, which is now saving our bacon--thanks! Then after all those runtimes and APIs were established, the Unicode consortium realized 65,000 characters wasn't actually enough and blew up all those assumptions. Since then, things have been very uncomfortable in those systems, string-wise. Next-generation systems are incorporating the semantics of modern Unicode, but it'll take a long while to fix this--you can't just redefine a primitive type like "string" in an existing giant codebase.
  - dgreensp 9y ago
    Indeed! I didn't know Plan 9 was behind UTF-8. :) We've come a long, long way since the 90s and even the 00s. It's shocking and gratifying how well emojis work online. A great motivator for Unicode support, too. :)
kevinburke 9y ago
Thanks for describing the post contents as "wildly inaccurate." I guess the problem with explaining anything is you have to decide which abstractions are good enough for the concept you are trying to get across
- josteink 9y ago
  Sure. But this is a pet peeves of mine: strings are text. Bytes (and thus encodings) something you should only be concerned about when doing file or network IO. It's boundary-stuff, and none of your actual text-processing should depend on it. Consider the phrasing my way of shielding you from accusations about being entirely wrong ;)
  - tentaTherapist 9y ago
    Or memory "IO", which standard libraries are often bad at abstracting away.
  - fauigerzigerk 9y ago
    >strings are text I do not completely agree with that. Text is what strings are mostly used for, but I wouldn't call a base64 encoded image text. Text is something a person can read and make sense of. So I think it would be more accurate to say that a string is a sequence of characters (grapheme clusters in unicode speak).
  - burntsushi 9y ago
    > It's boundary-stuff, and none of your actual text-processing should depend on it. I strongly disagree. For high performance text search, it's critical that you deal with its in-memory representation explicitly. This violates your maxim that such things are only done at the boundaries. For example, if you're implementing substring search, the techniques you use will heavily depend on how your string is represented in memory. Is it UTF-16? UTF-8? A sequence of codepoints? A sequence of grapheme clusters, where each cluster is a sequence of codepoints? Each of these choices will require different substring search strategies if you care about squeezing the most juice out of the underlying hardware.
twotwotwo 9y ago
Eh, a string 'is' its semantic definition and its in-memory representation and I'm sure other things in other contexts. Multiple notions of the same thing coexist outside of computers too. I might be a bunch of cells and physiological systems to someone in biology or medicine, a legal person to the state, something else to a philosopher. Something like a language specification would have good reason to stick to talking about the semantics, but that isn't a reason to discard any source that uses 'is' to talk about a different aspect. For example, Go's authors have written explanatory posts both about the semantics of strings and their in-memory representations, and it would be weird to say one of those posts is Just Wrong if it says 'A string is [whichever view of strings is most relevant locally]'. Sometimes another way of looking at the system is clearly better for the immediate purpose (writing about memory-use optimizations you would need to know what's in memory) and sometimes discussions need to jump around and look at the different levels at different points (like here, where the post both fills folks in on interactions between various APIs and performance, which is a(n important!) aspect of the implementation).
slavik81 9y ago
That feels like kicking the can down the road. What is text?

tyingq 9y ago

Depends on the language. Some co-opt the string as a generic buffer too, and have no other suitable type to use.

dgreensp 9y ago

Exactly; a string is not conceptually a "series of bytes" -- yes, there are languages that have a series-of-bytes type called "string" for historical reasons, but that's irrelevant to how to think of the string type in JavaScript, or in an ideal world. One level of abstraction that works is to think of a string as a series of integers (which may be much larger than 65535) representing code points. Whether your VM uses a fixed-length or variable-length encoding to encode this series of integers should be irrelevant, aside from performance and memory consumption concerns. JavaScript's "charAt" and numeric indexing of strings violate this abstraction, making them not useful in Unicode-aware code. Of course, the "series of code points" abstraction has limits too, since what we think of as a single "character" can be composed of multiple code points, which calls into the question the whole concept of a "character," which is fuzzy in the first place. This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary. There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary. This line of thinking does not make sense to me, and I'm not sure what scheme is being proposed. First of all, "most strings" are representable as UTF-8? All strings are representable as UTF-8. And if you are reading data from a UTF-8 source and you don't want to pay the time and memory cost of decoding it and re-encoding it in a less space-efficient code, leave it as a Buffer of bytes! Don't decode it. Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8. Node does make you specify which encoding you want when converting between strings and Buffers, but it's always embraced UTF-8 as its primary encoding, as far as I'm aware. Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16. This is kind of silly. Java made the same decision and was certainly not "invented" in ten man-days!

wrs 9y ago
Right, that was one of the obviously-correct representations at the time. When the pioneering major Unicode-based systems (later Mac OSes, Windows NT, Java, etc.) were being built, the Unicode consortium thought code points were just 16 bits, so two-byte characters were a fine solution, trading off some memory but retaining "normal" string characteristics like O(1) indexing and easy size calculation. Pretty much everybody went with that, except Plan 9, which nobody was paying attention to as they invented UTF-8, which is now saving our bacon--thanks! Then after all those runtimes and APIs were established, the Unicode consortium realized 65,000 characters wasn't actually enough and blew up all those assumptions. Since then, things have been very uncomfortable in those systems, string-wise. Next-generation systems are incorporating the semantics of modern Unicode, but it'll take a long while to fix this--you can't just redefine a primitive type like "string" in an existing giant codebase.
- dgreensp 9y ago
  Indeed! I didn't know Plan 9 was behind UTF-8. :) We've come a long, long way since the 90s and even the 00s. It's shocking and gratifying how well emojis work online. A great motivator for Unicode support, too. :)

kevinburke 9y ago

Thanks for describing the post contents as "wildly inaccurate." I guess the problem with explaining anything is you have to decide which abstractions are good enough for the concept you are trying to get across

josteink 9y ago
Sure. But this is a pet peeves of mine: strings are text. Bytes (and thus encodings) something you should only be concerned about when doing file or network IO. It's boundary-stuff, and none of your actual text-processing should depend on it. Consider the phrasing my way of shielding you from accusations about being entirely wrong ;)
- tentaTherapist 9y ago
  Or memory "IO", which standard libraries are often bad at abstracting away.
- fauigerzigerk 9y ago
  >strings are text I do not completely agree with that. Text is what strings are mostly used for, but I wouldn't call a base64 encoded image text. Text is something a person can read and make sense of. So I think it would be more accurate to say that a string is a sequence of characters (grapheme clusters in unicode speak).
- burntsushi 9y ago
  > It's boundary-stuff, and none of your actual text-processing should depend on it. I strongly disagree. For high performance text search, it's critical that you deal with its in-memory representation explicitly. This violates your maxim that such things are only done at the boundaries. For example, if you're implementing substring search, the techniques you use will heavily depend on how your string is represented in memory. Is it UTF-16? UTF-8? A sequence of codepoints? A sequence of grapheme clusters, where each cluster is a sequence of codepoints? Each of these choices will require different substring search strategies if you care about squeezing the most juice out of the underlying hardware.

twotwotwo 9y ago

Eh, a string 'is' its semantic definition and its in-memory representation and I'm sure other things in other contexts. Multiple notions of the same thing coexist outside of computers too. I might be a bunch of cells and physiological systems to someone in biology or medicine, a legal person to the state, something else to a philosopher. Something like a language specification would have good reason to stick to talking about the semantics, but that isn't a reason to discard any source that uses 'is' to talk about a different aspect. For example, Go's authors have written explanatory posts both about the semantics of strings and their in-memory representations, and it would be weird to say one of those posts is Just Wrong if it says 'A string is [whichever view of strings is most relevant locally]'. Sometimes another way of looking at the system is clearly better for the immediate purpose (writing about memory-use optimizations you would need to know what's in memory) and sometimes discussions need to jump around and look at the different levels at different points (like here, where the post both fills folks in on interactions between various APIs and performance, which is a(n important!) aspect of the implementation).

slavik81 9y ago

That feels like kicking the can down the road. What is text?