4 ms·

I don't think this article is very good. It seems to make the "newbie unicode error" of assuming that strings "have" an encoding (or that strings "are" UTF-8) a

by 0x0 9y ago

I don't think this article is very good. It seems to make the "newbie unicode error" of assuming that strings "have" an encoding (or that strings "are" UTF-8) and thinking of bytes in string. For example in the JSON paragraph, it makes the cardinal sin of referring to "... creating UTF-8 strings". No such thing! Strings are an array of integer unicode code points. Stop thinking about bytes at this level. The internal representation of strings and chars does not matter, because you as a programmer ever only see integer code points. Encoding only enters the picture the moment you want to convert your string to or from a byte array (for example to write to disk or send over the network). The encoding, such as "UTF-8", then specifies how to map between the array of abstract code points to an array of 8bit bytes.

alangpierce 9y ago
Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a language like that, you should probably have a basic understanding of how UTF-16 works. I'd say a JavaScript programmer should think of things this way: * JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points. * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. * Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.
- 0x0 9y ago
  It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)
  - burntsushi 9y ago
    > was fairly counter-productive with all its talk about "UTF-8 strings" Eh. I agree and disagree. I agree in the sense that the phrase "UTF-8 string" is generally a misnomer and is a good signal that there's some confusion somewhere, but I don't find think I find it as damning as you do. In particular, not all languages represent strings as sequences of codepoints, and instead make their internal UTF-8 byte representation a first class part of their string API. Two languages that come to mind are Go and Rust, where Go's conventionally uses UTF-8 and Rust uses enforced UTF-8. But in both cases, accessing the raw bytes is not only a standard operation, but is necessary whenever you want to do high performance string processing. That is, if someone said Go/Rust had "UTF-8 strings," that wouldn't be altogether wrong. UTF-8 is a first class aspect of both string APIs, while both provide standard functions you'd expect from Unicode strings.
- chrismorgan 9y ago
  An important subtlety: JavaScript doesn’t use UTF-16 for its strings; it uses UCS-2 code units. But internally, JavaScript engines can use whatever encoding they like, though most use UCS-2 most of the time. (Servo, for example, represents documents in WTF-8 rather than the conventional UCS-2. This is good for memory efficiency and certain other things, though bad for arbitrary indexing which is rare.) Python 2 is in the same boat concerning internal representation (it will use UTF-16 much of the time), but it doesn’t expose any UTF-16ness in the API.
- dom0 9y ago
  > * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. Don't do that (the array part). Guaranteeing O(1) indexing isn't very useful but precludes efficient storage.
sw00pur 9y ago
>No such thing! Strings are an array of integer unicode code points. I'd argue that, generally, strings are simply arrays of chars, which are bytes. THe failure here, was keeping the name "string" for what are arrays of codepoints instead of bytes.
- wolf550e 9y ago
  C does not own the word "string". A string is a piece if text. It is not a byte array. Unicode strings are arrays of code points which are 21bit numbers. If the API requires fast subscript (it usually does) then they would be UTF-32 or three-codepoints-in-int64, otherwise more compact internal representation is possible. If you don't require supporting subscript and allow only iteration over list of code points then in-memory representation of strings can be more compact. It can use UTF-8 or even SCSU or BOCU1. Some languages use polymorphic unicode strings which store ascii if the value is all-ascii and switch to something else if it isn't (python3.3 and factor come to mind).
- jmull 9y ago
  I'm going to argue a little differently... In C, strings were always semantically a sequence of characters (as they are commonly defined elsewhere). For a while a character was one byte, so the distinction was unimportant (and became blurred). A char was both a character and a byte. A string was both a sequence of characters and an array of characters... and an array of "char"s, and an array of bytes. Code -- and programmers! -- became dependent on these equivalency assumptions. Once it became clear we could no longer pretend that a maximum of 256 characters was tenable (less actually, since the use of 0-31 for control/separation/termination had become standard) we were left with conflict, leading to a variety of uncomfortable choices and compromises. One such conflict is "char"... should it retain its semantics or its size (one byte)? The last time I developed seriously in C or C++ it had retained its size, but lost its semantics -- a char IS a byte now. (That was a while ago, I don't know if that's changed -- it sounds like from your post that it hasn't.) I guess UTF-8 has won out in C and C++ (and elsewhere) so now, while a char is byte, a C/C++ string is: (1) an array of char/bytes; (2) a sequence of characters. The thing that's been dropped is that a string is no longer an array of characters. (In case there's confusion: here, "array" means an ordered sequence of elements of uniform size with O(1) random access, while sequence is just an ordered sequence of elements that doesn't necessarily offer O(1) random access or elements with uniform size.)
deleted 9y ago
[deleted]
paulddraper 9y ago
> It seems to make the "newbie unicode error" of assuming that strings "have" an encoding You imply that there is some universal "string" definition, when there is no such thing. Literally, "string" just means sequence. Beyond that, it depends on the context. * C++ std::string is a sequence of char (aka byte, defined to be 8+ bits). * Swift string is a sequence of Unicode graphemes. * Ruby String is a sequence of bytes/octects, with an attached (and mutable) encoding. * Python 2 str is a sequence of bytes/octets * Python 3 str is a sequence of Unicode code points * ECMAScript string (and Java java.lang.String) is a sequence of UTF-16 code units. Moreover, the entire idea of "Strings are an array of integer unicode code points" is confined to Unicode. It doesn't say anything about other character sets, e.g. ASCII, ISO 8859-1, or Windows-1252. (Though AFAIK Unicode is a superset of those particular three.) So...I think you can safely say "Unicode strings are sequences of Unicode code points."

by 0x0 9y ago

alangpierce 9y ago

Unfortunately, JavaScript (and Java and Python 2 and other languages) uses UTF-16 for its strings, and leaks that information to the programmer, so if you use a language like that, you should probably have a basic understanding of how UTF-16 works. I'd say a JavaScript programmer should think of things this way: * JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points. * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. * Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.

0x0 9y ago
It is true that Javascript, like many other languages (Java, Win32 wide chars, etc) has to deal with the problem that they assumed unicode code points could not exceed the integer value 65535, so you have to deal with the surrogate pairs. So I guess that is one way to "encode" all planes of unicode in a backward compatible way. Bytes don't enter the picture though! It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)
- burntsushi 9y ago
  > was fairly counter-productive with all its talk about "UTF-8 strings" Eh. I agree and disagree. I agree in the sense that the phrase "UTF-8 string" is generally a misnomer and is a good signal that there's some confusion somewhere, but I don't find think I find it as damning as you do. In particular, not all languages represent strings as sequences of codepoints, and instead make their internal UTF-8 byte representation a first class part of their string API. Two languages that come to mind are Go and Rust, where Go's conventionally uses UTF-8 and Rust uses enforced UTF-8. But in both cases, accessing the raw bytes is not only a standard operation, but is necessary whenever you want to do high performance string processing. That is, if someone said Go/Rust had "UTF-8 strings," that wouldn't be altogether wrong. UTF-8 is a first class aspect of both string APIs, while both provide standard functions you'd expect from Unicode strings.
chrismorgan 9y ago
An important subtlety: JavaScript doesn’t use UTF-16 for its strings; it uses UCS-2 code units. But internally, JavaScript engines can use whatever encoding they like, though most use UCS-2 most of the time. (Servo, for example, represents documents in WTF-8 rather than the conventional UCS-2. This is good for memory efficiency and certain other things, though bad for arbitrary indexing which is rare.) Python 2 is in the same boat concerning internal representation (it will use UTF-16 much of the time), but it doesn’t expose any UTF-16ness in the API.
dom0 9y ago
> * Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding. Don't do that (the array part). Guaranteeing O(1) indexing isn't very useful but precludes efficient storage.

sw00pur 9y ago

>No such thing! Strings are an array of integer unicode code points. I'd argue that, generally, strings are simply arrays of chars, which are bytes. THe failure here, was keeping the name "string" for what are arrays of codepoints instead of bytes.

wolf550e 9y ago
C does not own the word "string". A string is a piece if text. It is not a byte array. Unicode strings are arrays of code points which are 21bit numbers. If the API requires fast subscript (it usually does) then they would be UTF-32 or three-codepoints-in-int64, otherwise more compact internal representation is possible. If you don't require supporting subscript and allow only iteration over list of code points then in-memory representation of strings can be more compact. It can use UTF-8 or even SCSU or BOCU1. Some languages use polymorphic unicode strings which store ascii if the value is all-ascii and switch to something else if it isn't (python3.3 and factor come to mind).
jmull 9y ago
I'm going to argue a little differently... In C, strings were always semantically a sequence of characters (as they are commonly defined elsewhere). For a while a character was one byte, so the distinction was unimportant (and became blurred). A char was both a character and a byte. A string was both a sequence of characters and an array of characters... and an array of "char"s, and an array of bytes. Code -- and programmers! -- became dependent on these equivalency assumptions. Once it became clear we could no longer pretend that a maximum of 256 characters was tenable (less actually, since the use of 0-31 for control/separation/termination had become standard) we were left with conflict, leading to a variety of uncomfortable choices and compromises. One such conflict is "char"... should it retain its semantics or its size (one byte)? The last time I developed seriously in C or C++ it had retained its size, but lost its semantics -- a char IS a byte now. (That was a while ago, I don't know if that's changed -- it sounds like from your post that it hasn't.) I guess UTF-8 has won out in C and C++ (and elsewhere) so now, while a char is byte, a C/C++ string is: (1) an array of char/bytes; (2) a sequence of characters. The thing that's been dropped is that a string is no longer an array of characters. (In case there's confusion: here, "array" means an ordered sequence of elements of uniform size with O(1) random access, while sequence is just an ordered sequence of elements that doesn't necessarily offer O(1) random access or elements with uniform size.)

deleted 9y ago

[deleted]

paulddraper 9y ago

> It seems to make the "newbie unicode error" of assuming that strings "have" an encoding You imply that there is some universal "string" definition, when there is no such thing. Literally, "string" just means sequence. Beyond that, it depends on the context. * C++ std::string is a sequence of char (aka byte, defined to be 8+ bits). * Swift string is a sequence of Unicode graphemes. * Ruby String is a sequence of bytes/octects, with an attached (and mutable) encoding. * Python 2 str is a sequence of bytes/octets * Python 3 str is a sequence of Unicode code points * ECMAScript string (and Java java.lang.String) is a sequence of UTF-16 code units. Moreover, the entire idea of "Strings are an array of integer unicode code points" is confined to Unicode. It doesn't say anything about other character sets, e.g. ASCII, ISO 8859-1, or Windows-1252. (Though AFAIK Unicode is a superset of those particular three.) So...I think you can safely say "Unicode strings are sequences of Unicode code points."