7 ms·

Looking at the .NET parts of the manifesto, I just have to roll my eyes: Both C# and Java offer a 16 bit char type, which is less than a Unicode character, con

Looking at the .NET parts of the manifesto, I just have to roll my eyes: Both C# and Java offer a 16 bit char type, which is less than a Unicode character, congratulations. The .NET indexer str[i] works in units of the internal representation, hence a leaky abstraction once again. Substring methods will happily return an invalid string, cutting a non-BMP character in parts. While theoretically true, for most practical purposes, this reeks of a USA/American/English bias and lack of real world experience. You know what? I want to know that the text "ØÆÅ" is three characters long. I dont want to know that it's a 6-byte array once encoded to UTF8. Anywhere in my code telling me this is 6 characters is lying, not to mention a violation of numerous business-requirements. When I work with text I want to work with text and never the byte-stream it will eventually be encoded to. I want to work on top of an abstraction which lets me treat text as text. Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. And in all cases where it doesn't, it offers me numerous advantages over the PHPesque, amateurish and incorrect approach of treating everything as a dumb byte-array. It's not. It's text in my program. It's text rendered on your screen. It's just a byte-array when we send it over the wire, so stop trying to pretend text isn't text. This manifesto is wildly misguided.

derefr 13y ago
I fundamentally agree, but I don't think even you understand the implications of what you're saying. Though you say "characters", You're probably still thinking about Unicode codepoints here, not characters. A real character-oriented API would require an arbitrary number of bytes to store each character (a character being an arbitrary vector of codepoints), and would require a font to be associated to each region of codepoints--because it's the font's decision of how to render a sequence of codepoints, so it's the font that determines how many characters you get. (For an edge-case example: http://symbolset.com/ http://symbolset.com/) ...and saying that, it'd still be a good idea.
- chipsy 13y ago
  When I learned of the difficulty mapping between code points and characters, I realized that Unicode is a standard nobody will ever (knowingly) implement correctly. Even if everyone has access to a font API, there'll probably be bugs in the fonts for all eternity.
  - derefr 13y ago
    (I half-considered adding this to my comment above, but it didn't quite fit.) If we were serious about a character-oriented API, we definitely wouldn't want to introduce character rendering rules into places like the kernel. But I don't think we'd necessarily have to. The best solution, I think, would be to decompose fonts into two pieces: 1. a character map (a mapping from paramaterized codepoint sequences to single entities known to the font), and 2. a graphemes file (the way to actually draw each character.) The graphemes file would be what people would continue to think of as "the font." And the graphemes file would specify the character map it uses, in much the same way an XML/SGML document specifies a DTD. As with DTDs, the text library built into the OS would have a well-known core set of character maps built in, and allow others to be retrieved and cached when referenced by URL. The core set would become treated something like root CAs or timezone data are now: bundled artifacts that get updated pretty frequently by the package manager.
- TheLoneWolfling 13y ago
  Yes. I've got a toy language that I've been working on that uses a rope to store characters internally. Everything works on "logical characters" - arbitrary vectors of codepoints, as you say. There's still a number of edge cases I have yet to work out as to what exactly is considered a character, though. (I just added support for a single code point encoding multiple characters, for example.) I'm not so sure that making thing reliant on a font would be the best way to solve that, though. I'd intuitively say that there should be less coupling between rendering choices and internal encoding than that. The twist is that each node in the rope can only store characters of the same physical length in bytes (and same number of logical characters per physical character). This means that in the typical case (most characters require the same number of bytes to encode) it doesn't add too too much overhead. Still not something I would consider as the base String type for a lower-level language, though. There are a few simple optimizations that I have yet to do (encode smaller characters as what would be ordinarily be invalid longer encodings, if it makes sense (a single one-byte character in the middle of a bunch of two-byte characters, for example), that sort of thing.) It seems to work fairly well, so far. Or at least it tends to give "common-sensical" results, and avoids a large chunk of worst-case behavior that standard "prettified character array" strings have.
bjourne 13y ago
Well, he doesn't say that one should use byte count instead of code point count as string length. That's a strawman and not in the manifesto. What he is saying though is that one should prefer to count grapheme clusters instead of characters since the definition of the latter is incredibly vague.
drbawb 13y ago
They're not arguing that the length [of the bytestream] is what the user cares most about. I thought they covered this quite well in the "Myths" section on "counting coded characters or codepoints is important."[0] From that section they explain that for text manipulation [e.g: cursor position & manipulation of text under the cursor] the programmer should be counting grapheme clusters; whereas for storage [memory & disk] concerns the programmer _should_ care about the number of codepoints. They go on to say that counting the number of characters is up to the rendering engine and is completely unrelated to the number of codepoints. I don't think the _manifesto itself_ is wildly misguided. In that section they were merely pointing out _how .NET and Java currently report string length._ At least as I read the manifesto: they seem to believe that counting codepoints is useful _but orthogonal_ to counting characters. [0]: http://www.utf8everywhere.org/#myth.strlen http://www.utf8everywhere.org/#myth.strlen
golergka 13y ago
I don't understand how what you said contradicts the points in the manifesto. C# strings _try_ to look like they show you the characters and not bytes, but fail in that regard.
mikeash 13y ago
In what scenario, exactly, do you want to know that the text "ØÆÅ" is three characters long? That's almost never a useful question to ask or a useful answer to have.
- derefr 13y ago
  Lexicographic ordering and sorting; edit distance; spelling correction (really, anything involving prefixes/tries)...
  - haberman 13y ago
    According to the article, UTF-8 sorts lexicographically the same as UTF-32, implying that there is no need to know the length or character boundaries for ordering/sorting. The others would be good reasons though.
  - mikeash 13y ago
    All of those need pretty deep knowledge of Unicode in general and the language you're working with in particular. Counting "characters" (really, code points) is going to be the least of your worries.
- szabba 13y ago
  When writing wc?
marssaxman 13y ago
Quite the contrary: if you are treating text as anything but a dumb byte-array, you are probably doing it wrong. Unicode elements are not characters, but code points. Your example text "ØÆÅ" could be legitimately represented as either three or four code points, depending on whether the Å character is precomposed (U+00C5) or decomposed (U+0041 U+030A).
ender7 13y ago
Be careful when flinging bias accusations around; the author is not actually from the US/UK [0]. In addition, while .NET methods will work correctly for "ØÆÅ", they have incorrect behavior when it comes to any characters that lie outside the BMP, which includes many CJK characters. So, .NET only meets your requirements if you never interact with such languages. [0] http://www.utf8everywhere.org/#faq.anglophile http://www.utf8everywhere.org/#faq.anglophile
haberman 13y ago
I don't think you understood the statement you are quoting. The manifesto is not in any way advocating that you use byte length as a substitute for string length. (It does argue that string length is not very commonly necessary, and has unclear semantics because of the multiple different definitions for "character", but these arguments are unrelated to the statement you quoted.) Here's the point the author was making, which you missed. Take a non-BMP code point like '𝄞', which is U+1D11E. In UTF-16 this is represented by code units D834 and DD1E. If you try to use a "substring" operation to take the first "character" of this with a C#/.NET substring operation, you will get an invalid string, since D834 by itself is not a valid UTF-16 string. > I want to work on top of an abstraction which lets me treat text as text. If you think that UTF-16 will let you say string[i] and always get the i'th character, you are mistaken. That is one of the main points of the essay. > Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. If you write your apps this way, then you don't really support Unicode, you just support the BMP, without combining characters.
- mark-r 13y ago
  One of the advantages of working in UTF-8 is that if your code is broken, you find out about it right away - as soon as someone enters an accented character. If your UTF-16 code is broken you might not find it in testing.
enneff 13y ago
How many characters is "é́́"? To our eyes, one. But it's actually four code points (the e and three acute modifiers). This is the case whether you're using UTF-8 or UTF-16. Regardless of the encoding you choose, you need to understand how it represents characters. Also FYI, most programs don't display text on the screen. To most programs, text is a sequence of bytes to be shuffled around, and nothing more. Display and text manipulation is the minority case.

derefr 13y ago

I fundamentally agree, but I don't think even you understand the implications of what you're saying. Though you say "characters", You're probably still thinking about Unicode codepoints here, not characters. A real character-oriented API would require an arbitrary number of bytes to store each character (a character being an arbitrary vector of codepoints), and would require a font to be associated to each region of codepoints--because it's the font's decision of how to render a sequence of codepoints, so it's the font that determines how many characters you get. (For an edge-case example: http://symbolset.com/ http://symbolset.com/) ...and saying that, it'd still be a good idea.

chipsy 13y ago
When I learned of the difficulty mapping between code points and characters, I realized that Unicode is a standard nobody will ever (knowingly) implement correctly. Even if everyone has access to a font API, there'll probably be bugs in the fonts for all eternity.
- derefr 13y ago
  (I half-considered adding this to my comment above, but it didn't quite fit.) If we were serious about a character-oriented API, we definitely wouldn't want to introduce character rendering rules into places like the kernel. But I don't think we'd necessarily have to. The best solution, I think, would be to decompose fonts into two pieces: 1. a character map (a mapping from paramaterized codepoint sequences to single entities known to the font), and 2. a graphemes file (the way to actually draw each character.) The graphemes file would be what people would continue to think of as "the font." And the graphemes file would specify the character map it uses, in much the same way an XML/SGML document specifies a DTD. As with DTDs, the text library built into the OS would have a well-known core set of character maps built in, and allow others to be retrieved and cached when referenced by URL. The core set would become treated something like root CAs or timezone data are now: bundled artifacts that get updated pretty frequently by the package manager.
TheLoneWolfling 13y ago
Yes. I've got a toy language that I've been working on that uses a rope to store characters internally. Everything works on "logical characters" - arbitrary vectors of codepoints, as you say. There's still a number of edge cases I have yet to work out as to what exactly is considered a character, though. (I just added support for a single code point encoding multiple characters, for example.) I'm not so sure that making thing reliant on a font would be the best way to solve that, though. I'd intuitively say that there should be less coupling between rendering choices and internal encoding than that. The twist is that each node in the rope can only store characters of the same physical length in bytes (and same number of logical characters per physical character). This means that in the typical case (most characters require the same number of bytes to encode) it doesn't add too too much overhead. Still not something I would consider as the base String type for a lower-level language, though. There are a few simple optimizations that I have yet to do (encode smaller characters as what would be ordinarily be invalid longer encodings, if it makes sense (a single one-byte character in the middle of a bunch of two-byte characters, for example), that sort of thing.) It seems to work fairly well, so far. Or at least it tends to give "common-sensical" results, and avoids a large chunk of worst-case behavior that standard "prettified character array" strings have.

bjourne 13y ago

Well, he doesn't say that one should use byte count instead of code point count as string length. That's a strawman and not in the manifesto. What he is saying though is that one should prefer to count grapheme clusters instead of characters since the definition of the latter is incredibly vague.

drbawb 13y ago

They're not arguing that the length [of the bytestream] is what the user cares most about. I thought they covered this quite well in the "Myths" section on "counting coded characters or codepoints is important."[0] From that section they explain that for text manipulation [e.g: cursor position & manipulation of text under the cursor] the programmer should be counting grapheme clusters; whereas for storage [memory & disk] concerns the programmer _should_ care about the number of codepoints. They go on to say that counting the number of characters is up to the rendering engine and is completely unrelated to the number of codepoints. I don't think the _manifesto itself_ is wildly misguided. In that section they were merely pointing out _how .NET and Java currently report string length._ At least as I read the manifesto: they seem to believe that counting codepoints is useful _but orthogonal_ to counting characters. [0]: http://www.utf8everywhere.org/#myth.strlen http://www.utf8everywhere.org/#myth.strlen

golergka 13y ago

I don't understand how what you said contradicts the points in the manifesto. C# strings _try_ to look like they show you the characters and not bytes, but fail in that regard.

mikeash 13y ago

In what scenario, exactly, do you want to know that the text "ØÆÅ" is three characters long? That's almost never a useful question to ask or a useful answer to have.

derefr 13y ago
Lexicographic ordering and sorting; edit distance; spelling correction (really, anything involving prefixes/tries)...
- haberman 13y ago
  According to the article, UTF-8 sorts lexicographically the same as UTF-32, implying that there is no need to know the length or character boundaries for ordering/sorting. The others would be good reasons though.
- mikeash 13y ago
  All of those need pretty deep knowledge of Unicode in general and the language you're working with in particular. Counting "characters" (really, code points) is going to be the least of your worries.
szabba 13y ago
When writing wc?

marssaxman 13y ago

Quite the contrary: if you are treating text as anything but a dumb byte-array, you are probably doing it wrong. Unicode elements are not characters, but code points. Your example text "ØÆÅ" could be legitimately represented as either three or four code points, depending on whether the Å character is precomposed (U+00C5) or decomposed (U+0041 U+030A).

ender7 13y ago

Be careful when flinging bias accusations around; the author is not actually from the US/UK [0]. In addition, while .NET methods will work correctly for "ØÆÅ", they have incorrect behavior when it comes to any characters that lie outside the BMP, which includes many CJK characters. So, .NET only meets your requirements if you never interact with such languages. [0] http://www.utf8everywhere.org/#faq.anglophile http://www.utf8everywhere.org/#faq.anglophile

haberman 13y ago

I don't think you understood the statement you are quoting. The manifesto is not in any way advocating that you use byte length as a substitute for string length. (It does argue that string length is not very commonly necessary, and has unclear semantics because of the multiple different definitions for "character", but these arguments are unrelated to the statement you quoted.) Here's the point the author was making, which you missed. Take a non-BMP code point like '𝄞', which is U+1D11E. In UTF-16 this is represented by code units D834 and DD1E. If you try to use a "substring" operation to take the first "character" of this with a C#/.NET substring operation, you will get an invalid string, since D834 by itself is not a valid UTF-16 string. > I want to work on top of an abstraction which lets me treat text as text. If you think that UTF-16 will let you say string[i] and always get the i'th character, you are mistaken. That is one of the main points of the essay. > Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. If you write your apps this way, then you don't really support Unicode, you just support the BMP, without combining characters.

mark-r 13y ago
One of the advantages of working in UTF-8 is that if your code is broken, you find out about it right away - as soon as someone enters an accented character. If your UTF-16 code is broken you might not find it in testing.

enneff 13y ago

How many characters is "é́́"? To our eyes, one. But it's actually four code points (the e and three acute modifiers). This is the case whether you're using UTF-8 or UTF-16. Regardless of the encoding you choose, you need to understand how it represents characters. Also FYI, most programs don't display text on the screen. To most programs, text is a sequence of bytes to be shuffled around, and nothing more. Display and text manipulation is the minority case.