4 ms·

I think it's even better to take this one step further and have your default "character" actually be a grapheme[1]. In almost any case where you're dealing with

I think it's even better to take this one step further and have your default "character" actually be a grapheme[1]. In almost any case where you're dealing with individual character boundaries you want to split things on the grapheme level, not the code-point level. This doesn't matter much for (normalized) western European text, but if the language in question needs to use separate diacritical code points you'll likely end up with hanging accents in the like. Swift is the only language I know of that has grapheme clusters as the default unit of character, I'd love to see it in more places. [1]: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries

lelf 10y ago
Perl6 does support them natively too.
- SwellJoe 10y ago
  Perl has historically had excellent Unicode support. I remember going from Perl to Python (like a decade ago), and being annoyed at how messy Unicode support still was. Ruby, too, lacked good Unicode support, for many years after Perl had it pretty good. But, Perl 6 definitely gets it more right than other implementations I've seen.
MrBuddyCasino 10y ago
Never understood that either. Why is this so rare? Even in technical discussions like this one, some people will look at you funny upon hearing this suggestion.
- jcranmer 10y ago
  Navigating a UTF-8 string on codepoint level is a fairly simple algorithm, since UTF-8 is self-synchronizing. This means it can easily be done without relying on external libraries or data files. It's also stable with respect to Unicode version--it always produces the same result independent of what version of the Unicode tables you use. Moving to grapheme cluster boundaries means that the algorithm may work incorrectly if you input a string of Unicode N+1 to an implementation that only supports Unicode N. It also makes the "increment character" function very complicated. In the UTF-8 version, this looks roughly like: char *advance(char *str) { uint8_t c = (uint8_t)*str; /* Count the number of leading 1's */ int num1s = __builtin_clz(~c) - 24; if (num1s == 0) return str + 1; return str + num1s; } Grapheme-based indexing looks like this: char *advance_grapheme(char *str) { while (true) { uint32_t codepoint = read_codepoint(str); str = advance(str); uint32_t nextCodepoint = read_codepoint(str); /* This is typically something like table[table2[codepoint >> 4] * 16 + codepoint & 15]; */ GraphemeClusterBreak left = lookupProp(codepoint); GraphemeClusterBreak right = lookupProp(nextCodepoint); /* Several rules based on left versus right... */ } return str; } See the vast difference in the two implementations? It's a lot of complexity, and it's worth asking if that complexity needs to be built into the main library (strings are a fundamental datatype in any language). It's also important to note that it's questionable whether such a feature implemented by default is going to actually fix naive programmers' code--if you read UTR #29 carefully, you'll notice that something like क्ष will consist of two grapheme clusters (क् and ष), which is arguably incorrect. Internationalization is often tied heavily to GUI and, especially for problems like grapheme clusters, it arguably makes more sense for toolkits to implement and deal with the problems themselves and provide things like "text input widget" primitives to programmers rather than encouraging users to try to implement it themselves.
  - mtviewdave 10y ago
    History has shown that, when it comes to strings, developers have a hard time getting even something as simple as null-termination correct. If grapheme handling is complex, that's an argument for having it implemented by a small team of experts exactly once. The resulting abstraction might not be leak-proof, but then no abstraction is.
  - MrBuddyCasino 10y ago
    Thanks for the detailed response. This looks like a performance problem, big lookup tables trash the cashes. I with mtviewdave that more complexity means this should really be in the std lib. Java's charAt(int i) is misleading at best.
  - lomnakkus 10y ago
    > (strings are a fundamental datatype in any language) (Probably a bit "unfair" to "pounce" on an off-hand paranthetical like this, but I'm in a bit of a pedantic mood...) This is not true for e.g. Haskell. In Haskell it's defined as [Char], i.e. a list of characters. (Of course the Haskell community is suffering from that decision, but that's another story.) I'm not sure why strings would need to be a fundamendal type, though. Sure, they would probably be part of the standard library for almost all languages, but they don't need to be "magical" in the way most fundamental types (int, etc.) are.
asveikau 10y ago
1 grapheme = 1 unit of datatype is a good start. There is a lot of complexity in Unicode, but the mental model of a "char" is what most developers are thinking, largely blind to that complexity. Kudos to anyone trying to make that more "common sense" model closer to reality. However... How does a language or library attempting to abstract this part (like swift might) deal with the other, unrelated annoying aspects of Unicode? Even if 1 glyph is 1 "char", and we normalize all the inputs, there is still, say ... Bi-di text.
- johncolanduoni 10y ago
  I don't think there is an easy answer to this. Even just pure RTL text is hard to support: IIRC Android and iOS only recently made all the built-in views support it. This is complicated by the fact that most developer teams don't have nor can afford to hire someone familiar with RTL to implement this kind of stuff.
  - asveikau 10y ago
    I don't think the rules in Unicode are simple for anyone. I have seen professional developers who are native speakers of RTL languages screw it up. All this stuff about the directionality of a paragraph and the embedded markers... It boggles the mind. But a truly international product should be getting it right. Sad mismatch there. PS: bit of trivia that people forget these days, Win32 has been supporting it for longer than android/iOS.
Joeri 10y ago
Since all other systems have standardized on code points this would lead to subtle incompatibilities. For example, checking for length prior to inserting in a database must be done in code points. What i find more frustrating is how the documentation for many systems describes the basic unit of text as a character, without specifying whether a code point or grapheme is meant, and without leading people to an explanation of the difference. There is still a lot of software that processes unicode text incorrectly, not because it is difficult to do so, but because nobody told the developer how things should be done.
int_19h 10y ago
I don't think there should even be a "default" character at all. In some cases, codepoints are the right choice; in others, graphemes are. If we make this explicit - e.g. by forbidding to iterate strings directly, but providing functions/methods to extract codepoints and graphemes as iterable sequences - the programmer has to make a conscious choice every time they iterate.
- johncolanduoni 10y ago
  In principle I'd agree with you, but the response from most people learning a language that treats strings like that is likely to be "You can't iterate over strings? And it doesn't even have a character type!" I'm not saying people are stupid: I had never really thought about the difference between a code point and grapheme until relatively recently when I had to do some low-level text layout stuff and it became important. My point is that I don't think throwing people into the deep end and expecting them to grok the codepoint/grapheme division before they get to use the language is likely to be productive. Defaulting to graphemes carries the advantage that if a programmer who doesn't know much about languages that require more Unicode finesse uses it purely intuitively, they'll get things right in a lot of cases. Using codepoints, on the other hand, makes it easy to put text through a grinder while doing relatively innocent things. Unfortunately while we're discussing the finer points of using graphemes and codepoints in APIs a million supplementary characters were brutally eviscerated by code running on languages that haven't quite gotten past UCS-2 XD
  - int_19h 10y ago
    I think that throwing people into the deep end is the only way to get them to do it right in this case. Defaulting to graphemes is too often the wrong answer, as well (e.g. you don't want that in a parser). Really, this is not dissimilar to "what do you mean, there are more letters than A to Z?" issue that plagued software written in US back before Unicode became dominant. The way we (my perspective on this is as a native speaker of a language with a non-Latin alphabet) have eventually solved it is by basically forcing Unicode onto those people. It broke their simple and convenient picture of the world, and replaced it with something much more complicated. But it was necessary. My position is that letting programmers get away with a simplistic view of text processing (by allowing defaults that "mostly" work) is what creates those issues. So adjusting the abstractions such that they expose more of the underlying complexity is a good thing. People SHOULD believe that doing text processing the right way is hard, because it is.
raiph 10y ago
> Swift is the only language I know of that has grapheme clusters as the default unit of character Also Elixir and Perl 6. For a bit more info see https://news.ycombinator.com/item?id= https://news.ycombinator.com/item?id=

lelf 10y ago

Perl6 does support them natively too.

SwellJoe 10y ago
Perl has historically had excellent Unicode support. I remember going from Perl to Python (like a decade ago), and being annoyed at how messy Unicode support still was. Ruby, too, lacked good Unicode support, for many years after Perl had it pretty good. But, Perl 6 definitely gets it more right than other implementations I've seen.

MrBuddyCasino 10y ago

Never understood that either. Why is this so rare? Even in technical discussions like this one, some people will look at you funny upon hearing this suggestion.

jcranmer 10y ago
Navigating a UTF-8 string on codepoint level is a fairly simple algorithm, since UTF-8 is self-synchronizing. This means it can easily be done without relying on external libraries or data files. It's also stable with respect to Unicode version--it always produces the same result independent of what version of the Unicode tables you use. Moving to grapheme cluster boundaries means that the algorithm may work incorrectly if you input a string of Unicode N+1 to an implementation that only supports Unicode N. It also makes the "increment character" function very complicated. In the UTF-8 version, this looks roughly like: char *advance(char *str) { uint8_t c = (uint8_t)*str; /* Count the number of leading 1's */ int num1s = __builtin_clz(~c) - 24; if (num1s == 0) return str + 1; return str + num1s; } Grapheme-based indexing looks like this: char *advance_grapheme(char *str) { while (true) { uint32_t codepoint = read_codepoint(str); str = advance(str); uint32_t nextCodepoint = read_codepoint(str); /* This is typically something like table[table2[codepoint >> 4] * 16 + codepoint & 15]; */ GraphemeClusterBreak left = lookupProp(codepoint); GraphemeClusterBreak right = lookupProp(nextCodepoint); /* Several rules based on left versus right... */ } return str; } See the vast difference in the two implementations? It's a lot of complexity, and it's worth asking if that complexity needs to be built into the main library (strings are a fundamental datatype in any language). It's also important to note that it's questionable whether such a feature implemented by default is going to actually fix naive programmers' code--if you read UTR #29 carefully, you'll notice that something like क्ष will consist of two grapheme clusters (क् and ष), which is arguably incorrect. Internationalization is often tied heavily to GUI and, especially for problems like grapheme clusters, it arguably makes more sense for toolkits to implement and deal with the problems themselves and provide things like "text input widget" primitives to programmers rather than encouraging users to try to implement it themselves.
- mtviewdave 10y ago
  History has shown that, when it comes to strings, developers have a hard time getting even something as simple as null-termination correct. If grapheme handling is complex, that's an argument for having it implemented by a small team of experts exactly once. The resulting abstraction might not be leak-proof, but then no abstraction is.
- MrBuddyCasino 10y ago
  Thanks for the detailed response. This looks like a performance problem, big lookup tables trash the cashes. I with mtviewdave that more complexity means this should really be in the std lib. Java's charAt(int i) is misleading at best.
- lomnakkus 10y ago
  > (strings are a fundamental datatype in any language) (Probably a bit "unfair" to "pounce" on an off-hand paranthetical like this, but I'm in a bit of a pedantic mood...) This is not true for e.g. Haskell. In Haskell it's defined as [Char], i.e. a list of characters. (Of course the Haskell community is suffering from that decision, but that's another story.) I'm not sure why strings would need to be a fundamendal type, though. Sure, they would probably be part of the standard library for almost all languages, but they don't need to be "magical" in the way most fundamental types (int, etc.) are.

asveikau 10y ago

1 grapheme = 1 unit of datatype is a good start. There is a lot of complexity in Unicode, but the mental model of a "char" is what most developers are thinking, largely blind to that complexity. Kudos to anyone trying to make that more "common sense" model closer to reality. However... How does a language or library attempting to abstract this part (like swift might) deal with the other, unrelated annoying aspects of Unicode? Even if 1 glyph is 1 "char", and we normalize all the inputs, there is still, say ... Bi-di text.

johncolanduoni 10y ago
I don't think there is an easy answer to this. Even just pure RTL text is hard to support: IIRC Android and iOS only recently made all the built-in views support it. This is complicated by the fact that most developer teams don't have nor can afford to hire someone familiar with RTL to implement this kind of stuff.
- asveikau 10y ago
  I don't think the rules in Unicode are simple for anyone. I have seen professional developers who are native speakers of RTL languages screw it up. All this stuff about the directionality of a paragraph and the embedded markers... It boggles the mind. But a truly international product should be getting it right. Sad mismatch there. PS: bit of trivia that people forget these days, Win32 has been supporting it for longer than android/iOS.

Joeri 10y ago

Since all other systems have standardized on code points this would lead to subtle incompatibilities. For example, checking for length prior to inserting in a database must be done in code points. What i find more frustrating is how the documentation for many systems describes the basic unit of text as a character, without specifying whether a code point or grapheme is meant, and without leading people to an explanation of the difference. There is still a lot of software that processes unicode text incorrectly, not because it is difficult to do so, but because nobody told the developer how things should be done.

int_19h 10y ago

I don't think there should even be a "default" character at all. In some cases, codepoints are the right choice; in others, graphemes are. If we make this explicit - e.g. by forbidding to iterate strings directly, but providing functions/methods to extract codepoints and graphemes as iterable sequences - the programmer has to make a conscious choice every time they iterate.

johncolanduoni 10y ago
In principle I'd agree with you, but the response from most people learning a language that treats strings like that is likely to be "You can't iterate over strings? And it doesn't even have a character type!" I'm not saying people are stupid: I had never really thought about the difference between a code point and grapheme until relatively recently when I had to do some low-level text layout stuff and it became important. My point is that I don't think throwing people into the deep end and expecting them to grok the codepoint/grapheme division before they get to use the language is likely to be productive. Defaulting to graphemes carries the advantage that if a programmer who doesn't know much about languages that require more Unicode finesse uses it purely intuitively, they'll get things right in a lot of cases. Using codepoints, on the other hand, makes it easy to put text through a grinder while doing relatively innocent things. Unfortunately while we're discussing the finer points of using graphemes and codepoints in APIs a million supplementary characters were brutally eviscerated by code running on languages that haven't quite gotten past UCS-2 XD
- int_19h 10y ago
  I think that throwing people into the deep end is the only way to get them to do it right in this case. Defaulting to graphemes is too often the wrong answer, as well (e.g. you don't want that in a parser). Really, this is not dissimilar to "what do you mean, there are more letters than A to Z?" issue that plagued software written in US back before Unicode became dominant. The way we (my perspective on this is as a native speaker of a language with a non-Latin alphabet) have eventually solved it is by basically forcing Unicode onto those people. It broke their simple and convenient picture of the world, and replaced it with something much more complicated. But it was necessary. My position is that letting programmers get away with a simplistic view of text processing (by allowing defaults that "mostly" work) is what creates those issues. So adjusting the abstractions such that they expose more of the underlying complexity is a good thing. People SHOULD believe that doing text processing the right way is hard, because it is.

raiph 10y ago

> Swift is the only language I know of that has grapheme clusters as the default unit of character Also Elixir and Perl 6. For a bit more info see https://news.ycombinator.com/item?id= https://news.ycombinator.com/item?id=