5 ms·

One of the major difficulties with Unicode handling is not just that there are poor implementations out there with legacy baggage, but a lot of poor advice as w

by btn 13y ago

One of the major difficulties with Unicode handling is not just that there are poor implementations out there with legacy baggage, but a lot of poor advice as well (or well-meaning advice that seems correct, but misses some corner case or some language). For example, this article wants to count "graphemes", and the author goes through three versions of an algorithm to account for surrogate pairs and various combining marks. All seems well in the test cases the author shows, but combining marks are only one class of codepoints that can join to form a grapheme, and the algorithm will fail for other grapheme clusters such as 'நி' (Tamil letter NA + Tamil Vowel Sign I), or Hangul made of conjoining Jamo (such as '깍': 'ᄁ' + 'ᅡ' + 'ᆨ'), or other control characters. Luckily, the Unicode Technical Committee has figured this out for you, and UAX#29 provides an algorithm for determining grapheme cluster boundaries [1]. Yes, it's long and technical, it has many cases (and exceptions) to handle, and it can't be expressed compactly in two lines of JavaScript; but it will give you a well-defined and understood answer for all scripts in Unicode. [1] http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Bounda...

giovannibajo1 13y ago
One thing I never understand is why it is so important to count graphemes. I read dozens (hundreds?) of Unicode-related blogpost for many different languages, with long debates and discussions about the hurdles of counting graphemes, but they always forget to explain why one should need it; it's just assumed that it's important or interesting. This specific post just says: "Let's say you want to count the number of symbols in a given string, for example. How would you go about it?" and then go into a multi-page explanation, which is even incomplete (as you correctly noticed). I can't remember many cases in which it's been useful to count graphemes, in my programming activity. I usually need to either: 1) count the number of bytes of the Unicode encoding I'm using / going to use, for the purpose of low-level stuff like buffers/sockets/memory/etc. 2) ask a graphic library to tell me how big the string will be on the screen, in pixels (with the given fonts, layout, hints, and whatnot). Counting graphemes only sounds useful for things like command-line terminal; e.g.: if I were to make a command-line user interface (ala getopt()) which automatically wordwraps text in the usage screen at the 80-th column, I would need to count graphemes, in the unlikely case I had to support Tamil or Korean for such a specialistic case. tl;dr: counting grapheme is a very complicated problem you probably don't need to ever solve.
- btn 13y ago
  Counting graphemes may be over-used, but needing to know their boundaries is important (and leads naturally to counting). For example, when you hit "delete" in a text editor, you'll probably want it to delete whole graphemes (and similarly for text selection); if you're doing text truncation, you may measure it by pixels, but you'll want to chop off the excess bytes at a grapheme boundary. in the unlikely case I had to support Tamil or Korean for such a specialistic case. Why is it "unlikely" that you would want your software to support users of other languages?
  - pavlov 13y ago
    In the case of a delete action in a text editor, are you sure that deleting the whole grapheme is actually what the Tamil or Korean user wants? You mentioned the following examples in your grandparent post: - 'நி' (Tamil letter NA + Tamil Vowel Sign I) - Hangul made of conjoining Jamo (such as '깍': 'ᄁ' + 'ᅡ' + 'ᆨ') I don't speak either language, but it doesn't seem unreasonable to me that pressing Delete would delete just the vowel sign in Tamil, or just the last component within the Hangul character. In fact, that might be just what the user wants?
    - taejo 13y ago
      > I don't speak either language, but it doesn't seem unreasonable to me that pressing Delete would delete just the vowel sign in Tamil, or just the last component within the Hangul character. In fact, that might be just what the user wants? My Korean is pretty poor, but I think that's exactly what one wants. If you mistype a letter, you want to retype that letter, not the whole syllable. However, this should work uniformly: it shouldn't matter if the syllable is represented as a single codepoint or made up of comjoining jamo.
      - yew 13y ago
        If the Hangul and Tamil constructs are anything like ligatures (e.g. fi in the Latin alphabet), I would imagine that's the case most of the time. Plus lots of special rules for which glyphs to treat as single symbols and which to decompose (e.g. & is technically a ligature but almost never decomposed).
mathias 13y ago
Excellent point. I was aware http://mathiasbynens.be/notes/javascript-unicode#accounting-for-other-combining-marks http://mathiasbynens.be/notes/javascript-unicode#accounting-... is not a fully accurate/bulletproof solution (never said it was, either). I’ve added a note to the article now, linking to UAX#29. Thanks!

by btn 13y ago

giovannibajo1 13y ago

One thing I never understand is why it is so important to count graphemes. I read dozens (hundreds?) of Unicode-related blogpost for many different languages, with long debates and discussions about the hurdles of counting graphemes, but they always forget to explain why one should need it; it's just assumed that it's important or interesting. This specific post just says: "Let's say you want to count the number of symbols in a given string, for example. How would you go about it?" and then go into a multi-page explanation, which is even incomplete (as you correctly noticed). I can't remember many cases in which it's been useful to count graphemes, in my programming activity. I usually need to either: 1) count the number of bytes of the Unicode encoding I'm using / going to use, for the purpose of low-level stuff like buffers/sockets/memory/etc. 2) ask a graphic library to tell me how big the string will be on the screen, in pixels (with the given fonts, layout, hints, and whatnot). Counting graphemes only sounds useful for things like command-line terminal; e.g.: if I were to make a command-line user interface (ala getopt()) which automatically wordwraps text in the usage screen at the 80-th column, I would need to count graphemes, in the unlikely case I had to support Tamil or Korean for such a specialistic case. tl;dr: counting grapheme is a very complicated problem you probably don't need to ever solve.

btn 13y ago
Counting graphemes may be over-used, but needing to know their boundaries is important (and leads naturally to counting). For example, when you hit "delete" in a text editor, you'll probably want it to delete whole graphemes (and similarly for text selection); if you're doing text truncation, you may measure it by pixels, but you'll want to chop off the excess bytes at a grapheme boundary. in the unlikely case I had to support Tamil or Korean for such a specialistic case. Why is it "unlikely" that you would want your software to support users of other languages?
- pavlov 13y ago
  In the case of a delete action in a text editor, are you sure that deleting the whole grapheme is actually what the Tamil or Korean user wants? You mentioned the following examples in your grandparent post: - 'நி' (Tamil letter NA + Tamil Vowel Sign I) - Hangul made of conjoining Jamo (such as '깍': 'ᄁ' + 'ᅡ' + 'ᆨ') I don't speak either language, but it doesn't seem unreasonable to me that pressing Delete would delete just the vowel sign in Tamil, or just the last component within the Hangul character. In fact, that might be just what the user wants?
  - taejo 13y ago
    > I don't speak either language, but it doesn't seem unreasonable to me that pressing Delete would delete just the vowel sign in Tamil, or just the last component within the Hangul character. In fact, that might be just what the user wants? My Korean is pretty poor, but I think that's exactly what one wants. If you mistype a letter, you want to retype that letter, not the whole syllable. However, this should work uniformly: it shouldn't matter if the syllable is represented as a single codepoint or made up of comjoining jamo.
    - yew 13y ago
      If the Hangul and Tamil constructs are anything like ligatures (e.g. fi in the Latin alphabet), I would imagine that's the case most of the time. Plus lots of special rules for which glyphs to treat as single symbols and which to decompose (e.g. & is technically a ligature but almost never decomposed).

mathias 13y ago

Excellent point. I was aware http://mathiasbynens.be/notes/javascript-unicode#accounting-for-other-combining-marks http://mathiasbynens.be/notes/javascript-unicode#accounting-... is not a fully accurate/bulletproof solution (never said it was, either). I’ve added a note to the article now, linking to UAX#29. Thanks!