4 ms·
I found that even a suggestion to use "grapheme clusters" is misleading. People like to think that there is one kind of grapheme clusters, namely one specified
by lifthrasiir 7y ago
I found that even a suggestion to use "grapheme clusters" is misleading. People like to think that there is one kind of grapheme clusters, namely one specified in the UAX #29 [1], but that's just the default and the UAX is pretty much clear about it! Consider an `ij` digraph in Dutch [2] that should count as a single character as a motivating example. "Code points", or rather "Unicode scalar values" are formally defined and not changing; "grapheme clusters" are generally locale-dependent.
I think that we should ask what people want to do with "characters" instead:
- If you want an array of very short strings, use an array. Do not abuse strings. Recent languages even don't like it.
- If you want a string that is not too long when displayed, first and foremost try to resolve that on the actual display (say, with CSS). If that's really impossible, pick a font and actually measure. Make sure that you are using the actual size being printed---the bounding box is not linearly scaled when the font size changes.
- If that display is a terminal, you may also consider using the East Asian Width [3]. But keep in mind that the default width of "Ambiguous" characters greatly varies across locales (Asians tend to prefer dual-width ambiguous characters, for example).
- If you want a string that is not too long when stored, use the encoded byte size. Oh, do you want to make sure that certain languages are not disadvantaged? Do your research to determine the appropriate limit per language then, but I bet you would be much better with generous limits (cough cough Twitter cough).
- (While this is not counting characters, for the completeness,) if you want to navigate a string with a cursor or a keyboard, then default grapheme clusters may actually fit a bill, as that might be the best possible! Someone will obviously complain though.
- If you have to, uh, really, really count characters, the chance is that you can assume a particular script and/or language. Reject others and use the local convention. You may want to read about the Unicode Script Property [4].
[1] https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
[2] https://en.wikipedia.org/wiki/IJ_(digraph) https://en.wikipedia.org/wiki/IJ_(digraph)
[3] https://www.unicode.org/reports/tr11/ https://www.unicode.org/reports/tr11/
[4] https://www.unicode.org/reports/tr24/ https://www.unicode.org/reports/tr24/
- nabla9 7y agoYou are correct. The lesson is that code points are low level representation and the semantics does not transfer across languages in Unicode. If you don't know the context you can get characters that don't render correctly or you may split string into parts in the wrong UTF-8 strings can be treated as ASCII or Latin-1 replacements. If you want to deal with full Unicode with all cases you need locale. Exercise for the reader: Try to write radix tree data and rope data structure that works with every language in all cases with Unicode. http://cldr.unicode.org/ http://cldr.unicode.org/ is your friend.
- scatters 7y agoIf you want to navigate a string, you should stop "inside" compatibility ligatures (fi, U+FB01 being the canonical example).
- lifthrasiir 7y agoThat is why default grapheme clusters only concern boundaries aligned with "starters" or base characters (and thus are preserved after canonical normalizations). U+FB01 is only compatibly decomposable and harder to deal efficiently.