3 ms·

What I recall of the standard algorithm is that it does include font/user-agent/locale pieces in its calculations and "zero ambiguity" is a bit of a stretch, un

What I recall of the standard algorithm is that it does include font/user-agent/locale pieces in its calculations and "zero ambiguity" is a bit of a stretch, unfortunately. Even the overview includes this note: > Note: Font-based information may be required to determine the appropriate unit to use for UI purposes First of all, grapheme counting depends on normalization and there's like 6 Unicode normalization algorithms to consider, depending on locale and intended use/display style. Keep in mind that grapheme counting includes things like natural ligatures which English has always had a rough time counting. Things like `fi` and `ft` that sometimes form single visual graphemes from the font's perspective for some serif fonts but are always supposed to be counted as two graphemes from the user's perspective. Relatedly, one of the simpler parts of the Unicode standard algorithm is that the ZWJ always signals a grapheme cluster/merged grapheme, so per the standard algorithm the Ninja Cat is always a single grapheme even though the usual emoji rules for all the non-Microsoft fonts that don't include Cat+ZWJ+Job sequences ("cats with jobs emojis") fallback to two display characters to try to get the idea across, the spec does say that they should still count that as only a single grapheme even when not presented as such. (ETA Aside: I think the cats with jobs emojis are great and fun and should be wider adopted outside of just Microsoft. Why should only the people emojis have jobs?)

Dylan16807 4y ago
> First of all, grapheme counting depends on normalization and there's like 6 Unicode normalization algorithms to consider, depending on locale and intended use/display style. It says "The boundary specifications are stated in terms of text normalized according to Normalization Form NFD (see Unicode Standard Annex #15, “Unicode Normalization Forms” [UAX15]). In practice, normalization of the input is not required." And there's also "Even in Normalization Form NFC, a syllable block may contain a precomposed Hangul syllable in the middle." So are you sure normalization matters? I'm excluding K normalizations since they're not for general use and alter a bunch of characters on purpose. But also that's only NFC/NFD/NFKC/NFKD, are there two others or was "six" just a misremembering?
- rurban 4y ago
  You forgot FCC and FCD. In practice only NFC is used and sometimes NFD for speed
- WorldMaker 4y ago
  "like 6" was intentional hyperbole, but I appreciate the technical clarifications. > I'm excluding K normalizations since they're not for general use "Not for general use" doesn't mean that they aren't somebody's use and still a complicating factor in everything else.
  - Dylan16807 4y ago
    Nobody that cares about keeping text intact uses K. And you don't have to care about how they count characters. It won't affect your system, and they're probably counting something dumb. NFC/NFD will give you the same graphemes. Or to put it another way, there's only two real normalizations, and there's some weird junk off to the side.
crazygringo 4y ago
Here's the spec: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... And nope, it doesn't depend at all on font, user agent, or locale. Nor does it depend on normalization (that's explicitly stated). And like you say, a ligature for 'fi' is two graphemes. All of this is for the purposes of text selection, counting characters, etc. You're right there are additional considerations in rendering/UX, particularly with font ligatures. But that operates at a different level from graphemes. (The spec also explains that more sophisticated grapheme definitions can be constructed for locales. But "The Unicode definitions of grapheme clusters are defaults: not meant to exclude the use of more sophisticated definitions of tailored grapheme clusters where appropriate." So it works as a single shared canonical reference.)
- WorldMaker 4y ago
  That's my point, though, the algorithm defines "defaults" and "starting points" then suggests places like font/user agent/locale that may need to make more sophisticated judgement calls. That's very different from "an unambiguous algorithm that applies in all circumstances". I included this note from that very report: > Note: Font-based information may be required to determine the appropriate unit to use for UI purposes, such as identification of boundaries for first-letter paragraph styling. You fixated on this additional pull quote: > The Unicode definitions of grapheme clusters are defaults: not meant to exclude the use of more sophisticated definitions of tailored grapheme clusters where appropriate. Such definitions may more precisely match the user expectations within individual languages for given processes. I would add additional emphasis on the word default there which the document in other places contrasts to tailored as the preferred terminology between "simple algorithm rules of thumb" that this particular algorithm provides and user-focused/locale-dependent grapheme considerations (which still do vary a lot). For instance, this bit on tailored graphemes: > Grapheme clusters can be tailored to meet further requirements. Such tailoring is permitted, but the possible rules are outside of the scope of this document. One example of such a tailoring would be for the aksaras, or orthographic syllables, used in many Indic scripts. (And a later table of more examples of tailored graphemes that the "default grapheme" algorithm will not catch the subtleties of.) As for relying on normalization, I may have misread this particular pull quote: > A key feature of default Unicode grapheme clusters (both legacy and extended) is that they remain unchanged across all canonically equivalent forms of the underlying text. Thus the boundaries remain unchanged whether the text is in NFC or NFD. I read that as implying that algorithm input needed to be NFC or NFD (though there is clearly no difference to the algorithm which of the two forms it is in), not that the algorithm should also just as well work on unnormalized inputs. Rereading it again, I'm still not sure if the algorithm is well-defined on unnormalized inputs, but I can see how it reads as a possible implication that it might work. (Maybe?)

Dylan16807 4y ago

> First of all, grapheme counting depends on normalization and there's like 6 Unicode normalization algorithms to consider, depending on locale and intended use/display style. It says "The boundary specifications are stated in terms of text normalized according to Normalization Form NFD (see Unicode Standard Annex #15, “Unicode Normalization Forms” [UAX15]). In practice, normalization of the input is not required." And there's also "Even in Normalization Form NFC, a syllable block may contain a precomposed Hangul syllable in the middle." So are you sure normalization matters? I'm excluding K normalizations since they're not for general use and alter a bunch of characters on purpose. But also that's only NFC/NFD/NFKC/NFKD, are there two others or was "six" just a misremembering?

rurban 4y ago
You forgot FCC and FCD. In practice only NFC is used and sometimes NFD for speed
WorldMaker 4y ago
"like 6" was intentional hyperbole, but I appreciate the technical clarifications. > I'm excluding K normalizations since they're not for general use "Not for general use" doesn't mean that they aren't somebody's use and still a complicating factor in everything else.
- Dylan16807 4y ago
  Nobody that cares about keeping text intact uses K. And you don't have to care about how they count characters. It won't affect your system, and they're probably counting something dumb. NFC/NFD will give you the same graphemes. Or to put it another way, there's only two real normalizations, and there's some weird junk off to the side.

crazygringo 4y ago

Here's the spec: https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... And nope, it doesn't depend at all on font, user agent, or locale. Nor does it depend on normalization (that's explicitly stated). And like you say, a ligature for 'fi' is two graphemes. All of this is for the purposes of text selection, counting characters, etc. You're right there are additional considerations in rendering/UX, particularly with font ligatures. But that operates at a different level from graphemes. (The spec also explains that more sophisticated grapheme definitions can be constructed for locales. But "The Unicode definitions of grapheme clusters are defaults: not meant to exclude the use of more sophisticated definitions of tailored grapheme clusters where appropriate." So it works as a single shared canonical reference.)

WorldMaker 4y ago
That's my point, though, the algorithm defines "defaults" and "starting points" then suggests places like font/user agent/locale that may need to make more sophisticated judgement calls. That's very different from "an unambiguous algorithm that applies in all circumstances". I included this note from that very report: > Note: Font-based information may be required to determine the appropriate unit to use for UI purposes, such as identification of boundaries for first-letter paragraph styling. You fixated on this additional pull quote: > The Unicode definitions of grapheme clusters are defaults: not meant to exclude the use of more sophisticated definitions of tailored grapheme clusters where appropriate. Such definitions may more precisely match the user expectations within individual languages for given processes. I would add additional emphasis on the word default there which the document in other places contrasts to tailored as the preferred terminology between "simple algorithm rules of thumb" that this particular algorithm provides and user-focused/locale-dependent grapheme considerations (which still do vary a lot). For instance, this bit on tailored graphemes: > Grapheme clusters can be tailored to meet further requirements. Such tailoring is permitted, but the possible rules are outside of the scope of this document. One example of such a tailoring would be for the aksaras, or orthographic syllables, used in many Indic scripts. (And a later table of more examples of tailored graphemes that the "default grapheme" algorithm will not catch the subtleties of.) As for relying on normalization, I may have misread this particular pull quote: > A key feature of default Unicode grapheme clusters (both legacy and extended) is that they remain unchanged across all canonically equivalent forms of the underlying text. Thus the boundaries remain unchanged whether the text is in NFC or NFD. I read that as implying that algorithm input needed to be NFC or NFD (though there is clearly no difference to the algorithm which of the two forms it is in), not that the algorithm should also just as well work on unnormalized inputs. Rereading it again, I'm still not sure if the algorithm is well-defined on unnormalized inputs, but I can see how it reads as a possible implication that it might work. (Maybe?)