4 ms·

I included ligatures as a logically separate concept that needs to be accounted for when counting the "displayed width" of a string and how you handle cursor po

I included ligatures as a logically separate concept that needs to be accounted for when counting the "displayed width" of a string and how you handle cursor position and selection logic. However, ligatures are a part of grapheme clustering and it isn't entirely independent: Examples above and elsewhere include ligatures that have Unicode encodings (ex: ﬁ). Ligatures have been a part of character encodings and have affected grapheme clustering since EBCDIC (directly from which Unicode inherits a lot of its single codepoint Latin ligatures). There are way too many debates if normal forms should encode ligatures or always decompose them or some other strategy. The normal forms (NFC and NFD, NFKC and NFKD) themselves are embedded as a step of several of the standard grapheme clustering algorithms. Some people think ligatures should be entirely left to fonts and Unicode ligatures a relic of the IBM bitmap font past. Some fonts think it would be nice if some ligatures were more directly encoded and normal forms could do a first pass for them. Unicode has had portions of the specification on both sides of that argument. It generally leans away from the backward compatibility ligatures and generally normalizes them to decomposition, especially in locales like "en-us" these days, but it doesn't always and in every locale. (All of that is before you start to consider languages that are almost nothing but long sequences of ligatures, including but not limited to Arabic.) You can't do grapheme clustering and entirely ignore ligatures. You certainly can't count "display width" or "display position" without paying attention to ligatures, which was the point of bringing them up alongside mentions of grapheme clustering length and why it is insufficient for "display position" and "display width".

dhosek 11mo ago
Umm, you’re confusing things here. When you said “several of the standard grapheme clustering algorithms” that indicated the first element of confusion. There is one grapheme clustering algorithm in Unicode.¹ Ligature combinations, whether encoded in Unicode or not are independent of the concept of graphemes and are very much font-dependent, as is whether or not a grapheme cluster is displayed as multiple characters or a single character (any two consecutive regional indicators are considered a single grapheme, whether they map to a national flag or not and the set of national flags and whether those flags are even displayed or the output is just the regional indicator symbol is up to the font). Normalization does not play a role in grapheme clustering at all, which is why á and a+acute accent are both considered single graphemes. I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). The question about dealing with ligatures for non-Latin scripts generally came down to compatibility with existing systems more than anything else. This is why, for example, the Devanagari and Thai encodings which both have vowel markings which can occur before, after or around a consonant handle the sequence of input differently. Assembly of jambo into characters in Hangul is another case where theoretically, one could have all the displayed characters handled through ligatures and not encode the syllables directly (which is, in fact, how most modern Korean is encoded as evidenced by Korean wikipedia), but because the original Korean font encoding has all the syllables encoded in it² those syllables are part of Unicode as well. But the bottom line here, is that you seem to be confusing terminology a lot here. You can very much do grapheme clustering without paying attention to ligatures, and rules for things like normalized composition/decomposition are entirely independent of grapheme clustering (the K-forms of both, manage transitions like ¹ to 1 or ﬁ to fi and represent a one way transition). ⸻ 1. I wrote a Rust library implementing this and I’ve been following Unicode since it was a proposal from Microsoft and Apple in the 90s, so I know a little about this. 2. I think the more accurate term would be “most” of the syllables as I’ve seen additional syllables added to Unicode over time.
- WorldMaker 11mo ago
  My point has been grapheme clustering doesn't matter as a useful length on its own, especially not for display length. The technical specifics/technical corrections are getting into weeds that are orthogonal to point of the discussion in the first place. > I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). EBCDIC had multiple "code pages" to handle various international encodings. (DOS and Windows inherited the "code page" concept from IBM mainframes.) Of the code pages EBCDIC supported in the mainframe era, many were "Publishing" code pages intended for "pretty printing" text to printers that supported presumably and primarily bitmap fonts. A low ID example of such is IBM Code Page (CCSID) 361: https://web.archive.org/web/20130121103957/http://www-03.ibm.com/systems/resources/systems_i_software_globalization_pdf_cp00361z.pdf https://web.archive.org/web/20130121103957/http://www-03.ibm... You can see that code page includes fi, fl, ff, ffi, ij, among others even less common in English text. Most but not all of the IBM Code Pages were a part of the earliest Unicode standards encoding efforts.
  - dhosek 11mo ago
    There were numerous other standardized code pages beyond those. ij is a digraph traditionally used in Dutch and was present in at least one of the ECMA code pages and isn’t really a publication ligature (it’s more on par with, e.g., the dz digraph for Croatian). Once upon a time I had a four inch binder with printouts of every single ECMA-accepted codepage which in addition to all the national/regional 8-bit encodings also included a number of Asian 16-bit encodings including three separate Chinese encodings (PRC, Taiwan, Hong Kong).

dhosek 11mo ago

Umm, you’re confusing things here. When you said “several of the standard grapheme clustering algorithms” that indicated the first element of confusion. There is one grapheme clustering algorithm in Unicode.¹ Ligature combinations, whether encoded in Unicode or not are independent of the concept of graphemes and are very much font-dependent, as is whether or not a grapheme cluster is displayed as multiple characters or a single character (any two consecutive regional indicators are considered a single grapheme, whether they map to a national flag or not and the set of national flags and whether those flags are even displayed or the output is just the regional indicator symbol is up to the font). Normalization does not play a role in grapheme clustering at all, which is why á and a+acute accent are both considered single graphemes. I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). The question about dealing with ligatures for non-Latin scripts generally came down to compatibility with existing systems more than anything else. This is why, for example, the Devanagari and Thai encodings which both have vowel markings which can occur before, after or around a consonant handle the sequence of input differently. Assembly of jambo into characters in Hangul is another case where theoretically, one could have all the displayed characters handled through ligatures and not encode the syllables directly (which is, in fact, how most modern Korean is encoded as evidenced by Korean wikipedia), but because the original Korean font encoding has all the syllables encoded in it² those syllables are part of Unicode as well. But the bottom line here, is that you seem to be confusing terminology a lot here. You can very much do grapheme clustering without paying attention to ligatures, and rules for things like normalized composition/decomposition are entirely independent of grapheme clustering (the K-forms of both, manage transitions like ¹ to 1 or ﬁ to fi and represent a one way transition). ⸻ 1. I wrote a Rust library implementing this and I’ve been following Unicode since it was a proposal from Microsoft and Apple in the 90s, so I know a little about this. 2. I think the more accurate term would be “most” of the syllables as I’ve seen additional syllables added to Unicode over time.

WorldMaker 11mo ago
My point has been grapheme clustering doesn't matter as a useful length on its own, especially not for display length. The technical specifics/technical corrections are getting into weeds that are orthogonal to point of the discussion in the first place. > I’m puzzled about the assertion of EBCDIC having Latin ligatures because I never encountered them in my IBM mainframe days and a casual search didn’t turn any up. The only 8-bit encoding that I was aware of that included any ligatures was Mac extended ASCII which included fi and fl (well, putting aside TeX’s 7-bit encoding which was only used by TeX and because of its using x20 to hold the acute accent was unusable by other applications which expected a blank character in that spot). EBCDIC had multiple "code pages" to handle various international encodings. (DOS and Windows inherited the "code page" concept from IBM mainframes.) Of the code pages EBCDIC supported in the mainframe era, many were "Publishing" code pages intended for "pretty printing" text to printers that supported presumably and primarily bitmap fonts. A low ID example of such is IBM Code Page (CCSID) 361: https://web.archive.org/web/20130121103957/http://www-03.ibm.com/systems/resources/systems_i_software_globalization_pdf_cp00361z.pdf https://web.archive.org/web/20130121103957/http://www-03.ibm... You can see that code page includes fi, fl, ff, ffi, ij, among others even less common in English text. Most but not all of the IBM Code Pages were a part of the earliest Unicode standards encoding efforts.
- dhosek 11mo ago
  There were numerous other standardized code pages beyond those. ij is a digraph traditionally used in Dutch and was present in at least one of the ECMA code pages and isn’t really a publication ligature (it’s more on par with, e.g., the dz digraph for Croatian). Once upon a time I had a four inch binder with printouts of every single ECMA-accepted codepage which in addition to all the national/regional 8-bit encodings also included a number of Asian 16-bit encodings including three separate Chinese encodings (PRC, Taiwan, Hong Kong).