9 ms·

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." Like so many other things in Unicode, the correct answer is use-case dependent. (And for this reason, String iteration should be based on codepoints--it's the fundamental level on which Unicode works, and whatever algorithm you want to use to derive the correct answer for your purpose will be based on codepoint iteration. hsivonen's article (https://hsivonen.fi/string-length/ https://hsivonen.fi/string-length/), linked in this one, does try to explain why extended grapheme clusters is the wrong primitive to use in a language.)

pif 3y ago
> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".
- mattnewton 3y ago
  > Display size in a monospace font Some clusters are going to be multiple characters wide. > thing that gets deleted when you hit backspace Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke. Look into how Korean works.
- jcranmer 3y ago
  See, e.g., https://github.com/xi-editor/xi-editor/issues/655 https://github.com/xi-editor/xi-editor/issues/655 for why backspace isn't the same as extended grapheme cluster. As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).
- RichieAHB 3y ago
  Here’s a good example of the test cases used for backspaces in Android[1]. It’s definitely more involved than just deleting a grapheme cluster. [1] https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/tests/coretests/src/android/text/method/BackspaceTest.java https://android.googlesource.com/platform/frameworks/base/+/...
- jfultz 3y ago
  A couple of cases I'm aware of... * Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing. * Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".
  - dwringer 3y ago
    I've always felt ligatures that condense two or more glyphs into something that takes up the space of only one in a monospace font are going beyond what a font should handle and into the realm of what an editor should do. I have several such visual substitutions set up in my .emacs but I don't use fonts that do them on their own.
    - trealira 3y ago
      What about ligatures that make ASCII characters display differently when in proximity, but still use the same number of columns? For example, when == is written, connect them to be a 2 column wide = instead. Or when === is written, display a three column equals sign, but it's three bars instead of two.
  - deleted 3y ago
    [deleted]
- orphea 3y ago
  If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".
  - kdmccormick 3y ago
    But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off. Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.
  - bombela 3y ago
    I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "<emoji>" and "<emoji>" to be deleted as one unit. edit: emojis are filtered by HN
    - pests 3y ago
      Even the emoji's that you create by combining multiple emojis? Type one emoji, then a second, it merges into one. What happens when you backspace?
      - bombela 3y ago
        I expect the full emoji to be deleted. Because it merged into one visual entity. I guess it's more ambiguous for some languages that can have long ligatures though.
      - xeyownt 3y ago
        I'd say delete the whole with backspace, but only the last if you undo.
  - Findecanor 3y ago
    Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.
- layer8 3y ago
  In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ﬀ”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).
  - jameshart 3y ago
    Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters. Latin: Katakana Full width: カタカナ Half width: ｶﾀｶﾅ (How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)
    - layer8 3y ago
      The background here is that both are contained in the Japanese Shift JIS character set, and Unicode provides roundtrip compatibility. And they are in Shift JIS because the half-width katakana were in the 8-bit JIS character set [0] used with text-mode displays where all characters have the same width. To preserve screen layout, these later had to be distinguished from full-width katakana. [0] https://en.wikipedia.org/wiki/JIS_X_0201 https://en.wikipedia.org/wiki/JIS_X_0201
    - deleted 3y ago
      [deleted]
- yeputons 3y ago
  Here is a full article of such examples: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... Discussion on HN: https://news.ycombinator.com/item?id=31858311 https://news.ycombinator.com/item?id=31858311
- barkingcat 3y ago
  characters that alter their appearance to be one or more display units depending on the characters that are next to it (before and after). that would be a very crude example, but these types of characters appear all the time in human language
- MatmaRex 3y ago
  ﷽ 𒐫 𒈙 ꧅
- wbl 3y ago
  If I have a text with niqqudim I am going to want to think of the niqqudim differently when editing despite the fact they are entwined with the consonants.
raphlinus 3y ago
Agreed. And one more consideration is that (extended) grapheme cluster boundaries vary from one version of Unicode to another, and also allow for "tailoring." For example, should "อำ" be one grapheme cluster or two? It's two on Android but one by Unicode recommendation and is the behavior on mac. So in applications where a query such as length needs to have one definitive answer which cannot change by context, counting (extended) grapheme clusters is the wrong way to go.
- lifthrasiir 3y ago
  In fact, the name "extended" grapheme cluster should give it away. There was already a major revision to UAX #29 so that the original version is now referred as to "legacy". Your example is exactly this case: the second character, U+0E33 THAI CHARACTER SARA AM, prohibits a cluster boundary now but previously didn't [1]. [1] Relevant specifications: https://unicode.org/reports/tr29/#SpacingMark https://unicode.org/reports/tr29/#SpacingMark and https://unicode.org/reports/tr29/#GB9a https://unicode.org/reports/tr29/#GB9a
lucideer 3y ago
I'm not Korean but seeing that said of the Hangul example definitely made me pause - I doubt Koreans think of that example as a single grapheme (open to correction), though it is an excellent example all the same since it demonstrates the complexity of defining "units" consistently across language. It reminds me a little of Open Street Map's inconsistent administrative hierarchies ("states", "countries", "counties", etc. being represented at different administrative "levels" in their hierarchy for each geographical area), and how that hinders consistency in styling- font size, zoom levels, etc. being generally applied by level.
- lifthrasiir 3y ago
  As a native Korean, I can confirm that "각" is perceived as a single character. But the example itself is bad anyway because everyone use a precomposed form U+AC01 instead of U+1100 U+1161 U+11A8 instead (they are canonically equivalent). This is more clear when you also consider a compatibility representation "ㄱㅏㄱ" U+3131 U+314F U+3131, which is same to "각" after compatibility normalizations (NFKC or NFKD), but perceived as three atomic characters in general.
  - lucideer 3y ago
    Thanks for the clarification, that's interesting. My impression before was it would be considered a single "entity" (not the same as a roman-alphabetic word, but not a character either) containing 3 characters.
riggsdk 3y ago
There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single "symbol" when selecting text? Basically something that could help out users making simple text-editors. There are so many bad implementations out there that does it incorrectly so there must be some tools/libraries to help with this? Not only for actual applications but for people making games as well where you want users to enter names, chat or other text. Not all platforms make it easy (or possible) to embed a fully fledged text editing engine for those use-cases. I can imagine that typing a multi-code-point character manually by hand would allow the user to undo their typing mistake by a single backspace press when they are actively typing it, but after that if you return to the symbol and press backspace that it would delete the whole symbol (grapheme cluster). For example if you manually entered the code points for the various family combination emojis (mother, son, daughter) you could still correct it for a while - but after the fact the editor would only see it as a single symbol to be deleted with a single backspace press? Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. (Not sure that is the way in which you would normally type those characters but it seems possible to do it with unicode that way).
- layer8 3y ago
  Behavior that depends on whether you edited something else in between, or that depends on timing, is just bad. Either always backspace grapheme clusters, or else backspace characters, possibly NFC-normalized. I could also imagine having something like Shift+Backspace to backspace NFKD-normalized characters when normal Backspace deletes grapheme clusters. As for selection and cursor movement, grapheme clusters would seem to be the correct choice. Same for Delete. An editor may also support an “exploded” view of separate characters (like WordPerfect Reveal Codes) where you manipulate individual characters.
- jcranmer 3y ago
  Some platforms (e.g., Android) have methods specifically for asking how to edit a string following a backspace. However, there's no standard Unicode algorithm to answer the question (and I strongly suspect that it's something that's actually locale-dependent to a degree). On further reflection, probably the best starting point for string editing on backspace is to operate on codepoints, not grapheme clusters. For most written languages, the various elements that make up a character are likely to be separate codepoints. In Latin text, diacritics are generally precomposed (I mean, you can have a + diacritic as opposed to precomposed ä in theory, but the IME system is going to spit out ä anyways, even if dead keys are used). But if you have Indic characters or Hangul, the grapheme cluster algorithm is going to erroneously combine multiple characters into a single unit. The issue is that the biggest false positive for a codepoint-based algorithm is emoji, and if you're a monolingual speaker whose only exposure to complex written scripts is Unicode emoji, you're going to incorrectly generalize it for all written languages.
hgs3 3y ago
Everybody loves to debate what "character" means but nobody ever consults the standard. In the Unicode Standard a "character" is an abstract unit of textual data identified by a code point. The standard never refers to graphemes as "characters" but rather as user-perceived characters which the article omits.
astrange 3y ago
String iteration should be based on whatever you want to iterate on - bytes, codepoints, grapheme clusters, words or paragraphs. There's no reason to privilege any one of these, and Swift doesn't do this. "Length" is a meaningless query because of this, but you might want to default to whatever approximates width in a UI label, so that's grapheme clusters. Using codepoints mostly means you wish you were doing bytes.
- b3morales 3y ago
  > There's no reason to privilege any one of these, and Swift doesn't do this. Strange thing to say: Swift String count property is the count of extended grapheme clusters. The documentation is explicit: > A string is a collection of extended grapheme clusters, which approximate human-readable characters. [emphasis in original]
  - astrange 3y ago
    The length/count property was added after people asked for it, but it wasn't originally in the String revamp, and it provides iterators for all of the above. .count also only claims to be O(n) to discourage using it.
    - b3morales 3y ago
      That was almost seven years ago now. It has been the String API twice as long as it has not been the API.
haberman 3y ago
In that case, it sounds like `length` on Unicode strings simply shouldn't exist, since there is no obvious right answer for it. Instead there should be `codepointCount`, `graphemeCount`, etc.
- Etherlord87 3y ago
  You're absolutely correct! `length` is ambiguous - you shouldn't have a `time` argument in a `sleep` function; you should have `milliseconds` and/or `seconds` etc.
  - extraduder_ire 3y ago
    You could have a Duration argument though. The parallels of string length with the phrase "How long is a piece of string?"[0] make this apparent/amusing. I'm sure I'm not the first person to think that. [0]: https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string
    - Etherlord87 3y ago
      The `Duration` being a type that implements an interface removing the ambiguity? Like a `DateTime` object does? I think it might be useful to have a function returning a collection of information about text, how many unicode points, how many grapheme clusters, how many syllables, vowels, consonants, special characters… But for performance reasons you probably want separate functions that give you just one of these.
      - extraduder_ire 3y ago
        DateTime, Instant, and Duration are totally different things. I believe I was thinking of a class I remember seeing added to java at one point. Time is probably one of the few things in programming that's even more cursed than strings.
- kalleboo 3y ago
  There are basically 2 places where programmers mostly want the "length" of a string: 1. To save storage space or avoid pathological input, they want to limit the "length" of text input fields. E.g., not allow a name to be 4 KB long 2. To fit something on screen Developers mostly used to western languages can approximate both with "number of letters", but the correct answers are For 1. Limit to bytes to avoid people building infinite zalgo characters, but be intelligent about it - don't just crop the byte array not taking into account graphemes. For 2. This sucks, especially for the web, but the only really correct answer here is to render it and check. Did I miss any other common cases?
mananaysiempre 3y ago
> thing that gets deleted when you hit backspace Is there a canonical source for this part, by the way? Xi copied the logic from Android[1] (per the issue you linked downthread), which is reasonable given its heritage but seems suboptimal generally, and I vaguely remember that CLDR had something to say about this too, but I don’t know if there’s any sort of consensus on this problem that’s actually written down anywhere. [1] https://github.com/xi-editor/xi-editor/pull/837 https://github.com/xi-editor/xi-editor/pull/837
grumpyprole 3y ago
> And for this reason, String iteration should be based on codepoints Why not offer both and be clear about it? Rather than just "length", why not call them code points? The Python docs for "len" which can be called on a unicode string say "Return the length (the number of items) of an object.". It doesn't look like a clear and easy to use API to me.
- archgoon 3y ago
  If you insist that `len` shouldn't be defined on strings, and the default iterator should be undefined in python then: for c in "Hello": pass should throw an exception. Also if word[0] == 'H': pass should throw an exception. This would have been an extremely controversial suggestion when python3 came out to say the least. Codepoints is a natural way of defining unicode strings in python, and it mostly works the way you expect once you give it a bit of thought. It is lower level than, say, grapheme clusters, but its more well defined and it provides the proper primitives for dealing with all use cases.
  - grumpyprole 3y ago
    I would suggest that len works as the article suggests; and "Hello".codepoints gives the behaviour you want.
    - archgoon 3y ago
      [dead]
nerdbert 3y ago
> for this reason, String iteration should be based on codepoints This leads you to the problem where you'll get different results iterating over n a ï v e vs n a ̈ i v e And I can't see how that's ever going to be a useful outcome. If you normalize everything first, then you can sidestep this to some degree, but then in effect your normalization has turned codepoint iteration into grapheme iteration for most common Latin-script text characters.

pif 3y ago

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

mattnewton 3y ago
> Display size in a monospace font Some clusters are going to be multiple characters wide. > thing that gets deleted when you hit backspace Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke. Look into how Korean works.
jcranmer 3y ago
See, e.g., https://github.com/xi-editor/xi-editor/issues/655 https://github.com/xi-editor/xi-editor/issues/655 for why backspace isn't the same as extended grapheme cluster. As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).
RichieAHB 3y ago
Here’s a good example of the test cases used for backspaces in Android[1]. It’s definitely more involved than just deleting a grapheme cluster. [1] https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/tests/coretests/src/android/text/method/BackspaceTest.java https://android.googlesource.com/platform/frameworks/base/+/...
jfultz 3y ago
A couple of cases I'm aware of... * Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing. * Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".
- dwringer 3y ago
  I've always felt ligatures that condense two or more glyphs into something that takes up the space of only one in a monospace font are going beyond what a font should handle and into the realm of what an editor should do. I have several such visual substitutions set up in my .emacs but I don't use fonts that do them on their own.
  - trealira 3y ago
    What about ligatures that make ASCII characters display differently when in proximity, but still use the same number of columns? For example, when == is written, connect them to be a 2 column wide = instead. Or when === is written, display a three column equals sign, but it's three bars instead of two.
- deleted 3y ago
  [deleted]
orphea 3y ago
If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".
- kdmccormick 3y ago
  But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off. Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.
- bombela 3y ago
  I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "<emoji>" and "<emoji>" to be deleted as one unit. edit: emojis are filtered by HN
  - pests 3y ago
    Even the emoji's that you create by combining multiple emojis? Type one emoji, then a second, it merges into one. What happens when you backspace?
    - bombela 3y ago
      I expect the full emoji to be deleted. Because it merged into one visual entity. I guess it's more ambiguous for some languages that can have long ligatures though.
    - xeyownt 3y ago
      I'd say delete the whole with backspace, but only the last if you undo.
- Findecanor 3y ago
  Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.
layer8 3y ago
In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ﬀ”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).
- jameshart 3y ago
  Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters. Latin: Katakana Full width: カタカナ Half width: ｶﾀｶﾅ (How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)
  - layer8 3y ago
    The background here is that both are contained in the Japanese Shift JIS character set, and Unicode provides roundtrip compatibility. And they are in Shift JIS because the half-width katakana were in the 8-bit JIS character set [0] used with text-mode displays where all characters have the same width. To preserve screen layout, these later had to be distinguished from full-width katakana. [0] https://en.wikipedia.org/wiki/JIS_X_0201 https://en.wikipedia.org/wiki/JIS_X_0201
  - deleted 3y ago
    [deleted]
yeputons 3y ago
Here is a full article of such examples: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... Discussion on HN: https://news.ycombinator.com/item?id=31858311 https://news.ycombinator.com/item?id=31858311
barkingcat 3y ago
characters that alter their appearance to be one or more display units depending on the characters that are next to it (before and after). that would be a very crude example, but these types of characters appear all the time in human language
MatmaRex 3y ago
﷽ 𒐫 𒈙 ꧅
wbl 3y ago
If I have a text with niqqudim I am going to want to think of the niqqudim differently when editing despite the fact they are entwined with the consonants.

raphlinus 3y ago

Agreed. And one more consideration is that (extended) grapheme cluster boundaries vary from one version of Unicode to another, and also allow for "tailoring." For example, should "อำ" be one grapheme cluster or two? It's two on Android but one by Unicode recommendation and is the behavior on mac. So in applications where a query such as length needs to have one definitive answer which cannot change by context, counting (extended) grapheme clusters is the wrong way to go.

lifthrasiir 3y ago
In fact, the name "extended" grapheme cluster should give it away. There was already a major revision to UAX #29 so that the original version is now referred as to "legacy". Your example is exactly this case: the second character, U+0E33 THAI CHARACTER SARA AM, prohibits a cluster boundary now but previously didn't [1]. [1] Relevant specifications: https://unicode.org/reports/tr29/#SpacingMark https://unicode.org/reports/tr29/#SpacingMark and https://unicode.org/reports/tr29/#GB9a https://unicode.org/reports/tr29/#GB9a

lucideer 3y ago

I'm not Korean but seeing that said of the Hangul example definitely made me pause - I doubt Koreans think of that example as a single grapheme (open to correction), though it is an excellent example all the same since it demonstrates the complexity of defining "units" consistently across language. It reminds me a little of Open Street Map's inconsistent administrative hierarchies ("states", "countries", "counties", etc. being represented at different administrative "levels" in their hierarchy for each geographical area), and how that hinders consistency in styling- font size, zoom levels, etc. being generally applied by level.

lifthrasiir 3y ago
As a native Korean, I can confirm that "각" is perceived as a single character. But the example itself is bad anyway because everyone use a precomposed form U+AC01 instead of U+1100 U+1161 U+11A8 instead (they are canonically equivalent). This is more clear when you also consider a compatibility representation "ㄱㅏㄱ" U+3131 U+314F U+3131, which is same to "각" after compatibility normalizations (NFKC or NFKD), but perceived as three atomic characters in general.
- lucideer 3y ago
  Thanks for the clarification, that's interesting. My impression before was it would be considered a single "entity" (not the same as a roman-alphabetic word, but not a character either) containing 3 characters.

riggsdk 3y ago

There are libraries that help with iterating both code-points and grapheme clusters... - but are there any of them that can help decide what to do for example when pressing backspace given an input string and a cursor position? Or any other text editing behavior. This use-case-dependent behavior must have some "correct" behavior that is standardized somewhere? Like a way to query what should be treated like a single "symbol" when selecting text? Basically something that could help out users making simple text-editors. There are so many bad implementations out there that does it incorrectly so there must be some tools/libraries to help with this? Not only for actual applications but for people making games as well where you want users to enter names, chat or other text. Not all platforms make it easy (or possible) to embed a fully fledged text editing engine for those use-cases. I can imagine that typing a multi-code-point character manually by hand would allow the user to undo their typing mistake by a single backspace press when they are actively typing it, but after that if you return to the symbol and press backspace that it would delete the whole symbol (grapheme cluster). For example if you manually entered the code points for the various family combination emojis (mother, son, daughter) you could still correct it for a while - but after the fact the editor would only see it as a single symbol to be deleted with a single backspace press? Or typing 'o' + '¨' to produce 'ö' but realizing you wanted to type 'ô', there just one backspace press would revert it to 'o' again and you could press '^' to get the 'ô'. (Not sure that is the way in which you would normally type those characters but it seems possible to do it with unicode that way).

layer8 3y ago
Behavior that depends on whether you edited something else in between, or that depends on timing, is just bad. Either always backspace grapheme clusters, or else backspace characters, possibly NFC-normalized. I could also imagine having something like Shift+Backspace to backspace NFKD-normalized characters when normal Backspace deletes grapheme clusters. As for selection and cursor movement, grapheme clusters would seem to be the correct choice. Same for Delete. An editor may also support an “exploded” view of separate characters (like WordPerfect Reveal Codes) where you manipulate individual characters.
jcranmer 3y ago
Some platforms (e.g., Android) have methods specifically for asking how to edit a string following a backspace. However, there's no standard Unicode algorithm to answer the question (and I strongly suspect that it's something that's actually locale-dependent to a degree). On further reflection, probably the best starting point for string editing on backspace is to operate on codepoints, not grapheme clusters. For most written languages, the various elements that make up a character are likely to be separate codepoints. In Latin text, diacritics are generally precomposed (I mean, you can have a + diacritic as opposed to precomposed ä in theory, but the IME system is going to spit out ä anyways, even if dead keys are used). But if you have Indic characters or Hangul, the grapheme cluster algorithm is going to erroneously combine multiple characters into a single unit. The issue is that the biggest false positive for a codepoint-based algorithm is emoji, and if you're a monolingual speaker whose only exposure to complex written scripts is Unicode emoji, you're going to incorrectly generalize it for all written languages.

hgs3 3y ago

Everybody loves to debate what "character" means but nobody ever consults the standard. In the Unicode Standard a "character" is an abstract unit of textual data identified by a code point. The standard never refers to graphemes as "characters" but rather as user-perceived characters which the article omits.

astrange 3y ago

String iteration should be based on whatever you want to iterate on - bytes, codepoints, grapheme clusters, words or paragraphs. There's no reason to privilege any one of these, and Swift doesn't do this. "Length" is a meaningless query because of this, but you might want to default to whatever approximates width in a UI label, so that's grapheme clusters. Using codepoints mostly means you wish you were doing bytes.

b3morales 3y ago
> There's no reason to privilege any one of these, and Swift doesn't do this. Strange thing to say: Swift String count property is the count of extended grapheme clusters. The documentation is explicit: > A string is a collection of extended grapheme clusters, which approximate human-readable characters. [emphasis in original]
- astrange 3y ago
  The length/count property was added after people asked for it, but it wasn't originally in the String revamp, and it provides iterators for all of the above. .count also only claims to be O(n) to discourage using it.
  - b3morales 3y ago
    That was almost seven years ago now. It has been the String API twice as long as it has not been the API.

haberman 3y ago

In that case, it sounds like `length` on Unicode strings simply shouldn't exist, since there is no obvious right answer for it. Instead there should be `codepointCount`, `graphemeCount`, etc.

Etherlord87 3y ago
You're absolutely correct! `length` is ambiguous - you shouldn't have a `time` argument in a `sleep` function; you should have `milliseconds` and/or `seconds` etc.
- extraduder_ire 3y ago
  You could have a Duration argument though. The parallels of string length with the phrase "How long is a piece of string?"[0] make this apparent/amusing. I'm sure I'm not the first person to think that. [0]: https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string https://en.wiktionary.org/wiki/how_long_is_a_piece_of_string
  - Etherlord87 3y ago
    The `Duration` being a type that implements an interface removing the ambiguity? Like a `DateTime` object does? I think it might be useful to have a function returning a collection of information about text, how many unicode points, how many grapheme clusters, how many syllables, vowels, consonants, special characters… But for performance reasons you probably want separate functions that give you just one of these.
    - extraduder_ire 3y ago
      DateTime, Instant, and Duration are totally different things. I believe I was thinking of a class I remember seeing added to java at one point. Time is probably one of the few things in programming that's even more cursed than strings.
kalleboo 3y ago
There are basically 2 places where programmers mostly want the "length" of a string: 1. To save storage space or avoid pathological input, they want to limit the "length" of text input fields. E.g., not allow a name to be 4 KB long 2. To fit something on screen Developers mostly used to western languages can approximate both with "number of letters", but the correct answers are For 1. Limit to bytes to avoid people building infinite zalgo characters, but be intelligent about it - don't just crop the byte array not taking into account graphemes. For 2. This sucks, especially for the web, but the only really correct answer here is to render it and check. Did I miss any other common cases?

mananaysiempre 3y ago

> thing that gets deleted when you hit backspace Is there a canonical source for this part, by the way? Xi copied the logic from Android[1] (per the issue you linked downthread), which is reasonable given its heritage but seems suboptimal generally, and I vaguely remember that CLDR had something to say about this too, but I don’t know if there’s any sort of consensus on this problem that’s actually written down anywhere. [1] https://github.com/xi-editor/xi-editor/pull/837 https://github.com/xi-editor/xi-editor/pull/837

grumpyprole 3y ago

> And for this reason, String iteration should be based on codepoints Why not offer both and be clear about it? Rather than just "length", why not call them code points? The Python docs for "len" which can be called on a unicode string say "Return the length (the number of items) of an object.". It doesn't look like a clear and easy to use API to me.

archgoon 3y ago
If you insist that `len` shouldn't be defined on strings, and the default iterator should be undefined in python then: for c in "Hello": pass should throw an exception. Also if word[0] == 'H': pass should throw an exception. This would have been an extremely controversial suggestion when python3 came out to say the least. Codepoints is a natural way of defining unicode strings in python, and it mostly works the way you expect once you give it a bit of thought. It is lower level than, say, grapheme clusters, but its more well defined and it provides the proper primitives for dealing with all use cases.
- grumpyprole 3y ago
  I would suggest that len works as the article suggests; and "Hello".codepoints gives the behaviour you want.
  - archgoon 3y ago
    [dead]

nerdbert 3y ago

> for this reason, String iteration should be based on codepoints This leads you to the problem where you'll get different results iterating over n a ï v e vs n a ̈ i v e And I can't see how that's ever going to be a useful outcome. If you normalize everything first, then you can sidestep this to some degree, but then in effect your normalization has turned codepoint iteration into grapheme iteration for most common Latin-script text characters.