7 ms·

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in

by pif 3y ago

> An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." I'm sorry, but I fail to see how "This visually displays as a single unit" could ever differ from "Display size in a monospace font" or "Thing that gets deleted when you hit backspace".

mattnewton 3y ago
> Display size in a monospace font Some clusters are going to be multiple characters wide. > thing that gets deleted when you hit backspace Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke. Look into how Korean works.
jcranmer 3y ago
See, e.g., https://github.com/xi-editor/xi-editor/issues/655 https://github.com/xi-editor/xi-editor/issues/655 for why backspace isn't the same as extended grapheme cluster. As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).
RichieAHB 3y ago
Here’s a good example of the test cases used for backspaces in Android[1]. It’s definitely more involved than just deleting a grapheme cluster. [1] https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/tests/coretests/src/android/text/method/BackspaceTest.java https://android.googlesource.com/platform/frameworks/base/+/...
jfultz 3y ago
A couple of cases I'm aware of... * Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing. * Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".
- dwringer 3y ago
  I've always felt ligatures that condense two or more glyphs into something that takes up the space of only one in a monospace font are going beyond what a font should handle and into the realm of what an editor should do. I have several such visual substitutions set up in my .emacs but I don't use fonts that do them on their own.
  - trealira 3y ago
    What about ligatures that make ASCII characters display differently when in proximity, but still use the same number of columns? For example, when == is written, connect them to be a 2 column wide = instead. Or when === is written, display a three column equals sign, but it's three bars instead of two.
- deleted 3y ago
  [deleted]
orphea 3y ago
If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".
- kdmccormick 3y ago
  But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off. Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.
- bombela 3y ago
  I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "<emoji>" and "<emoji>" to be deleted as one unit. edit: emojis are filtered by HN
  - pests 3y ago
    Even the emoji's that you create by combining multiple emojis? Type one emoji, then a second, it merges into one. What happens when you backspace?
    - bombela 3y ago
      I expect the full emoji to be deleted. Because it merged into one visual entity. I guess it's more ambiguous for some languages that can have long ligatures though.
    - xeyownt 3y ago
      I'd say delete the whole with backspace, but only the last if you undo.
- Findecanor 3y ago
  Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.
layer8 3y ago
In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ﬀ”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).
- jameshart 3y ago
  Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters. Latin: Katakana Full width: カタカナ Half width: ｶﾀｶﾅ (How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)
  - layer8 3y ago
    The background here is that both are contained in the Japanese Shift JIS character set, and Unicode provides roundtrip compatibility. And they are in Shift JIS because the half-width katakana were in the 8-bit JIS character set [0] used with text-mode displays where all characters have the same width. To preserve screen layout, these later had to be distinguished from full-width katakana. [0] https://en.wikipedia.org/wiki/JIS_X_0201 https://en.wikipedia.org/wiki/JIS_X_0201
  - deleted 3y ago
    [deleted]
yeputons 3y ago
Here is a full article of such examples: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... Discussion on HN: https://news.ycombinator.com/item?id=31858311 https://news.ycombinator.com/item?id=31858311
barkingcat 3y ago
characters that alter their appearance to be one or more display units depending on the characters that are next to it (before and after). that would be a very crude example, but these types of characters appear all the time in human language
MatmaRex 3y ago
﷽ 𒐫 𒈙 ꧅
wbl 3y ago
If I have a text with niqqudim I am going to want to think of the niqqudim differently when editing despite the fact they are entwined with the consonants.

by pif 3y ago

mattnewton 3y ago

> Display size in a monospace font Some clusters are going to be multiple characters wide. > thing that gets deleted when you hit backspace Some clusters are meant to be composted of multiple keystrokes and a natural editing experience would allow users to delete the last stroke. Look into how Korean works.

jcranmer 3y ago

See, e.g., https://github.com/xi-editor/xi-editor/issues/655 https://github.com/xi-editor/xi-editor/issues/655 for why backspace isn't the same as extended grapheme cluster. As for "display size in monospace font", emojis and CJK characters are usually two units wide, not one (although, to be honest, there's a fair amount of bugs in the Unicode properties that define this).

RichieAHB 3y ago

Here’s a good example of the test cases used for backspaces in Android[1]. It’s definitely more involved than just deleting a grapheme cluster. [1] https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/tests/coretests/src/android/text/method/BackspaceTest.java https://android.googlesource.com/platform/frameworks/base/+/...

jfultz 3y ago

A couple of cases I'm aware of... * Coding ligatures often display as a single glyph (maybe occupying a single-width character space, or maybe spread out over multiple spaces), but are composed of multiple glyphs. The ligature may "look" like a single character for purposes of selection and cursoring, but it can act like multiple characters when subject to backspacing. * Similarly, I've seen keyboard interfaces for various languages (e.g., Hindi) where standard grapheme cluster rules bind together a group of code points, but the grapheme cluster was composed from multiple key presses (which typically add one code point each to the cluster). And in some such interfaces I've seen, the cluster can be decomposed by an equal number of backspace presses. I don't have a good sense of how much a monospaced Hindi font makes sense, but it's definitely a case where a "character" doesn't always act "character-like".

dwringer 3y ago
I've always felt ligatures that condense two or more glyphs into something that takes up the space of only one in a monospace font are going beyond what a font should handle and into the realm of what an editor should do. I have several such visual substitutions set up in my .emacs but I don't use fonts that do them on their own.
- trealira 3y ago
  What about ligatures that make ASCII characters display differently when in proximity, but still use the same number of columns? For example, when == is written, connect them to be a 2 column wide = instead. Or when === is written, display a three column equals sign, but it's three bars instead of two.
deleted 3y ago
[deleted]

orphea 3y ago

If you type "a", combine it with "´", then change your mind and hit backspace, you probably want to end up with "a" even through "á" was a thing "visually displayed as a single unit".

kdmccormick 3y ago
But then if I type "á" directly (through, say, a mobile keyboard) and hit backspace, I'd get "a", which doesn't seem terrible but does feel a little off. Seems like the right answer for codepoints vs graphemes, unfortunately, is dependent on the context.
bombela 3y ago
I expect to delete the character "á". And I prefer consistency too so I expect "œ" and "<emoji>" and "<emoji>" to be deleted as one unit. edit: emojis are filtered by HN
- pests 3y ago
  Even the emoji's that you create by combining multiple emojis? Type one emoji, then a second, it merges into one. What happens when you backspace?
  - bombela 3y ago
    I expect the full emoji to be deleted. Because it merged into one visual entity. I guess it's more ambiguous for some languages that can have long ligatures though.
  - xeyownt 3y ago
    I'd say delete the whole with backspace, but only the last if you undo.
Findecanor 3y ago
Most European keyboard layouts have it the other way around: first press a "dead key" for the diacritic mark and then the letter to apply it to. Where some layouts may require this method for some characters, another keyboard layout may have the same character on a dedicated key. The program receives the combined character as one unit, and does not need to be aware of different keyboard layouts.

layer8 3y ago

In terminals there is a distinction between single-width and double-width characters (east-asian characters, in particular). E.g. the three characters A美C would take up the width of four ASCII monospace characters, the “美” being double-width. Similarly, for composed characters like say the ligature “ﬀ”, you may want to backspace as if it was two “f”s (which logically it is, and decomposes to in NFKD normalization).

jameshart 3y ago
Unicode even has distinct full width and half width variants of Japanese katakana - where ‘full width’ is (in theory) as wide as two Latin characters. Latin: Katakana Full width: カタカナ Half width: ｶﾀｶﾅ (How that fixed width text looks in a web browser is anyone’s guess though. On iOS none of the Japanese kana stay on the fixed grid.)
- layer8 3y ago
  The background here is that both are contained in the Japanese Shift JIS character set, and Unicode provides roundtrip compatibility. And they are in Shift JIS because the half-width katakana were in the 8-bit JIS character set [0] used with text-mode displays where all characters have the same width. To preserve screen layout, these later had to be distinguished from full-width katakana. [0] https://en.wikipedia.org/wiki/JIS_X_0201 https://en.wikipedia.org/wiki/JIS_X_0201
- deleted 3y ago
  [deleted]

yeputons 3y ago

Here is a full article of such examples: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... Discussion on HN: https://news.ycombinator.com/item?id=31858311 https://news.ycombinator.com/item?id=31858311

barkingcat 3y ago

characters that alter their appearance to be one or more display units depending on the characters that are next to it (before and after). that would be a very crude example, but these types of characters appear all the time in human language

MatmaRex 3y ago

﷽ 𒐫 𒈙 ꧅

wbl 3y ago

If I have a text with niqqudim I am going to want to think of the niqqudim differently when editing despite the fact they are entwined with the consonants.