5 ms·

emoji width bugs mostly come down to how terminals interpret Unicode's "grapheme clusters" vs "codepoints" vs "display cells". emoji isn't one codepoint - it's

emoji width bugs mostly come down to how terminals interpret Unicode's "grapheme clusters" vs "codepoints" vs "display cells". emoji isn't one codepoint - it's often multiple joined by zero-width joiners, variation selectors, skin tone modifiers. so the terminal asks wcwidth(), gets 1 or 2, but the actual glyph might render wider or combine into a single shape. some emoji even change width depending on font. family emoji is like 7 codepoints, shows up as one glyph. most terminals don't track that. they just count codepoints and pray. unless terminal is using a grapheme-aware renderer and syncs with the font's shaping engine (like freetype or coretext), it'll always guess wrong. wezterm and kitty kinda parse it right often

crackalamoo 1y ago
Yeah, unfortunately I feel like despite all the advances in Unicode tech, my modern terminal (MacOS) still bugs out badly with emojis and certain special characters. I'm not sure how/when codepoints matter for wcwidth: my terminal handles many characters with more than one codepoint in UTF-8, like é and even Arabic characters, just fine.
- o11c 1y ago
  `wcwidth` works by assigning all codepoints (strictly, code units of whatever size `wchar_t` is on your system, but thankfully modern Unixen are sane) a width of -1 (error), 0 (combining), 1 (narrow), or 2 (wide). `wcswidth` could in theory work across multiple codepoints, but its API is braindead and cannot deal with partial errors. This is all from the perspective of what the application expects to output. What the terminal itself does might be something completely different - decomposed Hangul in particular tends to lead to rendering glitches in curses-based terminal programs. This is also different from what the (monospace) font expects to be rendered as. At least it has the excuse of not being able to call the system's `wcwidth`. Note that it is always a mistake to call an implementation of `wcwidth` other than the one provided by the OS, since that introduces additional mismatches, unless you are using a better API that calculates bounds rather than an exact width. I posted an oversimplified sketch (e.g. it doesn't include versioning) of that algorithm a while back ... https://news.ycombinator.com/item?id=43851532 https://news.ycombinator.com/item?id=43851532
  - PhilipRoman 1y ago
    As fallback, you can also just emit the character and see how far the cursor advanced via CSI 6n (try printf '\x1b[6n')
    - o11c 1y ago
      Doing that adds a lot of round trips, so you still really need to do the initial estimate. (also, probing for whether the terminal actually supports various features is nontrivial. At startup you can send the basic "identify the terminal" sequences (there are 2) and check the result with a timeout; subsequently you can make a request then follow it with the basic terminal id to see if you actually get what you requested. But remember you can get arbitrary normal input interspersed.)
duped 1y ago
Why do you need to sync with the shaping engine? TBH grapheme clusters are annoying but day 1 learning material for a text display widget that supports beyond ascii. It honestly irks me how many things just fuck it up, because it's not an intractably hard problem - just annoying enough to be intractable for people that are lazy (*). (*) the actually hard problem with grapheme clusters is that they're potentially unbounded in length and the standard is mutable, so your wcwidth() implementation needs to be updated along with standards to stay valid, particularly with emoji. This basically creates a software maintenance burden out of aether.
- inetknght 1y ago
  > This basically creates a software maintenance burden out of aether. So... basically all modern software?
- zarzavat 1y ago
  > Why do you need to sync with the shaping engine? GP explained already. Grapheme clusters ≠ glyphs. To find the number of glyphs you need the font. An emoji can render as one or two or three or more glyphs depending on what font the user has installed, because many emoji are formed by joining two or more emoji by a ZWJ) (Also even in a monospace font not all glyphs are of ﷽ equal width)
  - layer8 1y ago
    It's not the font that is deciding how emoji sequences are rendered. The renderer may decide based on which characters exist in the available fonts, but it doesn't have to. Same for glyph width in terminals. It wasn’t uncommon for non-double-width-aware terminals to only draw half an emoji in a regular-width cell.
    - zarzavat 1y ago
      How else are you going to render a sequence such as Emoji ZWJ Emoji other than as two glyphs, if no composed glyph is defined in the user's font? That's how it's supposed to be rendered, for backwards compatibility.
Joker_vD 1y ago
The main problem is not even if the terminal itself can track the grapheme width "correctly". It's a) the fonts suck; b) does the terminal user tracks the width correctly? About a): some fonts have the glyphs for e.g. the playing cards block that are 1.5 columns wide even though the code points themselves are defined to be Narrow. How do you render that properly? Then there are variation selectors: despite what some may think, they don't affect the East Asian Width of the preceding code point, so whether you print "\N{ALEMBIC}\N{VARIATION SELECTOR-15}" or "\N{ALEMBIC}\N{VARIATION SELECTOR-16}", it still, according to wcwidth(), takes 1 column; but fonts have glyphs that are, again, 1.5 and 2 cells wide. And then there is the elephant in the room problem b) which is management of cursor position. But the terminal, and the app that uses the terminal need to have exactly the same idea of where the cursor is, or e.g. readline can't reliably function, or colorful grep output. You need to know how many lines of text you've output (to be able to erase them properly), and whether the cursor is at the leftmost column (because of \b semantics) or at the rightmost column (because xenl is a thing) or neither. And no, requesting the cursor position report from the terminal doesn't really work, it's way too slow and it's interspersed with the user input. The TUI paradigm really breaks down completely the moment the client is unsure how its output affects the cursor movement in the terminal. And terminals don't help much either! Turning off autowrap is mostly useless (the excess output is not discared, it overwrites the rightmost column instead), the autobackwrap (to make \b go to the previous line from the leftmost column) is almost unsupported and has its own issues, there is no simple command/escape sequence to go to the rightmost column... Oh, and there is xenl behaviour, which has many different subtle variations, and which original VT100 didn't even properly have despite what terminfo manual page may tell you — you can try it with the terminal emulator mentioned in TFA for yourself: go to setup, press 4, 5, move with right arrow to the block 3 and turn the second bit in it on by pressing 6 so it looks like "3 0100", exit setup (what you did is put the temrinal into the local mode so you can input text to it from your keyboard and turned the autowrap on), then do ESC, print "[1;79Hab", do LINE-FEED, print "cd" — you'll see that there is an empty line which shouldn't really be there, and it is not there if you do e.g. printf "\033[1;1Hxx\033[1;79Hab\ncd" on xterm (ironic, given how xterm's maintainer prides themself on being very faithful to original VT100 behaviour) or any other modern terminal.
account42 1y ago
It's more down to whatever monospace font the terminal uses not having those emojis and the (likely proportional) font they come from giving them a different width.

crackalamoo 1y ago

Yeah, unfortunately I feel like despite all the advances in Unicode tech, my modern terminal (MacOS) still bugs out badly with emojis and certain special characters. I'm not sure how/when codepoints matter for wcwidth: my terminal handles many characters with more than one codepoint in UTF-8, like é and even Arabic characters, just fine.

o11c 1y ago
`wcwidth` works by assigning all codepoints (strictly, code units of whatever size `wchar_t` is on your system, but thankfully modern Unixen are sane) a width of -1 (error), 0 (combining), 1 (narrow), or 2 (wide). `wcswidth` could in theory work across multiple codepoints, but its API is braindead and cannot deal with partial errors. This is all from the perspective of what the application expects to output. What the terminal itself does might be something completely different - decomposed Hangul in particular tends to lead to rendering glitches in curses-based terminal programs. This is also different from what the (monospace) font expects to be rendered as. At least it has the excuse of not being able to call the system's `wcwidth`. Note that it is always a mistake to call an implementation of `wcwidth` other than the one provided by the OS, since that introduces additional mismatches, unless you are using a better API that calculates bounds rather than an exact width. I posted an oversimplified sketch (e.g. it doesn't include versioning) of that algorithm a while back ... https://news.ycombinator.com/item?id=43851532 https://news.ycombinator.com/item?id=43851532
- PhilipRoman 1y ago
  As fallback, you can also just emit the character and see how far the cursor advanced via CSI 6n (try printf '\x1b[6n')
  - o11c 1y ago
    Doing that adds a lot of round trips, so you still really need to do the initial estimate. (also, probing for whether the terminal actually supports various features is nontrivial. At startup you can send the basic "identify the terminal" sequences (there are 2) and check the result with a timeout; subsequently you can make a request then follow it with the basic terminal id to see if you actually get what you requested. But remember you can get arbitrary normal input interspersed.)

duped 1y ago

Why do you need to sync with the shaping engine? TBH grapheme clusters are annoying but day 1 learning material for a text display widget that supports beyond ascii. It honestly irks me how many things just fuck it up, because it's not an intractably hard problem - just annoying enough to be intractable for people that are lazy (*). (*) the actually hard problem with grapheme clusters is that they're potentially unbounded in length and the standard is mutable, so your wcwidth() implementation needs to be updated along with standards to stay valid, particularly with emoji. This basically creates a software maintenance burden out of aether.

inetknght 1y ago
> This basically creates a software maintenance burden out of aether. So... basically all modern software?
zarzavat 1y ago
> Why do you need to sync with the shaping engine? GP explained already. Grapheme clusters ≠ glyphs. To find the number of glyphs you need the font. An emoji can render as one or two or three or more glyphs depending on what font the user has installed, because many emoji are formed by joining two or more emoji by a ZWJ) (Also even in a monospace font not all glyphs are of ﷽ equal width)
- layer8 1y ago
  It's not the font that is deciding how emoji sequences are rendered. The renderer may decide based on which characters exist in the available fonts, but it doesn't have to. Same for glyph width in terminals. It wasn’t uncommon for non-double-width-aware terminals to only draw half an emoji in a regular-width cell.
  - zarzavat 1y ago
    How else are you going to render a sequence such as Emoji ZWJ Emoji other than as two glyphs, if no composed glyph is defined in the user's font? That's how it's supposed to be rendered, for backwards compatibility.

Joker_vD 1y ago

The main problem is not even if the terminal itself can track the grapheme width "correctly". It's a) the fonts suck; b) does the terminal user tracks the width correctly? About a): some fonts have the glyphs for e.g. the playing cards block that are 1.5 columns wide even though the code points themselves are defined to be Narrow. How do you render that properly? Then there are variation selectors: despite what some may think, they don't affect the East Asian Width of the preceding code point, so whether you print "\N{ALEMBIC}\N{VARIATION SELECTOR-15}" or "\N{ALEMBIC}\N{VARIATION SELECTOR-16}", it still, according to wcwidth(), takes 1 column; but fonts have glyphs that are, again, 1.5 and 2 cells wide. And then there is the elephant in the room problem b) which is management of cursor position. But the terminal, and the app that uses the terminal need to have exactly the same idea of where the cursor is, or e.g. readline can't reliably function, or colorful grep output. You need to know how many lines of text you've output (to be able to erase them properly), and whether the cursor is at the leftmost column (because of \b semantics) or at the rightmost column (because xenl is a thing) or neither. And no, requesting the cursor position report from the terminal doesn't really work, it's way too slow and it's interspersed with the user input. The TUI paradigm really breaks down completely the moment the client is unsure how its output affects the cursor movement in the terminal. And terminals don't help much either! Turning off autowrap is mostly useless (the excess output is not discared, it overwrites the rightmost column instead), the autobackwrap (to make \b go to the previous line from the leftmost column) is almost unsupported and has its own issues, there is no simple command/escape sequence to go to the rightmost column... Oh, and there is xenl behaviour, which has many different subtle variations, and which original VT100 didn't even properly have despite what terminfo manual page may tell you — you can try it with the terminal emulator mentioned in TFA for yourself: go to setup, press 4, 5, move with right arrow to the block 3 and turn the second bit in it on by pressing 6 so it looks like "3 0100", exit setup (what you did is put the temrinal into the local mode so you can input text to it from your keyboard and turned the autowrap on), then do ESC, print "[1;79Hab", do LINE-FEED, print "cd" — you'll see that there is an empty line which shouldn't really be there, and it is not there if you do e.g. printf "\033[1;1Hxx\033[1;79Hab\ncd" on xterm (ironic, given how xterm's maintainer prides themself on being very faithful to original VT100 behaviour) or any other modern terminal.

account42 1y ago

It's more down to whatever monospace font the terminal uses not having those emojis and the (likely proportional) font they come from giving them a different width.