3 ms·
The key observation to me here is that the character cell grid abstraction that terminals are built upon is poor fit for modern general-purpose multi-lingual te
by zokier 3y ago
The key observation to me here is that the character cell grid abstraction that terminals are built upon is poor fit for modern general-purpose multi-lingual text rendering. Works well enough with plain ASCII but anything beyond that gets increasingly messy
- lor_louis 3y agoI would argue that strings as an array of individually addressable 'characters' is a bad fit for modern multilingual text handling.
- _a_a_a_ 3y agoWhy? How does it fail?
- lor_louis 3y agoIn utf-8, bytes (uint8_t) may not represent a whole "code point". A code point being an individually meaningful element in utf-8 like a space an 'e' or a modifier code point like an accent or a ZWJ. Most utf-8 libraries will let you address individual code points but it might still garble the text if you split between an 'e' and a '`'. To prevent this, splitting should be done in between graphemes (sequences of code points that render like a single unit*). And even graphemes have their problems. Very interesting blog post about graphemes that parallels my experience writing a terminal text editor: https://mitchellh.com/writing/grapheme-clusters-in-terminals https://mitchellh.com/writing/grapheme-clusters-in-terminals * Event that is not a proper description of a grapheme And don't even get me started on regexes
- _a_a_a_ 3y agoYes, I understand a little about Unicode in this kind of problem, but a code point is an individual logical item even if it is composed of multiple bytes; being a kind of 'string' in itself. I should have asked more carefully, what would be a better system in your view? Thanks for the link, will check it out after Christmas.
- lor_louis 3y agoI personally believe that Swift's strings where graphemes are the smallest indexable unit are the gold standard for writing logic that might truncate multilingual text. It's still not perfect though, they add overhead and updates to Unicode might change behaviour so there's that but it should handle most cases gracefully.
- eviks 3y agoIt fails miserably even with ASCII, works only with a monospace variety of it, and non-monospace isn't modern...
- bee_rider 3y agoThat’s an interesting thought, IMO. These flexible width fonts are often thought of as localization-assistance features, to help fit other languages in the terminal world. But English also had to be modified to fit into fixed-width fonts. I wonder how the discussion would change if this technology had been popularized originally in another country.
- Per_Bothner 3y agoI am working on a re-write (https://github.com/xtermjs/xterm.js/issues/4800 https://github.com/xtermjs/xterm.js/issues/4800) of the xterm.js "BufferLine" data structures that (among other benefits) could potentially support variable-width fonts in the terminal. Instead of a 2-dimensional grid of character cells, the primary data structure will be a list of logical lines, each consisting of characters and attributes. A rather modest enhancement to the vt100/xterm escape sequences would be a mode where screen addressing would be in terms of logical characters and logical lines. The application should not need to know the width of characters or how lines are broken into screen rows. This makes use of variable-width fonts practical - which is necessary for decent rendering of many scripts. DomTerm's display engine (https://domterm.org https://domterm.org) does support lines containing a mix of monospace and variable width characters, and know how to wrap such lines. I'm working on long-term replacing DomTerm's native display engine with xterm.js (for speed and other reasons) which requires making xterm.js more versatile.