3 ms·
I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width cha
by PhilipRoman 3y ago
I cannot think of a single common case where grapheme cluster count is important. If you want to print them aligned to a terminal - guess what, double width characters exist, so the only reliable way is to print them first, measure the cursor movement using escape sequences, calculate length and erase the originally printed data.
Even for limiting input field sizes byte count is much better, as otherwise you are opening up yourself for unicode denial of service. I think the game Minecraft has such an exploit where you can fit in absurd amounts of utf-8 data (to the point of data corruption in multiplayer games) since it's limited by visual length.
My personal favorite dealing with UTF-8: pretend it's ascii and assume everything above 128 is an alphabetic character. It just works. For 99% of use cases it doesn't matter if the content is emojis, families of emojis, or ancient sumerian scripts. You can parse JSON and most other formats this way without caring about code points at all. The trend of unicodizing everything was a mistake, just treat strings as bytes and parse them as utf-8 only when you really need it (like when building a text editor or a browser engine from scratch).
- zkldi 3y agothis doesn't "just work" at all? utf-8 is a variable width encoding and if you treat it like ascii your software just isn't going to work outside of the ascii range.
- PhilipRoman 3y agoBut it does - the genius of utf-8 is that it was deliberately designed to be backwards compatible (it even preserves the ascii sorting order). You can run C programs written before utf-8 was invented with utf-8 inputs (unlike with the abomination that is utf-16). If a code point is outside the ascii range (0-127 inclusive), then it's utf-8 encoding is also guaranteed to not contain any ascii bytes. So as long as you treat anything inside 128-255 as "some unknown character", the utf-8 code points will be preserved and eventually displayed when the byte sequence is parsed as utf-8 by your terminal/browser/whatever.
- vlmutolo 3y agoI think tweets are restricted by number of extended grapheme clusters. And probably also by number of bytes.