2 ms·
Truncating mid-codepoint produces an invalid utf8 sequence, which some decoders will silently ignore or replace with <?> glyphs, and others will fail loudly. T
by Retr0id 2y ago
Truncating mid-codepoint produces an invalid utf8 sequence, which some decoders will silently ignore or replace with <?> glyphs, and others will fail loudly.
Truncating mid-grapheme-cluster can change semantics in unexpected ways. For example, the "family" emoji(s) can be encoded as a set of conjoined codepoints for each family member: https://stackoverflow.com/questions/49958287/printing-family-emoji-with-u200d-zero-width-joiner-directly-vs-via-list https://stackoverflow.com/questions/49958287/printing-family.... Depending on where you truncate, you might just get the father, which probably won't cause you any serious issues but it might cause confusion depending on context.
(IMHO, if you're truncating a way where this would matter, you should probably be truncating in screen-space, i.e., way higher up the stack)
- _gabe_ 2y agoYep, this all makes sense and is what I was thinking as well. I also think you’re right about truncation being a better idea in the rendering state rather than in the bytes themselves. Either way, truncation of strings being displayed to the user requires extra care no matter what language you’re using. As far as this being the reason most C code doesn’t handle UTF-8 though, I’m still skeptical.