5 ms·
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the stri
by jameshart 3y ago
Why are people so interested in ‘lengths’ of strings?
The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.
𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
- MadVikingGod 3y agoThe number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
- jameshart 3y agoAsk a human. Failing that, ask an AI. There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle. Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker. Strings are language. Language is hard.
- Dylan16807 3y agohttps://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
- eviks 3y agoCount by grapheme clusters and stop right before it exceeds the limit?
- zamadatix 3y agoIsn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.
- eviks 3y agostring length is a poorly defined concept, but yes, counting real characters (grapheme clusters) counts And what "various things" do you mean that are on top the char split?
- dhosek 3y agoThe rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.
- Pannoniae 3y agoWell don't ask me, ask the service operators who limit their fields to X bytes!:D Whatever their reasoning, if they want to limit strings to a certain length, they should make it consistent so the user won't run into problems where it's accepted in one place then it runs into an error or another part of the service refuses to accept it.
- zamadatix 3y agoSaying it's the ONLY thing anyone should care about is bound to over-generalize but I do find myself agreeing the vast majority of cases are really trying to guarantee something about byte or pixel dimensions. I think the reason we often end up defaulting to string length is, for most western text, it was often much easier to deal with and Emoji is, somewhat intentionally, a key driver in breaking that assumption even for those that don't care about supporting Cuneiform or have any idea or plans on trying to support other languages from the start.
- torstenvl 3y agoHow many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?
- jameshart 3y agoIt's possible that pressing backspace needs to ADD bytes.
- gumby 3y agoWhat is that case? Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters. Is there a real world case? I'd be excited to learn it.
- torstenvl 3y ago> What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)