3 ms·
> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very h
by xg15 7y ago
> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful.
So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m < n bytes.
How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when rendered? (E.g., the truncated string doesn't suddenly contain any glyphs or grapheme clusters that weren't in the original string)
- Tyr42 7y agohttps://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a521485a30890b1f79a0888fe107abe7 https://play.rust-lang.org/?version=stable&mode=debug&editio... Something like this? Check if each character pushes the byte total over the limit? I think this might fail for combining characters though.
- samatman 7y agoAvoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point. The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.
- smasher164 7y agoIt's not though. Replacing invalid byte sequences is not terribly difficult. https://golang.org/src/strings/strings.go?s=15854:15900#L627 https://golang.org/src/strings/strings.go?s=15854:15900#L627.
- samatman 7y agoWe are agreeing, the part I was indicating is difficult is 'displaying gibberish'. Knowing what constitutes a grapheme cluster is detailed and frequently changes.
- toast0 7y ago> How do I correctly truncate the string? Refuse to accept a string that is overlong, and require an interactive user (hopefully one literate in the language) to truncate it for you. In a non-interactive context, you can't.
- seppel 7y ago> How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when rendered? (E.g., the truncated string doesn't suddenly contain any glyphs or grapheme clusters that weren't in the original string) Cropping strings is a hard problem for ASCII strings as well. It can even be a security problem if the cropped part contains important information that alters the meaning of the first part (Something like "DELETE FROM table_name [WHERE condition]" or natural language where the cropped part is the condition or the negation). But even if you dont care about this: If you care about cropping visually nicely, you want some ellipsis at the end, you dont want to crop in the middle of the word (if possible), etc. In the end, you need some nice text processing anyway. Or you reject strings that are too long.
- naniwaduni 7y agoWhat do you think distinguishes the truncated string containing "glyphs or grapheme clusters" that weren't in the original string from the truncated string containing words that weren't in the original string? Is the latter somehow more acceptable? How about missing necessary context from the end of a sentence?
- sunyc 7y agouse U8_TRUNCATE macros in icu4c