4 ms·

The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one

The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?

jameshart 3y ago
Ask a human. Failing that, ask an AI. There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle. Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker. Strings are language. Language is hard.
Dylan16807 3y ago
https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
eviks 3y ago
Count by grapheme clusters and stop right before it exceeds the limit?
- zamadatix 3y ago
  Isn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.
  - eviks 3y ago
    string length is a poorly defined concept, but yes, counting real characters (grapheme clusters) counts And what "various things" do you mean that are on top the char split?
dhosek 3y ago
The rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.

jameshart 3y ago

Ask a human. Failing that, ask an AI. There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle. Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker. Strings are language. Language is hard.

Dylan16807 3y ago

https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/

eviks 3y ago

Count by grapheme clusters and stop right before it exceeds the limit?

zamadatix 3y ago
Isn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.
- eviks 3y ago
  string length is a poorly defined concept, but yes, counting real characters (grapheme clusters) counts And what "various things" do you mean that are on top the char split?

dhosek 3y ago

The rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.