6 ms·

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears sho

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII) (ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)

david_draco 3y ago
Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-locale.com/whitepaper/learn/learn-to-test.html https://web.archive.org/web/20160306060703/http://www.inter-... Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
- techdragon 3y ago
  I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.
jameshart 3y ago
Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. 󠁧󠁢󠁥󠁮󠁧󠁿 (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
- MadVikingGod 3y ago
  The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
  - jameshart 3y ago
    Ask a human. Failing that, ask an AI. There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle. Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker. Strings are language. Language is hard.
  - Dylan16807 3y ago
    https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
  - eviks 3y ago
    Count by grapheme clusters and stop right before it exceeds the limit?
kazinator 3y ago
When there are seven opinions on what the length is, you cannot miscount it, unless you somehow come up with some vastly unpopular eighth answer.

david_draco 3y ago

Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-locale.com/whitepaper/learn/learn-to-test.html https://web.archive.org/web/20160306060703/http://www.inter-... Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.

techdragon 3y ago
I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.

jameshart 3y ago

Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. 󠁧󠁢󠁥󠁮󠁧󠁿 (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.

MadVikingGod 3y ago
The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
- jameshart 3y ago
  Ask a human. Failing that, ask an AI. There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle. Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker. Strings are language. Language is hard.
- Dylan16807 3y ago
  https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/
- eviks 3y ago
  Count by grapheme clusters and stop right before it exceeds the limit?

kazinator 3y ago

When there are seven opinions on what the length is, you cannot miscount it, unless you somehow come up with some vastly unpopular eighth answer.