2 ms·

Prior to this article, I knew graphemes were a thing and that proper unicode software is supposed to count those instead of bytes or code points. I didn't know

Prior to this article, I knew graphemes were a thing and that proper unicode software is supposed to count those instead of bytes or code points. I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway. I'm therefore going to continue counting bytes. And comparing by memcmp. If the bytes look like unicode to some reader, fine. Opaque string as far as my software is concerned.

slimsag 3y ago
Two Unicode strings can be visually and semantically identical, but not byte-equal.
dundarious 3y ago
The point is that a byte focus will often frustrate users. e.g., a TUI with columns will have to truncate "long" strings in each column, and that truncation and column-separator arrangement really should be grapheme aware. e.g., a string search (for a name, let's say) should find Noël regardless of whether the user input ë via composing characters or the pre-composed version.
tonsky 3y ago
Good luck https://mastodon.online/@alexeyten@mas.to/111166351426290784 https://mastodon.online/@alexeyten@mas.to/111166351426290784
PeterisP 3y ago
Comparing by memcmp will result in false negatives unless you can ensure that all incoming text gets normalized to a particular canonical form.
- jandrese 3y ago
  Even then there are minefields in comparing text, especially case insensitive matching and supporting CKJ.
ssokolow 3y ago
> I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway. This is EXACTLY why Rust's standard library is blind to graphemes. Support for the case where your company requires a specially certified toolchain that lags five years behind Rust upstream is an explicit goal that they address by breaking the stuff that changes quickly out into minimal crates that can be audited, updated at a quicker pace without requiring toolchain updates, and which have the option to continue to support older compiler versions indefinitely.

slimsag 3y ago

Two Unicode strings can be visually and semantically identical, but not byte-equal.

dundarious 3y ago

The point is that a byte focus will often frustrate users. e.g., a TUI with columns will have to truncate "long" strings in each column, and that truncation and column-separator arrangement really should be grapheme aware. e.g., a string search (for a name, let's say) should find Noël regardless of whether the user input ë via composing characters or the pre-composed version.

tonsky 3y ago

Good luck https://mastodon.online/@alexeyten@mas.to/111166351426290784 https://mastodon.online/@alexeyten@mas.to/111166351426290784

PeterisP 3y ago

Comparing by memcmp will result in false negatives unless you can ensure that all incoming text gets normalized to a particular canonical form.

jandrese 3y ago
Even then there are minefields in comparing text, especially case insensitive matching and supporting CKJ.

ssokolow 3y ago

> I didn't know that unicode changes the definition of grapheme in backwards incompatible fashion annually, so software which works by grapheme count is probably inconsistent with other software using a different version of the standard anyway. This is EXACTLY why Rust's standard library is blind to graphemes. Support for the case where your company requires a specially certified toolchain that lags five years behind Rust upstream is an explicit goal that they address by breaking the stuff that changes quickly out into minimal crates that can be audited, updated at a quicker pace without requiring toolchain updates, and which have the option to continue to support older compiler versions indefinitely.