3 ms·

>With Unicode, sorting and case conversion are neigh impossible to do correctly Surely you mean that sorting correctly is impossible without Unicode? Otherwise

>With Unicode, sorting and case conversion are neigh impossible to do correctly Surely you mean that sorting correctly is impossible without Unicode? Otherwise you would have to hardcode the rules of sorting strings correctly in my language (and all other languages) yourself. Unless your preferred solution is "close my eyes and prefer non-ascii characters don't exist", then... I'm not a fan.

samatman 2y ago
Sorting is impossible to do correctly without knowledge of the language in which the text is written, because the collation rules for symbols differ between languages. Unicode, of course, defines those collation rules, and UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII. Case conversion is similar except the default rules do a very good job in general. But still, there are a few language-specific quirks and, again, you do have to know what language is involved to get those right. I'm agreeing with you, to be clear, just adding that a) Unicode isn't always enough, but it does a decent job if you don't know the language in advance, and that it defines the correct rules if you do know that.
- sgarland 2y ago
  > UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII This isn’t necessarily true beyond ASCII, and it depends entirely on the collation [0]. One need only to spend some time peering into the abyss that is RDBMS collation support [1] [2] to see the horror. [0]: http://www.unicode.org/reports/tr10/ http://www.unicode.org/reports/tr10/ [1]: https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets.html https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets... [2]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html
  - samatman 2y ago
    Well, no. It sorts according to the lexiographical order of Unicode. Earlier points before later points. How useful this is depends on the language, of course. I did say that as well. But Unicode was put together from legacy character encodings, and did what it could to preserve the order of those, so it's far from useless.

samatman 2y ago

Sorting is impossible to do correctly without knowledge of the language in which the text is written, because the collation rules for symbols differ between languages. Unicode, of course, defines those collation rules, and UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII. Case conversion is similar except the default rules do a very good job in general. But still, there are a few language-specific quirks and, again, you do have to know what language is involved to get those right. I'm agreeing with you, to be clear, just adding that a) Unicode isn't always enough, but it does a decent job if you don't know the language in advance, and that it defines the correct rules if you do know that.

sgarland 2y ago
> UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII This isn’t necessarily true beyond ASCII, and it depends entirely on the collation [0]. One need only to spend some time peering into the abyss that is RDBMS collation support [1] [2] to see the horror. [0]: http://www.unicode.org/reports/tr10/ http://www.unicode.org/reports/tr10/ [1]: https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets.html https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets... [2]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html
- samatman 2y ago
  Well, no. It sorts according to the lexiographical order of Unicode. Earlier points before later points. How useful this is depends on the language, of course. I did say that as well. But Unicode was put together from legacy character encodings, and did what it could to preserve the order of those, so it's far from useless.