3 ms·

The takeaway is not "unicode is overly complex", it's that an algorithm for sorting strings consisting of any possible human language characters is super convol

The takeaway is not "unicode is overly complex", it's that an algorithm for sorting strings consisting of any possible human language characters is super convoluted, and it's pretty amazing that unicode gives us a standard collation algorithm including localizations, that works remarkably well. But... is complicated enough to implement that many environments apparently haven't felt there was enough demand to do so?

nabla9 5y ago
Unicode is overly complex, but sorting is not the reason. Unicode is overly complex at grapheme cluster level.
pilif 5y ago
> algorithm for sorting strings consisting of any possible human language characters is super convoluted not only that: it's different depending on locale and sometimes context (in Germany there's a distinction of whether you sort for a phone book listing or not). Doing it right requires basically shipping a huge database with locale specific information which is seen as "bloat" by many projects and thus made optional or not available at all. >and it's pretty amazing that unicode gives us a standard collation algorithm including localizations, that works remarkably well. it's not good enough in many cases. Thankfully there's ICU, but many environments frown upon that because it's seen as bloated. If sorting works "fine" for US-ASCII, why link to a 60MB library?
- jrochkind1 5y ago
  I meant to include ICU in what I was saying works well when I said "unicode gives us a standard collation algorithm including localizations". ICU stands for "International Components for Unicode." ICU collation is an implementation of the Unicode Collation Algorithm (which is really a suite of algorithms, or an algorithm with various parameterized branches), using the Unicode Common Locale Data Repository for localization -- that's also unicode, the CLDR is a unicode thing too, part of unicode. It's all unicode. There are other implementations than ICU -- although maybe none as complete, because it is a lot/expensive, indeed. But ICU is an implementation of unicode algorithms. But for instance, unicode collation is also implemented -- in a way that will be the same as ICU modulo bugs or details the standard leaves up to implementations to vary -- in pure ruby by https://github.com/twitter/twitter-cldr-rb#sorting-collation https://github.com/twitter/twitter-cldr-rb#sorting-collation
shellac 5y ago
The only language / framework I've used that seemed to get this right is OpenStep / Cocoa, which is presumably why Swift is mentioned as being 'notable'. NSString would occasionally come up in performance analysis as being strangely slow compared with other languages, but this was because various operations like comparisons followed unicode.
jrochkind1 5y ago
PS: Did you know the unicode collation algorithms also includes a way to produce a sequence of bytes for a unicode string, such that sorting the produced sequence of bytes in strict byte order will produce the proper collation order for the original strings? Which is how you implement high-performance unicode collation sorting in say a db. Unicode is a very well-designed thing!
- dolmen 5y ago
  So, do you mind sharing the name of that transformation?

nabla9 5y ago

Unicode is overly complex, but sorting is not the reason. Unicode is overly complex at grapheme cluster level.

pilif 5y ago

> algorithm for sorting strings consisting of any possible human language characters is super convoluted not only that: it's different depending on locale and sometimes context (in Germany there's a distinction of whether you sort for a phone book listing or not). Doing it right requires basically shipping a huge database with locale specific information which is seen as "bloat" by many projects and thus made optional or not available at all. >and it's pretty amazing that unicode gives us a standard collation algorithm including localizations, that works remarkably well. it's not good enough in many cases. Thankfully there's ICU, but many environments frown upon that because it's seen as bloated. If sorting works "fine" for US-ASCII, why link to a 60MB library?

jrochkind1 5y ago
I meant to include ICU in what I was saying works well when I said "unicode gives us a standard collation algorithm including localizations". ICU stands for "International Components for Unicode." ICU collation is an implementation of the Unicode Collation Algorithm (which is really a suite of algorithms, or an algorithm with various parameterized branches), using the Unicode Common Locale Data Repository for localization -- that's also unicode, the CLDR is a unicode thing too, part of unicode. It's all unicode. There are other implementations than ICU -- although maybe none as complete, because it is a lot/expensive, indeed. But ICU is an implementation of unicode algorithms. But for instance, unicode collation is also implemented -- in a way that will be the same as ICU modulo bugs or details the standard leaves up to implementations to vary -- in pure ruby by https://github.com/twitter/twitter-cldr-rb#sorting-collation https://github.com/twitter/twitter-cldr-rb#sorting-collation

shellac 5y ago

The only language / framework I've used that seemed to get this right is OpenStep / Cocoa, which is presumably why Swift is mentioned as being 'notable'. NSString would occasionally come up in performance analysis as being strangely slow compared with other languages, but this was because various operations like comparisons followed unicode.

jrochkind1 5y ago

PS: Did you know the unicode collation algorithms also includes a way to produce a sequence of bytes for a unicode string, such that sorting the produced sequence of bytes in strict byte order will produce the proper collation order for the original strings? Which is how you implement high-performance unicode collation sorting in say a db. Unicode is a very well-designed thing!

dolmen 5y ago
So, do you mind sharing the name of that transformation?