5 ms·

The macOS LC_COLLATE hunt: Or why does sort order differently on macOS and Linux (2020)

97 points by g0xA52A2A 1y ago

deleted 1y ago
[deleted]
loeg 1y ago
(2020)
OptionOfT 1y ago
Updated link to the file as https://opensource.apple.com/source/adv_cmds/adv_cmds-118/usr-share-locale.tproj/colldef/la_LN.US-ASCII.src https://opensource.apple.com/source/adv_cmds/adv_cmds-118/us... doesn't work anymore: https://github.com/apple-oss-distributions/adv_cmds/blob/adv_cmds-118/usr-share-locale.tproj/colldef/la_LN.US-ASCII.src https://github.com/apple-oss-distributions/adv_cmds/blob/adv...
skopje 1y ago
So the ISO way is the right way, right?
- dataflow 1y ago
  I wondered the same. What's the right ordering?
- monerozcash 1y ago
  The right way is the one that you choose yourself and suits your needs. There's no default right answer to this, as the answer depends entirely on what you're sorting and how you want it sorted. Even for a given character set the "correct" alphabetical sorting is still locale dependent. And even knowing all that, "correct" programmatic sorting might still be essentially impossible. Some digraphs may be sorted differently depending on the specific word. For example A vs Aa, where Aa means Å. But Aa won't always necessarily mean Å, so good luck figuring that out.
asveikau 1y ago
Sorting is language specific even if you're restricted to languages using Latin characters. Eg. How do you sort N relative to Ñ? How do you treat the Turkish variations on the letter I? Doing a dumb sort by character or byte values is obviously the wrong call for any diacritics, but the right call may also depend on the language.
- dmurray 1y ago
  And that's why there are a hundred different possible values for LC_COLLATE, and it's completely normal that two popular Unix distributions picked different default values for that setting...right? It would have been reasonable to conclude the article a third of the way through, and say "sorting is locale-dependent, if what you value is consistent behaviour between different OSs (instead of sorting based on the user's preferences) you need to implement the sorting yourself."
  - harrall 1y ago
    LC_ALL=C which gives you consistent sorting behavior. The article does mention it but in passing.
- encom 1y ago
  Before the Danish language adopted the letter "å" (in 1948), the vowel was written as "aa". In the Danish alphabet, "å" is the last letter. Therefore a list of three Danish city names would be correctly sorted as: * Albertslund * Odense * Aarhus This feels like material for another Tom Scott video.
  - tpmoney 1y ago
    Not Tom Scott, but Dylan Beattie has done a handful of interesting talks[1] effectively on "there's no such thing as plain text" which in part covers this sort of thing. In fact, I think your Danish cities list is actually one of his examples. [1]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo https://www.youtube.com/watch?v=gd5uJ7Nlvvo
pjmlp 1y ago
Yet another one of those POSIX and ISO things that most people don't bother to know about. https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.html https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...
greesil 1y ago
It's not a stable sort?
o11c 1y ago
Minor note: on Debian (and possibly other distros), you don't have to use `locale-gen` to dynamically build things into `$complocaledir/locale-archive` (which, incidentally, can cause random breakage for programs that happen to start during system upgrades). The `locales-all` package works more like macOS. It's only a ~10MB download but unpacks to take ~250MB of disk space (these numbers will vary based on your libc version and packaging format). There are a lot of sparse arrays and UTF32 character data in compiled locales. Incidentally, the command to dump a locale's data is: LC_ALL=whatever locale -ck `locale | sed 's/=.*//; /LANG\|LC_ALL/d'`
1a527dd5 1y ago
Ask anyone who did a postgres upgrade. The words "collate" and "glibc" are enough to cause me to pause now. Learnt loads, never going to really use it again, but man do I understand the pain that causes now.
bluedino 1y ago
Now I'm remembering all the fun we had a long time ago with php websites that used an AS/400 for a data source. They didn't sort the same, and the mom and pop web dev shop that was hired to create the web site didn't understand the issue and hacked around it and failed.
kenada 1y ago
When I updated the Darwin SDK and source releases in nixpkgs last year, I tried using the FreeBSD locale data. It worked in a technical sense, but it broke things that depended on the quirks in the Apple’s locale data. That statement about compatibility is unfortunately true.
kbd 1y ago
In my Zsh startup on Mac I had to worry about collation, as I expected ~ to sort last (I have a directory prefixed with ~ to load plugins that need to be loaded last). Idk why a locale of utf-8 has it sorting differently, but I needed LC_COLLATE=C to have it sort as expected: # source all shell config export LC_COLLATE=C # ensure consistent sort, ~ at end for file in ~/bin/shell/**/*.(z|)sh; do source "$file"; done

97 points by g0xA52A2A 1y ago

deleted 1y ago

[deleted]

loeg 1y ago

(2020)

OptionOfT 1y ago

Updated link to the file as https://opensource.apple.com/source/adv_cmds/adv_cmds-118/usr-share-locale.tproj/colldef/la_LN.US-ASCII.src https://opensource.apple.com/source/adv_cmds/adv_cmds-118/us... doesn't work anymore: https://github.com/apple-oss-distributions/adv_cmds/blob/adv_cmds-118/usr-share-locale.tproj/colldef/la_LN.US-ASCII.src https://github.com/apple-oss-distributions/adv_cmds/blob/adv...

skopje 1y ago

So the ISO way is the right way, right?

dataflow 1y ago
I wondered the same. What's the right ordering?
monerozcash 1y ago
The right way is the one that you choose yourself and suits your needs. There's no default right answer to this, as the answer depends entirely on what you're sorting and how you want it sorted. Even for a given character set the "correct" alphabetical sorting is still locale dependent. And even knowing all that, "correct" programmatic sorting might still be essentially impossible. Some digraphs may be sorted differently depending on the specific word. For example A vs Aa, where Aa means Å. But Aa won't always necessarily mean Å, so good luck figuring that out.

asveikau 1y ago

Sorting is language specific even if you're restricted to languages using Latin characters. Eg. How do you sort N relative to Ñ? How do you treat the Turkish variations on the letter I? Doing a dumb sort by character or byte values is obviously the wrong call for any diacritics, but the right call may also depend on the language.

dmurray 1y ago
And that's why there are a hundred different possible values for LC_COLLATE, and it's completely normal that two popular Unix distributions picked different default values for that setting...right? It would have been reasonable to conclude the article a third of the way through, and say "sorting is locale-dependent, if what you value is consistent behaviour between different OSs (instead of sorting based on the user's preferences) you need to implement the sorting yourself."
- harrall 1y ago
  LC_ALL=C which gives you consistent sorting behavior. The article does mention it but in passing.
encom 1y ago
Before the Danish language adopted the letter "å" (in 1948), the vowel was written as "aa". In the Danish alphabet, "å" is the last letter. Therefore a list of three Danish city names would be correctly sorted as: * Albertslund * Odense * Aarhus This feels like material for another Tom Scott video.
- tpmoney 1y ago
  Not Tom Scott, but Dylan Beattie has done a handful of interesting talks[1] effectively on "there's no such thing as plain text" which in part covers this sort of thing. In fact, I think your Danish cities list is actually one of his examples. [1]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo https://www.youtube.com/watch?v=gd5uJ7Nlvvo

pjmlp 1y ago

Yet another one of those POSIX and ISO things that most people don't bother to know about. https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap07.html https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...

greesil 1y ago

It's not a stable sort?

o11c 1y ago

Minor note: on Debian (and possibly other distros), you don't have to use `locale-gen` to dynamically build things into `$complocaledir/locale-archive` (which, incidentally, can cause random breakage for programs that happen to start during system upgrades). The `locales-all` package works more like macOS. It's only a ~10MB download but unpacks to take ~250MB of disk space (these numbers will vary based on your libc version and packaging format). There are a lot of sparse arrays and UTF32 character data in compiled locales. Incidentally, the command to dump a locale's data is: LC_ALL=whatever locale -ck `locale | sed 's/=.*//; /LANG\|LC_ALL/d'`

1a527dd5 1y ago

Ask anyone who did a postgres upgrade. The words "collate" and "glibc" are enough to cause me to pause now. Learnt loads, never going to really use it again, but man do I understand the pain that causes now.

bluedino 1y ago

Now I'm remembering all the fun we had a long time ago with php websites that used an AS/400 for a data source. They didn't sort the same, and the mom and pop web dev shop that was hired to create the web site didn't understand the issue and hacked around it and failed.

kenada 1y ago

When I updated the Darwin SDK and source releases in nixpkgs last year, I tried using the FreeBSD locale data. It worked in a technical sense, but it broke things that depended on the quirks in the Apple’s locale data. That statement about compatibility is unfortunately true.

kbd 1y ago

In my Zsh startup on Mac I had to worry about collation, as I expected ~ to sort last (I have a directory prefixed with ~ to load plugins that need to be loaded last). Idk why a locale of utf-8 has it sorting differently, but I needed LC_COLLATE=C to have it sort as expected: # source all shell config export LC_COLLATE=C # ensure consistent sort, ~ at end for file in ~/bin/shell/**/*.(z|)sh; do source "$file"; done