5 ms·

It's not Unicode sorting is hard, it's mostly because CJK, ideograms, emojis, and many other non-alphabetical languages do not have order. It's not a bug, it's

by est 5y ago

It's not Unicode sorting is hard, it's mostly because CJK, ideograms, emojis, and many other non-alphabetical languages do not have order. It's not a bug, it's by design.

lifthrasiir 5y ago
> [...] CJK, ideograms, emojis, and many other non-alphabetical languages do not have order. Uh, no? Most if not all scripts do have letter order, CJK(V) "ideographs" included. The actual problem is twofold: the letter order might not be unique, and the word order might be completely different from the letter order. CJK(V) "ideographs" exhibit both problems---their word order is phonetic, while letter order is based on shapes and different dictionaries use different rules---and it is very common in Chinese and Japanese that the string meant to be sortable has pronunciation attached (e.g. Wikipedia). Also, believe it or not, even emojis do have order too [1], albeit arbitrary one. [1] https://unicode.org/emoji/charts/emoji-ordering.html https://unicode.org/emoji/charts/emoji-ordering.html
- naniwaduni 5y ago
  Even English a few edge cases where collation order doesn't match letter order: titles are typically ordered ignoring initial A/The, names are often sorted by surname (even when written with given name first); famously, M'/Mc are are traditionally sorted with Mac, along with a host of special cases that get gleefully thrown out in this era of doing very dumb very fast.
gumby 5y ago
It's not sensible to call out non-alphabetic (by which I imagine you exclude all phonemic systems including abjads, syllabaries, etc all of which have full character order. Even in Latin scripts, sorting order is language-specific and can even be domain specific (e.g. some languages sort phone books differently from dictionaries). For "CJKV" systems there are ordering methods such as radicals and, as in European languages, ordering can depend on what you want to accomplish.
minikomi 5y ago
Speaking for Japanese, there is certainly an "order" for words, but that order is based on the readings of words, which may be difficult to ascertain from just the string. https://en.m.wikipedia.org/wiki/Goj%C5%ABon https://en.m.wikipedia.org/wiki/Goj%C5%ABon Eg. Words starting with 一人 can be read as both いちにん (ichinin as in 一人前 - food for one person) or ひとり (hitori - 一人暮らし - living alone)
- glandium 5y ago
  Also some independent words can have different readings. Eg. 紅葉 is both こうよう and もみじ.
  - naniwaduni 5y ago
    大人気 is my favourite of those, as a particularly long ambiguous homograph where the two common readings don't even have overlapping meanings.
    - glandium 5y ago
      Indeed, that one can be funny especially when split over a line ending. You start reading it as one, and no, it was the other. I've had this experience several times with this specific word.
    - beeforpork 5y ago
      (1) 大人気 (おとなげ, otonage) - maturity (2) 大人気 (だいにんき, daininki) - very popular
esjeon 5y ago
FYI, Korean alphabet (Hangul) is fully ordered[1], and Korean unicode block[2] strictly follows it. [1]: https://en.wikipedia.org/wiki/Hangul#Alphabetic_order https://en.wikipedia.org/wiki/Hangul#Alphabetic_order [2]: https://unicode.org/charts/PDF/UAC00.pdf https://unicode.org/charts/PDF/UAC00.pdf
- est 5y ago
  Hangul has order because it's enumerable. Hanja on the other hand, does not have order and it's not enumerable. (not enumerable as in written text as strokes can be combined and invented on the fly, enumerable if you think it's a finite charset on computers)
  - yorwba 5y ago
    New characters can be invented on the fly, but not every random scribble would be considered a character. In practice, only a handful of different stroke types are used, which gives even newly invented nonsense characters a recognizable visual style. For any finite stroke count, there's only a finite number of combinations, so the infinte set of characters with unbounded stroke count can be enumerated. It can also be sorted using the radical and stroke count, although then you can't enumerate it in order because there's an infinite number of possible characters for each radical.
- oehtXRwMkIs 5y ago
  It's so annoying how "CJK" groups Hangul with completely different writing systems. And even makes people think they're similar.
  - gumby 5y ago
    Well, Hanja can also appear inline in otherwise Hangul texts. Vietnamese has a similar issue though their use of an extended Roman alphabet ended up in a slightly different place, Unicode-wise. I don't remember the decision but I believe it might have been at the request of some RoK people. There might also be an issue if old Hanja-only texts need to be handled differently from Hanzi rendering/processing. That would not surprise me. Still, I agree it seems pretty weird.
nitwit005 5y ago
Even English has no defined ordering for all characters. What is the ordering of punctuation marks? Do upper case or lower case letters go first? How do you deal with white space characters? ASCII ordering is just an arbitrary standard that mapped numbers to letters so people could make computer hardware interoperate. It's not always a sensible sort ordering.
- mjevans 5y ago
  The way most native English speakers (and writers) that I know, including myself, would sort such a set is to ignore the set (regexp) \W . The 'C' (blind bit sorting) Locale is more sensitive and a suitable default for simple machine focused environments. E.G. of sorting how English humans tend to: `A string with non-letters ignores the non-letters. Ace
- tyingq 5y ago
  Pedantic, I know, but the original 1963 ASCII document did speak a lot about collation and ordering. Perhaps a lot of it doesn't make sense today, but the sort order wasn't accidental. http://www.bahaistudies.net/asma/ascii.pdf http://www.bahaistudies.net/asma/ascii.pdf
- thaumasiotes 5y ago
  > Even English has no defined ordering for all characters. What is the ordering of punctuation marks? Do upper case or lower case letters go first? How do you deal with white space characters? This isn't really true; there are well-defined, standardized answers to all of those: - For purposes of alphabetizing, punctuation marks are completely ignored, as if they weren't present. "Stop me!" and "Stop, me!" are identical strings. - Case is also ignored. "AAAAAGH" and "aaaAagh" are identical strings. - Spaces are not ignored. Sort order is determined by the text before the first space. Shorter sorts before longer. "A bird", "a cat", and "a deer" all sort under "a" and above "ab". Note that these are rules for English, not rules for arbitrary sequences of binary data. They look the way they do because they model the data as a sequence of English words.
  - kps 5y ago

by est 5y ago

It's not Unicode sorting is hard, it's mostly because CJK, ideograms, emojis, and many other non-alphabetical languages do not have order. It's not a bug, it's by design.

lifthrasiir 5y ago

> [...] CJK, ideograms, emojis, and many other non-alphabetical languages do not have order. Uh, no? Most if not all scripts do have letter order, CJK(V) "ideographs" included. The actual problem is twofold: the letter order might not be unique, and the word order might be completely different from the letter order. CJK(V) "ideographs" exhibit both problems---their word order is phonetic, while letter order is based on shapes and different dictionaries use different rules---and it is very common in Chinese and Japanese that the string meant to be sortable has pronunciation attached (e.g. Wikipedia). Also, believe it or not, even emojis do have order too [1], albeit arbitrary one. [1] https://unicode.org/emoji/charts/emoji-ordering.html https://unicode.org/emoji/charts/emoji-ordering.html

naniwaduni 5y ago
Even English a few edge cases where collation order doesn't match letter order: titles are typically ordered ignoring initial A/The, names are often sorted by surname (even when written with given name first); famously, M'/Mc are are traditionally sorted with Mac, along with a host of special cases that get gleefully thrown out in this era of doing very dumb very fast.

gumby 5y ago

It's not sensible to call out non-alphabetic (by which I imagine you exclude all phonemic systems including abjads, syllabaries, etc all of which have full character order. Even in Latin scripts, sorting order is language-specific and can even be domain specific (e.g. some languages sort phone books differently from dictionaries). For "CJKV" systems there are ordering methods such as radicals and, as in European languages, ordering can depend on what you want to accomplish.

minikomi 5y ago

Speaking for Japanese, there is certainly an "order" for words, but that order is based on the readings of words, which may be difficult to ascertain from just the string. https://en.m.wikipedia.org/wiki/Goj%C5%ABon https://en.m.wikipedia.org/wiki/Goj%C5%ABon Eg. Words starting with 一人 can be read as both いちにん (ichinin as in 一人前 - food for one person) or ひとり (hitori - 一人暮らし - living alone)

glandium 5y ago
Also some independent words can have different readings. Eg. 紅葉 is both こうよう and もみじ.
- naniwaduni 5y ago
  大人気 is my favourite of those, as a particularly long ambiguous homograph where the two common readings don't even have overlapping meanings.
  - glandium 5y ago
    Indeed, that one can be funny especially when split over a line ending. You start reading it as one, and no, it was the other. I've had this experience several times with this specific word.
  - beeforpork 5y ago
    (1) 大人気 (おとなげ, otonage) - maturity (2) 大人気 (だいにんき, daininki) - very popular

esjeon 5y ago

FYI, Korean alphabet (Hangul) is fully ordered[1], and Korean unicode block[2] strictly follows it. [1]: https://en.wikipedia.org/wiki/Hangul#Alphabetic_order https://en.wikipedia.org/wiki/Hangul#Alphabetic_order [2]: https://unicode.org/charts/PDF/UAC00.pdf https://unicode.org/charts/PDF/UAC00.pdf

est 5y ago
Hangul has order because it's enumerable. Hanja on the other hand, does not have order and it's not enumerable. (not enumerable as in written text as strokes can be combined and invented on the fly, enumerable if you think it's a finite charset on computers)
- yorwba 5y ago
  New characters can be invented on the fly, but not every random scribble would be considered a character. In practice, only a handful of different stroke types are used, which gives even newly invented nonsense characters a recognizable visual style. For any finite stroke count, there's only a finite number of combinations, so the infinte set of characters with unbounded stroke count can be enumerated. It can also be sorted using the radical and stroke count, although then you can't enumerate it in order because there's an infinite number of possible characters for each radical.
oehtXRwMkIs 5y ago
It's so annoying how "CJK" groups Hangul with completely different writing systems. And even makes people think they're similar.
- gumby 5y ago
  Well, Hanja can also appear inline in otherwise Hangul texts. Vietnamese has a similar issue though their use of an extended Roman alphabet ended up in a slightly different place, Unicode-wise. I don't remember the decision but I believe it might have been at the request of some RoK people. There might also be an issue if old Hanja-only texts need to be handled differently from Hanzi rendering/processing. That would not surprise me. Still, I agree it seems pretty weird.

nitwit005 5y ago

Even English has no defined ordering for all characters. What is the ordering of punctuation marks? Do upper case or lower case letters go first? How do you deal with white space characters? ASCII ordering is just an arbitrary standard that mapped numbers to letters so people could make computer hardware interoperate. It's not always a sensible sort ordering.

mjevans 5y ago
The way most native English speakers (and writers) that I know, including myself, would sort such a set is to ignore the set (regexp) \W . The 'C' (blind bit sorting) Locale is more sensitive and a suitable default for simple machine focused environments. E.G. of sorting how English humans tend to: `A string with non-letters ignores the non-letters. Ace
tyingq 5y ago
Pedantic, I know, but the original 1963 ASCII document did speak a lot about collation and ordering. Perhaps a lot of it doesn't make sense today, but the sort order wasn't accidental. http://www.bahaistudies.net/asma/ascii.pdf http://www.bahaistudies.net/asma/ascii.pdf
thaumasiotes 5y ago
> Even English has no defined ordering for all characters. What is the ordering of punctuation marks? Do upper case or lower case letters go first? How do you deal with white space characters? This isn't really true; there are well-defined, standardized answers to all of those: - For purposes of alphabetizing, punctuation marks are completely ignored, as if they weren't present. "Stop me!" and "Stop, me!" are identical strings. - Case is also ignored. "AAAAAGH" and "aaaAagh" are identical strings. - Spaces are not ignored. Sort order is determined by the text before the first space. Shorter sorts before longer. "A bird", "a cat", and "a deer" all sort under "a" and above "ab". Note that these are rules for English, not rules for arbitrary sequences of binary data. They look the way they do because they model the data as a sequence of English words.
- kps 5y ago