6 ms·

A detail skimmed over in the article, but one which has significant importance is that; "one code point in unicode does not necessarily map to one character on

A detail skimmed over in the article, but one which has significant importance is that; "one code point in unicode does not necessarily map to one character on the screen." A "character" can, and often does, get constructed from multiple code points. This doesn't help an already complicated sorting issue (who knew "sort these names alphabetically" could be an ambiguous statement).

dhosek 5y ago
That said, the Unicode standard does a great job of tackling that question and providing a framework that can cover all the various language standards (and even with a fully normalized Unicode string you still need to do things like handle ch as a single character for sorting in Czech or Spanish).
- pezezin 5y ago
  In Spanish, "ch" and "ll" stopped being considered a single character for sorting purposes in 1994. In have some dictionaries printed before that date, and I find them quite confusing...
  - dhosek 5y ago
    As someone who learned his Spanish before 1994 (and whose dictionary also predates that time), I was unaware of this until now.
- Toutouxc 5y ago
  As a Czech I would love "ch" to go fuck itself for that exact reason.
- oldsecondhand 5y ago
  In Hungarian "dzs", "zs" and "sz" are so called compund letters, however most software treats them as separate characters, because it would be a pain in the ass do sorting the correct way. (It would either require changing the input method, or software would need to be aware of etymology.) It's only sorted the right way in paper dictionaries.
  - dhosek 5y ago
    Actually, it just needs to be context sensitive. If I have an English document and I have an index which has an entry for Zsigmond, it would be placed between Zebra and Zygote. But, if I have a Hungarian document, then the correct ordering would be to have Zsigmond after Zebra and Zygote. Any correctly-implemented program that deals with sorting should not be assuming it can just sort on code point¹ and Hungarian is one of the standard collations that is part of the Unicode reference implementations so it shouldn't be a problem. The fact that a lot of software is badly-implemented is not an excuse for software to continue to be so. I did a quick check and saw that I can specify Hungarian in MySQL as a collation for a table and I also know that it's available in Java and even JavaScript. ⸻ 1. This sorting is wrong not just for Hungarian but for every language including English which would expect, e.g., naïve to be sorted between nag and nanny and not after nay.
Gigachad 5y ago
It’s only complex if your language does not support Unicode properly. In something like rust or python you would just ask for the first character rather than the first byte. And the language will calculate what that is for you.
- chmod600 5y ago
  I'm not an expert, but your statement seems confusing and possibly misses the point of the parent.
  - deleted 5y ago
    [deleted]
- torstenvl 5y ago
  A 'character' meaning what? The first code point? The first non-combining code point? The first non-combining code point along with all associated combining code points? The first non-combining code point along with all associated combining code points modified to look like it would look in conjunction with surrounding non-combining code points along with their associated combining code points? The first displayable component of a code point? What are the following? the second character of sœur (o or the ligature?) the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the third character of есть (Cyrillic t with or without soft sign, which is always a separate code point and always displayed to the right but changes the sound?) the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the first character of ﷺ or ﷽ ? The main issue isn't programming language support, it's ambiguity in the concept of "character" and conventions about how languages treat them. Imagine how the letter i would be treated if Unicode were invented by a Turk. The fundamental issue here is that human communication is deeply nuanced in a way that does not lend itself well to systematic encoding or fast/naive algorithms on that encoding. Even in the plain ASCII range it's impossible to know how to render a word like "ANSCHLUSS" in lower case (or how many 'characters' such a word would have) without knowledge of the language, country of origin, and time period in which the word was written.
  - 5y ago
Twisell 5y ago
It's not skimmed the article have a whole pararagrah entitled "Unicode To The Confusion" about that. What you infer about sorting is better described as collation and is a separate set of problems. https://www.unicode.org/collation/ https://www.unicode.org/collation/ There can be multiple collations for the same encoding and even for the same language inside the same encoding. PS: As a matter of fact in some database like PostgreSQL you can define different collations for each columns if needed.
nanis 5y ago
> "one code point in unicode does not necessarily map to one character on the screen." Also, importantly, some characters are not represented at all. For example, there is no codepoint or combination of characters that distinguishes a capital Turkish dotless I from its identical looking but conceptually different sibling the Latin capital I. Similarly for capital Turkish dotted İ. I find it extremely weird that two codepoints couldn't have been spared, yet we have gradations of skin tone in emoji. One can use composition to deal with the round-trip from i -> İ -> i (within a closed ecosystem), but even that fails when it comes to ı -> I -> ı. A case in point are the product pages for my friend's book on Amazon. Compare "Sınır Ötesi" to "Sinir Ötesi". The former means "beyond borders" whereas the latter means "exceedingly irritating". Yet, because there is no unambiguous representation of the Turkish I's, the rendering makes an assumption on the basis of the domain. Note that all other non-US ASCII letters involved are rendered correctly. \c[PERSON FROWNING, ZERO WIDTH JOINER, PROGRAMMER] [tr]: https://www.amazon.com.tr/Gezging%C3%B6z-S%C4%B1n%C4%B1r-T%C3%BCrkiye-Miras%C4%B1-Rehberi/dp/6257631181/ https://www.amazon.com.tr/Gezging%C3%B6z-S%C4%B1n%C4%B1r-T%C... [us]: https://www.amazon.com/Gezging%C3%B6z-Sinir-T%C3%BCrkiye-Mirasi-Rehberi/dp/6257631181 https://www.amazon.com/Gezging%C3%B6z-Sinir-T%C3%BCrkiye-Mir...
- samatman 5y ago
  This is an outcome of the historic process which gave us modern Unicode. The Turkish alphabet ISO 8859-9 was published in 1988, and it doesn't distinguish between I and I, since why would it? It's a byte encoding designed specifically to accommodate Turkish. Unicode has a principle of adopting whatever makes it easy to convert from the preferred legacy encoding to Unicode, and for Turkish that meant using the same code point for I in both Turkish and practically everywhere else, since I is in ASCII and the bulk of Latin-based character encodings are based on ASCII. I point this out because it could indeed have been the case that they ended up with separate encodings, which would have solved your problem. But it wouldn't have solved the actual problem, which is the collation and casing are locale-specific according to Unicode, and indeed they must be. Example: in Dutch the letters 'ij' are one letter for collation purposes, as well as for casing, see https://en.wikipedia.org/wiki/IJsselstein https://en.wikipedia.org/wiki/IJsselstein for an example. This is of course impossible to get correct if the application doesn't know it's dealing with Dutch, which it often can't know. The very same problem you have with Turkish.
Someone 5y ago
> who knew "sort these names alphabetically" could be an ambiguous statement That already is ambiguous without considering text encodings or even computers. Sort order depends on locale (example, “in Lithuanian, "y" is sorted between "i" and "k"”) and within some locales even on the goal of the sorting (example: “in German dictionaries, "öf" would come before "of". In phone books the situation is the exact opposite.” (Both examples taken from https://unicode-org.github.io/icu/userguide/collation/ https://unicode-org.github.io/icu/userguide/collation/, which has many more examples) Also, on that “one code point in unicode does not necessarily map to one character on the screen”, I think it’s important to use consistent terminology. The screen doesn’t show characters but glyphs.
- Mlller 5y ago
  > in German dictionaries, "öf" would come before "of". I am only aware of “of” before “öf”, e.g. in Grimmʼs dictionary, the greatest dictionary of German itself, “offen” ‘open (adj.), openly’ comes before “öffen”, a variant of “öffnen” ‘to open’. When I have to sort strings in German, I make a sortstring by throwing out diacritics (via Unicode Normal Form “K”anonical Decomposition) and changing “ß” to “s”. Then I order by a tuple of this sortstring and the original string.
- bruce511 5y ago
  This is exactly my point. Growing up I thought alphabetical order was pretty sorted, then I grew up, and discovered that not only are there more characters, but people don't even agree on the _ordering_ of characters. Thus alphabetical order becomes ambiguous depending on where you are. _then_ you discover some symbol sets / languages don't have an order at all. Oh the world is complicated. I wish I was 4 again.
  - Ekaros 5y ago
    And then you come across dictionary or like and discover that yeah, sometimes things aren't sorted as strings, but weird things are done. Like "the" word, or person with some titles... While ofc. the title itself is sorted in that position...

dhosek 5y ago

That said, the Unicode standard does a great job of tackling that question and providing a framework that can cover all the various language standards (and even with a fully normalized Unicode string you still need to do things like handle ch as a single character for sorting in Czech or Spanish).

pezezin 5y ago
In Spanish, "ch" and "ll" stopped being considered a single character for sorting purposes in 1994. In have some dictionaries printed before that date, and I find them quite confusing...
- dhosek 5y ago
  As someone who learned his Spanish before 1994 (and whose dictionary also predates that time), I was unaware of this until now.
Toutouxc 5y ago
As a Czech I would love "ch" to go fuck itself for that exact reason.
oldsecondhand 5y ago
In Hungarian "dzs", "zs" and "sz" are so called compund letters, however most software treats them as separate characters, because it would be a pain in the ass do sorting the correct way. (It would either require changing the input method, or software would need to be aware of etymology.) It's only sorted the right way in paper dictionaries.
- dhosek 5y ago
  Actually, it just needs to be context sensitive. If I have an English document and I have an index which has an entry for Zsigmond, it would be placed between Zebra and Zygote. But, if I have a Hungarian document, then the correct ordering would be to have Zsigmond after Zebra and Zygote. Any correctly-implemented program that deals with sorting should not be assuming it can just sort on code point¹ and Hungarian is one of the standard collations that is part of the Unicode reference implementations so it shouldn't be a problem. The fact that a lot of software is badly-implemented is not an excuse for software to continue to be so. I did a quick check and saw that I can specify Hungarian in MySQL as a collation for a table and I also know that it's available in Java and even JavaScript. ⸻ 1. This sorting is wrong not just for Hungarian but for every language including English which would expect, e.g., naïve to be sorted between nag and nanny and not after nay.

Gigachad 5y ago

It’s only complex if your language does not support Unicode properly. In something like rust or python you would just ask for the first character rather than the first byte. And the language will calculate what that is for you.

chmod600 5y ago
I'm not an expert, but your statement seems confusing and possibly misses the point of the parent.
- deleted 5y ago
  [deleted]
torstenvl 5y ago
A 'character' meaning what? The first code point? The first non-combining code point? The first non-combining code point along with all associated combining code points? The first non-combining code point along with all associated combining code points modified to look like it would look in conjunction with surrounding non-combining code points along with their associated combining code points? The first displayable component of a code point? What are the following? the second character of sœur (o or the ligature?) the second character of حبيبي (the canonical form ب or the contextual form ﺒ ?) the third character of есть (Cyrillic t with or without soft sign, which is always a separate code point and always displayed to the right but changes the sound?) the first character of 실례합니다 (Korean phoneme or syllabic grapheme?) the first character of ﷺ or ﷽ ? The main issue isn't programming language support, it's ambiguity in the concept of "character" and conventions about how languages treat them. Imagine how the letter i would be treated if Unicode were invented by a Turk. The fundamental issue here is that human communication is deeply nuanced in a way that does not lend itself well to systematic encoding or fast/naive algorithms on that encoding. Even in the plain ASCII range it's impossible to know how to render a word like "ANSCHLUSS" in lower case (or how many 'characters' such a word would have) without knowledge of the language, country of origin, and time period in which the word was written.
- 5y ago

Twisell 5y ago

It's not skimmed the article have a whole pararagrah entitled "Unicode To The Confusion" about that. What you infer about sorting is better described as collation and is a separate set of problems. https://www.unicode.org/collation/ https://www.unicode.org/collation/ There can be multiple collations for the same encoding and even for the same language inside the same encoding. PS: As a matter of fact in some database like PostgreSQL you can define different collations for each columns if needed.

nanis 5y ago

> "one code point in unicode does not necessarily map to one character on the screen." Also, importantly, some characters are not represented at all. For example, there is no codepoint or combination of characters that distinguishes a capital Turkish dotless I from its identical looking but conceptually different sibling the Latin capital I. Similarly for capital Turkish dotted İ. I find it extremely weird that two codepoints couldn't have been spared, yet we have gradations of skin tone in emoji. One can use composition to deal with the round-trip from i -> İ -> i (within a closed ecosystem), but even that fails when it comes to ı -> I -> ı. A case in point are the product pages for my friend's book on Amazon. Compare "Sınır Ötesi" to "Sinir Ötesi". The former means "beyond borders" whereas the latter means "exceedingly irritating". Yet, because there is no unambiguous representation of the Turkish I's, the rendering makes an assumption on the basis of the domain. Note that all other non-US ASCII letters involved are rendered correctly. \c[PERSON FROWNING, ZERO WIDTH JOINER, PROGRAMMER] [tr]: https://www.amazon.com.tr/Gezging%C3%B6z-S%C4%B1n%C4%B1r-T%C3%BCrkiye-Miras%C4%B1-Rehberi/dp/6257631181/ https://www.amazon.com.tr/Gezging%C3%B6z-S%C4%B1n%C4%B1r-T%C... [us]: https://www.amazon.com/Gezging%C3%B6z-Sinir-T%C3%BCrkiye-Mirasi-Rehberi/dp/6257631181 https://www.amazon.com/Gezging%C3%B6z-Sinir-T%C3%BCrkiye-Mir...

samatman 5y ago
This is an outcome of the historic process which gave us modern Unicode. The Turkish alphabet ISO 8859-9 was published in 1988, and it doesn't distinguish between I and I, since why would it? It's a byte encoding designed specifically to accommodate Turkish. Unicode has a principle of adopting whatever makes it easy to convert from the preferred legacy encoding to Unicode, and for Turkish that meant using the same code point for I in both Turkish and practically everywhere else, since I is in ASCII and the bulk of Latin-based character encodings are based on ASCII. I point this out because it could indeed have been the case that they ended up with separate encodings, which would have solved your problem. But it wouldn't have solved the actual problem, which is the collation and casing are locale-specific according to Unicode, and indeed they must be. Example: in Dutch the letters 'ij' are one letter for collation purposes, as well as for casing, see https://en.wikipedia.org/wiki/IJsselstein https://en.wikipedia.org/wiki/IJsselstein for an example. This is of course impossible to get correct if the application doesn't know it's dealing with Dutch, which it often can't know. The very same problem you have with Turkish.

Someone 5y ago

> who knew "sort these names alphabetically" could be an ambiguous statement That already is ambiguous without considering text encodings or even computers. Sort order depends on locale (example, “in Lithuanian, "y" is sorted between "i" and "k"”) and within some locales even on the goal of the sorting (example: “in German dictionaries, "öf" would come before "of". In phone books the situation is the exact opposite.” (Both examples taken from https://unicode-org.github.io/icu/userguide/collation/ https://unicode-org.github.io/icu/userguide/collation/, which has many more examples) Also, on that “one code point in unicode does not necessarily map to one character on the screen”, I think it’s important to use consistent terminology. The screen doesn’t show characters but glyphs.

Mlller 5y ago
> in German dictionaries, "öf" would come before "of". I am only aware of “of” before “öf”, e.g. in Grimmʼs dictionary, the greatest dictionary of German itself, “offen” ‘open (adj.), openly’ comes before “öffen”, a variant of “öffnen” ‘to open’. When I have to sort strings in German, I make a sortstring by throwing out diacritics (via Unicode Normal Form “K”anonical Decomposition) and changing “ß” to “s”. Then I order by a tuple of this sortstring and the original string.
bruce511 5y ago
This is exactly my point. Growing up I thought alphabetical order was pretty sorted, then I grew up, and discovered that not only are there more characters, but people don't even agree on the _ordering_ of characters. Thus alphabetical order becomes ambiguous depending on where you are. _then_ you discover some symbol sets / languages don't have an order at all. Oh the world is complicated. I wish I was 4 again.
- Ekaros 5y ago
  And then you come across dictionary or like and discover that yeah, sometimes things aren't sorted as strings, but weird things are done. Like "the" word, or person with some titles... While ofc. the title itself is sorted in that position...