4 ms·

The biggest problem with this "not the true text" issue is when coders encounter unicode. A lot of coders, those who have worked in primarily english countries

The biggest problem with this "not the true text" issue is when coders encounter unicode. A lot of coders, those who have worked in primarily english countries see ascii as utf-8 and the difference is invisible. They can go decades being oblivious to topics like encodings and mappings and display. So it can be surprising to them when they start dealing with European characters for the first time. They view the text in one place (like an editor which treats the file as utf-8) and another (their program) which treats the text as ASCII. It's hard to explain to them that "when I look at it" isn't a universal truth, it also matters how the "look at it program" chooses to interpret, and display, it.

rustybolt 2y ago
In some cases pretending everything is ASCII is the sane thing to do. With Unicode, sorting and case conversion are neigh impossible to do correctly. While there are algorithms for collating codepoints into (extended) grapheme clusters, there is still a lot of freedom, so while there are wrong ways to do it there is no canonical right way.
- poincaredisk 2y ago
  >With Unicode, sorting and case conversion are neigh impossible to do correctly Surely you mean that sorting correctly is impossible without Unicode? Otherwise you would have to hardcode the rules of sorting strings correctly in my language (and all other languages) yourself. Unless your preferred solution is "close my eyes and prefer non-ascii characters don't exist", then... I'm not a fan.
  - samatman 2y ago
    Sorting is impossible to do correctly without knowledge of the language in which the text is written, because the collation rules for symbols differ between languages. Unicode, of course, defines those collation rules, and UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII. Case conversion is similar except the default rules do a very good job in general. But still, there are a few language-specific quirks and, again, you do have to know what language is involved to get those right. I'm agreeing with you, to be clear, just adding that a) Unicode isn't always enough, but it does a decent job if you don't know the language in advance, and that it defines the correct rules if you do know that.
    - sgarland 2y ago
      > UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII This isn’t necessarily true beyond ASCII, and it depends entirely on the collation [0]. One need only to spend some time peering into the abyss that is RDBMS collation support [1] [2] to see the horror. [0]: http://www.unicode.org/reports/tr10/ http://www.unicode.org/reports/tr10/ [1]: https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets.html https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets... [2]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html
      - samatman 2y ago
        Well, no. It sorts according to the lexiographical order of Unicode. Earlier points before later points. How useful this is depends on the language, of course. I did say that as well. But Unicode was put together from legacy character encodings, and did what it could to preserve the order of those, so it's far from useless.
- teddyh 2y ago
  “Sorting is hard in other languages, so I would like to force everybody to only use the characters from my language, no characters from any other language. This will make it easy for me.”
  - Etherlord87 2y ago
    English is the international language, and latin characters have a special 'canonical' status in computer science, so you're heavily strawmaning here...
  - DrillShopper 2y ago
    Laziness, Impatience, and Hubris
elric 2y ago
The same is true for all aspects of I18N and L11N. From keyboard layouts to date formats. I've seen tools that expect US qwerty use hard-coded shortcuts that are impossible to type on different layouts. Something assumptions and asses.
- 7bit 2y ago
  I hate Microsoft for that. They created a truly great date and time formatting thing in Windows where I can set a system encoding, a system language, a user language, a date/time "language" and configure individual things like the thousands-separator (. or ,) Only to have obvious junior devs throw all of that out the Windows and now most things are dictated by the users region. Yes, I live in Germany and the region is important for region-locked shit on the app store. That does not fucking mean I want my OS talk to me in German. And on M365, I want German date formatting (DD.MM.YYYY), but when M365 is in English, you cannot select that date formatting, because someone thought that Americans would never need that. Fuck all of these ignorant bastards!
  - tmm84 2y ago
    As a user who is in an asian region I have all my machines set to that region. However, from time to time there are cases where dates that need to be in USA format are automatically converted to match my region. Go back, edit them, save and they're formatted again. This isn't consistent enough of a problem for me so I don't have a list of programs that do it.
  - elric 2y ago
    Agreed. I'm in Belgium. I get really annoyed when applications or websites randomly decide to talk French at me. Or Dutch. Or even bloody German. My browser is set to Accept-Language: en. I wish people would realise that country is not a trustworthy indicator for language.

rustybolt 2y ago

In some cases pretending everything is ASCII is the sane thing to do. With Unicode, sorting and case conversion are neigh impossible to do correctly. While there are algorithms for collating codepoints into (extended) grapheme clusters, there is still a lot of freedom, so while there are wrong ways to do it there is no canonical right way.

poincaredisk 2y ago
>With Unicode, sorting and case conversion are neigh impossible to do correctly Surely you mean that sorting correctly is impossible without Unicode? Otherwise you would have to hardcode the rules of sorting strings correctly in my language (and all other languages) yourself. Unless your preferred solution is "close my eyes and prefer non-ascii characters don't exist", then... I'm not a fan.
- samatman 2y ago
  Sorting is impossible to do correctly without knowledge of the language in which the text is written, because the collation rules for symbols differ between languages. Unicode, of course, defines those collation rules, and UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII. Case conversion is similar except the default rules do a very good job in general. But still, there are a few language-specific quirks and, again, you do have to know what language is involved to get those right. I'm agreeing with you, to be clear, just adding that a) Unicode isn't always enough, but it does a decent job if you don't know the language in advance, and that it defines the correct rules if you do know that.
  - sgarland 2y ago
    > UTF-8 sorts lexicographically using the same naïve byte comparison which works for ASCII This isn’t necessarily true beyond ASCII, and it depends entirely on the collation [0]. One need only to spend some time peering into the abyss that is RDBMS collation support [1] [2] to see the horror. [0]: http://www.unicode.org/reports/tr10/ http://www.unicode.org/reports/tr10/ [1]: https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets.html https://dev.mysql.com/doc/refman/8.4/en/charset-unicode-sets... [2]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html
    - samatman 2y ago
      Well, no. It sorts according to the lexiographical order of Unicode. Earlier points before later points. How useful this is depends on the language, of course. I did say that as well. But Unicode was put together from legacy character encodings, and did what it could to preserve the order of those, so it's far from useless.
teddyh 2y ago
“Sorting is hard in other languages, so I would like to force everybody to only use the characters from my language, no characters from any other language. This will make it easy for me.”
- Etherlord87 2y ago
  English is the international language, and latin characters have a special 'canonical' status in computer science, so you're heavily strawmaning here...
- DrillShopper 2y ago
  Laziness, Impatience, and Hubris

elric 2y ago

The same is true for all aspects of I18N and L11N. From keyboard layouts to date formats. I've seen tools that expect US qwerty use hard-coded shortcuts that are impossible to type on different layouts. Something assumptions and asses.

7bit 2y ago
I hate Microsoft for that. They created a truly great date and time formatting thing in Windows where I can set a system encoding, a system language, a user language, a date/time "language" and configure individual things like the thousands-separator (. or ,) Only to have obvious junior devs throw all of that out the Windows and now most things are dictated by the users region. Yes, I live in Germany and the region is important for region-locked shit on the app store. That does not fucking mean I want my OS talk to me in German. And on M365, I want German date formatting (DD.MM.YYYY), but when M365 is in English, you cannot select that date formatting, because someone thought that Americans would never need that. Fuck all of these ignorant bastards!
- tmm84 2y ago
  As a user who is in an asian region I have all my machines set to that region. However, from time to time there are cases where dates that need to be in USA format are automatically converted to match my region. Go back, edit them, save and they're formatted again. This isn't consistent enough of a problem for me so I don't have a list of programs that do it.
- elric 2y ago
  Agreed. I'm in Belgium. I get really annoyed when applications or websites randomly decide to talk French at me. Or Dutch. Or even bloody German. My browser is set to Accept-Language: en. I wish people would realise that country is not a trustworthy indicator for language.