5 ms·

One thing that is very unintuitive with normalization is that MacOS is much more aggressive with normalizing Unicode than Windows or Linux distros. Even if you

One thing that is very unintuitive with normalization is that MacOS is much more aggressive with normalizing Unicode than Windows or Linux distros. Even if you copy and paste non-normalized text into a text box in safari on Mac, it will be normalized before it gets posted to the server. This leads to strange issues with string matching.

codesnik 3y ago
I was really surprised when realized that at least in hpfs cyrillics is normalized too. For example, no russian ever thinks that Й is a И with some diacritics. It's a different letter on it's own right. But mac normalizes it into two codepoints.
- anamexis 3y ago
  Well, there's no expectation in unicode that something viewed as a letter in its own right should use a single codepoint.
- asveikau 3y ago
  I dislike explaining string compares to monolingual English speakers who are programmers. Similar to this phenomenon of Й/И is people who think ñ and n should compare equally, or ç and c, or that the lowercase of I is always i (or that case conversion is locale-independent). In something like a code review, people will think you're insane for pointing out that this type of assumption might not hold. Actually, come to think of it, explaining localization bugs at all is a tough task in general.
  - iforgotpassword 3y ago
    Well, I do like this behavior for search though. I don't want to install a new keyboard layout just to be able to search for a Spanish word.
    - david-gpu 3y ago
      Is the convenience of a few foreigners searching for something more important than the convenience of the many native speakers searching for the same? Maybe we should start modifying the search behavior of English words to make them more convenient for non-native speakers as well. We could start by making "bed aidia" match "bad idea", since both sound similar to my foreign ears.
      - MrJohz 3y ago
        In fairness, for search, allowing multiple ways of typing the same thing is probably the best choice: you can prioritise true matches, where the user has typed the correct form of the letter, but also allow for more visual based matches. (Correcting common typos is also very convenient even for native speakers of a language — and of course a phonetic search that actually produced good results would be wonderful, albeit I suspect practically very difficult given just how many ways of writing a given pronunciation there might be!)
        david-gpu 3y ago
        As a counterexample, conflating two different glyphs as if they were the same can lead to the inability to search for a particular term. E.g. in Spanish these two words (cono, coño) have very different meanings. If I'm searching for one I don't want to see results pertaining to the other one. It would be like searching for "sheet" and getting results for "shit".
        MrJohz 3y ago
        It depends on how the search is implemented exactly and what the context is, but assuming I've searched for "cono", I would expect results that directly match "cono" to come first, then results that also match "coño". Similarly to how I'd expect to still get reasonable results if I type "beleive" instead of "believe". That said, this is obviously pretty context-dependent, in some settings it will make more sense to do an exact-match search, in which case you'd want to differentiate n and ñ (while still handling different possible unicode variants of ñ if those exist).
  - yxhuvud 3y ago
    Or that sort order is locale independent. Swedish is a good example here as åäö are sorted at the end, and where until 2006 w was sorted as v. And then it changed and w is now considered a letter of its own.
  - makeitdouble 3y ago
    The general reaction I've see until now was "meh, we have to make compromises (don't make me rewrite this for people I'll probably never meet)" Diacritics exacerbate this so much as they can be shared between two language yet have different rules/handling. French typically has a decent amount and they're meaningful but traditionally ignores them for comparison (in the dictionary for instance). That makes it more difficult for a dev to have an intuitive feeling of where it matters and where it doesn't.
  - koliber 3y ago
    These are different letters for people who speak the language and treating them the same in some usage seems weird. At the same time, sometimes words containing those letters might show up in context where the user is not familiar with that language. Such users might not know how to enter those letters. They might not even have the capability to type those letters with their installed keyboard layouts. If they are searching for content that contains such letters (e.g. a first name), normalizing them to the visually-closest ASCII is a sensible choice, even if it makes no sense to the speakers of the language. It's important to understand a situation from different perspectives. It's not about coming up with a single correct interpretation that makes logical sense. It about making a system work in least-surprising ways to all classes of users.
- bawolff 3y ago
  Normalization isn't based on what language the text is. NFC just means never use combining characters if possible, and NFD means always use combining characters if possible. It has nothing to do with whether something is a "real" letter in a specific language or not. The whether or not something is a "real" letter vs a letter with a modifier, more comes into play in the unicode collation algorithm, which is a separate thing.
creshal 3y ago
MacOS creates so many normalization problems in mixed environments that it's not even funny any more. No common server-side CMS etc. can deal with it, so the more Macs you add to an organization, the more problems you get with inconsistent normalization in your content. (And indeed, CMSes shouldn't have to second-guess users' intentions - diacretics and umlauts are pronounced differently and I should be able to encode that difference, e.g. to better cue TTS.) And, of course, the Apple fanboys will just shrug and suggest you also convert the rest of the organization to Apple devices, after all, if Apple made a choice, it can't be wrong.
- fauigerzigerk 3y ago
  I'm not sure I understand. On the one hand you seem to be saying that users should be able to choose which normalisation form to use (not sure why). On the other hand you're unhappy about macOS sending NFD. If it's a user choice then CMSs have to be able to deal with all normalisation forms anyway and shouldn't care one bit whether macOS sends NFD or NFC. Mac users could of course complain about their choice not being honoured by macOS but that's of no concern to CMSs.
  - creshal 3y ago
    > On the other hand you're unhappy about macOS sending NFD. Because MacOS always uses it, regardless of the user's intention, so it decomposes umlauts into diaereses (despite them having different meanings and pronunciations) and mangles cyrillic, and probably more problems I haven't yet run into.
    - kps 3y ago
      Unicode doesn't have ‘umlauts’, and (with a few unfortunate exceptions) doesn't care about meanings and pronunciations. From the Unicode perspective, what you're talking about is the difference between Unicode Normalization Form C: U+00FC LATIN SMALL LETTER U WITH DIAERESIS and Unicode Normalization Form D: U+0075 LATIN SMALL LETTER U U+0308 COMBINING DIAERESIS Unicode calls these two forms ‘canonically equivalent’.
- 3y ago
ttepasse 3y ago
Unfun normalisation fact: You can’t have a file named "ss" and a file named "ß" in the same folder in Mac OS.
- yxhuvud 3y ago
  So what happens if someone puts those two in a git repo and a Mac user checks out the folder?
  - staplung 3y ago
    git clone https://github.com/ghurley/encodingtest Cloning into 'encodingtest'... remote: Enumerating objects: 9, done. remote: Counting objects: 100% (9/9), done. remote: Compressing objects: 100% (5/5), done. remote: Total 9 (delta 1), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (9/9), done. Resolving deltas: 100% (1/1), done. warning: the following paths have collided (e.g. case-sensitive paths on a case-insensitive filesystem) and only one from the same colliding group is in the working tree: 'ss' 'ß'
    - Twisol 3y ago
      I have this issue on occasion with older mixed C/C++ codebases that use `.c` for C files and `.C` for C++ files. Maddening.
      - Athas 3y ago
        I never understood the popularity of the '.C' extension for C++ files. I have my own preference (.cpp), but it's essentially arbitrary compared to most other common alternatives (.cxx, .c++). The '.C' extension is the only one that just seems worse (this case sensitivity issue, and just general confusion given how similar '.c' looks to '.C'). But even more than that, I just don't get how C++ turns into 'C' at all. It seems actively misleading.
        keketi 3y ago
sorenjan 3y ago
I sometimes see texts where ä is rendered as a¨, i.e. with the dots next to the a instead of above it even though it's a completely different letter and not a version of a. I managed to track the issue down to MacOS' normalization, but it has happened on big national newspapers' websites and similar. I haven't seen it in a while, maybe Firefox on Windows renders it better or maybe various publishing tools have fixed it. It looks really unprofessional which is a bit strange since I thought Apple prides themselves on their typography.
- iforgotpassword 3y ago
  I have that in gnome terminal. The dots always end up on the letter after, not before. At least makes it easy to spot filenames in decomposed form so I can fix them.
- aidos 3y ago
  I have never see that on all my years on a Mac (though admittedly I’m not dealing in languages where I encounter it often). I’m assuming there’s an issue with the gpos table in the font you’re using so the dots aren’t negative shifted into position as they should be?
  - sorenjan 3y ago
    Well the point is that ä is one character, not two. It shouldn't be "a with two dots on it", it should be ä. It's its own letter with its own key on Swedish keyboards. MacOS apparently normalizes it to be two characters, and then somewhere in the publishing chain it gets mangled and end up as a¨. I have no doubt that it looked ok on the author's Mac. It's been a while since I last saw it, but it wasn't because of the font since it was published on a Swedish newspaper's website and other texts worked fine.
    - aidos 3y ago
      A single Unicode codepoint could be represented in a couple of different ways (either decomposed into 2 or as 1). Assume it’s the single codepoint representation. The font you’re using can (and probably will) rewrite it as 2 glyphs using the GSUB table. This makes sense because it’s a more efficient way to store the drawing operations. The GPOS table is then responsible for handling the offset to put things in their right place. Main point is that it’s up to the font to move things about. Now, that may not be what was going on in your case at all but it’s possible.
yxhuvud 3y ago
On the other hand, stuff written on macs are a lot more likely to require normalization in the first place.
dathinab 3y ago
While this (probably) still applies to Apple UI elements when they switched to APFS they stopped doing Unicode normalization on filesystem level. So now on macOS you can have a very mixed bag with some programs normalizing, some not (it's a bug) and many expecting normalized file names. So it's kinda like other Linux now except a lot of dev assuming normalization is happening (and in some cases still is when the string passes through certain APIs). Worse due to normalization now being somewhat application/framework dependent and often going beyond basic Unicode normalization it can lead to quite not so funny bugs. But luckily most users will never run into any of this bugs even if the use characters which might need normalization.

codesnik 3y ago

I was really surprised when realized that at least in hpfs cyrillics is normalized too. For example, no russian ever thinks that Й is a И with some diacritics. It's a different letter on it's own right. But mac normalizes it into two codepoints.

anamexis 3y ago
Well, there's no expectation in unicode that something viewed as a letter in its own right should use a single codepoint.
asveikau 3y ago
I dislike explaining string compares to monolingual English speakers who are programmers. Similar to this phenomenon of Й/И is people who think ñ and n should compare equally, or ç and c, or that the lowercase of I is always i (or that case conversion is locale-independent). In something like a code review, people will think you're insane for pointing out that this type of assumption might not hold. Actually, come to think of it, explaining localization bugs at all is a tough task in general.
- iforgotpassword 3y ago
  Well, I do like this behavior for search though. I don't want to install a new keyboard layout just to be able to search for a Spanish word.
  - david-gpu 3y ago
    Is the convenience of a few foreigners searching for something more important than the convenience of the many native speakers searching for the same? Maybe we should start modifying the search behavior of English words to make them more convenient for non-native speakers as well. We could start by making "bed aidia" match "bad idea", since both sound similar to my foreign ears.
    - MrJohz 3y ago
      In fairness, for search, allowing multiple ways of typing the same thing is probably the best choice: you can prioritise true matches, where the user has typed the correct form of the letter, but also allow for more visual based matches. (Correcting common typos is also very convenient even for native speakers of a language — and of course a phonetic search that actually produced good results would be wonderful, albeit I suspect practically very difficult given just how many ways of writing a given pronunciation there might be!)
      - david-gpu 3y ago
        As a counterexample, conflating two different glyphs as if they were the same can lead to the inability to search for a particular term. E.g. in Spanish these two words (cono, coño) have very different meanings. If I'm searching for one I don't want to see results pertaining to the other one. It would be like searching for "sheet" and getting results for "shit".
        MrJohz 3y ago
        It depends on how the search is implemented exactly and what the context is, but assuming I've searched for "cono", I would expect results that directly match "cono" to come first, then results that also match "coño". Similarly to how I'd expect to still get reasonable results if I type "beleive" instead of "believe". That said, this is obviously pretty context-dependent, in some settings it will make more sense to do an exact-match search, in which case you'd want to differentiate n and ñ (while still handling different possible unicode variants of ñ if those exist).
- yxhuvud 3y ago
  Or that sort order is locale independent. Swedish is a good example here as åäö are sorted at the end, and where until 2006 w was sorted as v. And then it changed and w is now considered a letter of its own.
- makeitdouble 3y ago
  The general reaction I've see until now was "meh, we have to make compromises (don't make me rewrite this for people I'll probably never meet)" Diacritics exacerbate this so much as they can be shared between two language yet have different rules/handling. French typically has a decent amount and they're meaningful but traditionally ignores them for comparison (in the dictionary for instance). That makes it more difficult for a dev to have an intuitive feeling of where it matters and where it doesn't.
- koliber 3y ago
  These are different letters for people who speak the language and treating them the same in some usage seems weird. At the same time, sometimes words containing those letters might show up in context where the user is not familiar with that language. Such users might not know how to enter those letters. They might not even have the capability to type those letters with their installed keyboard layouts. If they are searching for content that contains such letters (e.g. a first name), normalizing them to the visually-closest ASCII is a sensible choice, even if it makes no sense to the speakers of the language. It's important to understand a situation from different perspectives. It's not about coming up with a single correct interpretation that makes logical sense. It about making a system work in least-surprising ways to all classes of users.
bawolff 3y ago
Normalization isn't based on what language the text is. NFC just means never use combining characters if possible, and NFD means always use combining characters if possible. It has nothing to do with whether something is a "real" letter in a specific language or not. The whether or not something is a "real" letter vs a letter with a modifier, more comes into play in the unicode collation algorithm, which is a separate thing.

creshal 3y ago

MacOS creates so many normalization problems in mixed environments that it's not even funny any more. No common server-side CMS etc. can deal with it, so the more Macs you add to an organization, the more problems you get with inconsistent normalization in your content. (And indeed, CMSes shouldn't have to second-guess users' intentions - diacretics and umlauts are pronounced differently and I should be able to encode that difference, e.g. to better cue TTS.) And, of course, the Apple fanboys will just shrug and suggest you also convert the rest of the organization to Apple devices, after all, if Apple made a choice, it can't be wrong.

fauigerzigerk 3y ago
I'm not sure I understand. On the one hand you seem to be saying that users should be able to choose which normalisation form to use (not sure why). On the other hand you're unhappy about macOS sending NFD. If it's a user choice then CMSs have to be able to deal with all normalisation forms anyway and shouldn't care one bit whether macOS sends NFD or NFC. Mac users could of course complain about their choice not being honoured by macOS but that's of no concern to CMSs.
- creshal 3y ago
  > On the other hand you're unhappy about macOS sending NFD. Because MacOS always uses it, regardless of the user's intention, so it decomposes umlauts into diaereses (despite them having different meanings and pronunciations) and mangles cyrillic, and probably more problems I haven't yet run into.
  - kps 3y ago
    Unicode doesn't have ‘umlauts’, and (with a few unfortunate exceptions) doesn't care about meanings and pronunciations. From the Unicode perspective, what you're talking about is the difference between Unicode Normalization Form C: U+00FC LATIN SMALL LETTER U WITH DIAERESIS and Unicode Normalization Form D: U+0075 LATIN SMALL LETTER U U+0308 COMBINING DIAERESIS Unicode calls these two forms ‘canonically equivalent’.
3y ago

ttepasse 3y ago

Unfun normalisation fact: You can’t have a file named "ss" and a file named "ß" in the same folder in Mac OS.

yxhuvud 3y ago
So what happens if someone puts those two in a git repo and a Mac user checks out the folder?
- staplung 3y ago
  git clone https://github.com/ghurley/encodingtest Cloning into 'encodingtest'... remote: Enumerating objects: 9, done. remote: Counting objects: 100% (9/9), done. remote: Compressing objects: 100% (5/5), done. remote: Total 9 (delta 1), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (9/9), done. Resolving deltas: 100% (1/1), done. warning: the following paths have collided (e.g. case-sensitive paths on a case-insensitive filesystem) and only one from the same colliding group is in the working tree: 'ss' 'ß'
  - Twisol 3y ago
    I have this issue on occasion with older mixed C/C++ codebases that use `.c` for C files and `.C` for C++ files. Maddening.
    - Athas 3y ago
      I never understood the popularity of the '.C' extension for C++ files. I have my own preference (.cpp), but it's essentially arbitrary compared to most other common alternatives (.cxx, .c++). The '.C' extension is the only one that just seems worse (this case sensitivity issue, and just general confusion given how similar '.c' looks to '.C'). But even more than that, I just don't get how C++ turns into 'C' at all. It seems actively misleading.
      - keketi 3y ago

sorenjan 3y ago

I sometimes see texts where ä is rendered as a¨, i.e. with the dots next to the a instead of above it even though it's a completely different letter and not a version of a. I managed to track the issue down to MacOS' normalization, but it has happened on big national newspapers' websites and similar. I haven't seen it in a while, maybe Firefox on Windows renders it better or maybe various publishing tools have fixed it. It looks really unprofessional which is a bit strange since I thought Apple prides themselves on their typography.

iforgotpassword 3y ago
I have that in gnome terminal. The dots always end up on the letter after, not before. At least makes it easy to spot filenames in decomposed form so I can fix them.
aidos 3y ago
I have never see that on all my years on a Mac (though admittedly I’m not dealing in languages where I encounter it often). I’m assuming there’s an issue with the gpos table in the font you’re using so the dots aren’t negative shifted into position as they should be?
- sorenjan 3y ago
  Well the point is that ä is one character, not two. It shouldn't be "a with two dots on it", it should be ä. It's its own letter with its own key on Swedish keyboards. MacOS apparently normalizes it to be two characters, and then somewhere in the publishing chain it gets mangled and end up as a¨. I have no doubt that it looked ok on the author's Mac. It's been a while since I last saw it, but it wasn't because of the font since it was published on a Swedish newspaper's website and other texts worked fine.
  - aidos 3y ago
    A single Unicode codepoint could be represented in a couple of different ways (either decomposed into 2 or as 1). Assume it’s the single codepoint representation. The font you’re using can (and probably will) rewrite it as 2 glyphs using the GSUB table. This makes sense because it’s a more efficient way to store the drawing operations. The GPOS table is then responsible for handling the offset to put things in their right place. Main point is that it’s up to the font to move things about. Now, that may not be what was going on in your case at all but it’s possible.

yxhuvud 3y ago

On the other hand, stuff written on macs are a lot more likely to require normalization in the first place.

dathinab 3y ago

While this (probably) still applies to Apple UI elements when they switched to APFS they stopped doing Unicode normalization on filesystem level. So now on macOS you can have a very mixed bag with some programs normalizing, some not (it's a bug) and many expecting normalized file names. So it's kinda like other Linux now except a lot of dev assuming normalization is happening (and in some cases still is when the string passes through certain APIs). Worse due to normalization now being somewhat application/framework dependent and often going beyond basic Unicode normalization it can lead to quite not so funny bugs. But luckily most users will never run into any of this bugs even if the use characters which might need normalization.