6 ms·

What was the root cause and what was the fix?

a1371 5y ago
We had unicode characters in strings in our database. When those were being fetched by the front-end on Safari, the browser was making them a different character. This meant if the exact same string was going back to our server after being in Safari, any kind of string-matching was failing. So let's say you were looking up something based on the string, it was not there. We did some unicode testing on different platforms and this was never an issue. Also to note these weren't complicated emojis or anything like that (e.g. character & skin-tone). Just usual characters with accents that are not in English. The resolution for now has been using String.Prototype.normalize() with some tricks. It works for our case, but obviously it is not a silver bullet. This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-feef5ea42407 https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...
- microtonal 5y ago
  You said I always hear rosy thing about how polished the Apple ecosystem is. But this more sounds like a bug in your software. Many accented characters have two representations in Unicode, so you always have to normalize strings before making comparisons or use a comparison function that takes different encodings into account.
  - a1371 5y ago
    We learned that the hard way. Still, the basic dev expectation is to get the same thing out that they have put in. It shouldn't sound outrageous that we expected to get the same representation. That's all.
    - kccqzy 5y ago
      An alternative basic dev expectation is that given two different ways to represent the same thing, the software only gives you one canonical way. Fix your expectations.
    - marcellus23 5y ago
      With all due respect, software development is complicated and it sounds like you just weren’t aware of this aspect of it. Which is fine. But it’s a bug in _your_ software, not Safari.
  - cosarara 5y ago
    It sounds like this would be a nightmare in Unix filenames; even if everything in my system is perfectly clean utf-8, I could be trying to open "café.txt" and get a No such file or directory error because the input has é encoded a different way from what is saved on disk. You would need to list the directory contents, decode everything, perform the comparison in Unicode code points, and pray there is no more than one match. Any web UI that deals with files would have this kind of issues.
    - microtonal 5y ago
      Yeah. Though it really depends on the system. Some older filesystems, like HFS+ would normalize filenames. In APFS Apple decided not to normalize file names, but this caused a lot of issues. Later they have made APFS normalization-insensitive (so you cannot store two files with the same name, but different normalizations). Different normalizations are handled transparently by macOS frameworks. On Linux with btrfs: $ touch schön $ cat $(echo -e "scho\u0308n") cat: schön: No such file or directory $ touch $(echo -e "scho\u0308n") $ ls -l total 0 -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön $ ls | hexdump -c 0000000 s c h o 314 210 n \n s c h 303 266 n \n Unicode, loads of fun :). Edit: updated to make this copy-pastable.
      - r-w 5y ago
        I wonder if ZFS falls prey to the same issue. I've heard good things about it otherwise.
- lilyball 5y ago
  That's not a UTF-8 issue. That's Unicode. And macOS is not incorrect here. > So let's say you were looking up something based on the string, it was not there. What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail. And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type ﬁ (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi. > This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-feef5ea42407 https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee... That article flips NFC and NFD. > The resolution for now has been using String.Prototype.normalize() with some tricks. Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.
  - a1371 5y ago
    Thanks for trying to be helpful. Of course, I over-simplified our 4 years of development and the specialized field we are in. There isn't really a database involved here. I am sure there are many more correct ways for us to do certain things, that said, I respectfully disagree and macOS is the issue here. When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.
    - 5y ago
- iforgotpassword 5y ago
  Never compare unicode strings on the binary level. Or if you do, convert both sides to same same form, which you basically did in the end.
- KarlKemp 5y ago
  Safari still defaults to something other than utf8.
  - a1371 5y ago
    Yes, and this threw us off because we thought we weren't communicating the encoding correctly with Safari. A day later we finally realized that's not the issue.
  - josephg 5y ago
    So does javascript, in all web browsers and environments.
    - colejohnson66 5y ago
      JavaScript’s DOMString is basically defined as UTF-16.
- ksec 5y ago
  To be fair it reads to me a Unicode problem rather than macOS.

We had unicode characters in strings in our database. When those were being fetched by the front-end on Safari, the browser was making them a different character. This meant if the exact same string was going back to our server after being in Safari, any kind of string-matching was failing. So let's say you were looking up something based on the string, it was not there. We did some unicode testing on different platforms and this was never an issue. Also to note these weren't complicated emojis or anything like that (e.g. character & skin-tone). Just usual characters with accents that are not in English. The resolution for now has been using String.Prototype.normalize() with some tricks. It works for our case, but obviously it is not a silver bullet. This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-feef5ea42407 https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee...

microtonal 5y ago
You said I always hear rosy thing about how polished the Apple ecosystem is. But this more sounds like a bug in your software. Many accented characters have two representations in Unicode, so you always have to normalize strings before making comparisons or use a comparison function that takes different encodings into account.
- a1371 5y ago
  We learned that the hard way. Still, the basic dev expectation is to get the same thing out that they have put in. It shouldn't sound outrageous that we expected to get the same representation. That's all.
  - kccqzy 5y ago
    An alternative basic dev expectation is that given two different ways to represent the same thing, the software only gives you one canonical way. Fix your expectations.
  - marcellus23 5y ago
    With all due respect, software development is complicated and it sounds like you just weren’t aware of this aspect of it. Which is fine. But it’s a bug in _your_ software, not Safari.
- cosarara 5y ago
  It sounds like this would be a nightmare in Unix filenames; even if everything in my system is perfectly clean utf-8, I could be trying to open "café.txt" and get a No such file or directory error because the input has é encoded a different way from what is saved on disk. You would need to list the directory contents, decode everything, perform the comparison in Unicode code points, and pray there is no more than one match. Any web UI that deals with files would have this kind of issues.
  - microtonal 5y ago
    Yeah. Though it really depends on the system. Some older filesystems, like HFS+ would normalize filenames. In APFS Apple decided not to normalize file names, but this caused a lot of issues. Later they have made APFS normalization-insensitive (so you cannot store two files with the same name, but different normalizations). Different normalizations are handled transparently by macOS frameworks. On Linux with btrfs: $ touch schön $ cat $(echo -e "scho\u0308n") cat: schön: No such file or directory $ touch $(echo -e "scho\u0308n") $ ls -l total 0 -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön -rw-r--r-- 1 daniel daniel 0 Dec 17 12:00 schön $ ls | hexdump -c 0000000 s c h o 314 210 n \n s c h 303 266 n \n Unicode, loads of fun :). Edit: updated to make this copy-pastable.
    - r-w 5y ago
      I wonder if ZFS falls prey to the same issue. I've heard good things about it otherwise.
lilyball 5y ago
That's not a UTF-8 issue. That's Unicode. And macOS is not incorrect here. > So let's say you were looking up something based on the string, it was not there. What this tells me is you're storing unicode text in a database without using unicode-aware string comparisons. This means that regardless of macOS, you could run into this issue just by having someone submit text in a manner that uses a different normalization form than whatever you've been using, your lookup will fail. And it's not just NFC vs NFD. There's also NFKC and NFKD, which are the compatibility forms. For example, if I type ﬁ (U+FB01 LATIN SMALL LIGATURE FI), that has the same NFC and NFD forms, but in NFKC and NFKD it becomes fi. > This article helped: https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-feef5ea42407 https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-fee... That article flips NFC and NFD. > The resolution for now has been using String.Prototype.normalize() with some tricks. Please fix your database to use a proper unicode-aware comparison instead. I don't know what database you're using but this may involve simply telling it what the text encoding of the column is. This is especially true when you start thinking about compatibility normalization (i.e. NFKC/NFKD). I believe the compatibility forms are generally appropriate for string comparisons (e.g. if someone searches for "ﬁ" they should get results for "fi"; try it out in your browser's search field right now if you like) but you certainly don't want to store your text that way. And since you're presumably relying your database to do the lookup for you, this means you need your database to be unicode-aware, otherwise you have no choice but to store the compatibility canonicalization form in order to get search to be correct.
- a1371 5y ago
  Thanks for trying to be helpful. Of course, I over-simplified our 4 years of development and the specialized field we are in. There isn't really a database involved here. I am sure there are many more correct ways for us to do certain things, that said, I respectfully disagree and macOS is the issue here. When you can be compatible with everything else, please try to be. MacOS isn't in this case. I would rather stand on the shoulder of giants who spare my staff from having to learn about "NFKC" and "NFKD". It's nicer to spend that time with our families. That's my humble opinion and personal values of course.
  - 5y ago
iforgotpassword 5y ago
Never compare unicode strings on the binary level. Or if you do, convert both sides to same same form, which you basically did in the end.
KarlKemp 5y ago
Safari still defaults to something other than utf8.
- a1371 5y ago
  Yes, and this threw us off because we thought we weren't communicating the encoding correctly with Safari. A day later we finally realized that's not the issue.
- josephg 5y ago
  So does javascript, in all web browsers and environments.
  - colejohnson66 5y ago
    JavaScript’s DOMString is basically defined as UTF-16.
ksec 5y ago
To be fair it reads to me a Unicode problem rather than macOS.