4 ms·

Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovere

Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.

yuhong 10y ago
That is due to the HFS+ normalization that APFS eliminates. The filename is stored in UTF-16 and OS X converts it to UTF-8.
- frik 10y ago
  Is it stored "real UTF-16" or in USC2 wide-chars like in NTFS? The later would be a big problem.
  - yuhong 10y ago
    I believe HFS+ was created before Mac OS supported surrogates.
- eridius 10y ago
  Normalization and UTF-16 are completely orthogonal. It doesn't matter in the slightest whether the filesystem stores the filename in UTF-16 or UTF-8, either way it's still a sequence of Unicode scalar values, and that's what matters.
- kalleboo 10y ago
  HFS does a lot more than convert to UTF-8. E.g. it will normalize ö to o+(umlaut), using custom normalization rules that nobody else uses. If you tell HFS to store a file with an Ö in it, and then list the directory, the character Ö is nowhere to be found.
eridius 10y ago
What do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of the OS). So if you take a filename and convert it to NFD, it may not be the exact same byte sequence that you get if you plug that filename into HFS+ (but in most cases it will be). But whatever byte sequence you get from HFS+ is still going to be a valid Unicode sequence.
- caf 10y ago
  I suppose you could store the normalization table in the filesystem superblock, if you wanted the ability to update in future versions but still have those readable on older versions of the OS.
- therealmarv 10y ago
  I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there. The first one is recognized by Samba, the second one is invisible.
  - eridius 10y ago
    No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a combining acute mark. On HFS+, because it normalizes, if you try to pass either form to the filesystem, it will treat them the same. But on filesystems that don't normalize (which is most of them) they'll be treated as distinct files. As a result, depending on the input you provide and whether the tools in question do any normalization, you could end up with two files that look identical (e.g. your two "hällö" files) but are different unicode sequences under the hood. And none of this really has anything to do with UTF-8. UTF-8 is just a byte encoding scheme that can represent all valid unicode sequences.
    - therealmarv 10y ago
      Well one of this (the Apple one) is clearly wrong because Samba (which is not a niche software) does not work with it and this is critical for me. Just saying I'm not the only one who is confused ;)
      - 10y ago
grey-area 10y ago
Do you mean composed versus decomposed? Utf-8 allows both. Unfortunately this article doesn't explain it very well, here's a better link. https://docs.syncthing.net/advanced/folder-autonormalize.html https://docs.syncthing.net/advanced/folder-autonormalize.htm...
0x0 10y ago
If you are rsync'ing from HFS on a Mac to a Linux server, you can use "--iconv=UTF-8-MAC,UTF-8" to fix this problem.
- therealmarv 10y ago
  Yes, I used that too :) Also some similar options on sshfs. But I'm glad I discovered this all before transferring critical data :)

yuhong 10y ago

That is due to the HFS+ normalization that APFS eliminates. The filename is stored in UTF-16 and OS X converts it to UTF-8.

frik 10y ago
Is it stored "real UTF-16" or in USC2 wide-chars like in NTFS? The later would be a big problem.
- yuhong 10y ago
  I believe HFS+ was created before Mac OS supported surrogates.
eridius 10y ago
Normalization and UTF-16 are completely orthogonal. It doesn't matter in the slightest whether the filesystem stores the filename in UTF-16 or UTF-8, either way it's still a sequence of Unicode scalar values, and that's what matters.
kalleboo 10y ago
HFS does a lot more than convert to UTF-8. E.g. it will normalize ö to o+(umlaut), using custom normalization rules that nobody else uses. If you tell HFS to store a file with an Ö in it, and then list the directory, the character Ö is nowhere to be found.

eridius 10y ago

What do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of the OS). So if you take a filename and convert it to NFD, it may not be the exact same byte sequence that you get if you plug that filename into HFS+ (but in most cases it will be). But whatever byte sequence you get from HFS+ is still going to be a valid Unicode sequence.

caf 10y ago
I suppose you could store the normalization table in the filesystem superblock, if you wanted the ability to update in future versions but still have those readable on older versions of the OS.
therealmarv 10y ago
I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there. The first one is recognized by Samba, the second one is invisible.
- eridius 10y ago
  No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a combining acute mark. On HFS+, because it normalizes, if you try to pass either form to the filesystem, it will treat them the same. But on filesystems that don't normalize (which is most of them) they'll be treated as distinct files. As a result, depending on the input you provide and whether the tools in question do any normalization, you could end up with two files that look identical (e.g. your two "hällö" files) but are different unicode sequences under the hood. And none of this really has anything to do with UTF-8. UTF-8 is just a byte encoding scheme that can represent all valid unicode sequences.
  - therealmarv 10y ago
    Well one of this (the Apple one) is clearly wrong because Samba (which is not a niche software) does not work with it and this is critical for me. Just saying I'm not the only one who is confused ;)
    - 10y ago

grey-area 10y ago

Do you mean composed versus decomposed? Utf-8 allows both. Unfortunately this article doesn't explain it very well, here's a better link. https://docs.syncthing.net/advanced/folder-autonormalize.html https://docs.syncthing.net/advanced/folder-autonormalize.htm...

0x0 10y ago

If you are rsync'ing from HFS on a Mac to a Linux server, you can use "--iconv=UTF-8-MAC,UTF-8" to fix this problem.

therealmarv 10y ago
Yes, I used that too :) Also some similar options on sshfs. But I'm glad I discovered this all before transferring critical data :)