3 ms·

I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or

I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there. The first one is recognized by Samba, the second one is invisible.

eridius 10y ago
No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a combining acute mark. On HFS+, because it normalizes, if you try to pass either form to the filesystem, it will treat them the same. But on filesystems that don't normalize (which is most of them) they'll be treated as distinct files. As a result, depending on the input you provide and whether the tools in question do any normalization, you could end up with two files that look identical (e.g. your two "hällö" files) but are different unicode sequences under the hood. And none of this really has anything to do with UTF-8. UTF-8 is just a byte encoding scheme that can represent all valid unicode sequences.
- therealmarv 10y ago
  Well one of this (the Apple one) is clearly wrong because Samba (which is not a niche software) does not work with it and this is critical for me. Just saying I'm not the only one who is confused ;)
  - eridius 10y ago
    It's not wrong. There are multiple ways to represent a lot of filenames (in particular, names with accents, umlauts, or graves). It's very plausible that the input methods on different systems are producing different sequences (composed vs decomposed). If you're storing a file with a composed sequence on your server, then copy it to your Mac's drive, then copy it back, you may end up with a duplicate file because it would have ended up as a decomposed sequence on your local drive. But this doesn't make it wrong. You can reproduce this exact behavior with other systems too simply by changing the local filename to be decomposed. The only Mac-specific part here is that HFS+ will automatically decompose the filename.

eridius 10y ago

No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a combining acute mark. On HFS+, because it normalizes, if you try to pass either form to the filesystem, it will treat them the same. But on filesystems that don't normalize (which is most of them) they'll be treated as distinct files. As a result, depending on the input you provide and whether the tools in question do any normalization, you could end up with two files that look identical (e.g. your two "hällö" files) but are different unicode sequences under the hood. And none of this really has anything to do with UTF-8. UTF-8 is just a byte encoding scheme that can represent all valid unicode sequences.

therealmarv 10y ago
Well one of this (the Apple one) is clearly wrong because Samba (which is not a niche software) does not work with it and this is critical for me. Just saying I'm not the only one who is confused ;)
- eridius 10y ago
  It's not wrong. There are multiple ways to represent a lot of filenames (in particular, names with accents, umlauts, or graves). It's very plausible that the input methods on different systems are producing different sequences (composed vs decomposed). If you're storing a file with a composed sequence on your server, then copy it to your Mac's drive, then copy it back, you may end up with a duplicate file because it would have ended up as a decomposed sequence on your local drive. But this doesn't make it wrong. You can reproduce this exact behavior with other systems too simply by changing the local filename to be decomposed. The only Mac-specific part here is that HFS+ will automatically decompose the filename.