4 ms·

From what I could understand when this bit me, OSX uses a different normalization form from what Linux uses (I'm not even sure of what that means, but hey). OSX

by Wilya 14y ago

From what I could understand when this bit me, OSX uses a different normalization form from what Linux uses (I'm not even sure of what that means, but hey). OSX uses NFD, while pretty much everyone else uses NFC (by default). So, if you use scp (Finder and the UI seem to convert things on-the-fly) to move files with UTF-8 filenames (mine where of korean origin), you'll end up puzzled because you're supposed to have some UTF-8 on both sides, but it's not the same UTF-8. I don't disagree that it might be a corner case that doesn't concern a lot of people, but it's not like I was doing black magic.

pifflesnort 14y ago
> OSX uses a different normalization form from what Linux uses (I'm not even sure of what that means, but hey). http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_form... > OSX uses NFD, while pretty much everyone else uses NFC (by default). It depends on the file system. HFS+ uses UTF-8 NFD, as you noted. On Linux, ext4 uses whatever encoding your tools happen to use, as long as it's an 8-bit ASCII superset capable of representing / and \0. On the Mac, this means you can rely on a consistent file encoding, and you never wind up with weird encoding issues when using different software, sharing files between machines, or when writing code to parse file names. On Linux, this means you're at the whim of whatever happened to write the file. There's no real way of telling -what- the file name encoding might be, and if you wind up with files with mismatched encodings, you just wind up with garbage. I think Apple chose the right approach; it's the only way for it to be possible for things to 'just work' > I don't disagree that it might be a corner case that doesn't concern a lot of people, but it's not like I was doing black magic. Well, technically you were doing black magic, in that: - scp(1) is not file name encoding-aware. - The Linux ext* file systems do not provide a guaranteed encoding If you used a name-encoding aware tool/protocol (such as modern network file systems), and you have consistent and correct encoding on the server (there's no gaurantee with Linux), then transferring files between machines should always "just work".
- mturmon 14y ago
  Thanks for this insightful comment. I hadn't known how encodings in filenames are dealt with by CLI utilities. It looks like some versions/builds of rsync are encoding-aware (e.g., http://blog.gimpe.com/2009/09/13/tip-use-rsync-to-convert-filenames-utf-8-latin-1iso-8859-1/ http://blog.gimpe.com/2009/09/13/tip-use-rsync-to-convert-fi...), although some recent builds may not be (http://cygwin.com/ml/cygwin/2011-05/msg00467.html http://cygwin.com/ml/cygwin/2011-05/msg00467.html).

by Wilya 14y ago

pifflesnort 14y ago

> OSX uses a different normalization form from what Linux uses (I'm not even sure of what that means, but hey). http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_form... > OSX uses NFD, while pretty much everyone else uses NFC (by default). It depends on the file system. HFS+ uses UTF-8 NFD, as you noted. On Linux, ext4 uses whatever encoding your tools happen to use, as long as it's an 8-bit ASCII superset capable of representing / and \0. On the Mac, this means you can rely on a consistent file encoding, and you never wind up with weird encoding issues when using different software, sharing files between machines, or when writing code to parse file names. On Linux, this means you're at the whim of whatever happened to write the file. There's no real way of telling -what- the file name encoding might be, and if you wind up with files with mismatched encodings, you just wind up with garbage. I think Apple chose the right approach; it's the only way for it to be possible for things to 'just work' > I don't disagree that it might be a corner case that doesn't concern a lot of people, but it's not like I was doing black magic. Well, technically you were doing black magic, in that: - scp(1) is not file name encoding-aware. - The Linux ext* file systems do not provide a guaranteed encoding If you used a name-encoding aware tool/protocol (such as modern network file systems), and you have consistent and correct encoding on the server (there's no gaurantee with Linux), then transferring files between machines should always "just work".

mturmon 14y ago
Thanks for this insightful comment. I hadn't known how encodings in filenames are dealt with by CLI utilities. It looks like some versions/builds of rsync are encoding-aware (e.g., http://blog.gimpe.com/2009/09/13/tip-use-rsync-to-convert-filenames-utf-8-latin-1iso-8859-1/ http://blog.gimpe.com/2009/09/13/tip-use-rsync-to-convert-fi...), although some recent builds may not be (http://cygwin.com/ml/cygwin/2011-05/msg00467.html http://cygwin.com/ml/cygwin/2011-05/msg00467.html).