3 ms·

Would be interesting to know why Apple made the decision not to normalize on a file system level then. Just an argument based on separation of concerns?

by foota 9y ago

vetinari 9y ago
Normalizing on the file system level in the past was a mistake. Normalizing Unicode is not an easy task, it belongs into userspace and not into a kernel driver. Other systems consider filename a string of octets and interpretation what these octets mean is left for userspace to decide. For Linus Torvalds colorful opinions on HFS+, see here: https://plus.google.com/+JunioCHamano/posts/1Bpaj3e3Rru https://plus.google.com/+JunioCHamano/posts/1Bpaj3e3Rru (in the comments).
- rurban 9y ago
  Others have different opinions. Normalization of Unicode identifiers is necessary for identifiers to be identifiable. Mostly this is only done for domain names, I also do it for identifiers in my programming language, and consequently path names and directory entries should also identifiable. The garbage in - garbage out camp completely ignores all unicode security concerns and simply shifts blame to user-space. I applauded Apple to use normalization in HFS+, but the APFS integration is just horrible. 1st it makes unicode paths insecure again, and 2nd it's inconsistent. NFD over NFC is simply trading data over CPU. NFC requires 3 passes over a string, NFD only 2, whilst NFD is usually a few bytes longer. Linus talking about NFD corrupting data is of course nonsense. He should stay technical and only rant about things he has an idea about. The python3 NFKC normalization format is nonsense, but not really important. NFD is fine, because faster. Shifting interpretation of byte encodings from utf-8 to user-space is typical Linux non-sense, but he inherited the existing mess. Using utf-8 is miles better than bytes. Case-insensitivity in HFS+ is of course legacy nonsense. This should have been to target to get rid of, not normalization. The 2nd big security issue would be to forbid mixed scripts in a name (filename). I blogged about it here, and OP uses the same examples I used: http://perl11.org/blog/unicode-identifiers.html http://perl11.org/blog/unicode-identifiers.html Invisible combining marks, right-to-left overwrites or similar / spoofs security problems are only fixable with normalization and more TR31 (mixed scripts, confusables), though confusables can only be handled in user-space, not the FS.
- cryptonector 9y ago
  Normalization on CREATE was the mistake. ZFS does normalization on LOOKUP. This is much better, as it preserves whatever form you used, but is normalization-form insensitive, which is precisely what users need (and would want, if only they could be expected to understand what the heck is going on!).

Would be interesting to know why Apple made the decision not to normalize on a file system level then. Just an argument based on separation of concerns?

by foota 9y ago

Would be interesting to know why Apple made the decision not to normalize on a file system level then. Just an argument based on separation of concerns?

vetinari 9y ago

Normalizing on the file system level in the past was a mistake. Normalizing Unicode is not an easy task, it belongs into userspace and not into a kernel driver. Other systems consider filename a string of octets and interpretation what these octets mean is left for userspace to decide. For Linus Torvalds colorful opinions on HFS+, see here: https://plus.google.com/+JunioCHamano/posts/1Bpaj3e3Rru https://plus.google.com/+JunioCHamano/posts/1Bpaj3e3Rru (in the comments).

rurban 9y ago
Others have different opinions. Normalization of Unicode identifiers is necessary for identifiers to be identifiable. Mostly this is only done for domain names, I also do it for identifiers in my programming language, and consequently path names and directory entries should also identifiable. The garbage in - garbage out camp completely ignores all unicode security concerns and simply shifts blame to user-space. I applauded Apple to use normalization in HFS+, but the APFS integration is just horrible. 1st it makes unicode paths insecure again, and 2nd it's inconsistent. NFD over NFC is simply trading data over CPU. NFC requires 3 passes over a string, NFD only 2, whilst NFD is usually a few bytes longer. Linus talking about NFD corrupting data is of course nonsense. He should stay technical and only rant about things he has an idea about. The python3 NFKC normalization format is nonsense, but not really important. NFD is fine, because faster. Shifting interpretation of byte encodings from utf-8 to user-space is typical Linux non-sense, but he inherited the existing mess. Using utf-8 is miles better than bytes. Case-insensitivity in HFS+ is of course legacy nonsense. This should have been to target to get rid of, not normalization. The 2nd big security issue would be to forbid mixed scripts in a name (filename). I blogged about it here, and OP uses the same examples I used: http://perl11.org/blog/unicode-identifiers.html http://perl11.org/blog/unicode-identifiers.html Invisible combining marks, right-to-left overwrites or similar / spoofs security problems are only fixable with normalization and more TR31 (mixed scripts, confusables), though confusables can only be handled in user-space, not the FS.
cryptonector 9y ago
Normalization on CREATE was the mistake. ZFS does normalization on LOOKUP. This is much better, as it preserves whatever form you used, but is normalization-form insensitive, which is precisely what users need (and would want, if only they could be expected to understand what the heck is going on!).