3 ms·

Mac OS X's NFD conversion was the problem for me. The case insensitivity was, superficially anyway, not the problem - you get back the filename you initially wr

by ploxiln 12y ago

Mac OS X's NFD conversion was the problem for me. The case insensitivity was, superficially anyway, not the problem - you get back the filename you initially wrote (it just could overlap with another file that is not clearly defined because world languages are crazy). But NFD changes the filenames you wrote into a different byte sequence. I've use all linux systems for a while, recently added a macbook pro retina. I keep some personal directories of stuff backed up on multiple systems, along with a "manifest" of files and md5sums. NFD conversion meant that file names, which included correctly-encoded utf-8 (NFC ...) accented characters, changed between writing them, and "ls" of the directory. If I didn't already have experience with utf-16 and all sorts of unicode encoding and translation silliness, debugging this would have been impossible. The fix was to add this to my scripts: | perl -M'Unicode::Normalize' -M'open qw(:std :utf8)' -ne 'print NFC($_);' | man oh man was that hard to figure out though. Mac OS X is way way harder to make work right than linux. (See also disk utility)

MrBuddyCasino 12y ago
I get the problem, but don't you need canonical equivalence to make sure you don't allow files with filenames that look the same but which happen to have a different unicode byte sequence? Or is the problem specifically with NFC? I don't see how byte equality could be preserved the other normalisations.
- restalis 12y ago
  With or without Unicode's own meddling, characters may still look the same. Think of non-Latin alphabets that have letters defined in an identical fashion to those in Latin alphabet. Heck, even without other alphabets, the Latin letter O and the digit 0 are valid characters for filenames in any system I know of, and although looking the same they are yet different. (Having a bar, a point, or some other distinctive mark inside 0 is just a hack, the common description for the aspect of this digit makes no distinction from letter O.) This problem is not new, definitely not brought up by this Unicode thing.
  - MrBuddyCasino 12y ago
    True, we have seen this issue with fraudulent domain names, where they can pose a security issue. To me it seems normalisation at least reduces the attack surface, while not solving the issue completely. I am not qualified to judge whether or not the complexity increase is worth it. But what are the downsides - what is so bad about NFC?
- ploxiln 12y ago
  Well in my case, I don't have to worry about filename spoofing, or even filenames being too similar, for my own directory of records/archives. The strangeness of a normalizing filesystem, is that you can try to open a filename (with accent, NFC), and it works. Then, a readdir() doesn't list that exact filename byte sequence (you happen to get one that's extremely similar, but NFD). On linux it always does, there's no normalization, exactly what you open is exactly what's listed. This really is an awefully nice simplification, for this layer of abstraction. But you could argue that I'm just accustomed to the tradeoffs of it by this point.

by ploxiln 12y ago

MrBuddyCasino 12y ago

I get the problem, but don't you need canonical equivalence to make sure you don't allow files with filenames that look the same but which happen to have a different unicode byte sequence? Or is the problem specifically with NFC? I don't see how byte equality could be preserved the other normalisations.

restalis 12y ago
With or without Unicode's own meddling, characters may still look the same. Think of non-Latin alphabets that have letters defined in an identical fashion to those in Latin alphabet. Heck, even without other alphabets, the Latin letter O and the digit 0 are valid characters for filenames in any system I know of, and although looking the same they are yet different. (Having a bar, a point, or some other distinctive mark inside 0 is just a hack, the common description for the aspect of this digit makes no distinction from letter O.) This problem is not new, definitely not brought up by this Unicode thing.
- MrBuddyCasino 12y ago
  True, we have seen this issue with fraudulent domain names, where they can pose a security issue. To me it seems normalisation at least reduces the attack surface, while not solving the issue completely. I am not qualified to judge whether or not the complexity increase is worth it. But what are the downsides - what is so bad about NFC?
ploxiln 12y ago
Well in my case, I don't have to worry about filename spoofing, or even filenames being too similar, for my own directory of records/archives. The strangeness of a normalizing filesystem, is that you can try to open a filename (with accent, NFC), and it works. Then, a readdir() doesn't list that exact filename byte sequence (you happen to get one that's extremely similar, but NFD). On linux it always does, there's no normalization, exactly what you open is exactly what's listed. This really is an awefully nice simplification, for this layer of abstraction. But you could argue that I'm just accustomed to the tradeoffs of it by this point.