4 ms·

The prime number-one concern in kernel programming is managing complexity. Well, in most programming really, but in kernel programming unmanaged complexity lead

The prime number-one concern in kernel programming is managing complexity. Well, in most programming really, but in kernel programming unmanaged complexity leads to lost data and sometimes broken hardware instead of "just" crashes. Case-sensitivity is the easiest thing - you take a bytestring from userspace, you search for it exactly in the filesystem. Difficult to get wrong. Case-insensitivity for ASCII is slightly more complex - thanks to the clever people who designed ASCII, you can convert lower-case to upper-case by clearing a single bit. You don't want to always clear that bit, or else you'd get weirdness like "`" being the lowercase form of "@", so there's a couple of corner-cases to check. Case-sensitivity for Unicode is a giant mud-ball by comparison. There's no simple bit flip to apply, just a 66KB table of mappings[1] you have to hard-code. And that's not all! Changing the case of a Unicode string can change its length (ß -> SS), sometimes lower -> upper -> lower is not a round-trip conversion (ß -> SS -> ss), and some case-folding rules depend on locale (In Turkish, uppercase LATIN SMALL LETTER I is LATIN CAPITAL LETTER I WITH DOT ABOVE, not LATIN CAPITAL LETTER I like it is in ASCII). Oh, and since Unicode requires that LATIN SMALL LETTER E + COMBINING ACUTE ACCENT should be treated the same way as LATIN SMALL LETTER E WITH ACUTE, you also need to bring in the Unicode normalisation tables too. And keep them up-to-date with each new release of Unicode. So the last thing a kernel developer wants is Unicode support in a filesystem. [1]: http://www.unicode.org/Public/UNIDATA/CaseFolding.txt http://www.unicode.org/Public/UNIDATA/CaseFolding.txt

deleted 12y ago
[deleted]
Someone1234 12y ago
That's a really good and comprehensive answer, thanks.
- lnkmails 12y ago
  I second this. I was trying to understand the reasons and your answer helped me. It saved me a few hours. Thank you!
terminus 12y ago
Well put. However, I'd add that what sucks even more is that this is not just a kernel issue. Every single application needs to worry about this. In this particular instance, the vulnerability is caused by git not handling aliased names correctly.
cbd1984 12y ago
And even if you do things The Unicode Way, you'll get a constant stream of bug reports from people who don't like The Unicode Way, and want things done Their Way, which is, of course, The Right Way. There are as many Right Ways as there are people, of course, and no two are completely compatible, so if you want to Do The Right Thing, either go see a movie or write your application software to do it like that, because the OS kernel can only do it one way. And, of course, if you want applications to be able to do case folding, the filesystem needs to be case-sensitive so all the relevant information is preserved.
Someone 12y ago
"And keep them up-to-date with each new release of Unicode" I agree with the rest, but that is one thing you absolutely shouldn't do. Once there are disks 'out there' that were created with some idea about what case insensitivity is, your choice has been set in stone. The risk of dhanging any rule is simply too high. Somebody might start reading hat disk using the previous version of the file system. For example, the precursor to HFS+, HFS, like HFS+, kept directories sorted by name. However, it had a filename sorting bug that the Finder had to work around (see http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.html#UnicodeSubtleties http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.ht...)
- ori_b 12y ago
  > I agree with the rest, but that is one thing you absolutely shouldn't do. And now you have a file system that is case sensitive, BUT ONLY IN SOME UNICODE RANGES. Somehow, that's better than just not being case sensitive in the first place?
ploxiln 12y ago
Mac OS X's NFD conversion was the problem for me. The case insensitivity was, superficially anyway, not the problem - you get back the filename you initially wrote (it just could overlap with another file that is not clearly defined because world languages are crazy). But NFD changes the filenames you wrote into a different byte sequence. I've use all linux systems for a while, recently added a macbook pro retina. I keep some personal directories of stuff backed up on multiple systems, along with a "manifest" of files and md5sums. NFD conversion meant that file names, which included correctly-encoded utf-8 (NFC ...) accented characters, changed between writing them, and "ls" of the directory. If I didn't already have experience with utf-16 and all sorts of unicode encoding and translation silliness, debugging this would have been impossible. The fix was to add this to my scripts: | perl -M'Unicode::Normalize' -M'open qw(:std :utf8)' -ne 'print NFC($_);' | man oh man was that hard to figure out though. Mac OS X is way way harder to make work right than linux. (See also disk utility)
- MrBuddyCasino 12y ago
  I get the problem, but don't you need canonical equivalence to make sure you don't allow files with filenames that look the same but which happen to have a different unicode byte sequence? Or is the problem specifically with NFC? I don't see how byte equality could be preserved the other normalisations.
  - restalis 12y ago
    With or without Unicode's own meddling, characters may still look the same. Think of non-Latin alphabets that have letters defined in an identical fashion to those in Latin alphabet. Heck, even without other alphabets, the Latin letter O and the digit 0 are valid characters for filenames in any system I know of, and although looking the same they are yet different. (Having a bar, a point, or some other distinctive mark inside 0 is just a hack, the common description for the aspect of this digit makes no distinction from letter O.) This problem is not new, definitely not brought up by this Unicode thing.
    - MrBuddyCasino 12y ago

deleted 12y ago

[deleted]

Someone1234 12y ago

That's a really good and comprehensive answer, thanks.

lnkmails 12y ago
I second this. I was trying to understand the reasons and your answer helped me. It saved me a few hours. Thank you!

terminus 12y ago

Well put. However, I'd add that what sucks even more is that this is not just a kernel issue. Every single application needs to worry about this. In this particular instance, the vulnerability is caused by git not handling aliased names correctly.

cbd1984 12y ago

And even if you do things The Unicode Way, you'll get a constant stream of bug reports from people who don't like The Unicode Way, and want things done Their Way, which is, of course, The Right Way. There are as many Right Ways as there are people, of course, and no two are completely compatible, so if you want to Do The Right Thing, either go see a movie or write your application software to do it like that, because the OS kernel can only do it one way. And, of course, if you want applications to be able to do case folding, the filesystem needs to be case-sensitive so all the relevant information is preserved.

Someone 12y ago

"And keep them up-to-date with each new release of Unicode" I agree with the rest, but that is one thing you absolutely shouldn't do. Once there are disks 'out there' that were created with some idea about what case insensitivity is, your choice has been set in stone. The risk of dhanging any rule is simply too high. Somebody might start reading hat disk using the previous version of the file system. For example, the precursor to HFS+, HFS, like HFS+, kept directories sorted by name. However, it had a filename sorting bug that the Finder had to work around (see http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.html#UnicodeSubtleties http://dubeiko.com/development/FileSystems/HFSPLUS/tn1150.ht...)

ori_b 12y ago
> I agree with the rest, but that is one thing you absolutely shouldn't do. And now you have a file system that is case sensitive, BUT ONLY IN SOME UNICODE RANGES. Somehow, that's better than just not being case sensitive in the first place?

ploxiln 12y ago

Mac OS X's NFD conversion was the problem for me. The case insensitivity was, superficially anyway, not the problem - you get back the filename you initially wrote (it just could overlap with another file that is not clearly defined because world languages are crazy). But NFD changes the filenames you wrote into a different byte sequence. I've use all linux systems for a while, recently added a macbook pro retina. I keep some personal directories of stuff backed up on multiple systems, along with a "manifest" of files and md5sums. NFD conversion meant that file names, which included correctly-encoded utf-8 (NFC ...) accented characters, changed between writing them, and "ls" of the directory. If I didn't already have experience with utf-16 and all sorts of unicode encoding and translation silliness, debugging this would have been impossible. The fix was to add this to my scripts: | perl -M'Unicode::Normalize' -M'open qw(:std :utf8)' -ne 'print NFC($_);' | man oh man was that hard to figure out though. Mac OS X is way way harder to make work right than linux. (See also disk utility)

MrBuddyCasino 12y ago
I get the problem, but don't you need canonical equivalence to make sure you don't allow files with filenames that look the same but which happen to have a different unicode byte sequence? Or is the problem specifically with NFC? I don't see how byte equality could be preserved the other normalisations.
- restalis 12y ago
  With or without Unicode's own meddling, characters may still look the same. Think of non-Latin alphabets that have letters defined in an identical fashion to those in Latin alphabet. Heck, even without other alphabets, the Latin letter O and the digit 0 are valid characters for filenames in any system I know of, and although looking the same they are yet different. (Having a bar, a point, or some other distinctive mark inside 0 is just a hack, the common description for the aspect of this digit makes no distinction from letter O.) This problem is not new, definitely not brought up by this Unicode thing.
  - MrBuddyCasino 12y ago