5 ms·
Can someone explain to me why case insensitivity is a bad thing? Clearly Linus believes so but didn't explain why he believes so. Most UNIX and Linux systems s
by Someone1234 12y ago
Can someone explain to me why case insensitivity is a bad thing? Clearly Linus believes so but didn't explain why he believes so.
Most UNIX and Linux systems seem to have an "all lowercase" or "all uppercase" convention, so the fact that they have case sensitivity is often not utilised.
In fact the biggest reason you'd want case sensitivity off the top of my head is legacy support but that's just a circular argument (since you never really reach WHY it was that way originally, just that it was).
I guess based on what he talks about next he is worried about how case insensitivity interacts with other character sets (i.e. does it correctly change their case), but for most sets isn't the lower and upper case defined in the UNICODE language spec itself?
- bigdubs 12y agoI've run into issues with it from a programming point of view, you have to make sure to do a (slightly) more expensive case insensitive comparison when dealing with filenames.
- Someone1234 12y agoThis is only an issue when it is fake case insensitive. In the sense that it claims it is case insensitive but in fact is case sensitive (and returns case sensitive filenames). NTFS is a perfect example of this, it claims to be case insensitive but in fact stores everything in a case sensitive way (so you wind up doing things like ToUpper() on every filename). On a legitimately case insensitive filesystem both "etc" and "ETC" would return identical filenames as they would be stored identically in the file system's database (e.g. would return "etc" for both). So what you're really giving is not an example of why case insensitivity is bad, but why half measures are bad.
- Arnavion 12y ago>NTFS is a perfect example of this, it claims to be case insensitive Where does it claim this? In any case this is not true because... >but in fact stores everything in a case sensitive way This does not have to be true. NTFS can be case-sensitive or case-insensitive depending on which subsystem is accessing it. For example, while the POSIX subsystem for Windows was a thing, installing it had the side-effect of turning NTFS case-sensitive.
- bigdubs 12y agoPro-tip; never do .ToUpper() / .ToLower(), use foo.Equals(bar, StringComparison.OrdinalIgnoreCase), saves a string allocation also only loops over the strings once. For the truly anal.
- terminus 12y agoSpeaking from imperfect knowledge, I'd guess: case insensitivity implies allowing for aliasing (case aliasing for case insensitive and god-knows-what for Unicode insensitivity.) Which means that anywhere you handles names, you explicitly handle these aliases. Miss a spot (or an alias), and you have a security bug.
- dennisnedry 12y agoIn this case, the specific exploit Linus is referring to works by commiting a malicious .git file as .Git to your repo. Then somebody else on OSX clones your branch, and the bad .Git file will overwrite your .git file, causing a security breach.
- dennisnedry 12y agoThe exploit being referenced allows malicious code to be injected into a file that handles git, which is (.git). Now, normally Git does not allow you to overwrite (.git). Since HFS+ is case-insensitive, writing the file as (.Git) will overwrite the (.git) file and break security. Linus is saying that an uppercase and lowercase character are two different things, and not recognizing that can (and did) cause problems with security.
- Arnavion 12y agoHis argument is wrong. The issue is not with the filesystem being case-insensitive; the issue is that some strings which are not identical map to the same filename. Two strings differing in case is only one such example. Another example is the presence of codepoints which are ignored by the filesystem. The fix for git and Mercurial included the fix for this as well. Does Linus think all filesystems should allow all 256 byte sequences in every permutation as well? (He might, since that is the behavior of common Linux filesystems.) The proper mindset is to realize the actual issue (multiple strings map to the same filename) and use a "filename_according_to_program -> filename_according_to_filesystem" function everywhere when dealing with filenames, not to blame the filesystem.
- dennisnedry 12y agoYou hit the nail on the head, he does expect things to work almost exactly as they do on Linux. The thing about Linus if you've ever spent any time on reading the Linux Kernel mailing list is that he is extremely explosive and at times has really harsh analogies. But...he is usually right.
- daxelrod 12y agoThe question you need the filesystem to answer in this case is, "If I save a file with name A, will it be accessible via name B?" You can't assume that the filesystem has one canonical name from which the file will be accessed. EDIT: That actually isn't even enough, because the answer may vary by locale. Even worse, to solve this for Git's case, you need to know the answer 1) For every locale 2) For every supported filesystem
- thristian 12y agoThe prime number-one concern in kernel programming is managing complexity. Well, in most programming really, but in kernel programming unmanaged complexity leads to lost data and sometimes broken hardware instead of "just" crashes. Case-sensitivity is the easiest thing - you take a bytestring from userspace, you search for it exactly in the filesystem. Difficult to get wrong. Case-insensitivity for ASCII is slightly more complex - thanks to the clever people who designed ASCII, you can convert lower-case to upper-case by clearing a single bit. You don't want to always clear that bit, or else you'd get weirdness like "`" being the lowercase form of "@", so there's a couple of corner-cases to check. Case-sensitivity for Unicode is a giant mud-ball by comparison. There's no simple bit flip to apply, just a 66KB table of mappings[1] you have to hard-code. And that's not all! Changing the case of a Unicode string can change its length (ß -> SS), sometimes lower -> upper -> lower is not a round-trip conversion (ß -> SS -> ss), and some case-folding rules depend on locale (In Turkish, uppercase LATIN SMALL LETTER I is LATIN CAPITAL LETTER I WITH DOT ABOVE, not LATIN CAPITAL LETTER I like it is in ASCII). Oh, and since Unicode requires that LATIN SMALL LETTER E + COMBINING ACUTE ACCENT should be treated the same way as LATIN SMALL LETTER E WITH ACUTE, you also need to bring in the Unicode normalisation tables too. And keep them up-to-date with each new release of Unicode. So the last thing a kernel developer wants is Unicode support in a filesystem. [1]: http://www.unicode.org/Public/UNIDATA/CaseFolding.txt http://www.unicode.org/Public/UNIDATA/CaseFolding.txt
- deleted 12y ago[deleted]
- Someone1234 12y agoThat's a really good and comprehensive answer, thanks.
- lnkmails 12y agoI second this. I was trying to understand the reasons and your answer helped me. It saved me a few hours. Thank you!
- terminus 12y ago
- dunham 12y agoUpper/lower case is locale language sensitive. One specific example, the uppercase for "i" varies depending on whether you have a Turkish locale or not. String.toLowerCase() in Java has exceptions coded in for the "tr", "az", and "lt" locale languages. Some of these languages may cause the length of the string to change when upcased or downcased, too. Additionally, given a utf8 string, OSX will translate it into another, possibly different utf8 string before using it as a filename (the NFD normalization that Linus mentioned).
- kstenerud 12y agoYes, it would be nice to have universal "uppercase" and "lowercase" rules, but in the real world, collation rules are crazy, arbitrary, and damn near impossible to get right. Now try to get it right in EVERY locale around the world, because if you don't, you open a big gaping security hole or data corruption bug.
- thomasahle 12y agoIndeed "uppercase/lowercase" is a pretty anglocentric way phrase things. Who know what sort of cases somebody might come up with in their next language. "Normalization" is properly a more general term, but who says a normalization always exists?
- pmjordan 12y agoI don't know to what extent this affects HFS+, but case insensitivity in general is locale dependent. The classic example is the Turkish distinction between dotted and dotless I - http://en.m.wikipedia.org/wiki/Dotted_and_dotless_I http://en.m.wikipedia.org/wiki/Dotted_and_dotless_I. Quote: The dotless I, I ı, denotes the close back unrounded vowel sound (/ɯ/). Neither the upper nor the lower case version has a dot. The dotted I, İ i, denotes the close front unrounded vowel sound (/i/). Both the upper and lower case versions have a dot. End quote. I can imagine that being a nightmare, particularly if one user uses a Turkish locale and another on the same machine uses English. And all of that complexity in the kernel? Ouch. (Or HFS+ just behaves incorrectly for Turkish file names, not sure that's really better?)