6 ms·
There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which pe
by acdha 9y ago
There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers.
My position on this is mixed. I've had to write code to deal with normalization changes in archives and it's quite tedious. The HFS+ approach of normalization is in many ways the best choice considered in isolation but in practice it's really expensive to support since most filesystems, tools, APIs, etc. predate Unicode and everyone else chose the bag of bytes approach.
- catwell 9y agoNormalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article. On a side note, I wish Apple would have taken this opportunity to switch normalization from NFD to NFC, which basically everything else uses. The distinction causes complexity and often issues in software which share data between Apple platforms and other platforms, such as version control systems for instance. EDIT: according to pilif's comment, they did, which is awesome!
- acdha 9y agoDo you have any recent benchmarks showing a significant impact from normalization? I haven't seen that on anything in at least a decade and that was simply Red Hat shipping an ancient and completely unoptimized libicu. > Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article. Again, the article made a huge sweeping claim without supporting it. That's simply not true in either way – many apps on every OS don't handle this at all, some handle it consistently everywhere, and what a “Mac application” means varies widely from “clean, modern Cocoa” to “uses a lot of C, etc. libraries”, “cross platform C++”, “C# port”, “Electron shell”, etc. You can't make any statement which is true for every single one of those categories, much less for every code path which eventually results in a filesystem call. I've run into cases where something mostly worked until you hit their integrated ZIP, Git/SVN, etc. support and found a new way that a filename was constructed. My point wasn't that everything is fine but simply that this is complicated and no decision results in avoiding problems. Not normalizing allows for confusing visually-identical files; normalizing results in errors or data loss which will be blamed on the OS.
- vintagedave 9y ago> Again, the article made a huge sweeping claim without supporting it. Looks to me like the article made claims and backed them up with examples and screenshots. Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? Using its examples and concerns (Finder, console, and scripts)?
- acdha 9y ago> Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? I think you might to re-read my entire comment: note that I'm not arguing that the technical details are wrong, only that they're insufficient to support the huge “APFS is unusable” conclusion. As previously noted, Windows and Linux work the same way and they are used by more people in individual non-English locales than the total number of Mac users. Would you say “NTFS is unusable by non-English users” is a useful statement? There's plenty of room to say that a particular tool needs improvement, or that people making systems which copy or archive files should check for pathological cases, but it doesn't help anything to overstate the case so broadly.
- deong 9y agoThe issue isn't a "bag-of-bytes" filename model. The issue is a "bag-of-bytes" filename model combined with an inconsistent normalization scheme. It's not a problem on Windows or Linux filesystems because Windows and Linux don't provide a half-assed normalization scheme that lets me fairly easily create files that can't be accessed. If the Cocoa libraries did no normalization, then the resulting behavior might be obnoxious from a human-interface perspective, but I don't think the article would describe it as "little short of catastrophic". I'm sitting here on my US English keyboard typing scancodes that look just like they did in 1990, so I'm not the best authority on how big of a problem it really is, but I'd guess it's going to result in a lot of bugs. Anyone who's ever tried to use a Mac with a case-sensitive HFS+ partition should be able to tell you that programmers can't even "normalize" their filenames consistently strictly within their native language.
- 9y ago
- cryptonector 9y agoNormalization is not that expensive if you only care to do normalization-insensitive string comparison and string hashing. The reason it's not that expensive is: a) this requires no memory allocation, and b) most characters in most strings require no normalization! Notionally you just look at pairs of next codepoints, and if the second one isn't combining and the first one is canonical for the chosen NF (a very fast check for ASCII!), then there's no need to normalize the first, otherwise you gather the combining codepoints and normalize, producing one normalized character and restarting the process where you left off. Most of the time the first codepoint requires no normalization, so the fast path is fast -- not as fast as a normal strcmp() or memcmp(), but still pretty fast.
- Animats 9y agoNormalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. Why? It's done once per open, not once per I/O.
- cryptonector 9y agoIn HFS+ it's even only done once per-create, so pretty cheap. In ZFS it's once per-open()/stat()/and so on. But still, not at all on readdir(), and anyways, it's highly optimized. For an all ASCII filename the slow path is never taken, and for a mostly ASCII filename the slow path is only taken for non-ASCII codepoints that are followed by combining codepoints (that check is itself a slower-than-the-fast path, but still faster than the slowest path). Basically, it's noise.
- klodolph 9y agoNTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.
- rbanffy 9y ago> NTFS, by comparison, does the "bag of wchar_t" I threw up a little.
- aidenn0 9y agoThis is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense. Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.
- dietrichepp 9y agoThis timeline also meant that wchar_t was fixed as 16 bits on Windows for compatibility, but ended up as 32 bits on macOS and Unix. It can't really be removed from the C or C++ standards, but there's nothing you can do with wchar_t that's both useful and portable. Well-meaning developers learn how to support Unicode with wchar_t only to discover that wchar_t is the worst way to support Unicode. https://www.moria.us/articles/wchar-is-a-historical-accident/ https://www.moria.us/articles/wchar-is-a-historical-accident...
- rspeer 9y agoIt doesn't help that many developers are introduced to the basics of Unicode by a popular Joel on Software article from 2003, which recommended the use of wchar_t. [1] It was a very important article for 2003, but it should be honorably retired. Its plan of action isn't as good as the modern "UTF-8 Everywhere", and even its motivating examples are becoming less relevant: when was the last time you went to the 'Encoding' menu in your web browser and guessed which codepage the page author meant to use? When was the last time you worried about whether your e-mail was 7-bit clean? [1] https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- the_mitsuhiko 9y ago> I've had to write code to deal with normalization changes in archives and it's quite tedious. You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.
- acdha 9y agoExactly - and that matters to me in cases where the ability to say that someone copying files won't change the checksum on a manifest, which users find extremely confusing.
- deleted 9y ago[deleted]
- cryptonector 9y agoNot true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good). Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case. The way this works is that there's a) a string comparison function that can provide normalization- and/or case-insensitive comparisons, b) a character-at-a-time normalization function of sorts used for directory hashing (since ZFS hashes directories). This is much better than normalizing on create, which is destructive and obnoxious when the form you normalize to is not the more common form and you live in a sea of applications that don't do normalization.
- ridiculous_fish 9y agoThat's fascinating. What happens on lookup if there's two different files with canonically equivalent names?
- cryptonector 9y agoGreat question! You can't have two such files! Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.
- throwaway2048 9y agoso it infact does do normalization on file create then? How is it managing this without doing so?
- cryptonector 9y agoIt normalizes on LOOKUP, not create. On create it preserves the original form. Think of this as like case-preserving/case-insensitive behavior, but for form rather than case.
- 9y ago
- wnoise 9y ago> everyone else chose the bag of bytes approach. Sequence of bytes. "spiroagnew.txt" and "xtt.growapenis" are not the same filename.