3 ms·
> how hard it is to implement a modern filesystem that won't eat users' data. Why is that so hard? Because of edge-cases? Caching/Timing considerations?
by fn1 5y ago
> how hard it is to implement a modern filesystem that won't eat users' data.
Why is that so hard? Because of edge-cases? Caching/Timing considerations?
- eafer 5y agoThe main problem is simply that people really really really don't like losing their data after they saved it to disk. A simple app that corrupts its in-memory state once a year is probably acceptable. A filesystem that corrupts its on-disk state once a year is pure garbage. You basically need to aim for zero bugs. How hard this is, it depends on the filesystem. Something like FAT, for example, is pretty much designed for ease of implementation, with few edge cases. Modern filesystems are not like that at all, the data structures are very complicated, so they must be extremely well tested before they are good enough to use. That would probably require an fsck to check for subtle inconsistencies; in the case of APFS you can use mine, but it's still very incomplete. Apple's published fsck is not very thorough. As an example of the kind of problems to expect, I recall a bug in the Linux HFS+ driver. If you had a drive with lots of short filenames and lots of long filenames, and you started deleting the short filenames, eventually you would lose half of your files. This kind of things happen because HFS+ has variable-length keys in the index nodes of its trees, so deleting a record may trigger a complicated cascade of node splits. APFS inherited this feature, and it was very annoying to implement. But HFS+ is very well documented; APFS is not, and that doesn't help.
- hnlmorg 5y agoIt's worse than this. You not just need to aim for zero bugs, but zero bugs despite working with hardware that can degrade with use and who's firmware often does have bugs.
- trasz 5y agoHFS+ is open source, so you don’t even need to rewrite it from scratch.
- jscipione 5y agoAnd yet this didn't stop Apple from automatically converting HFS+ volumes to APFS in iOS 10.3 and macOS 10.13.0 soon after the APFS beta dropped in macOS 10.12.5 and it didn't stop Apple from requiring APFS for all volumes in macOS 10.14+. Apple must have been pretty confident that APFS was working reliably to be so bold.
- eafer 5y agoNot sure why you are telling me this, I don't know anything about Apple's internal development process. I assume they did run a lot of tests. But I recall at least one serious bug early on too[0]. [0] https://www.theregister.com/2018/02/16/apple_file_system_bug/ https://www.theregister.com/2018/02/16/apple_file_system_bug...
- bmn__ 5y agoThe big problems continue to be • C being a shitty language that does not force or even encourage programmers to handle errors • implementation knowledge about file system technology is generally stuck in the 1990s • disk controller hardware lying to the OS to make them appear more performant than they really are visit https://danluu.com https://danluu.com & ctrl+f "files"
- queuebert 5y agoCan you elaborate on the second point?
- bmn__ 5y agocourses offered by polytechnics and online learning platforms not being up-to-date, restricted and uneven access to expert implementers
- eafer 5y ago> C being a shitty language that does not force or even encourage programmers to handle errors I don't see where this is coming from. Most of the world's top filesystems are written in C, and they work just fine. Maybe other languages could get better results, but it's hard to say with so little data. > implementation knowledge about file system technology is generally stuck in the 1990s If you are talking about me, that might be true, I'm relatively new to this and still learning. But there's definitely people out there with some serious "implementation knowledge". And tools like xfstests did not exist in the 1990s, that makes a huge difference.