3 ms·

Kind of related, but I've had an idea for a while to make a user mode file system on Windows that uses a single backing file (maybe sqlite) in order to speed up

by slaymaker1907 6y ago

Kind of related, but I've had an idea for a while to make a user mode file system on Windows that uses a single backing file (maybe sqlite) in order to speed up builds since NTFS is very slow for large numbers of small files. Also would be nice to have integrity checking backed in. It amazes me how bad file systems are in 2021. No integrity checks, no transactions, slow/incorrect file watching, etc. I know these features are provided by some newer file systems like zfs and btrfs, but those are not used by most consumers and are mostly used on servers.

sgtnoodle 6y ago
I think part of it is the trade-off between durability and simplicity. 9 out of 10 users aren't going to be capable of configuring zfs or btrfs in a way that's actually more durable than FAT, NTFS or ext. When corruption inevitably occurs, the average person is much more likely to recover some of their data off of simpler filesystems than complex ones. "We saved 80% of your family photos by running FatFileFinderPro, that will be $250," is better to a consumer than, "Bummer, our highly advanced diagnostic tools say the drive is toast. That will be $250." OEMs could choose to solve this for operating systems and maybe they do on internal storage, but it doesn't help with external storage, which is once again what 9 out of 10 people consider to be a "hard drive". Hard drive manufactures therefore have a strong incentive to solve durability as best they can inside the drive units themselves. I don't actually worry about bitrot too much because I assume there's some sort of propietary error correction mechanism at the firmware level. I expect my data is either going to read out flawless, have gaping holes in it, or have the whole disk unit fail. I'm not going to blindly trust that such a mechanism exists, but it weighs into my risk analysis when making decisions on how to best protect my personal data.

sgtnoodle 6y ago

I think part of it is the trade-off between durability and simplicity. 9 out of 10 users aren't going to be capable of configuring zfs or btrfs in a way that's actually more durable than FAT, NTFS or ext. When corruption inevitably occurs, the average person is much more likely to recover some of their data off of simpler filesystems than complex ones. "We saved 80% of your family photos by running FatFileFinderPro, that will be $250," is better to a consumer than, "Bummer, our highly advanced diagnostic tools say the drive is toast. That will be $250." OEMs could choose to solve this for operating systems and maybe they do on internal storage, but it doesn't help with external storage, which is once again what 9 out of 10 people consider to be a "hard drive". Hard drive manufactures therefore have a strong incentive to solve durability as best they can inside the drive units themselves. I don't actually worry about bitrot too much because I assume there's some sort of propietary error correction mechanism at the firmware level. I expect my data is either going to read out flawless, have gaping holes in it, or have the whole disk unit fail. I'm not going to blindly trust that such a mechanism exists, but it weighs into my risk analysis when making decisions on how to best protect my personal data.