14 ms·
Bitrot and atomic COWs: Inside “next-gen” filesystems
- AndrewDucker 13y agoAnyone know how the latest versions of NTFS stand up against these?
- ghh 13y agoMicrosoft has introduced ReFS [1] as a potential successor to NTFS in Windows Server 2012. It has a few of the 'next-gen' filesystem features that the article mentions, such as integrity checking, but no copy-on-write and it's not feature-equivalent to NTFS yet. Also, you can't boot from it yet. [1] http://en.wikipedia.org/wiki/ReFS http://en.wikipedia.org/wiki/ReFS
- masklinn 13y agoIt seems to do COW and checksumming of metadata, but not of data. According to http://blogs.msdn.com/b/b8/archive/2012/01/16/building-the-next-generation-file-system-for-windows-refs.aspx http://blogs.msdn.com/b/b8/archive/2012/01/16/building-the-n... there's a feature "integrity stream" which is opt-in per-file (or per-subtree and inherited by all files) checksumming. It doesn't seem to do COW, but can be paired with "storage spaces"[0] from which bitrotted files can be recovered. [0] http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing-storage-for-scale-resiliency-and-efficiency.aspx http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing-s...
- Freaky 13y agoIntegrity streams are enabled by default on mirrored pools, as per your first link: > By default, when the /i switch is not specified, the behavior that the system chooses depends on whether the volume resides on a mirrored space. On a mirrored space, integrity is enabled because we expect the benefits to significantly outweigh the costs. Plus: > When this option, known as “integrity streams,” is enabled, ReFS always writes the file changes to a location different from the original one. This allocate-on-write technique ensures that pre-existing data is not lost due to the new write
- rincebrain 13y agoYou also can't store things like WSUS server's updates on it; I have no idea why. It rather mitigates the point, and frustrates me, that I can't use it for a lot of the things I'd like to be robust against such problems...
- higherpurpose 13y agoI want F2FS for my phones. It seems to make storage 50-100 percent faster compared to ext4 from the benchmarks I've seen. Motorola started using it in Moto X and Moto G, but I hope Google makes it the default for Android in the next Android version.
- pedrocr 13y agoF2FS sounds like a bit of a hack. It's basically a way to get performance out of the storage-provided flash translation layer that pretends the underlying storage isn't flash. It also doesn't have the fancy ZFS/btrfs features. I wonder if the COW that these do can be tuned to work well with flash devices.
- fsiefken 13y agojust backup your data to blueray (25 GB) or use PaperBack 1.0 for a 1 or more MB per A4 depending on compression. http://www.ollydbg.de/Paperbak/index.html http://www.ollydbg.de/Paperbak/index.html My girlfriend is already printing out the family pictures as she doesn't trust me keeping the backups of the jpegs.
- masklinn 13y agoThere's no guarantee that the data will reach your BR disk intact. Or that a writable BR will survive for years (let alone decades)
- Dylan16807 13y agoIt's pretty trivial to check that a write succeeded. Unless you mean getting corrupted before putting the data into the redundant/backup system, which can happen just as easily with ZFS or almost any setup.
- gvb 13y agoI would expect blueray to be worse life expectancy than DVD since it is more dense. DVDs have worse life expectancy than CDs. CDs have somewhere between poor and acceptable lifetimes. Good luck. Ref: * http://en.wikipedia.org/wiki/CD-R#Lifespan http://en.wikipedia.org/wiki/CD-R#Lifespan * Library of Congress "CD-R and DVD-R RW Longevity Research" http://www.loc.gov/preservation/scientists/projects/cd-r_dvd-r_rw_longevity.html http://www.loc.gov/preservation/scientists/projects/cd-r_dvd... The actual study: http://www.loc.gov/preservation/resources/rt/NIST_LC_OpticalDiscLongevity.pdf http://www.loc.gov/preservation/resources/rt/NIST_LC_Optical... * http://www.thexlab.com/faqs/opticalmedialongevity.html http://www.thexlab.com/faqs/opticalmedialongevity.html (references the NIST study above)
- anon4 13y ago1) I would never trust my data to something that degrades as easily as a plastic disc 2) I would never trust my data to a medium that can get destroyed if touched with your finger the wrong way and you're required to handle it manually to read the data on it
- chadly 13y agoIs bitrot something the average hard drive user even needs to worry about? I know the hard drives themselves at the hardware level implement checksums. Is it really necessary to also have it at the filesystem level? I am legitimately asking because I have a good 2TB of family photos on hard drives and spooky stories about random bits flipping freak me out.
- theatrus2 13y agoRunning ZFS for many years has shown, yes, this happens, and will continue to get worse as density goes up.
- olavgg 13y agoI've been running ZFS on several servers with tenfolds of TB's with data. I see checksum errors every month, from a single bit to several megabytes.
- booi 13y agoThat sounds like you have something else wrong. With ECC memory on server hardware, we've seen 0 checksum errors in the last 6 months and I've seen only 2 ever. A typical server has 136TB of raw hdd space and we get about 71TiB usable. It's about 80% full.
- sp332 13y agoDensity is exactly the problem. The chances of an unrecoverable error are around 1 per trillion! But oh wait, you have 2 trillion bits on that disk?
- fiatmoney 13y agoAbsolutely! Over the last ~10 years, with perhaps an average of 6-8 disks at any one time and relatively low intensity use, I've probably had 4 hard drives where the files had checksum errors, before complete failure. But you don't need anything fancy; 2TB is small enough though that you can just buy another HDD and back it up manually.
- wernerb 13y agoIf you are really concerned about bitrot, and raid is apparently "not the solution", generate your own parity files for important stuff: http://parchive.sourceforge.net/#clients http://parchive.sourceforge.net/#clients
- lmm 13y agoThat's very fiddly compared to using ZFS and getting it "for free".
- e40 13y agoUnless ZFS isn't easily available on the OS you are using.
- rbanffy 13y agoLinux has an easy to install port (http://zfsonlinux.org/ http://zfsonlinux.org/). I remember OSX as having to use it via FUSE. I guess that leaves only Windows. OS/2 had installable filesystems. Does Windows have something like it?
- Freaky 13y agoOS X has a native port - http://open-zfs.org/wiki/Distributions#ZFS-OSX http://open-zfs.org/wiki/Distributions#ZFS-OSX Windows has ReFS - http://en.wikipedia.org/wiki/ReFS http://en.wikipedia.org/wiki/ReFS
- rbanffy 13y agoI didn't mean like ZFS, but like installable filesystems. It should be possible, with some effort, make ZFS run on Windows. ReFS is beta-quality and lack several features ZFS has since its first production-grade release. It's not really an apples to apples comparison.
- wernerb 13y agoDepends on what you find fiddly. With par2 I can move the files to other filesystems without problems, not being bound to ZFS (bsd) means a lot to me. I will reconsider when Btrfs is stable.
- omh 13y agoAs a Solaris admin, it's a little strange to hear ZFS called a "next-gen" filesystem. (It's been around for at least 7-8 years!) But it's good to see these ideas getting usable in other operating systems, especially with the current state of Solaris licensing.
- yan 13y agoI think this is why there's a distinction between "new" and "next-gen". ZFS still has a lot of features and concepts that make most "current generation" file systems feel like toys.
- bensummers 13y agoAlso, Docker vs Zones. Linux still has a way to go. :-)
- MBCook 13y agoI'm using HFS+. It's a last-last gen filesystem if ever there was one. It's the only FS I've ever had silent corruption on, and it used to happen (10.4? 10.5? 10.6?) all the time. I'd kill to have Apple just buy a license to NTFS.
- groby_b 13y agoUh, you might want to look at NTFS performance for your given use case. If, say, you're a C++ programmer and your use case is reading lots of small files quickly and closing them again so you can compile, NTFS is a horror cabinet. HFS beats that quite handily (and gets obliterated by ExtFS2 performance.)
- anton_gogolev 13y agoListen to Belt and Suspenders [1] and Computational Skeuomorphism [2] episodes of Hypercritical for an excellent discussion of filesystems: what the hell do they do and how do they compare. [1]: http://5by5.tv/hypercritical/56 http://5by5.tv/hypercritical/56 [2]: http://5by5.tv/hypercritical/57 http://5by5.tv/hypercritical/57
- billyhoffman 13y ago+1 on Hypercritical's discussion on widely used file systems and their problems. When I saw this article on Ars I was surprised that John Siracusa wasn't the author. Somewhere he must be smiling.
- anton_gogolev 13y agoMy thoughts exactly. Listening to John, he somehow manages to refer to how sucky HFS+ is literally in every other episode of whatever podcast he happens to be on.
- acd 13y agoI think the next generation file system has CoW, are based on erasure codes and are internet distributed. For example your home NAS have InternetFS, now you can reach all your files where ever you are, all files in the home directory are always in sync. Failure of a hard drive does not matter. If your house burns down, your friends or someone else on the internet has the pieces you need to reconstruct your data. The next gen filesystem also have built in versioning via the CoW so you can always revert to an earlier version of an file. InternetFS(Encrypted, distributed, always in sync, cheap snapshots, p2p) Want to share Photos or movies with your friends and family? No problem just right click on the file, select the friend from a list, they see the file and they can view it on their computer. This next gen file system will be incredibly easy to use, cross platform. As simple to use as Facebook, Skype, Email.
- rakoo 13y agoSound to me like Tahoe-LAFS [0] on global scale. Shall we build this ? https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs
- xiaomai 13y agoI was under the impression that Tahoe-LAFS doesn't work offline. Is that not the case?
- rakoo 13y agoIndeed, but Tahoe-LAFS is not a replacement for your filesystem, rather a virtual place to backup and share files. Common use would be to have all files local, and a copy on your Tahoe-LAFS swarm.
- foobarqux 13y agoYou need to run special gateway servers, I don't think it uses transparent encryption and there isn't a global namespace (for sharing publicly or with non-static groups).
- 13y ago
- kibwen 13y agoStupid question: even if we use checksums and parity files and such to verify the integrity of our data, how do we verify that these integrity measures don't themselves become corrupted? Is it just "that's very unlikely to happen"?
- mnw21cam 13y agoBasically, yes. In the old days, integrity was ensured by using 1 bit in 8 parity, that is for every eight bits there would be an extra bit, and counting the bits set to 1 should always be even (or odd, depending). This meant that if there was a single bit flipped, you were guaranteed to detect the failure, but powerless to correct it. However, if you had two bits flipped, the parity would pass. Nowadays, we use something more like a hash function, where we take the block of 4kB or so and produce a 64-bit hash of it. In order for this check to fail, the failure has a 1 in 2^64 chance of guessing a correct checksum for the corrupted data, which is very unlikely to happen. Also, we have the possibility of using Reed-Solomon error correcting codes in some applications, which can not only detect errors but also correct them.
- steveklabnik 13y agoI've only had one cup of coffee so far, but... The problem would be if the checksum AND the data _both_ became corrupted in a way that the checksum were still valid. After all, if the checksum were to change, you'd get a failure, and then you could just verify that the data was not actually changed and/or lost. Not to mention that if you stored the checksum in two places, it'd be pretty easy to see that it was the checksum that itself changed.
- masklinn 13y ago> how do we verify that these integrity measures don't themselves become corrupted? In ZFS, a storage pool is a merkle tree[0], the checksums themselves are checksummed (as part of their ancestor blocks). The one risk is that the uberblock itself (the root of the tree holding a checksum for the whole thing) becomes corrupted, which is why IIRC zfs stores the last 128 revisions of the uberblock in 4 different physical locations[1]. [0] http://en.wikipedia.org/wiki/Merkle_tree http://en.wikipedia.org/wiki/Merkle_tree [1] sadly does not mean you can rollback to any of them, when a new uberblock is created (which is common) the previous's set of metadata (meta-object set MOS) becomes reclaimable, and once the MOS has been reclaimed/reused the corresponding uberblock is useless
- minikites 13y agoI have a small NAS4Free box at home with ZFS and automatic zpool scrubbing. Barring a physical disaster (fire, flood, etc), I expect my data to be safe indefinitely. It was pretty easy to set it up on an HP Microserver, I recommend it.
- scrumper 13y agoI have a Drobo in my Amazon cart, was about to pull the trigger until I read this article. Do you know if your NAS4Free solution a viable solution for a household full of Macs that need a shared Time Machine destination? I'm tempted: it's cheaper, and NAS4Free will keep evolving.
- minikites 13y agoUnfortunately, I have no idea about Time Machine, the one Mac I have is running 10.4 (pre Time Machine). SMB, SSH/SFTP (even set up key-based auth), regular FTP all work great. Transmission works great and has a decent web interface. UPS support seems to be okay, haven't really tested it. DLNA/UPnP (fuppes) crashes all the time and I only tried it with the very buggy VLC for iOS, so I'm not sure how stable it actually is.
- scrumper 13y agoOK thanks - you've set me on a big research path here, and hopefully saved me some money (and a few bits).
- mynegation 13y agoAnything that can run Samba server can be a target for Time Machine backup, google DIY Time Machine. I considered FreeNAS but figured it would be too much time to ste up and went with the Synology Intel box instead, could not be happier with it. Synology has step-by-step instructions for setting it up as a Time Machine backup target.
- dmd 13y ago
- transfire 13y agoHuge kudos to ZFS and btrfs, but I am very disappointed with one detail of next generations file systems: Why, oh why, do we still not have a file-type metadata field? We are still using silly file name extensions and magic mime-type detection. In the Age of Types (OOP and FP strong type systems) it only makes sense for the file system to do the same.
- chrismonsanto 13y agoI personally think the biggest problem with current file systems is lack of ACID semantics. It is a tragedy that you can't rollback a set of file system changes if a shell script fails halfway. Why should I have to bring out SQLite if I want transactions, but don't need relations? (Also I agree--yes MIME types in filesystem please)
- stouset 13y agoEven if such a thing were to exist, it would probably by necessity be built on some lower-level system that doesn't provide such guarantees… like a filesystem.
- wmf 13y agoIIRC in the old Tandem OS transactions were a basic service, even lower level than the file system. So their file system could use the transaction service to perform arbitrary transactions without much complexity. These days all the world's a VAX though.
- omh 13y agoDepending on exactly what you're doing, ZFS (and probably btrfs?) provide at least the Consistency & Durability of ACID, and maybe Isolation as well. But yes, it seems like it would be relatively straightforward to build Atomicity guarantees on top of a COW-based filesystem. I guess it's unlikely that Oracle would add that to ZFS now though - they don't tend to like discouraging the use of relational databases!
- 13y ago
- Freaky 13y agoBit worrying not to see a single mention of ECC memory when discussing protection from bitrot, especially with filesystems which depend on correctly functioning memory to provide the protections people expect from them. People sneer at me for being a stickler for it, but between 50GB of memory and 26TB of ZFS-protected storage, I see ECC corrections about as often as I see disk checksum errors - maybe half a dozen of each in the past year or two. Frankly I think it's idiotic it's not more common and better supported.
- pedrocr 13y agoI agree and it's one of the reasons it's much easier to go with AMD for cheap home servers/NAS (HP Microserver line is great for example) as Intel tends to reserve ECC for very expensive parts. What doesn't seem easy is to get ECC in laptops. I don't think even traditionally business-focused lines have it (e.g., Thinkpads).
- stassats 13y agoNowadays you can get ECC with intel for quite cheap: http://ark.intel.com/search/advanced/?s=t&MarketSegment=DT&ECCMemory=true http://ark.intel.com/search/advanced/?s=t&MarketSegment=DT&E...
- pedrocr 13y agoThat's good to know. It seems the issue is actually motherboard support. Newegg shows a single intel motherboard with ECC and no AMD ones. I suppose the low-end server hardware (e.g., HP Microservers) just tend to be AMD so it's easiest to get a cheap AMD server with ECC than an Intel one. I had assumed it was an intel issue but apparently not. It would be nice if the intel ultrabook standard started including ECC as well. One can dream...
- Freaky 13y agoOn LGA1150 you need a C22x chipset for ECC support: http://www.intel.com/content/www/us/en/chipsets/server-chipsets/server-chipset-c222-c224-c226.html http://www.intel.com/content/www/us/en/chipsets/server-chips... Prices for these start about 3x higher than other boards, their choice is extremely limited, and availability even more so :/
- curmudgeoned 13y agoBecause the author does seem to care about defining what COW means, because "atomic cow" doesn't readily google to its proper meaning, and because it doesn't seem to be otherwise mentioned here: COW: Copy-On-Write http://en.wikipedia.org/wiki/Copy-on-write http://en.wikipedia.org/wiki/Copy-on-write
- rodgerd 13y agoNothing about Hammer? Pity.
- visarga 13y agoIn our days, it can take 10+ hours to stream all the data off a multi TB disk. A single disk can contain an amazing quantity of data, thus, it can be very valuable and sensitive. I'd like a file system that duplicates sensitive data on the same drive. The file system data should be duplicated too, and marked in some way as to be able to reconstruct a disk after failure. I don't need to save on space, I only need safety. And keep in mind that it takes many hours to dump a single time the contents of a disk - so, it's practically inaccessible as a whole over short periods of time. For now, I get by with DropBox and TimeMachine but it's far from perfect. My photo collection alone is 1TB, so, no luck in backing it up in the cloud.
- bensummers 13y agoUse ZFS. Create a filesystem in your pool with copies=n to get n copies on the same disc. But still use mirrors or raidz to protect against failure of a drive.
- ac29 13y ago>My photo collection alone is 1TB, so, no luck in backing it up in the cloud. Amazon glacier would store that for $10/month, though the retrieval costs if you needed to restore the whole thing are a bit more complicated.
- rossng 13y agoI use BackBlaze for several hundred GB of data - and that's mostly over a 3Mb/s ADSL connection. It's been running for several years now and I can vouch for it being absolutely rock solid.
- bensummers 13y agoZFS is also wonderfully usable. It's a delight to admin. See http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-is-better-than-btrfs http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-...
- pedrocr 13y agoThat seems pretty out of date. I only follow btrfs lightly but at least the RAIDZ and send/receive points are wrong. Current btrfs supports those.
- cpncrunch 13y agoGiven that hard disks already do a CRC check when reading data (as far as I am aware), I don't see how this adds anything. The author is assuming that there is no hardware-level checksum or CRC, which I believe is incorrect.
- touristtam 13y agoSo, Btrfs+linux or ZFS+BSD for a home server ? I thought the former was not production ready. I am confused.