5 ms·
Zfs, to take one example, does do approximately what you want - the file system (or rather, the underlying block storage) has a tree structure, and changes to f
by barrkel 12y ago
Zfs, to take one example, does do approximately what you want - the file system (or rather, the underlying block storage) has a tree structure, and changes to files are propagated via copy-on-write rewrites up the tree, so that earlier snapshots still get to see the original file.
This does make rolling things back trivial.
It doesn't give you dedupe for free, though. Think about what would have to happen: every modification would mean rehashing the modified block (not a problem, that should happen anyway to verify integrity). Let's say we dedupe at the block level rather than the file level to avoid the need for more expensive hashing operations, and to increase the likelihood of actually sharing stuff. Now, to determine whether we can deallocate the new block, we need to look up the hash. So we need an index of every block on the file system by hash. That necessarily involves either a big chunk of memory or a bunch of random I/O. Both are at a premium for a filesystem - the former for cache, the latter for throughput.
If you just want file dedupe, it's a smaller problem, but is less likely to create gains - most people don't store many copies of the same file, unless they're in the third party file storage business. So it isn't really suited to a general file system. If this is something you want, you could periodically go through your file system, hash all files with only one link count in the inode, and hard link them using the hash as a file name, into an set of directories fanning out by hash prefix. There may be some wrinkles with permissions; I believe btrfs has a different kind of copy with copy-on-write semantics that might be useful here.
- ClashTheBunny 12y agocp --reflink
- eeZi 12y agoOne of the few feature btrs has which ZFS hasn't.
- wtallis 12y agoAt this point, I don't think it's fair to imply that btrfs is lagging behind ZFS. Yes, there are quite a few things that ZFS does better than btrfs, but btrfs isn't following in ZFS's footsteps and has some killer features that ZFS will never have, like on-the-fly changing between RAID modes and resizing arrays in either direction.
- ptman 12y agoZFS would need block pointer rewrites to implement those features, right? I don't think ZFS developers are opposed to that, but progress is just stalled. So I think "will never have" is a bit strongly worded.
- wtallis 12y agohttps://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s https://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s The above video is an explanation of a bunch of the barriers to implementing block pointer rewrites. The conclusion is that it would make the code a lot more complicated and break a lot of the layering, and probably make addition of other new features a lot harder. Even a standalone offline rewriting tool wouldn't necessarily be accepted into the OpenZFS codebase because of the maintenance burden. Their advice is that if you think you need that feature to solve a particular problem, you should be looking for a workaround to solve that problem without requiring the huge block pointer rewrite project (which nobody's working on), even if the workarounds have a significant and permanent performance impact. When you take into account how long the feature's been in demand and been on the roadmaps under "eventually", it's clearly not going to happen anytime soon and won't happen without a major change to how ZFS development is being done. It's not definitely impossible, but it's perpetually several years away from happening. With btrfs already having it's equivalent to that feature and stealing an ever-growing slice of the users who need that feature, it's probably never going to happen for ZFS.
- crest 12y agoBtrfs has a broken by design on disk format. Btrfs went with self describing checksums inside blocks instead of a Merkle-DAG with a round robin root. In doing so they ignored existing research at the start of the btrfs project and the only fix is to change the on disk format to add the children's checkums to the parent nodes.
- wtallis 12y agoThat doesn't sound "broken by design" to me, just less thoroughly safeguarded than ZFS but still more than almost any other filesystem. It's completely consistent with my claim that btrfs isn't trying to follow exactly in the footsteps of ZFS. (And it's not like Merkle trees don't have any tradeoffs.)
- allanjude 12y agoA feature similar to this was discussed at the OpenZFS Developers Summit earlier this month http://open-zfs.org/w/images/7/71/Fast_File_Cloning-Pavel_Zakharov.pptx http://open-zfs.org/w/images/7/71/Fast_File_Cloning-Pavel_Za...
- vidoc 12y agoI haven't used freebsd for 5-6 years, so I've never tried to use it with root on ZFS. Wondering if it suffers the same problems as btrfs when used as root: last time I tried that on Ubuntu, the fsync() performance made a lot of stuff just horribly slow (e.g: dpkg operations two orders of magnitude slower).
- ClashTheBunny 12y agoI don't know when this was fixed, but I have no longer had any problems with btrfs on root with ubuntu. I don't remember when it started working well, but I do remember those bad old days. I don't know if dpkg changed or if btrfs changed.
- laumars 12y agoI run FreeBSD with ZFS as root and it doesn't suffer from those issues. But then ZFS is quite significantly more mature than Btrfs anyway. It's also worth noting that ZFS has been available as a root file system long before FreeBSD added ZFS to RELEASE. I remember running OpenSolaris (and some of it's forks, eg Nexenta) with ZFS root about 6 years ago. Possibly longer actually. The issue with ZFS as root was more of a problem with the boot menu than the OS. OpenSolaris used GRUB where as FreeBSD obviously doesn't, so FreeBSD needs to either port their ZFS drivers to their bootloader, or employ a hacky method of having a UFS boot volume that then points to a ZFS root partition (which, sadly, is how FreeBSD currently works). Interestingly, since GRUB is GPL, it meant that technically there were GPL ZFS drivers even before Btrfs started life (never mind the various Linux ports of CDDL-licenced ZFS drivers that have appeared since). Albeit those GPL ZFS drivers were read only
- Freaky 12y ago> FreeBSD needs to either port their ZFS drivers to their bootloader, or employ a hacky method of having a UFS boot volume that then points to a ZFS root partition (which, sadly, is how FreeBSD currently works). A ZFS-aware loader hit CURRENT in late 2008, and a dedicated zfsloader for use from (gpt)zfsboot hit the stable branches in late 2009.
- marcosdumay 12y agoDeleting is not a time sensitive command anyway. But I don't think you'd have this problem. Just like we have an inode telling every block of a file, a filesystem like that would need a similar structure telling every hash of the file. When you delete a block, you look at this structure, the same way you look at an inode. The only thing missing is that you'll need a counter at the blocks. And this counter will create some synchronization problems that may turn out more important than saving disk space.