4 ms·
Are you really saving that much space doing that on a block level rather than a file level?
by boomboomsubban 7y ago
Are you really saving that much space doing that on a block level rather than a file level?
- secabeen 7y agoIf you are doing it on disk images for VMs or virtual desktops, yes.
- sliken 7y agoAgreed, but otherwise no.
- the8472 7y agoYes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
- boomboomsubban 7y ago>. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
- washadjeffmad 7y agoNot losing anything either, and less overhead. What is being argued here?
- the8472 7y agoThis has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
- boomboomsubban 7y agoYou can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
- Dylan16807 7y agoThat's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
- boomboomsubban 7y ago>fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
- Dylan16807 7y ago> Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions? > Why not? The files are kept in a hidden folder in the base directory. Those are read-only views, not real folders. > You don't gain [much] with offline block deduplication over file, though it is better. In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
- Dylan16807 7y agoI'm not sure what you're comparing to. Both filesystems do deduplication at a block level. But Btrfs can do reflink copies and ZFS can't. And Btrfs can do after-the-fact deduplication and ZFS can't.
- garmaine 7y agoHe's saying scan for files that are the same and hard-link them. Both filesystems can do this after-the-fact. Of course this only works if the data is read-only.
- Dylan16807 7y ago> Both filesystems can do this after-the-fact. It won't reclaim any space on ZFS if you're using snapshots.
- mrighele 7y agoI'm not sure the saving is much bigger. I would expect that in most cases if the two files differ most of the blocks will be different (e.g. add a byte to a file at the beginning, now no block is the same). I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.