3 ms·
Are there any deduplicating filesystems for Linux similar to the (very basic) NTFS deduplication in Windows? I’ve fiddled with quite a few Linux dedup solutions
by tatersolid 8y ago
Are there any deduplicating filesystems for Linux similar to the (very basic) NTFS deduplication in Windows? I’ve fiddled with quite a few Linux dedup solutions over the years but nothing seems to be production ready. Even ZFS isn’t that useful as it only chunks on multiples of the block size.
The basic feature set for deduplication in my book is non-block-aligned chunking with compression. NTFS has had transparent post-process dedup since 2012 and is pretty much ideal for file, VM, and backup servers. It just works in the background with little performance impact but huge space savings.
It feels dirty using Windows servers as NAS for our Linux machines, but that’s what we’re doing today.
- newnewpdro 8y agoWith XFS and reflink, out-of-band deduplication is totally possible and is a userspace [1] issue. But XFS is not doing anything to assist in accelerating the identification of duplicate blocks, instead it simply implements ioctls for what is essentially extent sharing. [1] https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove
- linsomniac 8y agobtrfs has some dedup options, but I just recently looked and they all seem like kludges. I'd really like HAMMER on Linux, but that doesn't seem to be going anywhere. And I don't really want to run DragonFly for my backup servers, for maintenance reasons.