4 ms·
I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline. ZFS's dedup uses grotesque
by LeoPanthera 10y ago
I think the filesystem is a fine place to do it, I just wish more filesystems would give us the option to do deduplication offline.
ZFS's dedup uses grotesque amounts of RAM since it insists on holding the dedup table in it and only doing dedup in realtime.
btrfs however allows you to scan the filesystem and do dedup of existing data later (even while it is mounted), using no RAM at all. This is one of a couple of places where I think btrfs is actually becoming a better-designed filesystem than ZFS.
- Skunkleton 10y agoThere are patches to BTRFS (actively developed) for inband dedup as well.
- jjobi 10y agoBTRFS has so many drawbacks using it as storage driver for docker. See the bottom of the page https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/ https://docs.docker.com/engine/userguide/storagedriver/btrfs.... Primary drawback being each container creates unique copy of the data in kernel page cache and taking away system resources from applications.
- jjobi 10y agoAgain, the focus of this blog is not about a de-duplication story, but about a new docker storage driver.
- LeoPanthera 10y agoWell thankfully open discussion is a feature of Hacker News, not a bug.