3 ms·

How does application-level deduplication compare to ZFS deduplication?

GrayShade 10y ago
Application-level dedup usually works at the file level, while ZFS dedups data blocks (of variable size). The ZFS way of doing it allows saving space on large files that are similar, like VM images. On the other hand it requires storing block hashes in memory. A ZFS dedup table entry is 320 bytes in size [1], so assuming 128 KB blocks that comes to 2.5 GB / TB. Moreover, the tables are kept in the ZFS cache (ARC) and they have to compete with other data and metadata. Since metadata is by default limited to 75% of ARC [2], you'll need at least 3.3 GB per TB and on the Internet you'll find even worse figures, like 20 GB [3]. [1] http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html http://www.oracle.com/technetwork/articles/servers-storage-a... [2] https://github.com/zfsonlinux/zfs/commit/9907cc1cc8c16fa2c7c03aa33264153ca2bdad6c https://github.com/zfsonlinux/zfs/commit/9907cc1cc8c16fa2c7c... [3] http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-ded...
- raattgift 10y ago
  tl;dr: I agree with you but would put it: DON'T DO DEDUP AT ALL, since even having all that RAM doesn't save you from real world problems. Ultimately, the deduplication table (DDT) is consulted on each write to any dataset or zvol where deduplication is not "off". The DDT layout on the underlying storage is effectively random. It is roughly an AVL tree with each node corresponding to a block checksum. While the tree is packed into as few underlying blocks as possible, as the DDT grows a larger number of seeks are needed to reach each node. This is important because back-to-back writes even of very similar data will need to consult spatially separated parts of the DDT. Bear in mind that the DDT is brought into ARC on a demand basis, i.e., all that memory you advise goes unused if there are no writes to a dedup!=off dataset or zvol. Where you have enough IOPS to service a couple of random reads per write, then you do not need the DDT in memory, so it's good that it is pushed out by hotter zfs metadata. This can be the case for high-4k-random-IOPS SSDs, for instance. On rotating media -- spinning disks -- you will NEVER have enough IOPS. [1] In perfectly ordinary circumstances, after a reboot or pool import or major arc shrink, the DDT will have to be brought in by future writes, which increases write latency by a factor of the track-to-track seek time for every write. The DDT reads are synchronous, affecting the entire TXG sync, so the extra latency affects all writes to the same pool, even to other (dedup=off) datasets and zvols. The extra seeking will also interfere with uncached asynchrnonous reads from the same pool. [2] After a crash it is wholly possible that the DDT has to be walked during the import phase, so your import will have to wait on many many track-to-track seeks before it becomes available; on openzfs, this is liable to stall all sync tasks for all pools (i.e., "zpool list" is likely to hang, "zfs snapshot" on another already-imported pool might.) This is especially likely in the presence of a deferred destroy. [3] Dataset and snapshot destroys require consulting the DDT for practically every DMU object in the snapshot or dataset; that's basically every file and directory at the POSIX layer, or the inside-the-zvol filesystem's metadata (and possibly many many ordinary blocks because of UNMAP). If your DDT isn't in RAM during this, that's multiple seeks per object. Additionally the DDT and the zfs metadata pointing to it needs to be updated as well, and that is almost never seek-free. [4] 2 & 3 combine: if you crash or reboot out of frustration while a destroy of a deduped dataset or zvol is in progress, your import will have to wait until the destroy is completed, only this time NONE of the DDT will be in main memory to start with. [5] resilvers and scrubs always walk the whole DDT first. If you have to replace a leaf vdev and you have a substantial DDT, you will have a bad time. [6] the times involved to deal with a deferred destroy or cold DDT walk can range from hours to weeks depending on the deduplicated data and the freespace at the time each write was first committed. Mitigations like persistent L2ARC, or vdev pinning (allocating a top level vdev for DDT metadata, for instance, so you can use a mirrored nvram to hold the DDT for a pool with rotating media holding the file data) have not yet landed in openzfs. On fast nvram for datasets that are read-mostly and which are highly deduplicatable, dedup=skein can be a win. However, even if seeks are virtually "free", heavy writing to such a dataset is almost certainly going to make one's ARC less effective for caching other zfs metadata and file data, and main RAM is almost always much much more expensive and scarce than solid state leaf vdev hardware. Additionally, the benefits of queueing on the hardware itself tend to evaporate because of the additional random read load (random reads are the enemy of throughput!). So even where DDT goes from merely survivable (which is never the case on media where random reads take on the order of milliseconds, because one fault will lead you to data and system unavailability) to pretty tolerable (a wide n-way-mirrored SSD-based pool) ZFS deduplication is unlikely to be worth turning on, and it's hard to back out of. It's better to focus on shared subtree strategies (zfs clone, zfs receive -o origin=snap, checksums that allow nopwrite, etc.) and compression rather than deduplication. And where you can't, it's better to spend the money adding more top level vdevs than it is to spend the time dealing with a situation where deduplication leads to unavailability.
throwawayish 10y ago
It doesn't, really. Applications can use much more domain knowledge to do the dedup, possibly with less resource usage or better speed (memory, eg. because an index to dedup with already exists, or CPU, eg. it can use existing constraints on data and doesn't need to hash all data). FS-level deduplication is... well. For classic FSes it does exist [1], but it's a pretty ugly kludge, and is pretty much inherently an off-line process that can't happen during writing the data the first time. For CoW FSes it's less of a problem, and eg. ZFS and btrfs can do it, but it's often not a exactly a great fit. (Although one could see their snapshotting features also as deduplication) [1] through extent sharing, ext4 does this, iirc experimental in XFS. duperemove uses this.

Application-level dedup usually works at the file level, while ZFS dedups data blocks (of variable size). The ZFS way of doing it allows saving space on large files that are similar, like VM images. On the other hand it requires storing block hashes in memory. A ZFS dedup table entry is 320 bytes in size [1], so assuming 128 KB blocks that comes to 2.5 GB / TB. Moreover, the tables are kept in the ZFS cache (ARC) and they have to compete with other data and metadata. Since metadata is by default limited to 75% of ARC [2], you'll need at least 3.3 GB per TB and on the Internet you'll find even worse figures, like 20 GB [3]. [1] http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html http://www.oracle.com/technetwork/articles/servers-storage-a... [2] https://github.com/zfsonlinux/zfs/commit/9907cc1cc8c16fa2c7c03aa33264153ca2bdad6c https://github.com/zfsonlinux/zfs/commit/9907cc1cc8c16fa2c7c... [3] http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-ded...

raattgift 10y ago
tl;dr: I agree with you but would put it: DON'T DO DEDUP AT ALL, since even having all that RAM doesn't save you from real world problems. Ultimately, the deduplication table (DDT) is consulted on each write to any dataset or zvol where deduplication is not "off". The DDT layout on the underlying storage is effectively random. It is roughly an AVL tree with each node corresponding to a block checksum. While the tree is packed into as few underlying blocks as possible, as the DDT grows a larger number of seeks are needed to reach each node. This is important because back-to-back writes even of very similar data will need to consult spatially separated parts of the DDT. Bear in mind that the DDT is brought into ARC on a demand basis, i.e., all that memory you advise goes unused if there are no writes to a dedup!=off dataset or zvol. Where you have enough IOPS to service a couple of random reads per write, then you do not need the DDT in memory, so it's good that it is pushed out by hotter zfs metadata. This can be the case for high-4k-random-IOPS SSDs, for instance. On rotating media -- spinning disks -- you will NEVER have enough IOPS. [1] In perfectly ordinary circumstances, after a reboot or pool import or major arc shrink, the DDT will have to be brought in by future writes, which increases write latency by a factor of the track-to-track seek time for every write. The DDT reads are synchronous, affecting the entire TXG sync, so the extra latency affects all writes to the same pool, even to other (dedup=off) datasets and zvols. The extra seeking will also interfere with uncached asynchrnonous reads from the same pool. [2] After a crash it is wholly possible that the DDT has to be walked during the import phase, so your import will have to wait on many many track-to-track seeks before it becomes available; on openzfs, this is liable to stall all sync tasks for all pools (i.e., "zpool list" is likely to hang, "zfs snapshot" on another already-imported pool might.) This is especially likely in the presence of a deferred destroy. [3] Dataset and snapshot destroys require consulting the DDT for practically every DMU object in the snapshot or dataset; that's basically every file and directory at the POSIX layer, or the inside-the-zvol filesystem's metadata (and possibly many many ordinary blocks because of UNMAP). If your DDT isn't in RAM during this, that's multiple seeks per object. Additionally the DDT and the zfs metadata pointing to it needs to be updated as well, and that is almost never seek-free. [4] 2 & 3 combine: if you crash or reboot out of frustration while a destroy of a deduped dataset or zvol is in progress, your import will have to wait until the destroy is completed, only this time NONE of the DDT will be in main memory to start with. [5] resilvers and scrubs always walk the whole DDT first. If you have to replace a leaf vdev and you have a substantial DDT, you will have a bad time. [6] the times involved to deal with a deferred destroy or cold DDT walk can range from hours to weeks depending on the deduplicated data and the freespace at the time each write was first committed. Mitigations like persistent L2ARC, or vdev pinning (allocating a top level vdev for DDT metadata, for instance, so you can use a mirrored nvram to hold the DDT for a pool with rotating media holding the file data) have not yet landed in openzfs. On fast nvram for datasets that are read-mostly and which are highly deduplicatable, dedup=skein can be a win. However, even if seeks are virtually "free", heavy writing to such a dataset is almost certainly going to make one's ARC less effective for caching other zfs metadata and file data, and main RAM is almost always much much more expensive and scarce than solid state leaf vdev hardware. Additionally, the benefits of queueing on the hardware itself tend to evaporate because of the additional random read load (random reads are the enemy of throughput!). So even where DDT goes from merely survivable (which is never the case on media where random reads take on the order of milliseconds, because one fault will lead you to data and system unavailability) to pretty tolerable (a wide n-way-mirrored SSD-based pool) ZFS deduplication is unlikely to be worth turning on, and it's hard to back out of. It's better to focus on shared subtree strategies (zfs clone, zfs receive -o origin=snap, checksums that allow nopwrite, etc.) and compression rather than deduplication. And where you can't, it's better to spend the money adding more top level vdevs than it is to spend the time dealing with a situation where deduplication leads to unavailability.

throwawayish 10y ago

It doesn't, really. Applications can use much more domain knowledge to do the dedup, possibly with less resource usage or better speed (memory, eg. because an index to dedup with already exists, or CPU, eg. it can use existing constraints on data and doesn't need to hash all data). FS-level deduplication is... well. For classic FSes it does exist [1], but it's a pretty ugly kludge, and is pretty much inherently an off-line process that can't happen during writing the data the first time. For CoW FSes it's less of a problem, and eg. ZFS and btrfs can do it, but it's often not a exactly a great fit. (Although one could see their snapshotting features also as deduplication) [1] through extent sharing, ext4 does this, iirc experimental in XFS. duperemove uses this.