12 ms·

This is great news. Among other incentives, ZFS has some truly excellent features for improving reliability. ZFS's built-in checksums, for example, can result

This is great news. Among other incentives, ZFS has some truly excellent features for improving reliability. ZFS's built-in checksums, for example, can result in much happier stories during the onset of disk failures: where a RAID array can quietly return incorrect sector contents without noticing, and be unable to correctly differentiate between the correct and not-so-correct sectors in the event of disk loss followed by disagreements discovered during rebuilds, ZFS simply does the right thing by making checks during normal operations, and uses the same checks to confidently do the right thing during recovery. And snapshotting. Oh, snapshotting. On the other hand, I've always wished we could get a modern re-take on ZFS. As anyone who's tried it will tell you: dedup in ZFS essentially doesn't work. ZFS, internally, is not built on content-addressable storage (or, it is, but since splitting of large files into blocks doesn't take any special actions to make similar blocks align perfectly, it doesn't have anywhere near the punch that it should). As a result, dedup operations that should be constant-time and zero memory overhead... aren't. Amazing though ZFS is, we've learned a lot about designing distributed and CAS storage since that groundwork was laid in ZFS. A new system that gets this right at heart would be monumental. Transporting snapshots (e.g. to other systems for backups... or to "resume" them (think pairing with CRIU containers)) could similarly be so much more powerful if only ZFS (or subsequent systems) can get content-addressable right on the same level that e.g. git does. `zfs send` can transport snapshots across the network to other storage pools -- amazing, right? It even has an incremental mode -- magic! In theory, this should be just like `git push` and `git fetch`: I should even be able to have, say n=3 machines, and have them all push snapshots of their filesystems to each other, and it should all dedup, right? And yet... as far as I can tell [1], the entire system is a footgun. Many operations break the ability receive incremental updates; if you thought you could make things topology agnostic... Mmm, may the force be with you. [1] https://gist.github.com/heavenlyhash/109b0b18df65579b498b https://gist.github.com/heavenlyhash/109b0b18df65579b498b -- These were my research notes on what kind of snapshot operations work, how they transport, etc. If you try to build anything using zfs send/recv, you may find these useful... and if anyone can find a shuffle of these commands with better outcomes I'd love to hear about it of course :)

ryao 11y ago
The deduplication code works, but each deduplication operation requires 3 serial IOS to lookup the information needed to check if deduplication is possible and if they are in aches, that becomes painful fast on storage with low IOPS. On my workstation where I have enough memory that the results of all of the lookups naturally fit and high IOPS storage, the deduplication code runs well. You would have a similar problem designing a system that perfectly deduplicates data at the record level if you tried.
- infogulch 11y ago
  I was thinking about this. To reduce both the huge ram usage and serial IOs you could use something similar to a Bloom filter to quickly test whether you should attempt to dedup a new block. If the filter says it's not a duplicate, then completely skip the standard (slow) dedup path. Bloom filters specifically have issues: they don't permit removing entries for one, and they're not really that efficient. But there's a paper about Cuckoo Filters which seems to solve both of these problems. For example: The "semi-sort" variant of the cuckoo filter benchmarked in the paper has a size of 192 MB and holds 128M items. So for 8kb blocks, it can dedup 1TB of blocks. More if you increase the block size or the size of the table. It has a 0.09% false-positive rate (!). I.e. unique blocks would use the slow path to test for duplication in vain only once in 1111 writes. The algorithm can perform 6 million lookups per second on the benchmark hardware. (2x Xeons at 2.27GHz, 12MB L3, 32 GB DRAM) This is assuming that the majority of writes are actually unique, and dedup is more of a "it would be nice" thing than essential. But for that case something like this would be a lot easier to implement and use far fewer resources. Just stick it in front of the existing dedup lookup and early-exit if the filter says it's not duplicate. [0]: https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
bryanlarsen 11y ago
Note that ZFS isn't magic. Even with ZFS's checksums on read, you should still be doing regular scrubs, just like you should be with LVM or btrfs. And once you have regular scrubs, checksums on read don't really add much.

ryao 11y ago

The deduplication code works, but each deduplication operation requires 3 serial IOS to lookup the information needed to check if deduplication is possible and if they are in aches, that becomes painful fast on storage with low IOPS. On my workstation where I have enough memory that the results of all of the lookups naturally fit and high IOPS storage, the deduplication code runs well. You would have a similar problem designing a system that perfectly deduplicates data at the record level if you tried.

infogulch 11y ago
I was thinking about this. To reduce both the huge ram usage and serial IOs you could use something similar to a Bloom filter to quickly test whether you should attempt to dedup a new block. If the filter says it's not a duplicate, then completely skip the standard (slow) dedup path. Bloom filters specifically have issues: they don't permit removing entries for one, and they're not really that efficient. But there's a paper about Cuckoo Filters which seems to solve both of these problems. For example: The "semi-sort" variant of the cuckoo filter benchmarked in the paper has a size of 192 MB and holds 128M items. So for 8kb blocks, it can dedup 1TB of blocks. More if you increase the block size or the size of the table. It has a 0.09% false-positive rate (!). I.e. unique blocks would use the slow path to test for duplication in vain only once in 1111 writes. The algorithm can perform 6 million lookups per second on the benchmark hardware. (2x Xeons at 2.27GHz, 12MB L3, 32 GB DRAM) This is assuming that the majority of writes are actually unique, and dedup is more of a "it would be nice" thing than essential. But for that case something like this would be a lot easier to implement and use far fewer resources. Just stick it in front of the existing dedup lookup and early-exit if the filter says it's not duplicate. [0]: https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf

bryanlarsen 11y ago

Note that ZFS isn't magic. Even with ZFS's checksums on read, you should still be doing regular scrubs, just like you should be with LVM or btrfs. And once you have regular scrubs, checksums on read don't really add much.