4 ms·

I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I f

I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I first started this project, 2TB hard drives were the largest available. My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache. Every few years, I try to do a better job of things but at this point, the best improvement would be radical simplification. ZFS has served very well in terms of reliability. I haven't lost data, and I've been able to catch lots of episodes of almost losing data. Or writing the wrong data. Not entirely sure how I'd replace it, if I want something that can spot bit rot and correct it. ZFS scrub.

roygbiv2 2y ago
Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.
- watersb 2y ago
  This archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives. While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.) Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational. The application leads to a big payoff for deduplication.
emptiestplace 2y ago
Cache or ZIL (SLOG device)?
- watersb 2y ago
  Both the ZIL and the L2ARC, plus a third "special" cache which aggregates small blocks and could hold the dedupe table. The ZIL is the "ZFS Intent Log", a log-structured ordered stream of file operations to be performed on the ZFS volume. If power goes out, or the disk controller goes away unexpectedly, this ZIL is the log that will get replayed to bring the volume back to a consistent state. I think. Usually the ZIL is on the same storage devices as the rest of the data. So a write to the ZIL has to wait for disk in the same line as everybody else. It might improve performance to give the ZIL its own, dedicated storage devices. NVMe is great, lower latency the better. Since the ZFS Intent Log gets flushed to disk every five seconds or so, a dedicated ZIL device doesn't have to be very big. But it has to be reliable and durable. Windows made small, durable NVMe cache drives a mainstream item for a while, when most laptops still used rotating hard drives. Optane NVMe at 16GB is cheap, like twenty bucks, buy three of them and use a mirrored pair of two for your ZIL. ---- Then there's the read cache, the ARC. I use 1TB mirrored NVMe devices. Finally, there's a "special" device that can for example be designated for use for intense things like the dedupe table (which Fast Dedupe is making smaller!).
  - emptiestplace 2y ago
    A couple things: - The ZIL is used exclusively for synchronous writes - critical for VMs, databases, NFS shares, and other applications requiring strict write consistency. Many conventional workloads won't benefit. Use `zilstat` to monitor. - The cheap 16GB Optane devices are indeed great in terms of latency, but they were designed primarily for read caching and have significantly limited write speeds. If you need better throughput, look for the larger Optane models which don't have these limitations. - SLOG doesn't need to be mirrored - the only risk is if your SLOG device fails at the exact moment your system crashes. While mirroring is reasonable for production systems, with these cheap 16GB Optanes you're just guaranteeing they'll wear out at the same time. You could kill one at a time instead. :) - As for those 1TB NVMe devices for read cache (L2ARC) - that's probably overkill unless you have a very specific use case. L2ARC actually consumes RAM to track what's in the cache, and that RAM might be better used for ARC (the main memory cache). L2ARC only makes sense when you have well-understood workload patterns and your ARC is consistently under pressure - like in a busy database server or similar high-traffic scenario. Use `arcstat` to monitor your cache hit ratios before deciding if you need L2ARC.

roygbiv2 2y ago

Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.

watersb 2y ago
This archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives. While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.) Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational. The application leads to a big payoff for deduplication.

emptiestplace 2y ago

Cache or ZIL (SLOG device)?

watersb 2y ago
Both the ZIL and the L2ARC, plus a third "special" cache which aggregates small blocks and could hold the dedupe table. The ZIL is the "ZFS Intent Log", a log-structured ordered stream of file operations to be performed on the ZFS volume. If power goes out, or the disk controller goes away unexpectedly, this ZIL is the log that will get replayed to bring the volume back to a consistent state. I think. Usually the ZIL is on the same storage devices as the rest of the data. So a write to the ZIL has to wait for disk in the same line as everybody else. It might improve performance to give the ZIL its own, dedicated storage devices. NVMe is great, lower latency the better. Since the ZFS Intent Log gets flushed to disk every five seconds or so, a dedicated ZIL device doesn't have to be very big. But it has to be reliable and durable. Windows made small, durable NVMe cache drives a mainstream item for a while, when most laptops still used rotating hard drives. Optane NVMe at 16GB is cheap, like twenty bucks, buy three of them and use a mirrored pair of two for your ZIL. ---- Then there's the read cache, the ARC. I use 1TB mirrored NVMe devices. Finally, there's a "special" device that can for example be designated for use for intense things like the dedupe table (which Fast Dedupe is making smaller!).
- emptiestplace 2y ago
  A couple things: - The ZIL is used exclusively for synchronous writes - critical for VMs, databases, NFS shares, and other applications requiring strict write consistency. Many conventional workloads won't benefit. Use `zilstat` to monitor. - The cheap 16GB Optane devices are indeed great in terms of latency, but they were designed primarily for read caching and have significantly limited write speeds. If you need better throughput, look for the larger Optane models which don't have these limitations. - SLOG doesn't need to be mirrored - the only risk is if your SLOG device fails at the exact moment your system crashes. While mirroring is reasonable for production systems, with these cheap 16GB Optanes you're just guaranteeing they'll wear out at the same time. You could kill one at a time instead. :) - As for those 1TB NVMe devices for read cache (L2ARC) - that's probably overkill unless you have a very specific use case. L2ARC actually consumes RAM to track what's in the cache, and that RAM might be better used for ARC (the main memory cache). L2ARC only makes sense when you have well-understood workload patterns and your ARC is consistently under pressure - like in a busy database server or similar high-traffic scenario. Use `arcstat` to monitor your cache hit ratios before deciding if you need L2ARC.