4 ms·

I recently got Time Machine working on a network drive that was ZFS underneath (deduplication turned on) and it didn't make a very big difference. Deduplicatio

I recently got Time Machine working on a network drive that was ZFS underneath (deduplication turned on) and it didn't make a very big difference. Deduplication would be nice, but I think it won't show a lot of benefit under many common workloads. Also, it can use up a ton of memory for larger drives (it would be a huge boon for SSDs, though).

wtallis 16y ago
Deduplication matters most for people who edit big files. In my case, I have some virtual machines with hard drive images stored on my main filesystem. If I didn't exclude them from Time Machine, they would completely fill my backup drive in a matter of days, but with block-level deduplication, there would only be slight overhead compared with storing the same data directly on the HFS+ file system instead of in the images.
ComputerGuru 16y ago
Really the biggest culprit is Mail.app - so this would depend on how big your Mail archive is.
- johnthedebs 16y ago
  The reason deduplication won't help much with large mail archives is that it operates at the block level. In order to be effective, you need to have two copies of the same data that are also divided along the same block boundaries (the second part is arguably the bigger problem). Also, many applications will already try avoiding duplicate data to varying degrees which makes dedup even less beneficial (backup utilities, VCSes, etc). If you are dealing with a lot of text, though (like in a mail archive, or with lots of source code) compression really shines. It has way less memory overhead, pretty low CPU overhead, and it can significantly increase disk throughput. You'll notice I was careful to say "under many common workloads" in my other post, it was because of all this. For most people, most of the time, there would just be no real benefit and a decent performance hit with dedup. Edit: to clarify
  - ComputerGuru 16y ago
    You're missing my point: it's not duplicate blocks in a single mail archive that I'm concerned about, it's duplicate blocks across multiple versions of the same mail archive. New mails are typically appended to the end of the archive, but the rest of it remains more or less the same across versions. Also, a well-written block-level deduplication algorithm will work regardless of boundary shifts. I designed and wrote the proprietary blocklevel algorithm used in Genie Timeline, and can tell you that it saves terabytes of data in corporations for just PSTs alone. And it doesn't rely on block boundaries, capable of detecting slided or shifted data.
    - johnthedebs 16y ago
      I'm admittedly not an expert, but isn't that a very different scenario? In other words, ZFS operates on blocks at a block level - it can't make any assumptions about the contents other than "this block is the same as this block" or "this block is not the same as this block." It can't (based on my understanding) realize that "this block is almost the same as this block" and shift things accordingly, because then it would be messing with application data that might be corrupted by the changes. If the application is making those shifts on the other hand, it would work just fine because the application could be smart enough to shuffle and unshuffle things in the most efficient way it knows how, but the file system doesn't have the luxury. If I'm still missing something, please let me know. Anyway, it'd be pretty straightforward to properly test this so maybe I'll do that.
    - wmf 16y ago
      Hasn't Mail.app been using maildirs for years? They should be pretty Time-Machine-friendly.

wtallis 16y ago

Deduplication matters most for people who edit big files. In my case, I have some virtual machines with hard drive images stored on my main filesystem. If I didn't exclude them from Time Machine, they would completely fill my backup drive in a matter of days, but with block-level deduplication, there would only be slight overhead compared with storing the same data directly on the HFS+ file system instead of in the images.

ComputerGuru 16y ago

Really the biggest culprit is Mail.app - so this would depend on how big your Mail archive is.

johnthedebs 16y ago
The reason deduplication won't help much with large mail archives is that it operates at the block level. In order to be effective, you need to have two copies of the same data that are also divided along the same block boundaries (the second part is arguably the bigger problem). Also, many applications will already try avoiding duplicate data to varying degrees which makes dedup even less beneficial (backup utilities, VCSes, etc). If you are dealing with a lot of text, though (like in a mail archive, or with lots of source code) compression really shines. It has way less memory overhead, pretty low CPU overhead, and it can significantly increase disk throughput. You'll notice I was careful to say "under many common workloads" in my other post, it was because of all this. For most people, most of the time, there would just be no real benefit and a decent performance hit with dedup. Edit: to clarify
- ComputerGuru 16y ago
  You're missing my point: it's not duplicate blocks in a single mail archive that I'm concerned about, it's duplicate blocks across multiple versions of the same mail archive. New mails are typically appended to the end of the archive, but the rest of it remains more or less the same across versions. Also, a well-written block-level deduplication algorithm will work regardless of boundary shifts. I designed and wrote the proprietary blocklevel algorithm used in Genie Timeline, and can tell you that it saves terabytes of data in corporations for just PSTs alone. And it doesn't rely on block boundaries, capable of detecting slided or shifted data.
  - johnthedebs 16y ago
    I'm admittedly not an expert, but isn't that a very different scenario? In other words, ZFS operates on blocks at a block level - it can't make any assumptions about the contents other than "this block is the same as this block" or "this block is not the same as this block." It can't (based on my understanding) realize that "this block is almost the same as this block" and shift things accordingly, because then it would be messing with application data that might be corrupted by the changes. If the application is making those shifts on the other hand, it would work just fine because the application could be smart enough to shuffle and unshuffle things in the most efficient way it knows how, but the file system doesn't have the luxury. If I'm still missing something, please let me know. Anyway, it'd be pretty straightforward to properly test this so maybe I'll do that.
  - wmf 16y ago
    Hasn't Mail.app been using maildirs for years? They should be pretty Time-Machine-friendly.