6 ms·

Please correct me if I'm wrong, but the "1GB ram per TB" is for the de-duplication table. ZFS' memory cache (ARC) by default uses up to 50% of system ram and f

Please correct me if I'm wrong, but the "1GB ram per TB" is for the de-duplication table. ZFS' memory cache (ARC) by default uses up to 50% of system ram and frees this under pressure. You can also set it lower.

jcalvinowens 5y ago
Yes, this is what I'm talking about. With respect to ARC, it gets along very poorly with the native page cache and memory allocator. I would add that BTRFS has no such requirements for deduplication.
- db48x 5y ago
  That’s because the btrfs deduplicator is quite a different animal. You could just as easily run fdupes or something similar. Meanwhile ZFS is doing something much more ambitious. In any case, I don’t think that either of them are worth using in most cases. Deduplication is just too expensive to be worth it.
  - Dylan16807 5y ago
    > You could just as easily run fdupes or something similar. If you only deduplicate whole files and soft links or hard links are suitable for your situation.
    - db48x 5y ago
      That’s not the aspect of the deduplication process that I was referring to. Btrfs only supports offline deduplication, where you periodically run a process that searches for duplicate blocks and combines them. Meanwhile ZFS supports online deduplication, where every block written is checked against an index to see if it is a duplicate of some existing block. That index is what takes up the memory that jcalvinowens complained about. When deduplication is turned on, ZFS keeps that whole index in memory all of the time. Meanwhile btrfs only needs that index to be loaded while the deduplicator is running, so it doesn’t show up most of the time. Personally I don’t think that either is really worth the cost; the benefit is so small that the cost isn’t worth paying.
      - Dylan16807 5y ago
        I know the ZFS deduplicator accomplishes more, I'm just pointing out that that if you want offline deduplication on ZFS, you can't have it.

jcalvinowens 5y ago

Yes, this is what I'm talking about. With respect to ARC, it gets along very poorly with the native page cache and memory allocator. I would add that BTRFS has no such requirements for deduplication.

db48x 5y ago
That’s because the btrfs deduplicator is quite a different animal. You could just as easily run fdupes or something similar. Meanwhile ZFS is doing something much more ambitious. In any case, I don’t think that either of them are worth using in most cases. Deduplication is just too expensive to be worth it.
- Dylan16807 5y ago
  > You could just as easily run fdupes or something similar. If you only deduplicate whole files and soft links or hard links are suitable for your situation.
  - db48x 5y ago
    That’s not the aspect of the deduplication process that I was referring to. Btrfs only supports offline deduplication, where you periodically run a process that searches for duplicate blocks and combines them. Meanwhile ZFS supports online deduplication, where every block written is checked against an index to see if it is a duplicate of some existing block. That index is what takes up the memory that jcalvinowens complained about. When deduplication is turned on, ZFS keeps that whole index in memory all of the time. Meanwhile btrfs only needs that index to be loaded while the deduplicator is running, so it doesn’t show up most of the time. Personally I don’t think that either is really worth the cost; the benefit is so small that the cost isn’t worth paying.
    - Dylan16807 5y ago
      I know the ZFS deduplicator accomplishes more, I'm just pointing out that that if you want offline deduplication on ZFS, you can't have it.