3 ms·

Yes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old file

Yes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.

boomboomsubban 7y ago
>. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
- washadjeffmad 7y ago
  Not losing anything either, and less overhead. What is being argued here?
- the8472 7y ago
  This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
  - boomboomsubban 7y ago
    You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
    - Dylan16807 7y ago
      That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
      - boomboomsubban 7y ago
        >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
        Dylan16807 7y ago
        > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions? > Why not? The files are kept in a hidden folder in the base directory. Those are read-only views, not real folders. > You don't gain [much] with offline block deduplication over file, though it is better. In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
        boomboomsubban 7y ago
        >I can't have a write to one file appear in a completely different file just because they had the same contents at one point. The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them. >Those are read-only views, not real folders. Then mount them writable, >But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.

boomboomsubban 7y ago

>. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.

washadjeffmad 7y ago
Not losing anything either, and less overhead. What is being argued here?
the8472 7y ago
This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
- boomboomsubban 7y ago
  You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
  - Dylan16807 7y ago
    That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
    - boomboomsubban 7y ago
      >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
      - Dylan16807 7y ago
        > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions? > Why not? The files are kept in a hidden folder in the base directory. Those are read-only views, not real folders. > You don't gain [much] with offline block deduplication over file, though it is better. In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
        boomboomsubban 7y ago
        >I can't have a write to one file appear in a completely different file just because they had the same contents at one point. The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them. >Those are read-only views, not real folders. Then mount them writable, >But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.