3 ms·

This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all

This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.

boomboomsubban 7y ago
You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
- Dylan16807 7y ago
  That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
  - boomboomsubban 7y ago
    >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
    - Dylan16807 7y ago
      > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions? > Why not? The files are kept in a hidden folder in the base directory. Those are read-only views, not real folders. > You don't gain [much] with offline block deduplication over file, though it is better. In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
      - boomboomsubban 7y ago
        >I can't have a write to one file appear in a completely different file just because they had the same contents at one point. The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them. >Those are read-only views, not real folders. Then mount them writable, >But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.
        Dylan16807 7y ago
        > this is a benefit of a CoW filesystem, not offline dedupe It's one method of gaining CoW. And ZFS does not support CoW between files. But whatever feature you lump it under, it's an advantage Btrfs has. > Then mount them writable, You can't.[1] And even if you could, you can't make a hard link between subvolumes. [1] You can 'clone' a snapshot to make a writable subvolume, but the original read-only snapshot can never be deleted while the clone exists, so this doesn't help you fix duplicates.
    - braindeath 7y ago
      > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. That's bullshit. hard links are the same exact inode, with all that entails. reflinking is at the data block level. Totally different.
      - boomboomsubban 7y ago
        How is linking based on file and linking based on block "totally different?"
        cmurf 7y ago
        Hard links are essentially the filename that points to an inode. Creating a hard link creates just another filename associated with the same inode. Once a hard link, always a hard link, until destroyed. Reflink has its own fs metadata including inode, with (initially) shared extents. Those shared extents can have their blocks individually and independently modified, per file. The point at which there are no more shared blocks, they're not reflinks.
        im3w1l 7y ago
        IIUC: If you hard link two files, changing one file changes the other. If you dedupe two blocks, it will copy-on-write, meaning that changing one block does not change the other. It's not an intrinsic property of file-dedupe vs block-dedupe. It's just how it's conventionally done.
        braindeath 7y ago
        hardlinks are paths pointing to the same inode - the same metadata, the same contents - they are the same file - just happen to have multiple addresses. Inodes that happen to share blocks are not the same file. ie totally different.

This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.

boomboomsubban 7y ago

You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.

Dylan16807 7y ago
That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
- boomboomsubban 7y ago
  >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
  - Dylan16807 7y ago
    > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. For files that are used by anything, they might be written to. I can't have a write to one file appear in a completely different file just because they had the same contents at one point. And what if my duplicate files have different owners or permissions? > Why not? The files are kept in a hidden folder in the base directory. Those are read-only views, not real folders. > You don't gain [much] with offline block deduplication over file, though it is better. In general, perhaps not. But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments.
    - boomboomsubban 7y ago
      >I can't have a write to one file appear in a completely different file just because they had the same contents at one point. The act of saving should destroy the link, though I know that isn't always the case. Still, this is a benefit of a CoW filesystem, not offline dedupe. Permissions may be a problem, I'm not great with them. >Those are read-only views, not real folders. Then mount them writable, >But for ZFS specifically you gain a lot by not using its deduplication system. It needs tons of memory and explodes files into fragments Live deduplication is valuable in some instances, otherwise there are still offline alternatives available.
      - Dylan16807 7y ago
        > this is a benefit of a CoW filesystem, not offline dedupe It's one method of gaining CoW. And ZFS does not support CoW between files. But whatever feature you lump it under, it's an advantage Btrfs has. > Then mount them writable, You can't.[1] And even if you could, you can't make a hard link between subvolumes. [1] You can 'clone' a snapshot to make a writable subvolume, but the original read-only snapshot can never be deleted while the clone exists, so this doesn't help you fix duplicates.
  - braindeath 7y ago
    > Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. That's bullshit. hard links are the same exact inode, with all that entails. reflinking is at the data block level. Totally different.
    - boomboomsubban 7y ago
      How is linking based on file and linking based on block "totally different?"
      - cmurf 7y ago
        Hard links are essentially the filename that points to an inode. Creating a hard link creates just another filename associated with the same inode. Once a hard link, always a hard link, until destroyed. Reflink has its own fs metadata including inode, with (initially) shared extents. Those shared extents can have their blocks individually and independently modified, per file. The point at which there are no more shared blocks, they're not reflinks.
      - im3w1l 7y ago
        IIUC: If you hard link two files, changing one file changes the other. If you dedupe two blocks, it will copy-on-write, meaning that changing one block does not change the other. It's not an intrinsic property of file-dedupe vs block-dedupe. It's just how it's conventionally done.
      - braindeath 7y ago
        hardlinks are paths pointing to the same inode - the same metadata, the same contents - they are the same file - just happen to have multiple addresses. Inodes that happen to share blocks are not the same file. ie totally different.