4 ms·

Do Btrfs or ext4 offer this?

wtallis 3y ago
This feature is basically the same as what underpins the reflink feature that btrfs has supported approximately forever and xfs has supported for at least several years.
- mustache_kimono 3y ago
  Does anyone know whether btrfs or XFS support reflinks from snapshot datasets?
  - Dylan16807 3y ago
    I can confirm BTRFS yes, but note that source and destination need to be on the same mount point before kernel 5.18
  - ComputerGuru 3y ago
    XFS doesn't have native snapshot support, though?
  - danudey 3y ago
    XFS doesn't have snapshot support, so the short answer there is no.
    - mustache_kimono 3y ago
      Shows what I know about XFS. Thanks!
      - plq 3y ago
        This is only a tangent given we are talking about snapshots and reflink, but just wanted to mention that LVM has snapshots, so if you need XFS snapshots, create the XFS filesystem on top of an LVM logical volume.
        danudey 3y ago
        It's worth noting that, while LVM can support snapshots for XFS (or any other filesystem, really), the performance penalty is almost entirely unacceptable (one benchmark[0] shows an up to 60% drop in write throughput), so ideally your process should be something like 1. Snapshot LVM volume 2. Mount the LVM volume and back the snapshot up somewhere else 3. Delete the snapshot as soon as possible Otherwise you're going to lose a ton of performance. [0] https://www.percona.com/blog/disaster-lvm-performance-in-snapshot-mode/ https://www.percona.com/blog/disaster-lvm-performance-in-sna...
      - PlutoIsAPlanet 3y ago
        You can get psuedo-snapshots on XFS with a tool like https://github.com/aravindavk/reflink-snapshot https://github.com/aravindavk/reflink-snapshot But, it still has to duplicate metadata which depending on the amount of files may cause inconsistency in the snapshot.
thrtythreeforty 3y ago
Btrfs yes, ext4 no (but I believe xfs does). This should end up being exposed through cp --reflink=always, so you could look up filesystem support for that.
- danudey 3y ago
  XFS does, I've used it for specifically this feature before.
- cmurf 3y ago
  cp defaults to reflink=auto for a while now, but of course some distros are way behind on coreutils.
dsr_ 3y ago
You can get a similar effect on top of any file system that supports hard links with rdfind ( https://rdfind.pauldreik.se/ https://rdfind.pauldreik.se/ ) -- but it's pretty slow. The Arch wiki says: "Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, bedup and btrfs-dedup. One may also want to merely deduplicate data on a file based level instead using e.g. rmlint, jdupes or dduper-git. For an overview of available features of those programs and additional information, have a look at the upstream Wiki entry. Furthermore, Btrfs developers are working on inband (also known as synchronous or inline) deduplication, meaning deduplication done when writing new data to the filesystem. Currently, it is still an experiment which is developed out-of-tree. Users willing to test the new feature should read the appropriate kernel wiki page."
- someplaceguy 3y ago
  > You can get a similar effect on top of any file system that supports hard links with rdfind ( https://rdfind.pauldreik.se/ https://rdfind.pauldreik.se/ ) -- but it's pretty slow. It's a similar effect only if you don't modify the files, I think. If you "clone" a file with a hard link and you modify the contents of one copy, the other copy would also be equally modified. As far as I understand this wouldn't happen with this type of block cloning: each copy of the file would be completely separate, except that they may (transparently) share data blocks on disk.

This feature is basically the same as what underpins the reflink feature that btrfs has supported approximately forever and xfs has supported for at least several years.

mustache_kimono 3y ago
Does anyone know whether btrfs or XFS support reflinks from snapshot datasets?
- Dylan16807 3y ago
  I can confirm BTRFS yes, but note that source and destination need to be on the same mount point before kernel 5.18
- ComputerGuru 3y ago
  XFS doesn't have native snapshot support, though?
- danudey 3y ago
  XFS doesn't have snapshot support, so the short answer there is no.
  - mustache_kimono 3y ago
    Shows what I know about XFS. Thanks!
    - plq 3y ago
      This is only a tangent given we are talking about snapshots and reflink, but just wanted to mention that LVM has snapshots, so if you need XFS snapshots, create the XFS filesystem on top of an LVM logical volume.
      - danudey 3y ago
        It's worth noting that, while LVM can support snapshots for XFS (or any other filesystem, really), the performance penalty is almost entirely unacceptable (one benchmark[0] shows an up to 60% drop in write throughput), so ideally your process should be something like 1. Snapshot LVM volume 2. Mount the LVM volume and back the snapshot up somewhere else 3. Delete the snapshot as soon as possible Otherwise you're going to lose a ton of performance. [0] https://www.percona.com/blog/disaster-lvm-performance-in-snapshot-mode/ https://www.percona.com/blog/disaster-lvm-performance-in-sna...
    - PlutoIsAPlanet 3y ago
      You can get psuedo-snapshots on XFS with a tool like https://github.com/aravindavk/reflink-snapshot https://github.com/aravindavk/reflink-snapshot But, it still has to duplicate metadata which depending on the amount of files may cause inconsistency in the snapshot.

thrtythreeforty 3y ago

Btrfs yes, ext4 no (but I believe xfs does). This should end up being exposed through cp --reflink=always, so you could look up filesystem support for that.

danudey 3y ago
XFS does, I've used it for specifically this feature before.
cmurf 3y ago
cp defaults to reflink=auto for a while now, but of course some distros are way behind on coreutils.

dsr_ 3y ago

You can get a similar effect on top of any file system that supports hard links with rdfind ( https://rdfind.pauldreik.se/ https://rdfind.pauldreik.se/ ) -- but it's pretty slow. The Arch wiki says: "Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, bedup and btrfs-dedup. One may also want to merely deduplicate data on a file based level instead using e.g. rmlint, jdupes or dduper-git. For an overview of available features of those programs and additional information, have a look at the upstream Wiki entry. Furthermore, Btrfs developers are working on inband (also known as synchronous or inline) deduplication, meaning deduplication done when writing new data to the filesystem. Currently, it is still an experiment which is developed out-of-tree. Users willing to test the new feature should read the appropriate kernel wiki page."

someplaceguy 3y ago
> You can get a similar effect on top of any file system that supports hard links with rdfind ( https://rdfind.pauldreik.se/ https://rdfind.pauldreik.se/ ) -- but it's pretty slow. It's a similar effect only if you don't modify the files, I think. If you "clone" a file with a hard link and you modify the contents of one copy, the other copy would also be equally modified. As far as I understand this wouldn't happen with this type of block cloning: each copy of the file would be completely separate, except that they may (transparently) share data blocks on disk.