5 ms·

One thing I like about btrfs is offline deduplication. So whenever you want you can search for duplications and then tell the filesystem about what you find.

by sliken 7y ago

One thing I like about btrfs is offline deduplication. So whenever you want you can search for duplications and then tell the filesystem about what you find. This avoids many of the performance and ram impacts of ZFS deduplication.

boomboomsubban 7y ago
Are you really saving that much space doing that on a block level rather than a file level?
- secabeen 7y ago
  If you are doing it on disk images for VMs or virtual desktops, yes.
  - sliken 7y ago
    Agreed, but otherwise no.
- the8472 7y ago
  Yes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
  - boomboomsubban 7y ago
    >. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
    - washadjeffmad 7y ago
      Not losing anything either, and less overhead. What is being argued here?
    - the8472 7y ago
      This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
      - boomboomsubban 7y ago
        You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
        Dylan16807 7y ago
        That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
        boomboomsubban 7y ago
        >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
- Dylan16807 7y ago
  I'm not sure what you're comparing to. Both filesystems do deduplication at a block level. But Btrfs can do reflink copies and ZFS can't. And Btrfs can do after-the-fact deduplication and ZFS can't.
  - garmaine 7y ago
    He's saying scan for files that are the same and hard-link them. Both filesystems can do this after-the-fact. Of course this only works if the data is read-only.
    - Dylan16807 7y ago
      > Both filesystems can do this after-the-fact. It won't reclaim any space on ZFS if you're using snapshots.
- mrighele 7y ago
  I'm not sure the saving is much bigger. I would expect that in most cases if the two files differ most of the blocks will be different (e.g. add a byte to a file at the beginning, now no block is the same). I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.
beerandt 7y ago
Man, I wish zfs had this. Or a periodic scan, maybe run with a flag as part of, or after, a scrub. The current implementation of dedup in zfs was the biggest disappointment I had in setting up my current storage box. Even with plenty of horsepower, I quickly turned it off and accepted the net reduction in capacity.
- toast0 7y ago
  There is a command, zdb -S pool [1], that scans the pool to generate a report about duplication --- of course, it doesn't make changes. [1] I put the wrong command here originally.
  - beerandt 7y ago
    As in, it reports any deduplication already implemented? Or actually detects duplicate blocks? Because that would be still be something, knowing how much deduplication would benefit you, if implemented.
    - agapon 7y ago
      https://www.freebsd.org/cgi/man.cgi?query=zdb&sektion=8 https://www.freebsd.org/cgi/man.cgi?query=zdb&sektion=8
    - toast0 7y ago
      Actually detect duplicate blocks. There's also commands to show the already implemented deduplication, of course.
  - agapon 7y ago
    You are thinking of zdb -S, of course.
londons_explore 7y ago
how does offline dedup work? Does it require reading all file data to locate duplicates? That sounds expensive to do anywhere except nearly idle systems. Any filesystem with data checksumming should only need to build an index of the checksums to find duplicates, which sounds far cheaper. Does btrfs do that?
- sliken 7y ago
  Well it's up to you. Generally you can look for duplicated blocks OR files, for any subset. So you could look at only files over 1 month. Or maybe any file over 2GB you could look for block level deduplication and for anything under 2GB you could look for file level deduplication. The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper. Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact." For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication. Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
  - londons_explore 7y ago
    Sounds like they don't use file data checksums... If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
    - sliken 7y ago
      Btrfs uses data and metadata checksums, those pre-existing checksums allow for checking for duplications without having to read the underlying files. So sounds like we are violently agreeing. Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums. ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive. Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.
mrighele 7y ago
Is the issue with ZFS deduplication really due to the fact that it is done online ? From what I understand the way that ZFS works, the amount of ram used to keep deduplication metadata would be necessary even if the deduplication was done offline, i.e. it is necessary also for read operations, not only for write operations.
- sliken 7y ago
  I believe the ZFS deduplication consumes a ton of ram (somewhere around 1GB of ram per TB of disk) to keep the block checksums in ram. Said list is checked for every block write. I don't believe it's necessary to check for reads, since the file metadata already tells you which blocks to look for. As a result ZFS with deduplication on uses a ton of ram and has slow writes. This is so big an impact that most folks just turn it off, despite the tempting lure of "free" disk space.
magicalhippo 7y ago
There was some talk about improving ZFS deduplication considerably by not trying to be perfect. That is, try to deduplicate against a subset of the data, not against all the data. This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops. Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance: https://www.youtube.com/watch?v=PYxFDBgxFS8 https://www.youtube.com/watch?v=PYxFDBgxFS8 http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf
gerdesj 7y ago
What I like about ext4 and xfs is that they are relatively simple and have a clearly defined set of capabilities that anyone can understand (within reason). They are not volume managers or anything else fancy (I have LVM/md etc for that). However, if I do have a machine that is not housed on a hypervisor with a SAN or other decent RAID then I still go for simple. I do use zfs in preference to ufs on my pfSense boxes running non RAID on mSATA type storage but that is not a scientific choice but based on some quite iffy hearsay. I have my opinion and you have yours. When it comes to filesystems then I want simple (and good backups.) Fast is nice but reliable is better. If I want faster I buy a bigger RAID controller or whatever.
- TheDong 7y ago
  Unfortunately ext4 and xfs are fast, but not reliable. They don't have data-checksumming, so it's very hard to know when you need to go to your backups. Their backup options are drastically more complex because there's no way to take an atomic snapshot. The complexity of data-integrity and backups are pushed a layer up, whereas more complex filesystems manage data-integrity and atomic snapshotting themselves, and can do so more easily due to existing at the correct layer.

by sliken 7y ago

boomboomsubban 7y ago

Are you really saving that much space doing that on a block level rather than a file level?

secabeen 7y ago
If you are doing it on disk images for VMs or virtual desktops, yes.
- sliken 7y ago
  Agreed, but otherwise no.
the8472 7y ago
Yes. For example when you're using snapshots but download new versions of something on the live volume then most installer software just overwrites the old files with nearly or fully identical new files. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Lots of software is not as CoW-friendly as it could be, offline dedup fixes that with relatively low overhead.
- boomboomsubban 7y ago
  >. Running an offline dedup will then go back and reintegrate the new files with the dupes on the snapshots. Running a dedupe based on file rather than block would do the same thing. You don't gain much if anything in your example using btrfs.
  - washadjeffmad 7y ago
    Not losing anything either, and less overhead. What is being argued here?
  - the8472 7y ago
    This has nothing to do with blocks or files. The point is that you can do that only when it's needed (not often) without paying the price (ZFS dedup tables) all the time.
    - boomboomsubban 7y ago
      You can do it when needed by file on any filesystem, fdupes is a common program used for the process. The only advantage btrfs has is searching for duplicate blocks rather than files. In the few cases where that saves a meaningful amount of space, ZFS's live deduplication would probably be worth the RAM.
      - Dylan16807 7y ago
        That's not right. * fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions * fdupes will not let you reconcile duplicates in snapshots on ZFS * ZFS deduplication is block-based
        boomboomsubban 7y ago
        >fdupes will let you make hard links on any filesystem, but hard links do the wrong thing when you edit them or need different permissions Hard links mostly work as you tell them to. You may need to set things up properly, but it's essentially the same thing being done. >fdupes will not let you reconcile duplicates in snapshots on ZFS Why not? The files are kept in a hidden folder in the base directory. >ZFS deduplication is block-based It is also live. You don't gain Mich with offline block deduplication over file, though it is better. Live deduplication is a feature even if unnecessary for most.
Dylan16807 7y ago
I'm not sure what you're comparing to. Both filesystems do deduplication at a block level. But Btrfs can do reflink copies and ZFS can't. And Btrfs can do after-the-fact deduplication and ZFS can't.
- garmaine 7y ago
  He's saying scan for files that are the same and hard-link them. Both filesystems can do this after-the-fact. Of course this only works if the data is read-only.
  - Dylan16807 7y ago
    > Both filesystems can do this after-the-fact. It won't reclaim any space on ZFS if you're using snapshots.
mrighele 7y ago
I'm not sure the saving is much bigger. I would expect that in most cases if the two files differ most of the blocks will be different (e.g. add a byte to a file at the beginning, now no block is the same). I think that block-level deduplication has more predictable performance though: for example if you write some blocks to a deduplicated file, with block-level deduplication you just write the new blocks, with file-level deduplication you have to rewrite the whole file.

beerandt 7y ago

Man, I wish zfs had this. Or a periodic scan, maybe run with a flag as part of, or after, a scrub. The current implementation of dedup in zfs was the biggest disappointment I had in setting up my current storage box. Even with plenty of horsepower, I quickly turned it off and accepted the net reduction in capacity.

toast0 7y ago
There is a command, zdb -S pool [1], that scans the pool to generate a report about duplication --- of course, it doesn't make changes. [1] I put the wrong command here originally.
- beerandt 7y ago
  As in, it reports any deduplication already implemented? Or actually detects duplicate blocks? Because that would be still be something, knowing how much deduplication would benefit you, if implemented.
  - agapon 7y ago
    https://www.freebsd.org/cgi/man.cgi?query=zdb&sektion=8 https://www.freebsd.org/cgi/man.cgi?query=zdb&sektion=8
  - toast0 7y ago
    Actually detect duplicate blocks. There's also commands to show the already implemented deduplication, of course.
- agapon 7y ago
  You are thinking of zdb -S, of course.

londons_explore 7y ago

how does offline dedup work? Does it require reading all file data to locate duplicates? That sounds expensive to do anywhere except nearly idle systems. Any filesystem with data checksumming should only need to build an index of the checksums to find duplicates, which sounds far cheaper. Does btrfs do that?

sliken 7y ago
Well it's up to you. Generally you can look for duplicated blocks OR files, for any subset. So you could look at only files over 1 month. Or maybe any file over 2GB you could look for block level deduplication and for anything under 2GB you could look for file level deduplication. The utilities can be written in any language, there's several popular ones called dupremove, bedup, btrfs-dedupe, bees, and dduper. Bedup specifically mentions "It integrates deeply with btrfs so that scans are incremental and low-impact." For my use I'd likely ignore all files less than a few weeks old for deduplication purposes. Any virtual machine images I'd use block level deduplication and everything else I'd use file level deduplication. Doesn't seem much different overhead wise than the normal scrub that linux software RAID, ZFS, and btrfs does anyways. I'm all for not storing old files twice, but would like to avoid the overhead of checking for deduplication on every write.
- londons_explore 7y ago
  Sounds like they don't use file data checksums... If they did, the dedup process for a multi tb drive would just be a few seconds, and the overhead would be so small it could be run on the whole drive every minute all day long.
  - sliken 7y ago
    Btrfs uses data and metadata checksums, those pre-existing checksums allow for checking for duplications without having to read the underlying files. So sounds like we are violently agreeing. Yes you can deduplicate 1TB quickly if you already have the data and metadata data checksums. ZFS (with dedupe on) checks the checksum of every written block against every existing block in the pool, thus the slow writes and being very ram intensive. Btrfs instead calculates data and metadata checksums for integrity then allows using them afterwards for deduplication. You could of course check for block level checksums for 100% of your storage. Or you could decide to dedupe only old files, only VMs, or only non-VMs. Or even a hybrid of block level for VMs, but file level for non-VMs.

mrighele 7y ago

Is the issue with ZFS deduplication really due to the fact that it is done online ? From what I understand the way that ZFS works, the amount of ram used to keep deduplication metadata would be necessary even if the deduplication was done offline, i.e. it is necessary also for read operations, not only for write operations.

sliken 7y ago
I believe the ZFS deduplication consumes a ton of ram (somewhere around 1GB of ram per TB of disk) to keep the block checksums in ram. Said list is checked for every block write. I don't believe it's necessary to check for reads, since the file metadata already tells you which blocks to look for. As a result ZFS with deduplication on uses a ton of ram and has slow writes. This is so big an impact that most folks just turn it off, despite the tempting lure of "free" disk space.

magicalhippo 7y ago

There was some talk about improving ZFS deduplication considerably by not trying to be perfect. That is, try to deduplicate against a subset of the data, not against all the data. This allows the dedup table (which stores the hashes of existing data to find identical blocks) to be limited in size to whatever you want, and especially prevent it from eating all the RAM. The tradeoff is that the smaller it is, the deduplication hit rate drops. Here's a talk by Matt Ahrens about it, which also includes some other tricks to improve dedup performance: https://www.youtube.com/watch?v=PYxFDBgxFS8 https://www.youtube.com/watch?v=PYxFDBgxFS8 http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf http://open-zfs.org/w/images/8/8d/ZFS_dedup.pdf

gerdesj 7y ago

What I like about ext4 and xfs is that they are relatively simple and have a clearly defined set of capabilities that anyone can understand (within reason). They are not volume managers or anything else fancy (I have LVM/md etc for that). However, if I do have a machine that is not housed on a hypervisor with a SAN or other decent RAID then I still go for simple. I do use zfs in preference to ufs on my pfSense boxes running non RAID on mSATA type storage but that is not a scientific choice but based on some quite iffy hearsay. I have my opinion and you have yours. When it comes to filesystems then I want simple (and good backups.) Fast is nice but reliable is better. If I want faster I buy a bigger RAID controller or whatever.

TheDong 7y ago
Unfortunately ext4 and xfs are fast, but not reliable. They don't have data-checksumming, so it's very hard to know when you need to go to your backups. Their backup options are drastically more complex because there's no way to take an atomic snapshot. The complexity of data-integrity and backups are pushed a layer up, whereas more complex filesystems manage data-integrity and atomic snapshotting themselves, and can do so more easily due to existing at the correct layer.