15 ms·

OpenZFS deduplication is good now and you shouldn't use it

454 points by type0 2y ago

nikisweeting 2y ago
I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools compress within a crawl to create wacz or warc.gz files, but I don't think anyone has tried to do compression across the entire database of all snapshots ever taken by a tool. Big thank you to all the people that worked on it! BTW has anyone tried a probabilistic dedup approach using soemthing like a bloom filter so you don't have to store the entire dedup table of hashes verbatim? Collect groups of ~100 block hashes into a bucket each, and store a hyper compressed representation in a bloom filter. On write, lookup the hash of the block to write in the bloom filter, and if a potential dedup hit is detected, walk the 100 blocks in the matching bucket manually to look for any identical hashes. In theory you could do this with layers of bloom filters with different resolutions and dynamically swap out the heavier ones to disk when memory pressure is too high to keep the high resolution ones in RAM. Allowing the accuracy of the bloom filter to be changed as a tunable parameter would let people choose their preference around CPU time/overhead:bytes saved ratio.
- uniqueuid 2y ago
  I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.
  - nikisweeting 2y ago
    WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.
    - uniqueuid 2y ago
      That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction. [edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler. https://support.archive-it.org/hc/en-us/articles/208001016-About-data-de-duplication https://support.archive-it.org/hc/en-us/articles/208001016-A...
      - nikisweeting 2y ago
        Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.
- mappu 2y ago
  Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives. dm-vdo has the same behaviour. You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)
  - nikisweeting 2y ago
    I should clarify I don't use WARCs at all with archivebox, it just stores raw files on the filsystem because I rely on ZFS for all my compression, so there is no offset alignment issue. The wget extractor within archivebox can produce WARCs as an output but no parts of ArchiveBox are built to rely on those, they are just one of the optional extractors that can be run.
- alchemist1e9 2y ago
  While a slightly different use case, I suspect you’d like zbackup if you don’t know about it.
deleted 2y ago
[deleted]
dark-star 2y ago
I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...
- wmf 2y ago
  Fixup block pointers is the one thing ZFS didn't want to do.
- magicalhippo 2y ago
  > and fixup the block pointers That's why. Due to reasons[1], ZFS does not have the capability to rewrite block pointers. It's been a long requested feature[2] as it would also allow for defragmentation. I've been thinking this could be solved using block pointer indirection, like virtual memory, at the cost of a bit of speed. But I'm by no means a ZFS developer, so there's surely something I'm missing. [1]: http://eworldproblems.mbaynton.com/posts/2014/zfs-block-pointer-rewrite-status-2013/index.html http://eworldproblems.mbaynton.com/posts/2014/zfs-block-poin... [2]: https://github.com/openzfs/zfs/issues/3582 https://github.com/openzfs/zfs/issues/3582
  - phongn 2y ago
    It looks like they’re playing more with indirection features now (created for vdev removal) for other features. One of the recent summit hackathons sketched out using indirect vdevs to perform rebalancing. Once you get a lot of snapshots, though, the indirection costs start to rise.
- olavgg 2y ago
  You can also use DragonFlyBSD with Hammer2, which supports both online and offline deduplication. It is very similar to ZFS in many ways. The big drawback though, is lack of file transfer protocols using RDMA. I've also heard there are some experimental branches that makes it possible to run Hammer2 on FreeBSD. But FreeBSD also lacks RDMA support. For FreeBSD 15, Chelsio has sponsored NVMe-oF target, and initiator support. I think this is just TCP though.
Wowfunhappy 2y ago
I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives. But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.
- magicalhippo 2y ago
  The author of the new file-based block cloning code had this in mind. A backround process would scan files and identify dupes, delete the dupes and replace them with cloned versions. There are of course edge cases to consider to avoid data loss, but I imagine it might come soon, either officially or as a third-party tool.
- Dylan16807 2y ago
  The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.
  - Wowfunhappy 2y ago
    You need block pointer rewrite for this?
    - Dylan16807 2y ago
      You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.
tilt_error 2y ago
If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?
- klysm 2y ago
  Kinda like log structured merge tree?
- 0x457 2y ago
  Because to make this work without a lot of copying, you would need to mutate things that ZFS absolutely does not want to make mutable.
- UltraSane 2y ago
  If the block to be written is already being stored then you will match the hash and the block won't have to be written. This can save a lot of write IO in real world use.
- magicalhippo 2y ago
  Keep in mind ZFS was created at a time when disks were glacial in comparison to CPUs. And, the fastest write is the one you don't perform, so you can afford some CPU time to check for duplicate blocks. That said, NVMe has changed that balance a lot, and you can afford a lot less before you're bottlenecking the drives.
klysm 2y ago
I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.
- magicalhippo 2y ago
  Internally ZFS is essentially an object store. There was some work which tried to expose it through an object store API. Sadly it seems to not have gone anywhere. Tried to find the talk but failed, was sure I had seen it on a Delveloper Summit but alas.
- UltraSane 2y ago
  Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?
  - mappu 2y ago
    High-density drives are usually zoned storage, and it's pretty difficult to implement the regular filesystem API on top of that with any kind of reasonable performance (device- vs host- managed SMR). The S3 API can work great on zones, but only because it doesn't let you modify an existing object without rewriting the whole thing, which is an extremely rough tradeoff.
  - lazide 2y ago
    It’s only a ‘disaster’ if you are using it exclusively programmatically and want to do special tuning. File systems are pretty good if you have a mix of human and programmatic uses, especially when the programmatic cases are not very heavy duty. The programmatic scenarios are often entirely human hostile, if you try to imagine what would be involved in actually using them. Like direct S3 access, for example.
  - deleted 2y ago
    [deleted]
  - perlgeek 2y ago
    One way it's a disaster is that file names (on Linux at least, haven't used Windows in a long time) are byte strings that can contain directory paths from different/multiple file systems. So if you have non-ASCII characters in your paths, encoding/decoding is guesswork, and at worst, differs from path segment to path segment, and there's no metadata attached which encoding to use.
tjwds 2y ago
Edit: disregard this, I was wrong and missed the comment deletion window.
- deleted 2y ago
  [deleted]
- gtirloni 2y ago
  HN will automatically redirect the submitter to a recent submission instead of allowing a new post... if it had a significant number of comments. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
kderbe 2y ago
I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this discussion with opinions on contrast ratios...) For less patient readers, note that the concise summary is at the bottom of the post, not the top.
- emptiestplace 2y ago
  It scrolls horizontally :(
  - going_north 2y ago
    It's because of this element in one of the final sections [1]: <code>kstat.zfs.<pool>.misc.ddt_stats_<checksum></code> Typesetting code on a narrow screen is tricky! [1] https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/#thats-a-lot-anything-else https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-...
  - ThePowerOfFuet 2y ago
    Not on Firefox on Android it doesn't.
    - dspillett 2y ago
      It does in chrome on android (1080 px wide screen, standard ppi & zoom levels) but not by enough that you see it on the main body text (scrolling just reveals more margin), so you might find it does for you too but not enough that you noticed. As it is scrolling here, though inconsequentially, it might be bad on a smaller device with less screen and/or other ppi settings.
- Aachen 2y ago
  That being: > As we’ve seen from the last 7000+ words, the overheads are not trivial. Even with all these changes, you still need to have a lot of deduplicated blocks to offset the weight of all the unique entries in your dedup table. [...] what might surprise you is how rare it is to find blocks eligible for deduplication are on most general purpose workloads. > But the real reason you probably don’t want dedup these days is because since OpenZFS 2.2 we have the BRT (aka “block cloning” aka “reflinks”). [...] it’s actually pretty rare these days that you have a write operation coming from some kind of copy operation, but you don’t know that came from a copy operation. [...] [This isn't] saving as much raw data as dedup would get me, though it’s pretty close. But I’m not spending a fortune tracking all those uncloned and forgotten blocks. > [Dedup is only useful if] you have a very very specific workload where data is heavily duplicated and clients can’t or won’t give direct “copy me!” signal The section labeled "summary" imo doesn't do the article justice by being fairly vague. I hope these quotes from near the end of the article give a more concrete idea of why (not) use it
bastloing 2y ago
Forget dedupe just use zfs compression, a lot more bang for your buck
- Joel_Mckay 2y ago
  Unless your data-set is highly compressed media files. In general, even during rsync operations one often turns off compression on large video files, as the compression operation has low or negative impact on storage/transfers while eating ram and cpu power. De-duplication is good for Virtual Machine OS images, as the majority of the storage cost is a replicated backing image. =3
  - bastloing 2y ago
    Compression is still king. Check out HP's Nimble storage arrays. Way quicker to do compression, fewer iops, and less overhead. Even when it misses, like video files, it's still a winner.
eek2121 2y ago
So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe. No, you won't save much on a client system. That isn't what the feature is made for.
- hinkley 2y ago
  When ZFS first came out I had visions of it being a turnkey RAID array replacement for nontechnical users. Pop out the oldest disk, pop in a new (larger one), wait for the pretty lights to change color. Done. It is very clear that consumer was never a priority, and so I wonder what the venn diagram is of 'client system' and 'zfs filesystem'. Not that big right?
- UltraSane 2y ago
  My reaction also. Dedupe is a must have for when you are storing hundreds of VMs. you WILL save so much data and inline dedupe will save a lot of write IO.
  - XorNot 2y ago
    It's an odd notion in the age of containers where dedupe is like, one of the core things we do (but stupidly: amongst dissimilar images there's definitely more identical files then different ones).
- edelbitter 2y ago
  I tried two of the most non-random archives I had and was disappointed just as the author. For mail archives, I got 10%. For entire filesystems, I got.. just as much as with any other COW. Because indeed, I duplicate them only once. Later shared blocks are all over the place.
- doublepg23 2y ago
  I assuming the author is aware why the feature exists since they state in the second sentence they funded the improvement over the course of two years?
UltraSane 2y ago
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. The effectiveness of dedupe is strongly affected by the size of the blocks being hashed, with the smaller the better. As the blocks get smaller the odds of having a matching block grow rapidly. In my experience 4KB is my preferred block size.
- acdha 2y ago
  > Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.
  - UltraSane 2y ago
    What software?
    - SteveNuts 2y ago
      Logrotate
    - acdha 2y ago
      Log rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.
      - UltraSane 2y ago
        That is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.
UltraSane 2y ago
Knowing that your storage has really good inline dedupe is awesome and will affect how you design systems. Solid dedupe lets you effectively treat multiple copies of data as symlinks.
nisten 2y ago
can someone smarter than me explain what happens when instead of the regular 4kb block size in kernel builds we use 16kb or 64kb block size or is that only for the memory part, i am confused. Will a larger block size make this thing good or bad?
- UltraSane 2y ago
  Generally the smaller the dedupe block the better as you are far more likely to find a matching block. But larger blocks will reduce the number of hashes you have to store. In my experience 4KB is the sweet spot to maximize how much data you save.
  - spockz 2y ago
    So in this case I think it would make sense to have a separate pool where you store large files like media so you can save on the dedup for them. Is there an inherent performance loss of using 64kB blocks on FS level when using storage devices that are 4kB under the hood?
    - nisten 2y ago
      Hmmm you might be able to do both no? Like the dedube is gonna run at the filesystem level but your memory security & ownership stuff is gonna run more efficiently. I am not sure.
tiffanyh 2y ago
OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?
- nikisweeting 2y ago
  - https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove - https://codeberg.org/jbruchon/jdupes https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/ https://www.jdupes.com/ - https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes - https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind
- acdha 2y ago
  The built in Photos duplicate feature is the best choice for most people: it’s not just generic file-level dedupe but smart enough to do things like take three versions of the same photo and pick the highest-quality one, which is great if you ever had something like a RAW/TIFF+JPEG workflow or mixed full res and thumbnails.
- spockz 2y ago
  Or better yet. A single photo I take of the kids will be stored in my camera roll. I will then share it with family using three different messengers. Now I have 4 copies. Each of the individual (recoded) are stored inside those messengers and also backed up. This even happens when sharing the same photo multiple times in different chats with the same messenger. Is there any way to do de duplication here? Or just outright delete all the derivatives?
- EraYaN 2y ago
  digiKam can dedupe on actual similarity (so different resizes and formats of the same image). But it does take some time to calculate all the hashes.
wpollock 2y ago
When the lookup key is a hash, there's no locality over the megabytes of the table. So don't all the extra memory accesses to support dedup affect the L1 and L2 caches? Has anyone at OpenZFS measured that? It also occurs to me that spacial locality on spinning rust disks might be affected, also affecting performance.
forrestthewoods 2y ago
My dream Git successor would use either dedupe or a simple cache plus copy-on-write so that repos can commit toolchains and dependencies and users wouldn’t need to worry about disk drive bloat. Maybe someday…
- fragmede 2y ago
  It does dedup using Sha-1 on entire files. you might try git-lfs for your usecase though.
  - forrestthewoods 2y ago
    Git LFS is a really really bad gross hack. It’s awful. https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/ https://www.forrestthewoods.com/blog/dependencies-belong-in-...
    - fragmede 2y ago
      It's quite functional and usable, now. so I'd agree with hack, just not the rest of your adjectives. That was a good read! I've been thinking a lot about what comes after git too. One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.
      - forrestthewoods 2y ago
        > One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo. That’s the problem that a virtual file system solves. When you clone a repo it only materializes files when they’re accessed. This is how my work repo operates. It’s great.
        fragmede 2y ago
        The problem I'm trying to avoid is having to dive down a hierarchy, and that not everyone needs or wants to know there is such a hierarchy. Like graphics design for foo-team only needs to have access to some subset of graphics. Arguably they could be given a symlink into the checkout, but the problem with that is of having a singular checkout. The problem I had with Google3 is that the tools weren't great at branching and didn't fit my workflow, which tends to involve multiple checkouts (or worktrees using git). being forced to checkout the root of the repo, and then having to manage a symlink on top of that is no good for users that don't need/want to manage the complexity of having a single machine-global checkout.
girishso 2y ago
Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.
- nikisweeting 2y ago
  How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically? There are a few different ways you could solve it but it depends on what final outcome you need.
  - girishso 2y ago
    Just reporting in some plain text format so I can manually delete the duplicates, or create some script to delete. I can't have like 10 external HDDs attached at the same time, so the tool needs to dump details (hashes?) somewhere on Mac HDD, and compare against those to find the duplicates.
    - nikisweeting 2y ago
      Here you go: cd /path/to/drive find . -type f -exec sha256sum {} + | sed -E 's/^([^ ]+) \./\1,/' >> ~/all_hashes.txt Run that for each drive, then when you're done run: sort ~/all_hashes.txt > ~/sorted_hashes.txt awk -F, 'NR==1{print;next} {print $0 | "sort | uniq -w64 -D"}' ~/sorted_hashes.txt > ~/non_unique_hashes.txt The output in ~/non_unique_hashes.txt will contain only the non-unique hashes that appear on more than one path.
- UltraSane 2y ago
  dupeGuru works pretty well.
hhdhdbdb 2y ago
Any timing attacks possible on a virtualized system using dedupe? Eg find out what my neighbours have installed. Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.
- aidenn0 2y ago
  I don't think you even need timing attacks if you can read the zpool statistics; you can ask for a histogram of deduped blocks. Guessing one byte at a time is not possible though because dedupe is block-level in ZFS.
  - beng-nl 2y ago
    Gosh, you’re likely right, but what if comparing the blocks (to decide on deduping) is a byte at a time and somehow that can be detected (with a timing channel or a uarch side channel)? Zfs likely compares the hash, but I think KSM doesn’t use hashes but memcmp (or something in that spirit) to avoid collisions. So just maybe… just maybe GP is onto something.. interesting fantasy ;-)
    - hhdhdbdb 2y ago
      Thanks for putting meat on the (speculitive) bone I threw out! Very interesting.
- UltraSane 2y ago
  VMWare ESXi used to dedupe RAM and had to disable this by default because of a security issue it caused that leaded data between VMs.
rodarmor 2y ago
General-purpose deduplication sounds good in theory but tends not to work out in practice. IPFS uses a rolling hash with variable-sized pieces, in an attempt to deduplicate data rysnc-style. However, in practice, it doesn't actually make a difference, and adds complexity for no reason.
rkagerer 2y ago
I'd love if dedicated hardware existing in disk controllers for calculating stuff like ECC could be enhanced to expose hashes of blocks to the system. Getting this for free for all your I/O would allow some pretty awesome things.
- UltraSane 2y ago
  That is a neat idea. Hard drives could do dedupe from the ECC they calculate for each sector. The main issue with that is that the current ECC is optimal for detecting bit errors but doesn't have the same kind of statistical guarantee of uniqueness that SHA256 or MetroHash has. You need to be VERY confident of the statistical properties of the hash used in dedupe if you are going to increment the ref count of the block hash instead of writing the data to disk.
watersb 2y ago
I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I first started this project, 2TB hard drives were the largest available. My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache. Every few years, I try to do a better job of things but at this point, the best improvement would be radical simplification. ZFS has served very well in terms of reliability. I haven't lost data, and I've been able to catch lots of episodes of almost losing data. Or writing the wrong data. Not entirely sure how I'd replace it, if I want something that can spot bit rot and correct it. ZFS scrub.
- roygbiv2 2y ago
  Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.
  - watersb 2y ago
    This archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives. While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.) Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational. The application leads to a big payoff for deduplication.
- emptiestplace 2y ago
  Cache or ZIL (SLOG device)?
  - watersb 2y ago
david_draco 2y ago
In addition to the copy_file_range discussion at the end, it would be great to be able to applying deduplication to selected files, identified by searching the filesystem for say >1MB files which have identical hash.
nabla9 2y ago
You should use: cp --reflink=auto You get file level deduplication. The command above performs a lightweight copy (ZFS clone in file level), where the data blocks are copied only when modified. Its a copy, not a hard link. The same should work in other copy-on-write transactional filesystems as well if they have reflink support.
simonjgreen 2y ago
We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]
- jamesbfb 2y ago
  Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!
  - rwarfield 2y ago
    Isn't it better to use `nix store optimise` for dedup of the nix store? The nix command has more knowledge of the structure of the nix store so should be able to do a better job with fewer resources. Also the store is immutable so you don't actually need reflinks - hard links are enough.
    - Filligree 2y ago
      It is, yeah, though you have to turn it on. I'm not actually sure why it's off by default.
      - amarshall 2y ago
        It’s off by default as it can make builds slower (regardless of platform)—you should test this if you care. There also are (or were) some bugs on macOS that would cause corruption.
        Filligree 2y ago
        That seems like the wrong default. Most people do very little building on their desktops; they get all their software from the cache.
  - 2y ago
nobrains 2y ago
What are the use cases where it makes sense to use de-dup? Backup comes to mind. What else?
- 3np 2y ago
  Many similar VM images or even live root filesystems.
- zitsarethecure 2y ago
  Flatpak uses OSTree dedup. https://blogs.gnome.org/wjjt/2021/11/24/on-flatpak-disk-usage-and-deduplication/ https://blogs.gnome.org/wjjt/2021/11/24/on-flatpak-disk-usag...
- UltraSane 2y ago
  VMware + All flash storage array + inline dedupe/compression = happy users.
merpkz 2y ago
I don't get it - many people here claim in this thread that VM base image deduplication is great use case for this. So lets assume there are couple of hundreds of VMs on a ZFS dataset with dedupe on, each of them ran by different people for different purposes entirely - some databases, some web frontends / backends, minio S3 storage or backups ect - this might save you those measly hundreds of megabytes for linux system files those VMs might have in common ( even though knowing how many linux versions are out there with different patch levels - unlikely ) it will still not be worth it considering ZFS will keep track of each users individual files - databases and backup files and whatnot - data which is almost guaranteed to be unique between users so it will completely miss the point of ZFS deduplication. What am I missing?
- jeroenhd 2y ago
  It largely depends on how you set up your environment. On my home server, most VMs consist of a few gigabytes of a base Linux system and then a couple of hundred megabytes of application code. Some of those VMs also store large amounts of data, but most of that data could be stored in something like a dedicated minio server and maybe a dedicated database server. I could probably get rid of a huge chunk of my used storage if I switched to a deduplicating system (but I have plenty of storage so I don't really need to). If you're selling VMs to customers then there's probably no advantage in using deduplication.
- 3np 2y ago
  In such a sevario you'd probably have several partitions. So dedupe activated on the root filesystem (/bin,/lib etc) but not for /home and /var.
xmodem 2y ago
We have a read-heavy zpool with some data that's used as part of our build process, on which we see a roughly 8x savings with dedup - and because of this ZFS dedup makes it economically viable for us to store the pool on NVMe rather than spinning rust.
- myself248 2y ago
  And being read-heavy, suboptimal performance at write time is an infrequent pain, I guess?
  - xmodem 2y ago
    Not even that - the data being written is coming straight from the network, and the pool has no issues keeping up.
BodyCulture 2y ago
I wanted to use ZFS badly, but of course all data must be encrypted. It was surprising to see how usage gets much more complicated than expected and so many people just don’t encrypt their data because things get wild then. Look, even Proxmox, which I totally expected to support encryption with default installation (it has „Enterprise“ on the website) does loose important features when trying to use with encryption. Also please study the issue tracker, there are a few surprising things I would not have expected to exist in a productive file system.
- eadmund 2y ago
  The best way to encrypt ZFS is to run unecrypted ZFS atop encrypted volumes (e.g. LUKS volumes). ZFS ‘encryption’ leaves too much in plaintext for my comfort.
  - BodyCulture 2y ago
    In the Proxmox forum some people tried this method and do not report big success. Can not recommend for production. Still the same picture, encryption seems to be not a first class citizen in ZFS land.
burnt-resistor 2y ago
Already don't use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren't assured to be trustworthy because of their low usage and prevalent lack of thorough, formalized QA. There's nothing wrong with experimentation, but it's necessary to have an accurate understanding of the risk budget for a given technology for a given use-case.
- volkadav 2y ago
  I sympathize with your concerns for stability and testing, but I think that you might reconsider things in open-source ZFS land. OpenZFS/ZoL have been merged since the 2.0 release several years back, and some very large (e.g. Netflix) environments use FreeBSD which in turn uses OpenZFS, as well as being in use by the various Illumos derivatives and such. It is true that there has been some feature divergence between Oracle ZFS and OpenZFS since the fork, but as I recall that was more "nice to haves" like fs-native encryption than essential reliability fixes, fwiw.
  - ComputerGuru 2y ago
    Don't disagree with your post but netflix doesn't use zfs for a couple of reasons, one of which is broken sendfile support (though that might be fixed soon!).
qwertox 2y ago
What happened to the issue with ZFS which occurred around half a year go? I never changed a thing (because it also had some cons) and am believing that as long as a ZFS scrub shows no errors, all is OK. Could I be not seeing a problem?
cmiller1 2y ago
So if a sweet spot exists where dedup is widely beneficial then: Is there an easy way to analyze your dataset to find if you're in this sweet spot? If so, is anyone working on some kind of automated partial dedup system where only portions of the filesystem are dedupped based on analysis of how beneficial it would be?
- mdaniel 2y ago
  I can't speak to the first one, but AIUI the ZFS way of thinking about the second one is to create a new filesystem and just mount it where you want it, versus "portions of the filesystem" which I doubt very seriously that ZFS allows. Bonus points that in that scenario, I would suspect the dedupe and compression would work even better since any such setup is likely to contain more homogeneous content (music, photos, etc)
- mlfreeman 2y ago
  Are there any tools that can run (even across network on another box) to analyze possible duplication at various block sizes? I am NOT interested in finding duplicate files, but duplicate slices within all my files overall. I can easily throw together code myself to find duplicate files. EDIT: I guess I’m looking for a ZFS/BTRFS/other dedupe preview tool that would say “you might save this much if you used this dedupe process.”
teilo 2y ago
Why are enterprise SANs so good at dedupe, but filesystems so bad? We use HPE Nimble (yeah, they changed the name recently but I can't be bothered to remember it), and the space savings are insane for the large filesystems we work with. And there is no performance hit. Some of this is straight up VM storage volumes for ESX virtual disks, some direct LUNs for our file servers. Our gains are upwards of 70%.
- growse 2y ago
  Totally naive question: is this better than you get than simply compressing? It's not 100% clear to me why explicit deduping blocks would give you any significant benefit over a properly chosen compression algorithm.
onnimonni 2y ago
I'm storing a lot of text documents (.html) which contain long similiar sections and are thus not copies but "partial copies". Would someone know if the fast dedup works also for this? Anything else I could be using instead?

454 points by type0 2y ago

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools compress within a crawl to create wacz or warc.gz files, but I don't think anyone has tried to do compression across the entire database of all snapshots ever taken by a tool. Big thank you to all the people that worked on it! BTW has anyone tried a probabilistic dedup approach using soemthing like a bloom filter so you don't have to store the entire dedup table of hashes verbatim? Collect groups of ~100 block hashes into a bucket each, and store a hyper compressed representation in a bloom filter. On write, lookup the hash of the block to write in the bloom filter, and if a potential dedup hit is detected, walk the 100 blocks in the matching bucket manually to look for any identical hashes. In theory you could do this with layers of bloom filters with different resolutions and dynamically swap out the heavier ones to disk when memory pressure is too high to keep the high resolution ones in RAM. Allowing the accuracy of the bloom filter to be changed as a tunable parameter would let people choose their preference around CPU time/overhead:bytes saved ratio.

uniqueuid 2y ago
I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.
- nikisweeting 2y ago
  WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.
  - uniqueuid 2y ago
    That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction. [edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler. https://support.archive-it.org/hc/en-us/articles/208001016-About-data-de-duplication https://support.archive-it.org/hc/en-us/articles/208001016-A...
    - nikisweeting 2y ago
      Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.
mappu 2y ago
Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives. dm-vdo has the same behaviour. You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)
- nikisweeting 2y ago
  I should clarify I don't use WARCs at all with archivebox, it just stores raw files on the filsystem because I rely on ZFS for all my compression, so there is no offset alignment issue. The wget extractor within archivebox can produce WARCs as an output but no parts of ArchiveBox are built to rely on those, they are just one of the optional extractors that can be run.
alchemist1e9 2y ago
While a slightly different use case, I suspect you’d like zbackup if you don’t know about it.

deleted 2y ago

[deleted]

dark-star 2y ago

I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...

wmf 2y ago
Fixup block pointers is the one thing ZFS didn't want to do.
magicalhippo 2y ago
> and fixup the block pointers That's why. Due to reasons[1], ZFS does not have the capability to rewrite block pointers. It's been a long requested feature[2] as it would also allow for defragmentation. I've been thinking this could be solved using block pointer indirection, like virtual memory, at the cost of a bit of speed. But I'm by no means a ZFS developer, so there's surely something I'm missing. [1]: http://eworldproblems.mbaynton.com/posts/2014/zfs-block-pointer-rewrite-status-2013/index.html http://eworldproblems.mbaynton.com/posts/2014/zfs-block-poin... [2]: https://github.com/openzfs/zfs/issues/3582 https://github.com/openzfs/zfs/issues/3582
- phongn 2y ago
  It looks like they’re playing more with indirection features now (created for vdev removal) for other features. One of the recent summit hackathons sketched out using indirect vdevs to perform rebalancing. Once you get a lot of snapshots, though, the indirection costs start to rise.
olavgg 2y ago
You can also use DragonFlyBSD with Hammer2, which supports both online and offline deduplication. It is very similar to ZFS in many ways. The big drawback though, is lack of file transfer protocols using RDMA. I've also heard there are some experimental branches that makes it possible to run Hammer2 on FreeBSD. But FreeBSD also lacks RDMA support. For FreeBSD 15, Chelsio has sponsored NVMe-oF target, and initiator support. I think this is just TCP though.

Wowfunhappy 2y ago

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives. But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.

magicalhippo 2y ago
The author of the new file-based block cloning code had this in mind. A backround process would scan files and identify dupes, delete the dupes and replace them with cloned versions. There are of course edge cases to consider to avoid data loss, but I imagine it might come soon, either officially or as a third-party tool.
Dylan16807 2y ago
The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.
- Wowfunhappy 2y ago
  You need block pointer rewrite for this?
  - Dylan16807 2y ago
    You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.

tilt_error 2y ago

If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?

klysm 2y ago
Kinda like log structured merge tree?
0x457 2y ago
Because to make this work without a lot of copying, you would need to mutate things that ZFS absolutely does not want to make mutable.
UltraSane 2y ago
If the block to be written is already being stored then you will match the hash and the block won't have to be written. This can save a lot of write IO in real world use.
magicalhippo 2y ago
Keep in mind ZFS was created at a time when disks were glacial in comparison to CPUs. And, the fastest write is the one you don't perform, so you can afford some CPU time to check for duplicate blocks. That said, NVMe has changed that balance a lot, and you can afford a lot less before you're bottlenecking the drives.

klysm 2y ago

I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.

magicalhippo 2y ago
Internally ZFS is essentially an object store. There was some work which tried to expose it through an object store API. Sadly it seems to not have gone anywhere. Tried to find the talk but failed, was sure I had seen it on a Delveloper Summit but alas.
UltraSane 2y ago
Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?
- mappu 2y ago
  High-density drives are usually zoned storage, and it's pretty difficult to implement the regular filesystem API on top of that with any kind of reasonable performance (device- vs host- managed SMR). The S3 API can work great on zones, but only because it doesn't let you modify an existing object without rewriting the whole thing, which is an extremely rough tradeoff.
- lazide 2y ago
  It’s only a ‘disaster’ if you are using it exclusively programmatically and want to do special tuning. File systems are pretty good if you have a mix of human and programmatic uses, especially when the programmatic cases are not very heavy duty. The programmatic scenarios are often entirely human hostile, if you try to imagine what would be involved in actually using them. Like direct S3 access, for example.
- deleted 2y ago
  [deleted]
- perlgeek 2y ago
  One way it's a disaster is that file names (on Linux at least, haven't used Windows in a long time) are byte strings that can contain directory paths from different/multiple file systems. So if you have non-ASCII characters in your paths, encoding/decoding is guesswork, and at worst, differs from path segment to path segment, and there's no metadata attached which encoding to use.

tjwds 2y ago

Edit: disregard this, I was wrong and missed the comment deletion window.

deleted 2y ago
[deleted]
gtirloni 2y ago
HN will automatically redirect the submitter to a recent submission instead of allowing a new post... if it had a significant number of comments. https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html

kderbe 2y ago

I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this discussion with opinions on contrast ratios...) For less patient readers, note that the concise summary is at the bottom of the post, not the top.

emptiestplace 2y ago
It scrolls horizontally :(
- going_north 2y ago
  It's because of this element in one of the final sections [1]: <code>kstat.zfs.<pool>.misc.ddt_stats_<checksum></code> Typesetting code on a narrow screen is tricky! [1] https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/#thats-a-lot-anything-else https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-...
- ThePowerOfFuet 2y ago
  Not on Firefox on Android it doesn't.
  - dspillett 2y ago
    It does in chrome on android (1080 px wide screen, standard ppi & zoom levels) but not by enough that you see it on the main body text (scrolling just reveals more margin), so you might find it does for you too but not enough that you noticed. As it is scrolling here, though inconsequentially, it might be bad on a smaller device with less screen and/or other ppi settings.
Aachen 2y ago
That being: > As we’ve seen from the last 7000+ words, the overheads are not trivial. Even with all these changes, you still need to have a lot of deduplicated blocks to offset the weight of all the unique entries in your dedup table. [...] what might surprise you is how rare it is to find blocks eligible for deduplication are on most general purpose workloads. > But the real reason you probably don’t want dedup these days is because since OpenZFS 2.2 we have the BRT (aka “block cloning” aka “reflinks”). [...] it’s actually pretty rare these days that you have a write operation coming from some kind of copy operation, but you don’t know that came from a copy operation. [...] [This isn't] saving as much raw data as dedup would get me, though it’s pretty close. But I’m not spending a fortune tracking all those uncloned and forgotten blocks. > [Dedup is only useful if] you have a very very specific workload where data is heavily duplicated and clients can’t or won’t give direct “copy me!” signal The section labeled "summary" imo doesn't do the article justice by being fairly vague. I hope these quotes from near the end of the article give a more concrete idea of why (not) use it

bastloing 2y ago

Forget dedupe just use zfs compression, a lot more bang for your buck

Joel_Mckay 2y ago
Unless your data-set is highly compressed media files. In general, even during rsync operations one often turns off compression on large video files, as the compression operation has low or negative impact on storage/transfers while eating ram and cpu power. De-duplication is good for Virtual Machine OS images, as the majority of the storage cost is a replicated backing image. =3
- bastloing 2y ago
  Compression is still king. Check out HP's Nimble storage arrays. Way quicker to do compression, fewer iops, and less overhead. Even when it misses, like video files, it's still a winner.

eek2121 2y ago

So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe. No, you won't save much on a client system. That isn't what the feature is made for.

hinkley 2y ago
When ZFS first came out I had visions of it being a turnkey RAID array replacement for nontechnical users. Pop out the oldest disk, pop in a new (larger one), wait for the pretty lights to change color. Done. It is very clear that consumer was never a priority, and so I wonder what the venn diagram is of 'client system' and 'zfs filesystem'. Not that big right?
UltraSane 2y ago
My reaction also. Dedupe is a must have for when you are storing hundreds of VMs. you WILL save so much data and inline dedupe will save a lot of write IO.
- XorNot 2y ago
  It's an odd notion in the age of containers where dedupe is like, one of the core things we do (but stupidly: amongst dissimilar images there's definitely more identical files then different ones).
edelbitter 2y ago
I tried two of the most non-random archives I had and was disappointed just as the author. For mail archives, I got 10%. For entire filesystems, I got.. just as much as with any other COW. Because indeed, I duplicate them only once. Later shared blocks are all over the place.
doublepg23 2y ago
I assuming the author is aware why the feature exists since they state in the second sentence they funded the improvement over the course of two years?

UltraSane 2y ago

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. The effectiveness of dedupe is strongly affected by the size of the blocks being hashed, with the smaller the better. As the blocks get smaller the odds of having a matching block grow rapidly. In my experience 4KB is my preferred block size.

acdha 2y ago
> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.
- UltraSane 2y ago
  What software?
  - SteveNuts 2y ago
    Logrotate
  - acdha 2y ago
    Log rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.
    - UltraSane 2y ago
      That is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.

UltraSane 2y ago

Knowing that your storage has really good inline dedupe is awesome and will affect how you design systems. Solid dedupe lets you effectively treat multiple copies of data as symlinks.

nisten 2y ago

can someone smarter than me explain what happens when instead of the regular 4kb block size in kernel builds we use 16kb or 64kb block size or is that only for the memory part, i am confused. Will a larger block size make this thing good or bad?

UltraSane 2y ago
Generally the smaller the dedupe block the better as you are far more likely to find a matching block. But larger blocks will reduce the number of hashes you have to store. In my experience 4KB is the sweet spot to maximize how much data you save.
- spockz 2y ago
  So in this case I think it would make sense to have a separate pool where you store large files like media so you can save on the dedup for them. Is there an inherent performance loss of using 64kB blocks on FS level when using storage devices that are 4kB under the hood?
  - nisten 2y ago
    Hmmm you might be able to do both no? Like the dedube is gonna run at the filesystem level but your memory security & ownership stuff is gonna run more efficiently. I am not sure.

tiffanyh 2y ago

OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?

nikisweeting 2y ago
- https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove - https://codeberg.org/jbruchon/jdupes https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/ https://www.jdupes.com/ - https://github.com/adrianlopezroche/fdupes https://github.com/adrianlopezroche/fdupes - https://github.com/pauldreik/rdfind https://github.com/pauldreik/rdfind
acdha 2y ago
The built in Photos duplicate feature is the best choice for most people: it’s not just generic file-level dedupe but smart enough to do things like take three versions of the same photo and pick the highest-quality one, which is great if you ever had something like a RAW/TIFF+JPEG workflow or mixed full res and thumbnails.
spockz 2y ago
Or better yet. A single photo I take of the kids will be stored in my camera roll. I will then share it with family using three different messengers. Now I have 4 copies. Each of the individual (recoded) are stored inside those messengers and also backed up. This even happens when sharing the same photo multiple times in different chats with the same messenger. Is there any way to do de duplication here? Or just outright delete all the derivatives?
EraYaN 2y ago
digiKam can dedupe on actual similarity (so different resizes and formats of the same image). But it does take some time to calculate all the hashes.

wpollock 2y ago

When the lookup key is a hash, there's no locality over the megabytes of the table. So don't all the extra memory accesses to support dedup affect the L1 and L2 caches? Has anyone at OpenZFS measured that? It also occurs to me that spacial locality on spinning rust disks might be affected, also affecting performance.

forrestthewoods 2y ago

My dream Git successor would use either dedupe or a simple cache plus copy-on-write so that repos can commit toolchains and dependencies and users wouldn’t need to worry about disk drive bloat. Maybe someday…

fragmede 2y ago
It does dedup using Sha-1 on entire files. you might try git-lfs for your usecase though.
- forrestthewoods 2y ago
  Git LFS is a really really bad gross hack. It’s awful. https://www.forrestthewoods.com/blog/dependencies-belong-in-version-control/ https://www.forrestthewoods.com/blog/dependencies-belong-in-...
  - fragmede 2y ago
    It's quite functional and usable, now. so I'd agree with hack, just not the rest of your adjectives. That was a good read! I've been thinking a lot about what comes after git too. One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.
    - forrestthewoods 2y ago
      > One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo. That’s the problem that a virtual file system solves. When you clone a repo it only materializes files when they’re accessed. This is how my work repo operates. It’s great.
      - fragmede 2y ago
        The problem I'm trying to avoid is having to dive down a hierarchy, and that not everyone needs or wants to know there is such a hierarchy. Like graphics design for foo-team only needs to have access to some subset of graphics. Arguably they could be given a symlink into the checkout, but the problem with that is of having a singular checkout. The problem I had with Google3 is that the tools weren't great at branching and didn't fit my workflow, which tends to involve multiple checkouts (or worktrees using git). being forced to checkout the root of the repo, and then having to manage a symlink on top of that is no good for users that don't need/want to manage the complexity of having a single machine-global checkout.

girishso 2y ago

Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.

nikisweeting 2y ago
How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically? There are a few different ways you could solve it but it depends on what final outcome you need.
- girishso 2y ago
  Just reporting in some plain text format so I can manually delete the duplicates, or create some script to delete. I can't have like 10 external HDDs attached at the same time, so the tool needs to dump details (hashes?) somewhere on Mac HDD, and compare against those to find the duplicates.
  - nikisweeting 2y ago
    Here you go: cd /path/to/drive find . -type f -exec sha256sum {} + | sed -E 's/^([^ ]+) \./\1,/' >> ~/all_hashes.txt Run that for each drive, then when you're done run: sort ~/all_hashes.txt > ~/sorted_hashes.txt awk -F, 'NR==1{print;next} {print $0 | "sort | uniq -w64 -D"}' ~/sorted_hashes.txt > ~/non_unique_hashes.txt The output in ~/non_unique_hashes.txt will contain only the non-unique hashes that appear on more than one path.
UltraSane 2y ago
dupeGuru works pretty well.

hhdhdbdb 2y ago

Any timing attacks possible on a virtualized system using dedupe? Eg find out what my neighbours have installed. Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.

aidenn0 2y ago
I don't think you even need timing attacks if you can read the zpool statistics; you can ask for a histogram of deduped blocks. Guessing one byte at a time is not possible though because dedupe is block-level in ZFS.
- beng-nl 2y ago
  Gosh, you’re likely right, but what if comparing the blocks (to decide on deduping) is a byte at a time and somehow that can be detected (with a timing channel or a uarch side channel)? Zfs likely compares the hash, but I think KSM doesn’t use hashes but memcmp (or something in that spirit) to avoid collisions. So just maybe… just maybe GP is onto something.. interesting fantasy ;-)
  - hhdhdbdb 2y ago
    Thanks for putting meat on the (speculitive) bone I threw out! Very interesting.
UltraSane 2y ago
VMWare ESXi used to dedupe RAM and had to disable this by default because of a security issue it caused that leaded data between VMs.

rodarmor 2y ago

General-purpose deduplication sounds good in theory but tends not to work out in practice. IPFS uses a rolling hash with variable-sized pieces, in an attempt to deduplicate data rysnc-style. However, in practice, it doesn't actually make a difference, and adds complexity for no reason.

rkagerer 2y ago

I'd love if dedicated hardware existing in disk controllers for calculating stuff like ECC could be enhanced to expose hashes of blocks to the system. Getting this for free for all your I/O would allow some pretty awesome things.

UltraSane 2y ago
That is a neat idea. Hard drives could do dedupe from the ECC they calculate for each sector. The main issue with that is that the current ECC is optimal for detecting bit errors but doesn't have the same kind of statistical guarantee of uniqueness that SHA256 or MetroHash has. You need to be VERY confident of the statistical properties of the hash used in dedupe if you are going to increment the ref count of the block hash instead of writing the data to disk.

watersb 2y ago

I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I first started this project, 2TB hard drives were the largest available. My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache. Every few years, I try to do a better job of things but at this point, the best improvement would be radical simplification. ZFS has served very well in terms of reliability. I haven't lost data, and I've been able to catch lots of episodes of almost losing data. Or writing the wrong data. Not entirely sure how I'd replace it, if I want something that can spot bit rot and correct it. ZFS scrub.

roygbiv2 2y ago
Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.
- watersb 2y ago
  This archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives. While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.) Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational. The application leads to a big payoff for deduplication.
emptiestplace 2y ago
Cache or ZIL (SLOG device)?
- watersb 2y ago

david_draco 2y ago

In addition to the copy_file_range discussion at the end, it would be great to be able to applying deduplication to selected files, identified by searching the filesystem for say >1MB files which have identical hash.

nabla9 2y ago

You should use: cp --reflink=auto You get file level deduplication. The command above performs a lightweight copy (ZFS clone in file level), where the data blocks are copied only when modified. Its a copy, not a hard link. The same should work in other copy-on-write transactional filesystems as well if they have reflink support.

simonjgreen 2y ago

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

jamesbfb 2y ago
Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!
- rwarfield 2y ago
  Isn't it better to use `nix store optimise` for dedup of the nix store? The nix command has more knowledge of the structure of the nix store so should be able to do a better job with fewer resources. Also the store is immutable so you don't actually need reflinks - hard links are enough.
  - Filligree 2y ago
    It is, yeah, though you have to turn it on. I'm not actually sure why it's off by default.
    - amarshall 2y ago
      It’s off by default as it can make builds slower (regardless of platform)—you should test this if you care. There also are (or were) some bugs on macOS that would cause corruption.
      - Filligree 2y ago
        That seems like the wrong default. Most people do very little building on their desktops; they get all their software from the cache.
- 2y ago

nobrains 2y ago

What are the use cases where it makes sense to use de-dup? Backup comes to mind. What else?

3np 2y ago
Many similar VM images or even live root filesystems.
zitsarethecure 2y ago
Flatpak uses OSTree dedup. https://blogs.gnome.org/wjjt/2021/11/24/on-flatpak-disk-usage-and-deduplication/ https://blogs.gnome.org/wjjt/2021/11/24/on-flatpak-disk-usag...
UltraSane 2y ago
VMware + All flash storage array + inline dedupe/compression = happy users.

merpkz 2y ago

I don't get it - many people here claim in this thread that VM base image deduplication is great use case for this. So lets assume there are couple of hundreds of VMs on a ZFS dataset with dedupe on, each of them ran by different people for different purposes entirely - some databases, some web frontends / backends, minio S3 storage or backups ect - this might save you those measly hundreds of megabytes for linux system files those VMs might have in common ( even though knowing how many linux versions are out there with different patch levels - unlikely ) it will still not be worth it considering ZFS will keep track of each users individual files - databases and backup files and whatnot - data which is almost guaranteed to be unique between users so it will completely miss the point of ZFS deduplication. What am I missing?

jeroenhd 2y ago
It largely depends on how you set up your environment. On my home server, most VMs consist of a few gigabytes of a base Linux system and then a couple of hundred megabytes of application code. Some of those VMs also store large amounts of data, but most of that data could be stored in something like a dedicated minio server and maybe a dedicated database server. I could probably get rid of a huge chunk of my used storage if I switched to a deduplicating system (but I have plenty of storage so I don't really need to). If you're selling VMs to customers then there's probably no advantage in using deduplication.
3np 2y ago
In such a sevario you'd probably have several partitions. So dedupe activated on the root filesystem (/bin,/lib etc) but not for /home and /var.

xmodem 2y ago

We have a read-heavy zpool with some data that's used as part of our build process, on which we see a roughly 8x savings with dedup - and because of this ZFS dedup makes it economically viable for us to store the pool on NVMe rather than spinning rust.

myself248 2y ago
And being read-heavy, suboptimal performance at write time is an infrequent pain, I guess?
- xmodem 2y ago
  Not even that - the data being written is coming straight from the network, and the pool has no issues keeping up.

BodyCulture 2y ago

I wanted to use ZFS badly, but of course all data must be encrypted. It was surprising to see how usage gets much more complicated than expected and so many people just don’t encrypt their data because things get wild then. Look, even Proxmox, which I totally expected to support encryption with default installation (it has „Enterprise“ on the website) does loose important features when trying to use with encryption. Also please study the issue tracker, there are a few surprising things I would not have expected to exist in a productive file system.

eadmund 2y ago
The best way to encrypt ZFS is to run unecrypted ZFS atop encrypted volumes (e.g. LUKS volumes). ZFS ‘encryption’ leaves too much in plaintext for my comfort.
- BodyCulture 2y ago
  In the Proxmox forum some people tried this method and do not report big success. Can not recommend for production. Still the same picture, encryption seems to be not a first class citizen in ZFS land.

burnt-resistor 2y ago

Already don't use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren't assured to be trustworthy because of their low usage and prevalent lack of thorough, formalized QA. There's nothing wrong with experimentation, but it's necessary to have an accurate understanding of the risk budget for a given technology for a given use-case.

volkadav 2y ago
I sympathize with your concerns for stability and testing, but I think that you might reconsider things in open-source ZFS land. OpenZFS/ZoL have been merged since the 2.0 release several years back, and some very large (e.g. Netflix) environments use FreeBSD which in turn uses OpenZFS, as well as being in use by the various Illumos derivatives and such. It is true that there has been some feature divergence between Oracle ZFS and OpenZFS since the fork, but as I recall that was more "nice to haves" like fs-native encryption than essential reliability fixes, fwiw.
- ComputerGuru 2y ago
  Don't disagree with your post but netflix doesn't use zfs for a couple of reasons, one of which is broken sendfile support (though that might be fixed soon!).

qwertox 2y ago

What happened to the issue with ZFS which occurred around half a year go? I never changed a thing (because it also had some cons) and am believing that as long as a ZFS scrub shows no errors, all is OK. Could I be not seeing a problem?

cmiller1 2y ago

So if a sweet spot exists where dedup is widely beneficial then: Is there an easy way to analyze your dataset to find if you're in this sweet spot? If so, is anyone working on some kind of automated partial dedup system where only portions of the filesystem are dedupped based on analysis of how beneficial it would be?

mdaniel 2y ago
I can't speak to the first one, but AIUI the ZFS way of thinking about the second one is to create a new filesystem and just mount it where you want it, versus "portions of the filesystem" which I doubt very seriously that ZFS allows. Bonus points that in that scenario, I would suspect the dedupe and compression would work even better since any such setup is likely to contain more homogeneous content (music, photos, etc)
mlfreeman 2y ago
Are there any tools that can run (even across network on another box) to analyze possible duplication at various block sizes? I am NOT interested in finding duplicate files, but duplicate slices within all my files overall. I can easily throw together code myself to find duplicate files. EDIT: I guess I’m looking for a ZFS/BTRFS/other dedupe preview tool that would say “you might save this much if you used this dedupe process.”

teilo 2y ago

Why are enterprise SANs so good at dedupe, but filesystems so bad? We use HPE Nimble (yeah, they changed the name recently but I can't be bothered to remember it), and the space savings are insane for the large filesystems we work with. And there is no performance hit. Some of this is straight up VM storage volumes for ESX virtual disks, some direct LUNs for our file servers. Our gains are upwards of 70%.

growse 2y ago
Totally naive question: is this better than you get than simply compressing? It's not 100% clear to me why explicit deduping blocks would give you any significant benefit over a properly chosen compression algorithm.

onnimonni 2y ago

I'm storing a lot of text documents (.html) which contain long similiar sections and are thus not copies but "partial copies". Would someone know if the fast dedup works also for this? Anything else I could be using instead?