2 ms·

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archiv

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools compress within a crawl to create wacz or warc.gz files, but I don't think anyone has tried to do compression across the entire database of all snapshots ever taken by a tool. Big thank you to all the people that worked on it! BTW has anyone tried a probabilistic dedup approach using soemthing like a bloom filter so you don't have to store the entire dedup table of hashes verbatim? Collect groups of ~100 block hashes into a bucket each, and store a hyper compressed representation in a bloom filter. On write, lookup the hash of the block to write in the bloom filter, and if a potential dedup hit is detected, walk the 100 blocks in the matching bucket manually to look for any identical hashes. In theory you could do this with layers of bloom filters with different resolutions and dynamically swap out the heavier ones to disk when memory pressure is too high to keep the high resolution ones in RAM. Allowing the accuracy of the bloom filter to be changed as a tunable parameter would let people choose their preference around CPU time/overhead:bytes saved ratio.

uniqueuid 2y ago
I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.
- nikisweeting 2y ago
  WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.
  - uniqueuid 2y ago
    That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction. [edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler. https://support.archive-it.org/hc/en-us/articles/208001016-About-data-de-duplication https://support.archive-it.org/hc/en-us/articles/208001016-A...
    - nikisweeting 2y ago
      Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.
mappu 2y ago
Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives. dm-vdo has the same behaviour. You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)
- nikisweeting 2y ago
  I should clarify I don't use WARCs at all with archivebox, it just stores raw files on the filsystem because I rely on ZFS for all my compression, so there is no offset alignment issue. The wget extractor within archivebox can produce WARCs as an output but no parts of ArchiveBox are built to rely on those, they are just one of the optional extractors that can be run.
alchemist1e9 2y ago
While a slightly different use case, I suspect you’d like zbackup if you don’t know about it.

uniqueuid 2y ago

I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.

nikisweeting 2y ago
WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.
- uniqueuid 2y ago
  That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction. [edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler. https://support.archive-it.org/hc/en-us/articles/208001016-About-data-de-duplication https://support.archive-it.org/hc/en-us/articles/208001016-A...
  - nikisweeting 2y ago
    Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.

mappu 2y ago

Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives. dm-vdo has the same behaviour. You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)

nikisweeting 2y ago
I should clarify I don't use WARCs at all with archivebox, it just stores raw files on the filsystem because I rely on ZFS for all my compression, so there is no offset alignment issue. The wget extractor within archivebox can produce WARCs as an output but no parts of ArchiveBox are built to rely on those, they are just one of the optional extractors that can be run.

alchemist1e9 2y ago

While a slightly different use case, I suspect you’d like zbackup if you don’t know about it.