Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the d

Ah cool, TIL, thanks for the link. I didn't realize that was possible. I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.