I had some archives which are heavily redundant (web mirrors & scrape mirrors by date), and so I looked into file order compression: https://www.gwern.net/Archi

I had some archives which are heavily redundant (web mirrors & scrape mirrors by date), and so I looked into file order compression: https://www.gwern.net/Archiving-URLs#sort---key-compression-trick https://www.gwern.net/Archiving-URLs#sort---key-compression-... On my particular use-case, the compression gains are enormous. It's particularly impressive because as far as any enduser is concerned, it's just like any other compressed XZ tarball.