4 ms·

What's the best way to store a bunch of .zip files that all share some data? Assume that I can't decompress or alter the zip files in any way. Basically I want

What's the best way to store a bunch of .zip files that all share some data? Assume that I can't decompress or alter the zip files in any way. Basically I want a shared encoding dictionary. Is there an easy solution for this? The use case is maintaining an archive of .crx files.

sph 3y ago
Not sure if I understand the question. Having a shared encoding dictionary is called compression. If you want one across many zip files that you cannot modify, then compress them into another archive.
- chatmasta 3y ago
  Yes, I have a bunch of zips I cannot modify. If I could uncompress them all, and zip them into one big archive, then it would be much smaller than if I put all the compressed zips into one big archive. This is because there's a lot more shared strings between uncompressed files across different zips, than there are shared strings between the compressed zips. (At least... this is my assumption based on how I understand the formats to work. I do need to measure and verify this.) So basically I'm wondering if there is some way to tell the "outer" zip to reuse the encoding dictionaries of each smaller zip, or to somehow intelligently merge their encoding dictionaries rather than treating each inner zip like an opaque blob.
  - teraflop 3y ago
    No, the ZIP format compresses each embedded file separately, so it doesn't matter how much commonality there is between different files in the same archive.
    - chatmasta 3y ago
      Wow, really? I always assumed that if I zipped a directory, it would use a shared dictionary to compress all the files. But I guess what you say makes sense, because otherwise it wouldn't be possible to extract just one file. Are there compression formats that do share an encoding dictionary across multiple files? I guess tar + gzip might do that?
      - teraflop 3y ago
        Yeah, if you compress a tar file with gzip (or bzip2 or zstd or whatever) then the compressor doesn't care about the file boundaries, so it will be able to take advantage of redundancies. However, those compressors generally only have a small context window, so they'll only be able to take advantage of relatively nearby redundancies in the archive. They won't help you if the common substrings are separated by many megabytes of unrelated data. So the order in which you pack the files matters. In theory, you could do a two-pass approach, where you first scan the entire set of files to create a (relatively small) shared dictionary that's as useful as possible, and then uses that dictionary to compress each file independently. I don't know of any archive format that does that, but you could roll your own using the zstd command line compressor.
Scaevolus 3y ago
There's not an _easy_ way to do this right now. Your best bet is a lossless transform that undoes the huffman coding in the zip files, converting the compressed streams from effectively uncorrelated bitstreams to largely similar byte streams, and then pass that through a large-window compression algorithm (zstd?). Similar techniques are used in ChromeOS for delta updates.
mappu 3y ago
Use precomp or antiz to losslessly pre-transform the zip, and then use solid compression (e.g. tar.xz or zpaq).
indrora 3y ago
Depending on implementation, zip supports Zstd compressed archives with dictionaries. You could maintain a shared dictionary among all the archives. If you have a way to post process the archives, you may be able to abuse multipart zip files to split them and reconstitute them later.

sph 3y ago

Not sure if I understand the question. Having a shared encoding dictionary is called compression. If you want one across many zip files that you cannot modify, then compress them into another archive.

chatmasta 3y ago
Yes, I have a bunch of zips I cannot modify. If I could uncompress them all, and zip them into one big archive, then it would be much smaller than if I put all the compressed zips into one big archive. This is because there's a lot more shared strings between uncompressed files across different zips, than there are shared strings between the compressed zips. (At least... this is my assumption based on how I understand the formats to work. I do need to measure and verify this.) So basically I'm wondering if there is some way to tell the "outer" zip to reuse the encoding dictionaries of each smaller zip, or to somehow intelligently merge their encoding dictionaries rather than treating each inner zip like an opaque blob.
- teraflop 3y ago
  No, the ZIP format compresses each embedded file separately, so it doesn't matter how much commonality there is between different files in the same archive.
  - chatmasta 3y ago
    Wow, really? I always assumed that if I zipped a directory, it would use a shared dictionary to compress all the files. But I guess what you say makes sense, because otherwise it wouldn't be possible to extract just one file. Are there compression formats that do share an encoding dictionary across multiple files? I guess tar + gzip might do that?
    - teraflop 3y ago
      Yeah, if you compress a tar file with gzip (or bzip2 or zstd or whatever) then the compressor doesn't care about the file boundaries, so it will be able to take advantage of redundancies. However, those compressors generally only have a small context window, so they'll only be able to take advantage of relatively nearby redundancies in the archive. They won't help you if the common substrings are separated by many megabytes of unrelated data. So the order in which you pack the files matters. In theory, you could do a two-pass approach, where you first scan the entire set of files to create a (relatively small) shared dictionary that's as useful as possible, and then uses that dictionary to compress each file independently. I don't know of any archive format that does that, but you could roll your own using the zstd command line compressor.

Scaevolus 3y ago

There's not an _easy_ way to do this right now. Your best bet is a lossless transform that undoes the huffman coding in the zip files, converting the compressed streams from effectively uncorrelated bitstreams to largely similar byte streams, and then pass that through a large-window compression algorithm (zstd?). Similar techniques are used in ChromeOS for delta updates.

mappu 3y ago

Use precomp or antiz to losslessly pre-transform the zip, and then use solid compression (e.g. tar.xz or zpaq).

indrora 3y ago

Depending on implementation, zip supports Zstd compressed archives with dictionaries. You could maintain a shared dictionary among all the archives. If you have a way to post process the archives, you may be able to abuse multipart zip files to split them and reconstitute them later.