5 ms·

One advantage of tar is that, because the format has no built-in support for compression or random access, the entire archive is compressed together. Similariti

One advantage of tar is that, because the format has no built-in support for compression or random access, the entire archive is compressed together. Similarities in adjacent files will improve the compression ratio. To support random access, the ZIP format must compress each file separately.

orlp 4y ago
That's just not true, at least theoretically speaking (I have no idea about ZIP's actual internals). One could do a two-pass algorithm to find common substrings to build a dictionary, store this shared dictionary once, and then compress the files individually referring to this shared dictionary for the common substrings.
- bick_nyers 4y ago
  As long as you have a pointer index in the header for the start/end of every individual file I don't see why that wouldn't work.
  - lazide 4y ago
    Well, except no zip decompression library I know supports anything like that.
- deleted 4y ago
  [deleted]
- adrianmonk 4y ago
  You definitely could. I did it for a proprietary format once. Each item was quite small (typically 100-ish bytes IIRC), and I needed random access. One approach would have been to split it into pages of a few kilobytes and separately compress those. But on this platform, I/O was primitive and very slow, and the CPU was also extremely slow. So it was advantageous to decompress only the necessary data. So I came up with a rudimentary custom compression algorithm. I scanned all entries for common strings and made a shared dictionary. Each entry was then stored as a sequence of indexes into the shared dictionary. Decompressing an entry was dead simple: read the sequence of dictionary indices and copy from the dictionary to the output. On a side note, the unexpected challenge of this method was how to choose which strings to put in the shared dictionary. I tried a few things and ended up with a heuristic that tries to balance favoring longer strings with favoring frequently occurring strings. I'm sure there are much better ways than that. If I had it to do over, I probably would have gone with the compressed pages approach simply because the implementation would've been done so much quicker. But I think I did achieve faster decompression and a better compression ratio.
xyzzyz 4y ago
With tar.gz, there is not much of a benefit: the window size is only 32KB. Now with LZMA, the situation is much different.
- masklinn 4y ago
  That can still be a significant benefit if the files are small, which is not uncommon. The 512 bytes header is a bit of an annoyance there (especially compared to zip’s 30+file name) but not that much. And then, obviously, one of the advantages of tar is you can compress with whatever you want.
  - IshKebab 4y ago
    Then just compress every 10MB or whatever independently. Probably a good idea for parallelism anyway.
    - masklinn 4y ago
      What are you talking about?
      - IshKebab 4y ago
        You two were debating the benefits of `tar.gz` style solid compression vs zip style independent compression of files. Zip allows random access while solid compression will give slightly better compression for small similar files. I was pointing out that you can have the best of both worlds by catting all the files together like `tar` but then compressing it in chunks instead of as one big stream. Plus compressing it in chunks allows you to decompress it in parallel.
IshKebab 4y ago
One disadvantage of tar is that because it has no built-in support for compression it has to give up the very important feature of random access for a trivial improvement in compression ratio. Anyway 7z can optionally do solid compression anyway. I've no idea why it isn't more popular the Linux world. I guess the only real reason is the CLI tools don't come installed by default.

orlp 4y ago

That's just not true, at least theoretically speaking (I have no idea about ZIP's actual internals). One could do a two-pass algorithm to find common substrings to build a dictionary, store this shared dictionary once, and then compress the files individually referring to this shared dictionary for the common substrings.

bick_nyers 4y ago
As long as you have a pointer index in the header for the start/end of every individual file I don't see why that wouldn't work.
- lazide 4y ago
  Well, except no zip decompression library I know supports anything like that.
deleted 4y ago
[deleted]
adrianmonk 4y ago
You definitely could. I did it for a proprietary format once. Each item was quite small (typically 100-ish bytes IIRC), and I needed random access. One approach would have been to split it into pages of a few kilobytes and separately compress those. But on this platform, I/O was primitive and very slow, and the CPU was also extremely slow. So it was advantageous to decompress only the necessary data. So I came up with a rudimentary custom compression algorithm. I scanned all entries for common strings and made a shared dictionary. Each entry was then stored as a sequence of indexes into the shared dictionary. Decompressing an entry was dead simple: read the sequence of dictionary indices and copy from the dictionary to the output. On a side note, the unexpected challenge of this method was how to choose which strings to put in the shared dictionary. I tried a few things and ended up with a heuristic that tries to balance favoring longer strings with favoring frequently occurring strings. I'm sure there are much better ways than that. If I had it to do over, I probably would have gone with the compressed pages approach simply because the implementation would've been done so much quicker. But I think I did achieve faster decompression and a better compression ratio.

xyzzyz 4y ago

With tar.gz, there is not much of a benefit: the window size is only 32KB. Now with LZMA, the situation is much different.

masklinn 4y ago
That can still be a significant benefit if the files are small, which is not uncommon. The 512 bytes header is a bit of an annoyance there (especially compared to zip’s 30+file name) but not that much. And then, obviously, one of the advantages of tar is you can compress with whatever you want.
- IshKebab 4y ago
  Then just compress every 10MB or whatever independently. Probably a good idea for parallelism anyway.
  - masklinn 4y ago
    What are you talking about?
    - IshKebab 4y ago
      You two were debating the benefits of `tar.gz` style solid compression vs zip style independent compression of files. Zip allows random access while solid compression will give slightly better compression for small similar files. I was pointing out that you can have the best of both worlds by catting all the files together like `tar` but then compressing it in chunks instead of as one big stream. Plus compressing it in chunks allows you to decompress it in parallel.

IshKebab 4y ago

One disadvantage of tar is that because it has no built-in support for compression it has to give up the very important feature of random access for a trivial improvement in compression ratio. Anyway 7z can optionally do solid compression anyway. I've no idea why it isn't more popular the Linux world. I guess the only real reason is the CLI tools don't come installed by default.