3 ms·

The tar.gz format is to combine a collection of files first as a single tar file and then compress the one file into gz. It's not compressing individual file a

• kbenson 10y ago
  > It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. Yes, that is the whole point of this discussion. This limitation causes problems for some. > What you described is the zip format. That is not the zip format. Zip is a collection of file objects with uncompressed headers which specify the metadata and size of the compressed individual file that follows. As such, you can easily scan through accessing the headers. > Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how the files are packaged. Create 1000 files with the content "foobarbaz". Create two archives, first by using zip and then by gzipping a tarball. They will not be comparable in size. A tar is basically concatenating the files together with headers between them. Compressing this allows the DEFLATE algorithm to work across a larger dataset, and thus more efficiently. The zip archive might actually be larger than the tar before compression. The trade-off here is that you get better compression characteristics from compressing one large file, but you don't get easy access to individual portions of the compressed file easily. > Also gzip supports on-the-fly stream-based compression and decompression so anything requires jumping to the end to do reading and writing while compression/decompresion is no go. The stream format works in blocks of up to 65k. That's the reason I said to read the last 65k of the compressed file. That should be guaranteed to contain at least one full block, which if I understand correctly you should be able to uncompress by itself. If the end of the tar archive is an index, and the footer of the index specifies the size of the index, you know whether you got the whole index in that last gzip block, or whether you need to get one or more prior gzip blocks (you can probably make an educated guess based on the index size). Once you have the index, which specifies file names and by offsets of the compressed file of the block at which the file/header starts, you can fairly efficiently index into large compressed files (again, assuming 65k max gzip block size and the ability to treat blocks independently). The benefit would be that since the index is just another file in the tar, you can uncompress the format from a plain tar command with no negative consequences besides an extra file being present (the index).
  • deleted 10y ago
    [deleted]
  • ww520 10y ago
    > The stream format works in blocks of up to 65k. Can you point to where in the gzip or DEFLATE spec having that stipulation? The gzip format doesn't store the compressed data length, nor the offset of the compressed data blobs. It just has the header and then the blobs of DEFLATE compressed data, one after another. That's why it's good for streaming. Same thing with DEFLATE. It's just a series of arbitrary size compressed data blocks. With a dictionary upfront for dynamic Huffman and no dictionary for static Huffman. You read as much data as you can to decompress until encountering the ending Huffman symbol, where you arrive at the next block. The 65K limit is for non-compress type block, which is not useful in the scheme of things. There's no well known magic signature bytes to search for a DEFLATE header.
    • kbenson 10y ago
      > Can you point to where in the gzip or DEFLATE spec having that stipulation? You are correct, I was misreading the stream format. There's a mode in it that mentioned 65k of data, and I was misreading what I saw (rather badly, at that). That said, it looks like, since deflate duplicate string reduction portion (LZ77 in gzip) references a sequence of bytes in the prior 32k, you could make sure to read at least 32k prior to the point you want to start looking at, and that would be covered. Unfortunately, the location of the huffman table looks to be the sticking point. If it was one large block, that would be easy, read some of the front, but it's probably not as easy as that, depending on how tar adds files. > The gzip format doesn't store the compressed data length, nor the offset of the compressed data blobs. I wasn't expecting it to store the offset, but I was incorrectly interpreting that it had a max length, which meant you could likely find it with high assurance. I'm going to look deeper into the specifics of the huffman table, where it can/must be within the block, and how tar handles adding files to a compressed archive (does it add another block, or extend the existing one?) and when compressing in general (does tar arbitrarily limit deflate stream block sizes?). I'm not super confident it's possible, otherwise I imagine someone would have done it.