5 ms·
It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible. One way would be to use the last file in the tar as the
by kbenson 10y ago
It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible.
One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and append to the compressed tar (add).
You can retrieve the index by starting at the end of the compressed archive, and reading backwards until you find a deflate header (at most 65k plus a few more bytes, since that's the size of a deflate chunk), If it's an indexed tar, the last file will be the index, and the end of the index will be a footer with the index size (so you know the maximum you'll need to seek back from the end). This isn't extremely efficient, but it is limited in scope, and helped by knowing the index size.
You could verify the index by checking some or all of the reported file byte offsets. Worst case scenario is small files with one or more per deflate chunk, and you would have to visit each chunk. This makes the worst case scenario equivalent to listing files an un-indexed tar.gz, plus the overhead of locating and reading the index (relatively small).
Uncompressing the archive as a regular tar.gz would result in a normal operation, with an additional file (the index) included.
I imagine this isn't popular is not because it hasn't been done, but because most people don't really need an index.
- bluedino 10y agoAfter writing one myself along with a guy on IRC who needed a 'quick way to access huge .tar files', I found one written in python: https://github.com/devsnd/tarindexer https://github.com/devsnd/tarindexer He ended up doing the smart thing and changing his archiving program to not use .tar files, which solved all of his problems, but for the 2TB of data he already had this worked rather well.
- kbenson 10y agoYeah, I almost wrote one myself in Perl a year or two ago. :) I was going for maximal tar compatibility, graceful fallback, low impact file additions, and the ability to add an index after the tar is created without rewriting it all. That's what I came up with off the top of my head. I did just search for indexed tar though, and it came up with dar[1], which I had heard of, but completely forgotten about. 1: http://dar.linux.free.fr/ http://dar.linux.free.fr/
- TimMeade 10y agoIf I recall correctly and it's been a while: The initial header is x bytes from the beginning of the file, or that you have to search for a known key string PKsomething. Then you have to go back and forth between that and the compressed data as you decompress. When compressing files you have to go back to the beginning of the zip file ( or disk1 in a multi disk archive) and update that table with the CRC info. Just was not practical for streaming for multiple reasons. Remember the original zip file format was created by phil in about 1987. I believe they just thought it easier to start over with a better design for gzip with the same compression algorithm.
- kbenson 10y agoYeah, the above is with an assumption that the archive is in deflate's stream format. I'm not sure off the top of my head how gzip's different levels relate to that, or what tar defaults to when compressing as the tar is created compared to a full file tar compress after it's created.
- dexterdog 10y agoThat can't be the case anymore because I implemented a streaming function a few years back that sends an unlimited number of files as a zip, but builds the zip on the fly streaming it to the client. Maybe it works because I am not using compression (files in my case are all JPG and/or compressed video so there is little benefit). But my process definitely starts streaming the zip right away and has to pull URLs on the fly to create the zip file.
- kbenson 10y agoI'm pretty sure you were using the deflate stream format. There may be other formats (maybe with different magic numbers in the header) that have slightly different capabilities and needs.
- TimMeade 10y agoThis cannot be a 'true' zip file. The central directory is at the end of the zip file. It's made up of information that is needed for the decompression. So if you are streaming it, you cannot decompress until the entire file is reassembled.
- ww520 10y agoThe tar.gz format is to combine a collection of files first as a single tar file and then compress the one file into gz. It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. What you described is the zip format. Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how the files are packaged. Also gzip supports on-the-fly stream-based compression and decompression so anything requires jumping to the end to do reading and writing while compression/decompresion is no go.
- kbenson 10y ago> It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. Yes, that is the whole point of this discussion. This limitation causes problems for some. > What you described is the zip format. That is not the zip format. Zip is a collection of file objects with uncompressed headers which specify the metadata and size of the compressed individual file that follows. As such, you can easily scan through accessing the headers. > Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how the files are packaged. Create 1000 files with the content "foobarbaz". Create two archives, first by using zip and then by gzipping a tarball. They will not be comparable in size. A tar is basically concatenating the files together with headers between them. Compressing this allows the DEFLATE algorithm to work across a larger dataset, and thus more efficiently. The zip archive might actually be larger than the tar before compression. The trade-off here is that you get better compression characteristics from compressing one large file, but you don't get easy access to individual portions of the compressed file easily. > Also gzip supports on-the-fly stream-based compression and decompression so anything requires jumping to the end to do reading and writing while compression/decompresion is no go. The stream format works in blocks of up to 65k. That's the reason I said to read the last 65k of the compressed file. That should be guaranteed to contain at least one full block, which if I understand correctly you should be able to uncompress by itself. If the end of the tar archive is an index, and the footer of the index specifies the size of the index, you know whether you got the whole index in that last gzip block, or whether you need to get one or more prior gzip blocks (you can probably make an educated guess based on the index size). Once you have the index, which specifies file names and by offsets of the compressed file of the block at which the file/header starts, you can fairly efficiently index into large compressed files (again, assuming 65k max gzip block size and the ability to treat blocks independently). The benefit would be that since the index is just another file in the tar, you can uncompress the format from a plain tar command with no negative consequences besides an extra file being present (the index).
- greglindahl 10y agoOne community that does use this technique (although the index is in a separate file) is the web archiving community's WARC file format. The Internet Archive Wayback Machine has ~ 13 petabytes of web archive stored that way, and you use it every time you look at a webpage in the Wayback. The file format is what you'd expect: each webpage is gzipped separately, the output is catted together, and the index stores the offset and length of each individual compressed webpage. https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive