3 ms·

Only tangentially related but the ZIP file format is designed with floppy disks in mind. A ZIP file puts its table of contents at the end of the file. The point

Only tangentially related but the ZIP file format is designed with floppy disks in mind. A ZIP file puts its table of contents at the end of the file. The point of this design was that if you had a slow floppy and wanted to add some files to zip file you could read the old table of contents, append some new files and write a new table of contents. Therefore you didn't have to re-write the entire file. Of course you might have some wasted space because you've got 2 tables of contents and might even have some older versions of files earlier in the zip file. That was a trade off though. Re-writing an entire floppy (or 10) was prohibitively slow. Unfortunately that makes zip files whole unsuited to streaming because according to the official standard you MUST read the table of contents at the end of the file to know which parts of the file correspond to the correct contents. Steaming the file and looking for headers is an invalid way to parse a zip file. A large number of engineers are unaware of this fact and sadly keep choosing zip for standards that would be better off with a format that can be streamed. (tar/gz can be streamed). Examples: Chrome's extension format. The W3Cs Widget format. I'm sure there are several others.

rarrrrrr 14y ago
Indeed. This is one of my favorite idioms for untarring a remote file locally: ssh remote_host cat archive.tar.gz | pv | tar zx Unless the contents are very large, that's almost always faster than rsyncing the remote directory locally, because both ends just stream. No coordinated start and stop for each little file. There is one big advantage of the zip file format: you can extract a single member without having to read the whole archive. It's usually just two seeks: seek to end, read TOC; seek to position, read member.
rogerbinns 14y ago
This is also the reason the self extractors are so easy. A zip file doesn't care what is at the beginning of the file so you can prepend anything you want. Note that streaming is pointless for the examples you give. You definitely want the content signed and integrity checked, and you can't do that until the whole thing is downloaded. If you did decide to start extracting before you have the whole thing then you'd need to ensure no race conditions or avenues of attack from malicious payloads. On a related tale, back when I worked on WAN optimizers I looked at Windows Explorer showing the contents of a zip file (folder view). Writing SMB/CIFS requests by hand it was possible to do so using 5 requests (which means 5 round trips). Explorer in Windows XP used 1,500 requests and Explorer in Vista used 3,000!
saurik 14y ago
(edit: I just noticed that rarrrrrr made the same argument in his comment, which I totally believe I did not have locally before leaving my response. Regardless, I went into more depth and provided a few related issues, so I don't feel my comment is sufficiently repetitive to delete.) While tar.gz can be streamed, if you happen to have the entire zip file already and want to extract a single item from it, you are screwed: you actually must scan through the entire file; this is both because gzip is not random access and because tar files have no index at all, they just concatenate files. This reason is actually insidious, because it also means that zip files are much more efficient containers if you need to be able to "browse into them", such as with a GUI. This means that they tend to be much better supported ("better" assuming .tar.gz files are supported at all) by graphical file managers. In comparison, the fact that you can't stream download and decompress the tar.gz on the fly is actually largely irrelevant: you aren't going to be using the Chrome extension until you've downloaded the entire package anyway; and for some file formats (Java jar files, for example) you don't intend to decompress the entire contents anyway. Ironically, the reason tar files are designed this way, is actually due to yet another ancient piece of storage technology: the tape drive. The name "tar" stands for "tape archive", and is a modified version of the file format used for "ar" that is better suited to tape drives, which were much slower than floppies if you needed to rewrite earlier content (if rewriting was possible at all, which I don't think it was on all tapes).
- greggman 14y ago
  Streaming is not unimportant for the examples I gave. Both are 1 step away from being replacements for flash apps. They are basically a webpage with all assets bundled together making them easy to distribute. Except, because a non-streamable format was chosen they can't make the leap to full flash replacements. Even as an extensions or widget, the fact that I can't stream them means I have to wait for them to download entirely before they can be used. I can certainly imagine a Chrome app (game) > 500meg that would be awesome if I could start interacting with as soon as 1-20 meg had been downloaded. That use case though was made harder because a non streamable format was chosen. Instead the app has to download some small boot loader and the manually stream the rest of its assets. With a little more foresight that could have been solved directly by the format chosen. As for tar/gz not being random access. There's nothing preventing implementations from caching locations as the file is downloaded or inserting a file as part of the tar that lists the locations.

rarrrrrr 14y ago

Indeed. This is one of my favorite idioms for untarring a remote file locally: ssh remote_host cat archive.tar.gz | pv | tar zx Unless the contents are very large, that's almost always faster than rsyncing the remote directory locally, because both ends just stream. No coordinated start and stop for each little file. There is one big advantage of the zip file format: you can extract a single member without having to read the whole archive. It's usually just two seeks: seek to end, read TOC; seek to position, read member.

rogerbinns 14y ago

This is also the reason the self extractors are so easy. A zip file doesn't care what is at the beginning of the file so you can prepend anything you want. Note that streaming is pointless for the examples you give. You definitely want the content signed and integrity checked, and you can't do that until the whole thing is downloaded. If you did decide to start extracting before you have the whole thing then you'd need to ensure no race conditions or avenues of attack from malicious payloads. On a related tale, back when I worked on WAN optimizers I looked at Windows Explorer showing the contents of a zip file (folder view). Writing SMB/CIFS requests by hand it was possible to do so using 5 requests (which means 5 round trips). Explorer in Windows XP used 1,500 requests and Explorer in Vista used 3,000!

saurik 14y ago

(edit: I just noticed that rarrrrrr made the same argument in his comment, which I totally believe I did not have locally before leaving my response. Regardless, I went into more depth and provided a few related issues, so I don't feel my comment is sufficiently repetitive to delete.) While tar.gz can be streamed, if you happen to have the entire zip file already and want to extract a single item from it, you are screwed: you actually must scan through the entire file; this is both because gzip is not random access and because tar files have no index at all, they just concatenate files. This reason is actually insidious, because it also means that zip files are much more efficient containers if you need to be able to "browse into them", such as with a GUI. This means that they tend to be much better supported ("better" assuming .tar.gz files are supported at all) by graphical file managers. In comparison, the fact that you can't stream download and decompress the tar.gz on the fly is actually largely irrelevant: you aren't going to be using the Chrome extension until you've downloaded the entire package anyway; and for some file formats (Java jar files, for example) you don't intend to decompress the entire contents anyway. Ironically, the reason tar files are designed this way, is actually due to yet another ancient piece of storage technology: the tape drive. The name "tar" stands for "tape archive", and is a modified version of the file format used for "ar" that is better suited to tape drives, which were much slower than floppies if you needed to rewrite earlier content (if rewriting was possible at all, which I don't think it was on all tapes).

greggman 14y ago
Streaming is not unimportant for the examples I gave. Both are 1 step away from being replacements for flash apps. They are basically a webpage with all assets bundled together making them easy to distribute. Except, because a non-streamable format was chosen they can't make the leap to full flash replacements. Even as an extensions or widget, the fact that I can't stream them means I have to wait for them to download entirely before they can be used. I can certainly imagine a Chrome app (game) > 500meg that would be awesome if I could start interacting with as soon as 1-20 meg had been downloaded. That use case though was made harder because a non streamable format was chosen. Instead the app has to download some small boot loader and the manually stream the rest of its assets. With a little more foresight that could have been solved directly by the format chosen. As for tar/gz not being random access. There's nothing preventing implementations from caching locations as the file is downloaded or inserting a file as part of the tar that lists the locations.