6 ms·

zip is a horribly designed format https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/ There were precedents of well designed f

zip is a horribly designed format https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/ There were precedents of well designed file formats long before .zip hit the scene. Of course there wasn't the internet so being aware of better format designs might not have been common

stonogo 6y ago
Computer communications predates the internet. We definitely knew about them. Nobody wanted to pay for them. Also, it was in everyone's best interest to informally standardize on a tool -- in those days, having to download an extraction tool in addition to the archive you wanted was a nontrivial cost (real monetary cost, not just opportunity cost of wasting more time). Once SEA decided to start a legal fight, consensus was reached pretty quickly that nobody wanted to be the one jerk left using ARC. There were semi-organized grassroots campaigns to get laggards to switch. The personal computer software world was less stringent about standards at the time, and the tool that worked well enough was always going to beat the tool that pleased the software engineers.
- greggman3 6y ago
  My comment wasn't about paying for formats. It was about designing formats. The zip format is extremely poorly designed. See linked article above
  - stonogo 6y ago
    I already know the zip format. Formats were not open then. You did not select archive tools based on the on-disk format it used; you selected archive tools based on a combination of software cost, compression performance, and network effects. Having multiple implementations of software that could work on the same archive format was a novelty which often resulted in legal battles, as the formats themselves were often considered proprietary information.
    - greggman3 6y ago
      You still seem to be arguing over compression formats. I'm not saying there were better or worse compression formats. I'm only saying (a) zip is a badly designed format ... see article on why (b) the concept of a well designed file format (related to compression or not) was already well known in 1989 when zip shipped.
- knome 6y ago
  The reference to "arc wars" in the original article seemed interesting. I did a quick search and was pleased to have found these at textfiles.com http://textfiles.com/computers/sea.txt http://textfiles.com/computers/sea.txt http://textfiles.com/computers/arcsuit.txt http://textfiles.com/computers/arcsuit.txt
  - 1vuio0pswjnm7 6y ago
    "SEA claimed trademark infringement on the name "ARC," and violation of their copyright on the "look and feel" of ARC's command-line user- interface, in addition to charging Katz with appropriating ARC program code." The "look and feel" of the command line interface. By the mid-1990s people seemed to assume this only applied to GUIs.
userbinator 6y ago
It's interesting to compare zip with lzh, which was very popular in Japan; they're both from roughly the same era, yet lzh definitely seems far more logically designed (and often outperformed zip in compression ratio too.)
- onefuncman 6y ago
  from my memory, both ARJ and LZH outperformed ZIP for my benchmarks. I used to stripe archives across as many floppy disks as I had until I eventually got a CD writer...
  - userbinator 6y ago
    ZIP commonly uses DEFLATE which is LZ with a 32k window + Huffman, whereas LZH went up to LZ with a 64k window + Huffman. I suspect the doubled window alone is responsible for much of the improvement. According to http://fileformats.archiveteam.org/wiki/ARJ http://fileformats.archiveteam.org/wiki/ARJ , ARJ uses something similar to LZH's lh6 algorithm, which is LZ with a 32k window + Huffman, so the difference could be due to ARJ's reduced file format overhead compared to ZIP, or perhaps slightly better Huffman codes.
deleted 6y ago
[deleted]
geocar 6y ago
> But, that contradicts 4.1.9 that says zip files maybe be streamed. If zip files can be streamed then both of the example above would fail because in the first case we'd see file B and in the second we'd see file A (old) before we saw that the central directory doesn't reference them. If you have to wait for the central directory before you can correctly use any of the entries then functionally you can not stream zip files. You cannot filter a particular file in a stream using UNIX fifos and constant memory, but you could replace the old file A with the second file A in the stream because it is later. In this way, yes you can consume a zip file in a streaming way, even if you have to wait until the end to pass any of the output to another process. TAR and CPIO have the same "feature".
- greggman3 6y ago
  No you can't because the definition of "streaming" is to use the data as it comes down the wire. So CMD.BAT format disk CMD.BAT rem format disk If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong.
  - geocar 6y ago
    > If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong. The output directory contains the second CMD.BAT when you're done processing, whether you use the streaming algorithm or you consult the TOC, because in the streaming algorithm you simply overwrite the second CMD.BAT (if you created it as a result of the first CMD.BAT) with the new contents.
    - greggman3 6y ago
      There's no point in "streaming" if you're not using the data as it arrives. That's the entire meaning of "streaming".
      - geocar 6y ago
        What else can you do? You can't wait for the "central directory" record at the end, because now you've missed everything (you can't stream backwards!)
gildas 6y ago
Out of curiosity, is there any reason you don't mention zip.js at all in the documentation of unzipit?
zmodem 6y ago
One big part about your post is about streaming. I don't think the ZIP format was ever designed for streaming, and there are no claims about streaming in the original app notes. Actually, the wording about streaming you refer to were only added to the app note last year, and as you rightly point out, it's not guaranteed to work.

stonogo 6y ago

Computer communications predates the internet. We definitely knew about them. Nobody wanted to pay for them. Also, it was in everyone's best interest to informally standardize on a tool -- in those days, having to download an extraction tool in addition to the archive you wanted was a nontrivial cost (real monetary cost, not just opportunity cost of wasting more time). Once SEA decided to start a legal fight, consensus was reached pretty quickly that nobody wanted to be the one jerk left using ARC. There were semi-organized grassroots campaigns to get laggards to switch. The personal computer software world was less stringent about standards at the time, and the tool that worked well enough was always going to beat the tool that pleased the software engineers.

greggman3 6y ago
My comment wasn't about paying for formats. It was about designing formats. The zip format is extremely poorly designed. See linked article above
- stonogo 6y ago
  I already know the zip format. Formats were not open then. You did not select archive tools based on the on-disk format it used; you selected archive tools based on a combination of software cost, compression performance, and network effects. Having multiple implementations of software that could work on the same archive format was a novelty which often resulted in legal battles, as the formats themselves were often considered proprietary information.
  - greggman3 6y ago
    You still seem to be arguing over compression formats. I'm not saying there were better or worse compression formats. I'm only saying (a) zip is a badly designed format ... see article on why (b) the concept of a well designed file format (related to compression or not) was already well known in 1989 when zip shipped.
knome 6y ago
The reference to "arc wars" in the original article seemed interesting. I did a quick search and was pleased to have found these at textfiles.com http://textfiles.com/computers/sea.txt http://textfiles.com/computers/sea.txt http://textfiles.com/computers/arcsuit.txt http://textfiles.com/computers/arcsuit.txt
- 1vuio0pswjnm7 6y ago
  "SEA claimed trademark infringement on the name "ARC," and violation of their copyright on the "look and feel" of ARC's command-line user- interface, in addition to charging Katz with appropriating ARC program code." The "look and feel" of the command line interface. By the mid-1990s people seemed to assume this only applied to GUIs.

userbinator 6y ago

It's interesting to compare zip with lzh, which was very popular in Japan; they're both from roughly the same era, yet lzh definitely seems far more logically designed (and often outperformed zip in compression ratio too.)

onefuncman 6y ago
from my memory, both ARJ and LZH outperformed ZIP for my benchmarks. I used to stripe archives across as many floppy disks as I had until I eventually got a CD writer...
- userbinator 6y ago
  ZIP commonly uses DEFLATE which is LZ with a 32k window + Huffman, whereas LZH went up to LZ with a 64k window + Huffman. I suspect the doubled window alone is responsible for much of the improvement. According to http://fileformats.archiveteam.org/wiki/ARJ http://fileformats.archiveteam.org/wiki/ARJ , ARJ uses something similar to LZH's lh6 algorithm, which is LZ with a 32k window + Huffman, so the difference could be due to ARJ's reduced file format overhead compared to ZIP, or perhaps slightly better Huffman codes.

deleted 6y ago

[deleted]

geocar 6y ago

> But, that contradicts 4.1.9 that says zip files maybe be streamed. If zip files can be streamed then both of the example above would fail because in the first case we'd see file B and in the second we'd see file A (old) before we saw that the central directory doesn't reference them. If you have to wait for the central directory before you can correctly use any of the entries then functionally you can not stream zip files. You cannot filter a particular file in a stream using UNIX fifos and constant memory, but you could replace the old file A with the second file A in the stream because it is later. In this way, yes you can consume a zip file in a streaming way, even if you have to wait until the end to pass any of the output to another process. TAR and CPIO have the same "feature".

greggman3 6y ago
No you can't because the definition of "streaming" is to use the data as it comes down the wire. So CMD.BAT format disk CMD.BAT rem format disk If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong.
- geocar 6y ago
  > If the TOC only points to the second CMD.BAT then executing the first one as you stream it is wrong. The output directory contains the second CMD.BAT when you're done processing, whether you use the streaming algorithm or you consult the TOC, because in the streaming algorithm you simply overwrite the second CMD.BAT (if you created it as a result of the first CMD.BAT) with the new contents.
  - greggman3 6y ago
    There's no point in "streaming" if you're not using the data as it arrives. That's the entire meaning of "streaming".
    - geocar 6y ago
      What else can you do? You can't wait for the "central directory" record at the end, because now you've missed everything (you can't stream backwards!)

gildas 6y ago

Out of curiosity, is there any reason you don't mention zip.js at all in the documentation of unzipit?

zmodem 6y ago

One big part about your post is about streaming. I don't think the ZIP format was ever designed for streaming, and there are no claims about streaming in the original app notes. Actually, the wording about streaming you refer to were only added to the app note last year, and as you rightly point out, it's not guaranteed to work.