4 ms·

ZIP does have bad ideas like mentioned there. I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). (

ZIP does have bad ideas like mentioned there. I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). (If you do not need the metadata, I like the Hamster archive format, which is: It is a sequence of lumps, where each lump consists of: null-terminated ASCII filename, 32-bit PDP-endian data size, and then the data. (That is the entire specification.)) (Another thing about ZIP: If you have a truncated ZIP file (possibly due to a disk error on a floppy disk), I have found that bsdtar can read it; other programs I have tried are unable to read truncated ZIP files.)

cesarb 5y ago
> I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). That has the bad property that it does not allow for easy random access. With the ZIP format, you can directly read an arbitrary file within the archive without having to decompress or even read the compressed data of any of the other files, and many applications of the ZIP format (for example, JAR archives) depend on that property. With .tar.gz and similar, you have to decompress all data before the desired file.
- idle_zealot 5y ago
  The solution here is to invert (like zip does). So rather than a .tar.gz you get a .tar full of .gzs.
  - LeoPanthera 5y ago
    This ruins your compression ratio, since references between files becomes impossible.
    - d110af5ccf 5y ago
      Zstd (https://facebook.github.io/zstd/ https://facebook.github.io/zstd/) provides more flexibility and supports user provided compression dictionaries - you can just place one at the front of the archive. Of course this doesn't fully solve the metadata issue and it's definitely more complex than just using the zip format. But on the plus side, it allows you to avoid using the zip format and the compression algorithm is significantly more performant and flexible.
      - stouset 5y ago
        Placing a user-supplied dictionary at the front of the archive is essentially how “normal” compression works. The advantage of a user-supplied dictionary is that it can be transmitted once, out of band. With this approach it can also be larger than what would make sense for an individual file and it can take advantage of similarity between files. But if you just send it along with the compressed data anyway, you’re not buying anything extra.
        d110af5ccf 5y ago
        Yes, I realize that it's manually reinventing the wheel. However the context here was supporting random access (and append, truncate, etc) while also using separate formats for archiving and compression. Such an approach is more work but provides clean separation and thus far more flexibility. Bear in mind that the alternative (in context) is throwing a bunch of independently compressed files at something like tar. In the event that the files are small, that will absolutely destroy the compression ratio. If the files are large enough or you only have a few then you could always omit the custom dictionary.
      - mort96 5y ago
        To manage this, you're essentially back to writing a tool which handles both compression and zipping, which is exactly what zzo38computer set out to avoid.
    - account42 5y ago
      (Unlimited) references between files are incompatible with random access.
  - j16sdiz 5y ago
    Tar does not have a good index either. It is designed for tapes, not random access disk
- zzo38computer 5y ago
  Yes, that is a valid point; for some applications, random access will be helpful. One possibility is to make each file as a separate compression block, which is possible if using concatenatable compression formats; then, the same format can be used for solid and for non-solid compression. However, then, you will need a index if you want random access. I don't know what compression formats have such a feature as a index, but I thought of some ideas of a (concatenatable) compression format; one possible thing to add is a optional index block (which links the compressed and uncompressed offsets of the beginning of blocks which do not depend on any previous blocks).
  - thrashh 5y ago
    Random access of ZIPs was helpful during the dial-up days because you could download select files instead of the whole ZIP file too. If something like tar.gz had been standard then, then that wouldn’t have been possible.
- yjftsjthsd-h 5y ago
  Obviously the solution is to make .gz.tar files:)
- Someone 5y ago
  A solution would be to reinvent the wheel invented by pixie and others (PDF has a TOC at the end, too, for example (in ascii, because PDF started life as a text-based format “postscript without the halting problem”)) introduce a convention that a table of contents gets appended as two files: - a file named “TOC” with a well-specified format that contains the byte offsets of each file in the archive - a file named “TOC-offset” with content the 8-byte offset of the “TOC” entry (keeping this separate means the file can be fixed-size. That makes reliably finding it easier) Preferably, both files also would have some checksum. The TOC itself could have a pointer to the previous TOC. If so, it could be a diff. Supporting tools could easily check for the presence of a TOC and either overwrite it with new files and wrote a new one, or just append new files and write a new TOC pair. Attackers could thwart this by creating an archive without a TOC that has a file ending in what looks like a TOC/TOC-offset pair. An alternative would be to write the TOC-offset file at the start. That’s more robust against such attacks, but requires the ability to overwrite the start of a file.
  - chriswarbo 5y ago
    There are dangers when allowing one TOC to refer to another, and the ability of offsets to cause self-reference, e.g. see Section 3 of http://www.ieee-security.org/TC/SPW2014/papers/5103a198.PDF http://www.ieee-security.org/TC/SPW2014/papers/5103a198.PDF In general, it's also a good idea to keep data and metadata distinct, to avoid potential conflicts (and hence exploits). In this case having the archive metadata as a file inside the archive seems fraught with dangers. We could avoid this by e.g. having two levels of archive, e.g. the first level contains TOC files and data files (e.g. tar files); the 'payload' files are kept in the data files. That way, there's no way for payload to be interpreted as metadata, or vice versa.
- tored 5y ago
  Random access is a nice feature of zip, for a system I worked with I made HTTP API that could read induvial files within a zip archive without unpacking, trivial to implement and still performant, so we didn't need to change anything in how these archives were stored.
userbinator 5y ago
32-bit PDP-endian data size That may have been sufficient 30 years ago...
GoblinSlayer 5y ago
>I like the Hamster archive format, which is: It is a sequence of lumps, where each lump consists of: null-terminated ASCII filename, 32-bit PDP-endian data size, and then the data. gzip format does that too, but gunzip extracts concatenated files as one file with everything concatenated.

cesarb 5y ago

> I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). That has the bad property that it does not allow for easy random access. With the ZIP format, you can directly read an arbitrary file within the archive without having to decompress or even read the compressed data of any of the other files, and many applications of the ZIP format (for example, JAR archives) depend on that property. With .tar.gz and similar, you have to decompress all data before the desired file.

idle_zealot 5y ago
The solution here is to invert (like zip does). So rather than a .tar.gz you get a .tar full of .gzs.
- LeoPanthera 5y ago
  This ruins your compression ratio, since references between files becomes impossible.
  - d110af5ccf 5y ago
    Zstd (https://facebook.github.io/zstd/ https://facebook.github.io/zstd/) provides more flexibility and supports user provided compression dictionaries - you can just place one at the front of the archive. Of course this doesn't fully solve the metadata issue and it's definitely more complex than just using the zip format. But on the plus side, it allows you to avoid using the zip format and the compression algorithm is significantly more performant and flexible.
    - stouset 5y ago
      Placing a user-supplied dictionary at the front of the archive is essentially how “normal” compression works. The advantage of a user-supplied dictionary is that it can be transmitted once, out of band. With this approach it can also be larger than what would make sense for an individual file and it can take advantage of similarity between files. But if you just send it along with the compressed data anyway, you’re not buying anything extra.
      - d110af5ccf 5y ago
        Yes, I realize that it's manually reinventing the wheel. However the context here was supporting random access (and append, truncate, etc) while also using separate formats for archiving and compression. Such an approach is more work but provides clean separation and thus far more flexibility. Bear in mind that the alternative (in context) is throwing a bunch of independently compressed files at something like tar. In the event that the files are small, that will absolutely destroy the compression ratio. If the files are large enough or you only have a few then you could always omit the custom dictionary.
    - mort96 5y ago
      To manage this, you're essentially back to writing a tool which handles both compression and zipping, which is exactly what zzo38computer set out to avoid.
  - account42 5y ago
    (Unlimited) references between files are incompatible with random access.
- j16sdiz 5y ago
  Tar does not have a good index either. It is designed for tapes, not random access disk
zzo38computer 5y ago
Yes, that is a valid point; for some applications, random access will be helpful. One possibility is to make each file as a separate compression block, which is possible if using concatenatable compression formats; then, the same format can be used for solid and for non-solid compression. However, then, you will need a index if you want random access. I don't know what compression formats have such a feature as a index, but I thought of some ideas of a (concatenatable) compression format; one possible thing to add is a optional index block (which links the compressed and uncompressed offsets of the beginning of blocks which do not depend on any previous blocks).
- thrashh 5y ago
  Random access of ZIPs was helpful during the dial-up days because you could download select files instead of the whole ZIP file too. If something like tar.gz had been standard then, then that wouldn’t have been possible.
yjftsjthsd-h 5y ago
Obviously the solution is to make .gz.tar files:)
Someone 5y ago
A solution would be to reinvent the wheel invented by pixie and others (PDF has a TOC at the end, too, for example (in ascii, because PDF started life as a text-based format “postscript without the halting problem”)) introduce a convention that a table of contents gets appended as two files: - a file named “TOC” with a well-specified format that contains the byte offsets of each file in the archive - a file named “TOC-offset” with content the 8-byte offset of the “TOC” entry (keeping this separate means the file can be fixed-size. That makes reliably finding it easier) Preferably, both files also would have some checksum. The TOC itself could have a pointer to the previous TOC. If so, it could be a diff. Supporting tools could easily check for the presence of a TOC and either overwrite it with new files and wrote a new one, or just append new files and write a new TOC pair. Attackers could thwart this by creating an archive without a TOC that has a file ending in what looks like a TOC/TOC-offset pair. An alternative would be to write the TOC-offset file at the start. That’s more robust against such attacks, but requires the ability to overwrite the start of a file.
- chriswarbo 5y ago
  There are dangers when allowing one TOC to refer to another, and the ability of offsets to cause self-reference, e.g. see Section 3 of http://www.ieee-security.org/TC/SPW2014/papers/5103a198.PDF http://www.ieee-security.org/TC/SPW2014/papers/5103a198.PDF In general, it's also a good idea to keep data and metadata distinct, to avoid potential conflicts (and hence exploits). In this case having the archive metadata as a file inside the archive seems fraught with dangers. We could avoid this by e.g. having two levels of archive, e.g. the first level contains TOC files and data files (e.g. tar files); the 'payload' files are kept in the data files. That way, there's no way for payload to be interpreted as metadata, or vice versa.
tored 5y ago
Random access is a nice feature of zip, for a system I worked with I made HTTP API that could read induvial files within a zip archive without unpacking, trivial to implement and still performant, so we didn't need to change anything in how these archives were stored.

userbinator 5y ago

32-bit PDP-endian data size That may have been sufficient 30 years ago...

GoblinSlayer 5y ago

>I like the Hamster archive format, which is: It is a sequence of lumps, where each lump consists of: null-terminated ASCII filename, 32-bit PDP-endian data size, and then the data. gzip format does that too, but gunzip extracts concatenated files as one file with everything concatenated.