3 ms·

> You might think that ZIP files would have a header, or at least a fixed-width footer ZIP files do have per file headers, which duplicate information in the d

by wahern 6y ago

> You might think that ZIP files would have a header, or at least a fixed-width footer ZIP files do have per file headers, which duplicate information in the directory, such as the file name. And malicious files have used this quirk to circumvent safeguards. See, e.g., WinRAR filename spoofing exploit: https://www.rapid7.com/db/modules/exploit/windows/fileformat/winrar_name_spoofing/ https://www.rapid7.com/db/modules/exploit/windows/fileformat... Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. Simply duplicating metadata can inadvertently create an explosion of failure cases and implementation complexity. Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules.

jandrese 6y ago
However if the metadata is duplicated you have the possibility of recovering data from a corrupted or incomplete archive.
klodolph 6y ago
> Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules. I can't agree with this. The reason you deal with inconsistencies and reconciliation rules is because, despite your efforts to the contrary, the world will give you inconsistent or incomplete copies of your data from time to time. A sure-fire way to ensure that your data is lost is to only keep one copy of it. At some place in your storage stack, you want some level of redundancy. The general idea for an archive format is that it is somewhat self-synchronizing, and somewhat possible to repair. All media has a non-zero error rate. And then there are all of the various reasons (software, human error, disk full, etc.) that a file might get accidentally truncated (which destroys the central directory in a zip file). You may not personally have to deal with the consequences of damaged media or similar types of failures, but for those that do, it's nice to have a data format where a stray error or two doesn't render the entire archive useless. If you are building on top of "reliable" storage systems (like cloud storage providers), you're just pushing the problem to lower levels in the stack... but there is less context at lower levels in the stack, which means recovery can't be very sophisticated.
- closeparen 6y ago
  This seems like a tension between the needs of genuine archival storage, and packaging of multi-file artifacts for web distribution. The latter use case dominates in the “archives” I actually encounter.
  - klodolph 6y ago
    You can create multiple formats and expect your users to understand the differences and select the correct one, but the 99% use case is non-technical users--so 99% of us are going to choose the default option. A bit of redundancy (a short signature and copy of file metadata) is a clear win here, at least for popular formats like Zip. In order for a format to BE the default option for non-technical users, it has to be reasonably good for most use cases, rather than optimized for one specific use case. If you are writing a package manager or backup program you might make a different choice, but how often do you find yourself writing a package manager?
lifthrasiir 6y ago
> Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. You might be very nervous to learn that most filesystems duplicate some metadata in one way or another. FAT, NTFS MFT, ext4 superblocks and group descriptors come to my mind. In some level you ought to have duplicates; it's a matter of where to place them. ZIP's choice made sense at the time of smaller and slower disks. (I do think that the modern standard ZIP should make some fields scrubbed to prevent compatibility problems, though.)

by wahern 6y ago

jandrese 6y ago

However if the metadata is duplicated you have the possibility of recovering data from a corrupted or incomplete archive.

klodolph 6y ago

> Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules. I can't agree with this. The reason you deal with inconsistencies and reconciliation rules is because, despite your efforts to the contrary, the world will give you inconsistent or incomplete copies of your data from time to time. A sure-fire way to ensure that your data is lost is to only keep one copy of it. At some place in your storage stack, you want some level of redundancy. The general idea for an archive format is that it is somewhat self-synchronizing, and somewhat possible to repair. All media has a non-zero error rate. And then there are all of the various reasons (software, human error, disk full, etc.) that a file might get accidentally truncated (which destroys the central directory in a zip file). You may not personally have to deal with the consequences of damaged media or similar types of failures, but for those that do, it's nice to have a data format where a stray error or two doesn't render the entire archive useless. If you are building on top of "reliable" storage systems (like cloud storage providers), you're just pushing the problem to lower levels in the stack... but there is less context at lower levels in the stack, which means recovery can't be very sophisticated.

closeparen 6y ago
This seems like a tension between the needs of genuine archival storage, and packaging of multi-file artifacts for web distribution. The latter use case dominates in the “archives” I actually encounter.
- klodolph 6y ago
  You can create multiple formats and expect your users to understand the differences and select the correct one, but the 99% use case is non-technical users--so 99% of us are going to choose the default option. A bit of redundancy (a short signature and copy of file metadata) is a clear win here, at least for popular formats like Zip. In order for a format to BE the default option for non-technical users, it has to be reasonably good for most use cases, rather than optimized for one specific use case. If you are writing a package manager or backup program you might make a different choice, but how often do you find yourself writing a package manager?

lifthrasiir 6y ago

> Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. You might be very nervous to learn that most filesystems duplicate some metadata in one way or another. FAT, NTFS MFT, ext4 superblocks and group descriptors come to my mind. In some level you ought to have duplicates; it's a matter of where to place them. ZIP's choice made sense at the time of smaller and slower disks. (I do think that the modern standard ZIP should make some fields scrubbed to prevent compatibility problems, though.)