4 ms·

I don't think you understand the reason for the ZIP archive file design. Back in the late 1980s, backup media for consumers was limited to mostly floppy disks,

by canucker2016 2y ago

I don't think you understand the reason for the ZIP archive file design. Back in the late 1980s, backup media for consumers was limited to mostly floppy disks, some users had tape/another hard disk. Say you had a variable number of files to compress and write out to a ZIP archive. IF you write out the central directory first, followed by all the individually possibly compressed and/or encrypted files, you'd have to calculate all the files to be archived, process them (compress and/or encrypt), write them out, then go back and update the info for the actual compressed values and offsets for the ZIP local entries. Now if you wanted to add files to the ZIP archive, the central directory will grow and push the following individual compressed/encrypted files further out and you'll have to update ALL the central directory entries since each entry includes an offset from the beginning of the disk - if the archive does not span multiple disks, this offset is from the start of the ZIP archive file. So that's one reason for why the ZIP central directory is placed at the end of the ZIP archive file. If you're streaming the output from a ZIP program, then placing the ZIP central dir at the start of the file is a non-starter since you can't rewind a stream to update the ZIP central directory entries. Why do some programs ignore the ZIP central directory as the ONE source of truth? Before SSDs and their minimal seek latency, coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory and proceed accordingly. If the central directory in the ZIP archive is corrupted or missing, the user could still recover the data for the individual files (if all the ZIP local entries are intact). In this case, ignoring the ZIP central dir and scanning sequentially for ZIP local entries is REQUIRED. The fault here is the security scanners. There's never been any guarantee that the ONLY data in the ZIP archive was only valid ZIP local file entries followed by the ZIP central directory. Between ZIP local file entries, one can place any data. Unzip programs don't care.

Spivak 2y ago
The more general principle is that single source of truth is not ideal for data storage where you're worried about corruption. There's a backup MBR on your hard disk at the end, your ext4 filesystem has many backups of your superblock. When it comes to user data the natural programmer instinct for "is exactly what I expect or fail" which is typically good design, falls to pragmatism where try your hardest to not lose data, partial results are better then nothing, is desired.
- tjoff 2y ago
  Having a backup copy isn't quite the same thing though. It is just a copy of the single source of truth. Not a different implementation or used for a different use case. Also trivial to verify.
unoti 2y ago
> coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory Would this have worked? Reserve a few bytes at the beginning of the archive at a fixed location offset from the start, and say "this is where we will write the offset to where the central directory will start." Then build the whole archive, writing the central directory at the end. Then seek back to that known offset at the start of the file and write the offset to the central directory. When creating the archive, we can write the central directory to a temp file, and then append that in to the end of the file we're building at the end, and fix up the offset. Seems like this strategy would enable us to both have a number of files in the archive that are known at the beginning, and also allow us to do a high-speed seek to the central directory when reading the archive. I imagine people thought about this idea and didn't do it for one reason or another. I can imagine why we didn't do that for Unix TAR-- most tape devices are a one-way write stream and don't have random access. But ZIP was designed for disks; I'm curious why this idea wouldn't have solved both problems.
- canucker2016 2y ago
  You forgot about the streaming case. ZIP creators can stream the archive out and never seek back earlier in the stream (send the data to another tool or a write-only medium). The central directory at the end of the archive fits that constraint. Any design where a placeholder needs to be updated later won't.

Spivak 2y ago

The more general principle is that single source of truth is not ideal for data storage where you're worried about corruption. There's a backup MBR on your hard disk at the end, your ext4 filesystem has many backups of your superblock. When it comes to user data the natural programmer instinct for "is exactly what I expect or fail" which is typically good design, falls to pragmatism where try your hardest to not lose data, partial results are better then nothing, is desired.

tjoff 2y ago
Having a backup copy isn't quite the same thing though. It is just a copy of the single source of truth. Not a different implementation or used for a different use case. Also trivial to verify.

unoti 2y ago

> coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory Would this have worked? Reserve a few bytes at the beginning of the archive at a fixed location offset from the start, and say "this is where we will write the offset to where the central directory will start." Then build the whole archive, writing the central directory at the end. Then seek back to that known offset at the start of the file and write the offset to the central directory. When creating the archive, we can write the central directory to a temp file, and then append that in to the end of the file we're building at the end, and fix up the offset. Seems like this strategy would enable us to both have a number of files in the archive that are known at the beginning, and also allow us to do a high-speed seek to the central directory when reading the archive. I imagine people thought about this idea and didn't do it for one reason or another. I can imagine why we didn't do that for Unix TAR-- most tape devices are a one-way write stream and don't have random access. But ZIP was designed for disks; I'm curious why this idea wouldn't have solved both problems.

canucker2016 2y ago
You forgot about the streaming case. ZIP creators can stream the archive out and never seek back earlier in the stream (send the data to another tool or a write-only medium). The central directory at the end of the archive fits that constraint. Any design where a placeholder needs to be updated later won't.