6 ms·

The other common archive, ZIP, has a standard. But it's a hot mess! To quote Wikipedia: > Tools that correctly read ZIP archives must scan for the end of centr

by tdeck 6y ago

The other common archive, ZIP, has a standard. But it's a hot mess! To quote Wikipedia: > Tools that correctly read ZIP archives must scan for the end of central directory record signature, and then, as appropriate, the other, indicated, central directory records. They must not scan for entries from the top of the ZIP file, because (as previously mentioned in this section) only the central directory specifies where a file chunk starts and that it has not been deleted. Scanning could lead to false positives, as the format does not forbid other data to be between chunks, nor file data streams from containing such signatures You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory. It's amazing to me that this is the format that won the compression wars and stuck around to this day.

LeoPanthera 6y ago
zip at least has a de-facto standard, which today is the "Info-ZIP" utility.
- chungy 6y ago
  Zip also has a real bonefide standard: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
  - greggman3 6y ago
    That spec is seriously underspecified https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/
    - chungy 6y ago
      Seems kind of a shame to tuck this away in a comments section on a (mostly) unrelated article. I would suggest you submit it as a main article for HN :)
- greggman3 6y ago
  not really. The zip spec says anything can come in before the "zip data". Info-Zip gives this example unzip unz552x3.exe unzipsfx.exe // extract the DOS SFX stub cat unzipsfx.exe yourzip.zip > yourDOSzip.exe // create the SFX archive zip -A yourDOSzip.exe // fix up internal offsets http://infozip.sourceforge.net/FAQ.html#unixSFX http://infozip.sourceforge.net/FAQ.html#unixSFX In other words, append any data you want, then fix the pointers in the central directory. There are tons of various zip utilities that will mess up depending on what you put in that self extractor including MacOS's built in finder based zip support. PKWare, the keepers of the documentation for the ZIP format refuse to specify that you must read the end of central directory. They do this by claiming you can stream a zip file. But you can't stream a zip file if you have to read the central directory because data you may have already used, "the definition of streaming", maybe not be listed in the central directory in which case you shouldn't have used it. In other words imagine each file in the zip is a compressed script and you are going to execute them. Imagine there are 2 scripts, the first one is "rm -rf /" and the second "ls /". The central directory only points to the 2nd script but if you stream you'll read the first script an execute it. The ZIP spec claims you can stream but clearly you can't.
wahern 6y ago
> You might think that ZIP files would have a header, or at least a fixed-width footer ZIP files do have per file headers, which duplicate information in the directory, such as the file name. And malicious files have used this quirk to circumvent safeguards. See, e.g., WinRAR filename spoofing exploit: https://www.rapid7.com/db/modules/exploit/windows/fileformat/winrar_name_spoofing/ https://www.rapid7.com/db/modules/exploit/windows/fileformat... Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. Simply duplicating metadata can inadvertently create an explosion of failure cases and implementation complexity. Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules.
- jandrese 6y ago
  However if the metadata is duplicated you have the possibility of recovering data from a corrupted or incomplete archive.
- klodolph 6y ago
  > Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules. I can't agree with this. The reason you deal with inconsistencies and reconciliation rules is because, despite your efforts to the contrary, the world will give you inconsistent or incomplete copies of your data from time to time. A sure-fire way to ensure that your data is lost is to only keep one copy of it. At some place in your storage stack, you want some level of redundancy. The general idea for an archive format is that it is somewhat self-synchronizing, and somewhat possible to repair. All media has a non-zero error rate. And then there are all of the various reasons (software, human error, disk full, etc.) that a file might get accidentally truncated (which destroys the central directory in a zip file). You may not personally have to deal with the consequences of damaged media or similar types of failures, but for those that do, it's nice to have a data format where a stray error or two doesn't render the entire archive useless. If you are building on top of "reliable" storage systems (like cloud storage providers), you're just pushing the problem to lower levels in the stack... but there is less context at lower levels in the stack, which means recovery can't be very sophisticated.
greggman3 6y ago
ZIP is a hot mess but that particular feature is not a bad idea, it's just poorly implemented / designed. The point was you could create a multi-disk zip (zip came out when we had 1.4meg floppy disks). If you zip up 10meg across 7 disks and you want to update the README of your 7 disks, by putting the central directory at the end you just ask the user to insert disk 7, read the central directory, append the new README, write a new central directory. With the directory at the start you'd have to re-write all 7 disks. With it at the end you only have to re-write disk 7 and only a small portion of the file. The only bad part of that particular feature in the zip implementation is the scanning part. The variable sized underspecified "Zip comment" that you have to scan through should have either come before the central directory OR there should have been a length or pointer to directory after it so no scanning needed.
- anyfoo 6y ago
  Also what would have been wrong with having a fixed size trailer/footer at the end of the file, pointing back to the (possibly newly appended) central directory, and even back to the previous trailer/footer if chaining is necessary/useful? No scanning necessary, then.
  - silvestrov 6y ago
    The PDF format does this correctly. PDF files end with: startxref 12345 %%EOF where 12345 is the file offset of the TOC.
- wilsonrocks 6y ago
  Many memories of PKZip asking me to insert the last disk of a set :) Usually when copying a game...
  - ungamedplayer 6y ago
    And disk (total / 2 + 1) was the corrupted one, you'd only find it mid way through.
yjftsjthsd-h 6y ago
Oh yeah, that's the trick that lets you shove a zip into other file formats and have it be valid, right?
st_goliath 6y ago
> You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory. Yes. ZIP has no header, it has a footer. What gets really messy is that the footer contains a variable length comment field, with the length stored before the field, so good look finding where the footer starts. And here we are just talking a about a correctly generated file. This also opens up the door to all kinds of intentional abuse.
tda 6y ago
So can you technically "hide" data in a ZIP file as a form of steganography? Just add it to the zip file but mark it as deleted. Then any common zip utility will skip it, but a specialized one could extract that extra information. This would be easy to miss as the size of a zip file is always different than the sum of the size of the files it contains. And of course googling zip file steganography reveals this is not a new idea: https://aroralakshay2014.medium.com/hide-it-lest-find-it-zip-steganography-af20f5b1cbc9 https://aroralakshay2014.medium.com/hide-it-lest-find-it-zip...
viraptor 6y ago
> It's amazing to me that this is the format that won the compression wars and stuck around to this day. The header at the end actually helped it a lot. It allowed the self-unpacking archives to exist.
- diggernet 6y ago
  And allowed adding a file to a zip without needing to copy the whole zip to add header space at the start. A mere optimization today, but absolutely critical when working with floppies.

by tdeck 6y ago

LeoPanthera 6y ago

zip at least has a de-facto standard, which today is the "Info-ZIP" utility.

chungy 6y ago
Zip also has a real bonefide standard: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
- greggman3 6y ago
  That spec is seriously underspecified https://games.greggman.com/game/zip-rant/ https://games.greggman.com/game/zip-rant/
  - chungy 6y ago
    Seems kind of a shame to tuck this away in a comments section on a (mostly) unrelated article. I would suggest you submit it as a main article for HN :)
greggman3 6y ago
not really. The zip spec says anything can come in before the "zip data". Info-Zip gives this example unzip unz552x3.exe unzipsfx.exe // extract the DOS SFX stub cat unzipsfx.exe yourzip.zip > yourDOSzip.exe // create the SFX archive zip -A yourDOSzip.exe // fix up internal offsets http://infozip.sourceforge.net/FAQ.html#unixSFX http://infozip.sourceforge.net/FAQ.html#unixSFX In other words, append any data you want, then fix the pointers in the central directory. There are tons of various zip utilities that will mess up depending on what you put in that self extractor including MacOS's built in finder based zip support. PKWare, the keepers of the documentation for the ZIP format refuse to specify that you must read the end of central directory. They do this by claiming you can stream a zip file. But you can't stream a zip file if you have to read the central directory because data you may have already used, "the definition of streaming", maybe not be listed in the central directory in which case you shouldn't have used it. In other words imagine each file in the zip is a compressed script and you are going to execute them. Imagine there are 2 scripts, the first one is "rm -rf /" and the second "ls /". The central directory only points to the 2nd script but if you stream you'll read the first script an execute it. The ZIP spec claims you can stream but clearly you can't.

wahern 6y ago

> You might think that ZIP files would have a header, or at least a fixed-width footer ZIP files do have per file headers, which duplicate information in the directory, such as the file name. And malicious files have used this quirk to circumvent safeguards. See, e.g., WinRAR filename spoofing exploit: https://www.rapid7.com/db/modules/exploit/windows/fileformat/winrar_name_spoofing/ https://www.rapid7.com/db/modules/exploit/windows/fileformat... Any time metadata is duplicated in a system (specifically, an interface or protocol), alarm bells go off for me. Simply duplicating metadata can inadvertently create an explosion of failure cases and implementation complexity. Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules.

jandrese 6y ago
However if the metadata is duplicated you have the possibility of recovering data from a corrupted or incomplete archive.
klodolph 6y ago
> Conversely, avoiding duplication of data is an example of failure-proof design--if it's not duplicated, you don't have to worry about inconsistencies and reconciliation rules. I can't agree with this. The reason you deal with inconsistencies and reconciliation rules is because, despite your efforts to the contrary, the world will give you inconsistent or incomplete copies of your data from time to time. A sure-fire way to ensure that your data is lost is to only keep one copy of it. At some place in your storage stack, you want some level of redundancy. The general idea for an archive format is that it is somewhat self-synchronizing, and somewhat possible to repair. All media has a non-zero error rate. And then there are all of the various reasons (software, human error, disk full, etc.) that a file might get accidentally truncated (which destroys the central directory in a zip file). You may not personally have to deal with the consequences of damaged media or similar types of failures, but for those that do, it's nice to have a data format where a stray error or two doesn't render the entire archive useless. If you are building on top of "reliable" storage systems (like cloud storage providers), you're just pushing the problem to lower levels in the stack... but there is less context at lower levels in the stack, which means recovery can't be very sophisticated.

greggman3 6y ago

ZIP is a hot mess but that particular feature is not a bad idea, it's just poorly implemented / designed. The point was you could create a multi-disk zip (zip came out when we had 1.4meg floppy disks). If you zip up 10meg across 7 disks and you want to update the README of your 7 disks, by putting the central directory at the end you just ask the user to insert disk 7, read the central directory, append the new README, write a new central directory. With the directory at the start you'd have to re-write all 7 disks. With it at the end you only have to re-write disk 7 and only a small portion of the file. The only bad part of that particular feature in the zip implementation is the scanning part. The variable sized underspecified "Zip comment" that you have to scan through should have either come before the central directory OR there should have been a length or pointer to directory after it so no scanning needed.

anyfoo 6y ago
Also what would have been wrong with having a fixed size trailer/footer at the end of the file, pointing back to the (possibly newly appended) central directory, and even back to the previous trailer/footer if chaining is necessary/useful? No scanning necessary, then.
- silvestrov 6y ago
  The PDF format does this correctly. PDF files end with: startxref 12345 %%EOF where 12345 is the file offset of the TOC.
wilsonrocks 6y ago
Many memories of PKZip asking me to insert the last disk of a set :) Usually when copying a game...
- ungamedplayer 6y ago
  And disk (total / 2 + 1) was the corrupted one, you'd only find it mid way through.

yjftsjthsd-h 6y ago

Oh yeah, that's the trick that lets you shove a zip into other file formats and have it be valid, right?

st_goliath 6y ago

> You might think that ZIP files would have a header, or at least a fixed-width footer, but they don't. Instead you're supposed to scan backward through the file looking for a magic number indicating you've found the central directory. Yes. ZIP has no header, it has a footer. What gets really messy is that the footer contains a variable length comment field, with the length stored before the field, so good look finding where the footer starts. And here we are just talking a about a correctly generated file. This also opens up the door to all kinds of intentional abuse.

tda 6y ago

So can you technically "hide" data in a ZIP file as a form of steganography? Just add it to the zip file but mark it as deleted. Then any common zip utility will skip it, but a specialized one could extract that extra information. This would be easy to miss as the size of a zip file is always different than the sum of the size of the files it contains. And of course googling zip file steganography reveals this is not a new idea: https://aroralakshay2014.medium.com/hide-it-lest-find-it-zip-steganography-af20f5b1cbc9 https://aroralakshay2014.medium.com/hide-it-lest-find-it-zip...

viraptor 6y ago

> It's amazing to me that this is the format that won the compression wars and stuck around to this day. The header at the end actually helped it a lot. It allowed the self-unpacking archives to exist.

diggernet 6y ago
And allowed adding a file to a zip without needing to copy the whole zip to add header space at the start. A mere optimization today, but absolutely critical when working with floppies.