14 ms·
Zip – How not to design a file format
- mjevans 5y agohttps://web.archive.org/web/20210722231119/https://games.greggman.com/game/zip-rant/ https://web.archive.org/web/20210722231119/https://games.gre... Edit: For anyone seeing this much later in time, the website got flooded when it hit the Hacker News main page. I grabbed the archive mirror of it from earlier today.
- IgorPartola 5y agoSince this is a frequent occurrence, wouldn’t it be neat if we could do this kind of “archive interesting article” thing using IPFS? Maybe even do it automatically when submitting it to HN such that it could tell HN itself to grab a copy?
- tablespoon 5y ago> Since this is a frequent occurrence, wouldn’t it be neat if we could do this kind of “archive interesting article” thing using IPFS? Maybe even do it automatically when submitting it to HN such that it could tell HN itself to grab a copy? Saving things to the Wayback Machine at the Internet Archive is a far superior solution than IPFS. IIRC, IPFS only keeps things as long as they're popular, which has famously resulted in things like most NFTs pointing to dead links.
- kkielhofner 5y agoWith IPFS this can be avoided by using IPFS pinning. It can be done on any node but usually the use of an IPFS pinning service from providers such as Pinata and/or Infura makes the most sense. It’s the best of both worlds - IPFS will still do it’s decentralized thing but with a pinning service even when individual IPFS nodes do garbage collection or otherwise expire an object locally they can always go fetch it from the persistent copy stored with the pining service.
- mukesh610 5y agoIf my understanding is correct, IPFS provides a way to only store files. Wayback Machine archives entire webpages, along with any CSS or JS (?) included. So building a webpage archival service over IPFS needs significantly more development effort.
- kkielhofner 5y agoYou're absolutely correct. I wasn't commenting on the appropriateness or suitability using IPFS for the purpose of archiving webpages. I was replying to parent that there is an IPFS solution for the expired objects/dead links issue.
- Karunamon 5y agoIIRC, wayback machine has a pretty wicked flaw in that if a domain adds a restrictive robots.txt at some point (say, a domain changes hands), then all prior archives for that domain are retroactively taken offline. Not a problem for archiving things just to keep them online under heavy load, but it's a big problem for future use. I'd prefer something like archive.is for longer-term storage.
- 1vuio0pswjnm7 5y agoWhat is a good example of a well-designed file format. It seems like file formats are often designed according to subjective opinions and tastes. Is there "universally-accepted" guidance to designing them.
- LeoPanthera 5y agoLzip, which is a better version of xz, is designed to be well designed. If you see what I mean. https://www.nongnu.org/lzip/lzip_talk_ghm_2019.html https://www.nongnu.org/lzip/lzip_talk_ghm_2019.html
- tyingq 5y agoI'd mostly settle for "documented well". Like the Sqlite file format: https://www.sqlite.org/fileformat.html https://www.sqlite.org/fileformat.html Also nice that it's platform independent. Same file works on big/small endian machines.
- nayuki 5y agoFittingly, the author of SQLite proposed an alternative to Zip files based on SQLite database files: https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html ; https://sqlite.org/sqlar/doc/trunk/README.md https://sqlite.org/sqlar/doc/trunk/README.md
- liuliu 5y agoAt this point, just don't design file format. Use flatbuffers. 1. It has good forward / backward compatibility stories; 2. Schema is your file format documentation; 3. Don't need to decode every parts; 4. Can skip for unknowns (if you use union type); 5. Many language bindings. The only downside, and this is a big one: validate a flatbuffers payload method doesn't have nearly as much language coverage as the encode / decode methods. (i.e. C++ has it, but a lot of other languages don't have validate method implemented). Case study - Apache Arrow uses flatbuffers to persist its metadata: https://github.com/apache/arrow/tree/master/format https://github.com/apache/arrow/tree/master/format
- sliken 5y ago
- deleted 5y ago[deleted]
- EvanAnderson 5y agoI was looking at the details of the ZIP format just last week when I discovered that a major performance issue was being caused by quirks of a ZIP implementation. Some knowledge of the ZIP format on the developer's part would have saved massive I/O loads (and WAN traffic). There were various "archive" ZIP files that had grown to several gigabytes each. The developer was using Info-ZIP "zip.exe" to "append" files to the archives multiple times per hour (>100). Like virtually every ZIP implementation, Info-ZIP just re-writes the entire file when appending. As a result, I was seeing multiple terabytes of I/O churn each day as new ZIP files were being created to replace the old ones (which was killing VM replication across a 500Mbps pipe, and is what drew my attention to begin with). While I would have loved to find a ZIP implementation that just appended the file and a new central directory I found it easier to just rename the ZIP files every day to minimize the churn. Still, I can't believe there isn't a ZIP implementation out there that can do this.
- bradleyjg 5y agoGzip files can just be straight concatenated together. There are some caveats around metadata but it’s an amazing property.
- yakubin 5y ago> How do you read a zip file? > This is undefined by the spec. > There are 2 obvious ways. > 1. Scan from the front, when you see an id for a record do the appropriate thing. > 2. Scan from the back, find the end-of-central-directory-record and then use it to read through the central directory, only looking at things the central directory references. I was recently bitten by this at work. I got a zip from someone and couldn't find inside the files that were supposed to be there. I asked a colleague, and they sent me a screenshot showing that the files were there, and that they didn't see the set of files that I saw. I listed the content of the zip using the "unzip -l" command. They used the engrampa GUI. At that point I looked at the hexdump of the file. What caught my eye was that I saw the zip magic number near the end of the zip, which was odd. The magic number was also present at the beginning of the file. At this point I suspected that someone used cat(1) to concatenate two zips together. I checked it with dd(1), extracting the sequence of bytes before the second occurrence of the zip magic number and the remainder into two separate files. And sure enough at that point both "unzip -l" and "engrampa" showed the same set of files, and both could show both zips correctly. Turns out engrampa was reading the file forwards, whereas unzip was reading the file backwards.
- fendy3002 5y agoThe fact that merging zip files with cat and do not produce error is amazing by itself. Now we'll wait until someone find / use a way to distribute hidden files using this method.
- cesarb 5y ago> Now we'll wait until someone find / use a way to distribute hidden files using this method. You mean like the GIFAR attack (https://en.wikipedia.org/wiki/GIFAR https://en.wikipedia.org/wiki/GIFAR)?
- deleted 5y ago[deleted]
- j16sdiz 5y agoThis is what the original SFX self-executable extractor use. Appends a zip file to a extractor exe. This is also why traditionally zip are read from backwards.
- cesarb 5y agoThere's also ISO/IEC 21320-1 (https://www.iso.org/standard/60101.html https://www.iso.org/standard/60101.html, available as a free download at https://standards.iso.org/ittf/PubliclyAvailableStandards/index.html https://standards.iso.org/ittf/PubliclyAvailableStandards/in...), which subsets APPNOTE.TXT by forbidding several less common features of the file format (like multiple volumes, encryption, or compression methods other than DEFLATE). Unfortunately, as far as I can see even that standard doesn't forbid self-extracting archives, non-empty zip file comments, or local file headers which don't match the central directory. It would be great if that standard were updated to also forbid these problematic features of the ZIP format, so that archive files created following that standard could be parsed without ambiguity. Then it would be a matter of updating other ZIP-using standards like OpenDocument to reference this simplified standard.
- zzo38computer 5y agoZIP does have bad ideas like mentioned there. I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). (If you do not need the metadata, I like the Hamster archive format, which is: It is a sequence of lumps, where each lump consists of: null-terminated ASCII filename, 32-bit PDP-endian data size, and then the data. (That is the entire specification.)) (Another thing about ZIP: If you have a truncated ZIP file (possibly due to a disk error on a floppy disk), I have found that bsdtar can read it; other programs I have tried are unable to read truncated ZIP files.)
- cesarb 5y ago> I think is better to use a separate format for archive and for compression (like .tar.gz is, and some others). That has the bad property that it does not allow for easy random access. With the ZIP format, you can directly read an arbitrary file within the archive without having to decompress or even read the compressed data of any of the other files, and many applications of the ZIP format (for example, JAR archives) depend on that property. With .tar.gz and similar, you have to decompress all data before the desired file.
- idle_zealot 5y agoThe solution here is to invert (like zip does). So rather than a .tar.gz you get a .tar full of .gzs.
- LeoPanthera 5y agoThis ruins your compression ratio, since references between files becomes impossible.
- d110af5ccf 5y agoZstd (https://facebook.github.io/zstd/ https://facebook.github.io/zstd/) provides more flexibility and supports user provided compression dictionaries - you can just place one at the front of the archive. Of course this doesn't fully solve the metadata issue and it's definitely more complex than just using the zip format. But on the plus side, it allows you to avoid using the zip format and the compression algorithm is significantly more performant and flexible.
- glennvtx 5y ago.PDF would like a word..
- raoof 5y agoapart from all negative point the positive point is that we can make .zip file act like a .iso file and .iso file act like .html file and .html act like a bash and .bat file and have a global executable format bootable.apk...zip.iso.html.bat that run on all platform I mean literally all platform.
- selfhoster11 5y agoIt's not that far-fetched for desktop platforms. This was actually implemented as the "αcτµαlly pδrταblε εxεcµταblε" [0] [0] https://justine.lol/ape.html https://justine.lol/ape.html
- raoof 5y agoI think ape currently modifies the file but my idea (that partially inspired by it) don't need to modify the executable you just double click on windows and `chmod +x` it on mac and linux and it will run, and rename the file to .apk to install on android same for ios. and we can also use webassembly to share code or even use PWA for all platform and also with a little effort we can make it bootable on all architecture.
- john_moscow 5y agoThe author is missing an obvious point that computing was way too different at the time the format was designed. You couldn't do a "find XXX | grep YYY | zip" pipeline; you would instead often add files to an archive one-by-one. So the format consists of appendable records sharing the same structure, and adding new files is simply a matter of appending records to the end. Now sometimes you would want to quickly locate and extract just one file out of the entire archive, and scanning the entire archive would be ridiculously slow on a system with <1MB/s throughput a some kilobytes of RAM. Hence, the central directory at the end of the archive. Some other time, the power would go down while you are appending to the archive, or a bad block would pop up on your HDD, hence redundancy between the central directory and the records. Also, writing archivers back then was very different from what we do now. There were no unit tests, no abstraction layers, and very limited debuggers. Software was hand-coded in assembly, and adding an extra field or a check here or there was much less trivial than it is now. You can always find something that was designed under completely different constraints and call it a bad design. But in reality it only shows that the author hasn't done enough research on the topic.
- rbobby 5y agoZip files on floppy disks!
- mrlonglong 5y agoThere were zip floppies with 120MiB which was glorious back in those days.
- PaulDavisThe1st 5y ago> You couldn't do a "find XXX | grep YYY | zip" pipeline; The format was developed in 1989, at which time find(1) and grep(1) and command lines pipes had been around for at least 10 years (more, depending on how you measure it). It wasn't that you couldn't run pipelines like this, but more that zip's creators didn't care about Unix. > Software was hand-coded in assembly I'm not sure what version of software history you've been reading, but I can assure you that I personally was not hand-coding assembly in 1989, even though I was writing device drivers, PostScript RIPs and other stuff at that time.
- edflsafoiewq 5y agoWhen scanning backwards, when you find 0x504b0506 (EOCD signature), you can also check that the length of comment field (20 bytes ahead) is correct and the offset to start of central directory (16 bytes ahead) points to the right signature to virtually eliminate the possibility of mistaking a spurious 0x504b0506 as the start of EOCD. Not sure exactly how it works for ZIP64 but it should be similar.
- pcr910303 5y agoI guess file formats also somewhat(?) suffer from the ‘Worse Is Better’[0] – a bunch of universal formats suffer from being not well-designed because it was a result of a popular program that worked at the right time… [0]: https://www.dreamsongs.com/RiseOfWorseIsBetter.html https://www.dreamsongs.com/RiseOfWorseIsBetter.html
- delhanty 5y agoAnother Zip article from early 2020 - historical but also covers quite a lot of technical ground: Zip Files: History, Explanation and Implementation Hans Wennborg https://www.hanshq.net/zip.html https://www.hanshq.net/zip.html https://news.ycombinator.com/item?id=22506451 https://news.ycombinator.com/item?id=22506451
- tinus_hn 5y agoProbably not a such a terrible design considering it was later used as the base, among others, for Java archives and all modern Office documents.
- soheil 5y agoMirror: https://arxiv.link/games.greggman.com/game/zip-rant/ https://arxiv.link/games.greggman.com/game/zip-rant/
- ramraj07 5y agoAnd Microsoft chose zip as the underlying format for all it's documents!
- superjan 5y agoGiven that apk’s, and office docs are zip files, the ambiguities might be used as an attack vector. You might be able to construct a file that looks different when it is audited versus when it is used/run, due to differences in the zip parsing.
- palsecam 5y agoTo get an idea on how to deal with ZIP files in code: — in C, minimal-style library: https://github.com/richgel999/miniz https://github.com/richgel999/miniz, look especially at https://github.com/richgel999/miniz/blob/master/miniz_zip.h https://github.com/richgel999/miniz/blob/master/miniz_zip.h — in JS, writing only and no compression: https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js https://github.com/PaulCapron/pwa2uwp/blob/master/src/zip.js (255 lines including blanks & comments)
- jart 5y agoYou can also read the redbean source code, since it's the web server that reads/writes to its own executable zip file as a native process. https://github.com/jart/cosmopolitan/blob/f3e28aa192d433c379b86df00d760d87da25a487/tool/net/redbean.c#L3246 https://github.com/jart/cosmopolitan/blob/f3e28aa192d433c379... Some ZIP APIs are designed in such a way that you can even use the POPCNT instruction to decode them.
- jopsen 5y agoAnyone know a good widely supported archive format? tar is nicely simple, if it weren't for the fact that there is so many legacy variants to support.
- tored 5y agoI only use zip because zip won. There exist other better archive formats, but what matters for me is availability of tooling, every OS has a zip implementation and every major programming language has zip library. But one good thing that RAR has is recovery records to protect against data corruption, nice feature for a long term archiving format. Not sure if other formats have this too.
- account42 5y ago> But one good thing that RAR has is recovery records to protect against data corruption, nice feature for a long term archiving format. Not sure if other formats have this too. This can be done with any format by having an additional .par2 file for the recovery information. [0] [0] https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- craigds 5y agore the streaming thing, I assume the intent is that a server can stream a zipfile trivially while generating its contents on the fly. and then write the central directory at the end. It can't easily be read while streaming, but it's a great choice for writing a stream
- hannob 5y agoI think what one needs to realize here is in what circumstances zip was born. It was ultimately a private person who founded a small company who wrote a tool for DOS and distributed it as shareware. There were other tools like it (ARJ and LHA were other popular ones). PKZIP disappeared, but its format ZIP became the de-facto standard and survived decades later, which was likely the result of some random circumstances. So it shouldn't be surprising that the format isn't necessary ideal looking decades back. (One could add that ZIP encryption is a big mess. The original ZIP encryption is insecure, there are two incompatible less insecure modes that still are vulnerable to malleability attacks and there's no way to have any modern form of encryption with an AEAD in ZIP.)
- Wildgoose 5y agoI always used LHA. I think I'm in good company seeing as Carmack used it to compress assets in both Doom and Quake.
- glenneroo 5y agoLHA was released as freeware and included some source code to help with porting, whereas PKZip was released as shareware and as the article mentions, it's hard to find actual implementation details around i.e. it was probably easier/cheaper/smarter to go with LHA. https://en.wikipedia.org/wiki/LHA_(file_format) https://en.wikipedia.org/wiki/LHA_(file_format)
- TimMeade 5y agoI think what is being forgotten here is the time the zip file format was created. It was more than 32 years ago as Phil started with arc. Also; there was no internet, no windows, no linux even. It was designed for BBS file transfer. Even in the late 80's and early 90's when it was expanded to the current format for new compression and multi-platform, some machines were big endian and others little endian and that is taken into account. Means had to be added to deal with "extended attributes". When Jean-loup Gailly and Mark Adler started infozip a lot of discussion was done to try to guess what the future would hold, but it was just that. Guessing. The fact we are still using it all is a major miracle.
- flatiron 5y agoMy thoughts exactly. Easy to sit atop an ivory tower to look down upon deflate 32 years later. No denying it’s usefulness though.
- herendin 5y agoThe author makes good points about the poor documentation and vagueness of details of the current Zip format. That is something worth criticizing. But I think criticizing the 30-year-old 'design' is pointless. So this is a good article with a bad title. What is much more helpful is suggesting, in detail, how Zip might be extended to remain backwards compatible while moving forward to adopt a more sensible future format. (Though 30 years later, some guy is going to write an article saying how terrible that new design is).
- ghaff 5y agoSmall nit. There was internet--but very few people had access to it. (And I'm guessing, don't remember, that ZIP wasn't used much or at all on internet FTP sites and that Unix-centric formats were used instead.)
- fallous 5y agoWhile I agree with your overall point regarding the importance of contextual history when looking at any design, one nit I would pick is your statement that there was no internet when Phil Katz created and PKWARE released PKZIP. First release was Feb of 1989, and I was most assuredly on the internet at that time, and in fact that is where I first encountered PKZIP. The 80s BBS scene was built around things like ARC, LHarc, and ZOO. ARC essentially got killed by its creators SEA thanks to their suing PKWARE over PKARC. Many BBS operators banned the use of ARC after that lawsuit in protest (a similar furor would erupt 5-6 years later over GIF).
- audiometry 5y agoGreggman… I swear I remember reading him blogging back in 2000? 2001? Maybe worked for Nintendo in Japan?
- ogurechny 5y agoWell, a rant about ancient format is a rant, but I guess that today a ZIP file is what a handful of well-known libraries agree upon, and it's okay — as long as you only need file archive functions, and not some data signing guarantees, file deduplication, mapping for random access, and everything else.
- dave_sid 5y agoI’m not designing a file format right now.
- MauranKilom 5y agoThe discussion here made me try opening a ZIP file that Windows considered corrupted with 7zip. And it worked! I guess that could imply that it was truncated in transfer? Edit: Extracting shows a warning regarding data after the last record. Yep, appears to have been truncated.
- nayuki 5y agoIt gets worse, because the DEFLATE compression format used in Zip has a number of unspecified behaviors: https://www.nayuki.io/page/unspecified-edge-cases-in-the-deflate-standard https://www.nayuki.io/page/unspecified-edge-cases-in-the-def...
- hinkley 5y ago> Reading an entire zip file's contents and writing out a brand new zip file could be an extremely slow process. That’s not the reason. Writing to disk and reading it back is bad, but keeping the whole file in memory is far worse and people seem to forget that routinely. If you read from the beginning, you can stream the decompression. If you are already taking care not to clobber important files on your filesystem, this isn’t much worse. The other advantage in the modern era of this format is that you can stream compression of the archive too. Need to send someone a bunch of files? You can start writing the file to the client while still compressing the data. If you’re very brave, you can start sending it while you’re still globbing the file system or pulling database records. This means you aren’t turning CPU time on the source into latency at the receiving end.
- thayne 5y agoAnd tar is a mire of slightly incompatible versions, that only supports streaming, not random access. I wish there was a modern alternative that was ubiquitous enough to be practical.
- egberts 5y agoAs an IDS/IPS architect, Zip (and many other compression) format is the narly data stream that necessitate HUGE buffering, just to get to the compressing coding table at the end of the file. I only wish that compression table was at the forefront of its file format.
- Fnoord 5y agoJust distribute some checksum and/or parity files with the zip file and you should be good to go.