6 ms·
From a security perspective, and as a programmer, I've never liked ZIP files precisely because there are two mechanisms to identify the contents, the per-file h
by wahern 2y ago
From a security perspective, and as a programmer, I've never liked ZIP files precisely because there are two mechanisms to identify the contents, the per-file header and the central directory. When you're defining a format, protocol, or w'ever, ideally there should be a single source of truth, a single valid & useable parse, etc; basically, the structure of the data or process should be intrinsically constraining. There shouldn't be a pathway for multiple implementations to produce different functional results, and ZIP archives are in my mind the archetype for getting this wrong. tar files aren't ideal, but in the abstract (ignoring issues with long file names) they don't have this problem. (tar files don't support random access, either, but better to rely on something suboptimal than something that's fundamentally broken.)
A similar security problem, though not as fundamentally baked into the format, is MIME parsing. The header section is supposed to be delimited from the body by an empty line (likewise for nested entities). But what if it's not? For better or worse, Sendmail was tolerant of the absence of an empty line and treated as headers everything up to the first line that didn't parse as a header or header continuation.[1] Later systems, like Postfix, originally copied this behavior. But Microsoft Exchange and Outlook are even more tolerant, yet in a much more horrendous way, by parsing as a header anything that looks like a Content-Type or related header immediately after the first empty line. They have similar hacks for other, similar violations. So today, depending on the receiving software, you can send messages that appear differently, including having different attachments. It's a security nightmare!
I not a Postel's Law hater, but ZIP archives and Microsoft's MIME parsing behaviors are just egregiously wrong and indefensible. And even if you think the Robustness Principle is inherently bad policy, you still have to design your formats, protocols, and systems to be as intrinsically constraining as possible. You can't rely on vendors adhering to a MUST rule in an RFC, unless it's unquestioningly crystal clear what the repercussions will be--everybody else will (because it's the natural and convenient thing to do) reject your output as trash and drop it on the floor immediately so violations never have a chance to get a foothold.
[1] MTAs don't necessarily need to care about MIME parsing, but Sendmail eventually gained features where parsing message contents mattered, setting the de facto norm (for those paying attention) until Microsoft came along.
- failbuffer 2y agoIn a similar vein, HTTP header smuggling attacks exploit differences in header parsing. For instance, a reverse proxy and a web server might handle repetition of headers or the presence of whitespace differently.
- kevin_thibedeau 2y agoThe central directory allows zip archives to be split across multiple files on separate media without needing to read them all in for selective extraction. Not particularly useful today but invaluable in the sneakernet era with floppies.
- thrdbndndn 2y agoStill useful today. Try to transmit a 100G file through any service is usually a pain especially if one end has non-stable Internet.
- SubiculumCode 2y agoI recently had to do this with about 700Gb, and yeah OneDrive hated that. I ended up concatenating tars together.
- rakoo 2y agoThat's a very bad way of solving that issue. If transmission is a problem, either use a proper retry-friendly protocol (such as bittorrent) or split the file. Using hacks on the data format just leads to additional pain
- meehai 2y agocouldn't agree more! We need to separate and design modules as unitary as possible: - zip should ARCHIVE/COMPRESS, i.e. reduce the file size and create a single file from the file system point of view. The complexity should go in the compression algorithm. - Sharding/sending multiple coherent pieces of the same file (zip or not) is a different module and should be handled by specialized and agnostic protocols that do this like the ones you mentioned. People are always doing tools that handle 2 or more use cases instead of following the UNIX principle to create generic and good single respectability tools that can be combined together (thus allowing a 'whitelist' of combinations which is safe). Quite frankly it's annoying and very often leads to issues such as this that weren't even thought in the original design because of the exponential problem of combining tools together.