3 ms·

WinRAR does it right, 7zip and Windows Explorer do it wrong according to https://en.m.wikipedia.org/wiki/ZIP_(file_format) https://en.m.wikipedia.org/wiki/ZIP_(

by nh2 2y ago

WinRAR does it right, 7zip and Windows Explorer do it wrong according to https://en.m.wikipedia.org/wiki/ZIP_(file_format) https://en.m.wikipedia.org/wiki/ZIP_(file_format) > only files specified in the central directory at the end of the file are valid. Scanning a ZIP file for local file headers is invalid (except in the case of corrupted archives)

leni536 2y ago
This contradicts the specification, which explicitly supports stream-processing zip files, which necessarily can't happen if your source of truth is the central directory record. Unless if you can wrap your stream processing in some kind of transaction that you can drop once you discover foul play.
- nh2 2y ago
  Source what you're referring to / explanation? In ZIP, later info wins. I don't see how that isn't always streamable.
  - leni536 2y ago
    Hmm, it appears that you are right. I vaguely remembered that zip was streamable, but it appears that it only means that it's stream writable, as in you can write zip file contents from a datastream of unknown size, and append the checksum and file size later in the zip file stream. However such a zip file is definitely not stream readable, as the local file header no longer contains the size of the following file data, so you can't know where to end a given file. So for reading you definitely have to locate the central directory record. In my defense the spec says[1]. > 4.3.2 Each file placed into a ZIP file MUST be preceded by a "local file header" record for that file. Each "local file header" MUST be accompanied by a corresponding "central directory header" record within the central directory section of the ZIP file. Then in 4.3.6 it describes the file format, which seems to be fundamentally incompatible to altering zip files by appending data, as the resulting file would not conform to this format. So basically some implementations (maybe opportunistically, relying on compressed sizes being available in the local file headers) stream read from the zip file, assuming that it's a valid zip file, but not validating. Some other implementations only use the central directory record at the end, but don't validate the file format either. A validating zip file parser should be possible, by locating the central directory record and checking that the referred files with their metadata fully cover the file contents of the zip files, without gaps and overlaps. But this probably won't win any benchmarks. [1] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
    - nh2 2y ago
      Good observations! Indeed: ZIP files are stream-writable, and some ZIP files are stream-readable, but not both: ZIP files that were stream-written are not steam-readable. Also, steaming unzipping always requires that by the time you arrive at the central directory, you delete so-far-unzipped files that don't have entries in it, as those were "deleted". > Then in 4.3.6 it describes the file format, which seems to be fundamentally incompatible to altering zip files by appending data, as the resulting file would not conform to this format. My interpretation of the spec and specifically of 4.3.6 is that it is informational for how ZIP files usually look, and that you may store arbitrary data in between files; such data then doesn't count as "files". This reading then does allow appending and concatenation. Unfortunately 4.3.6 does not have MUST/MAY wording so we don't really know if this reading was intended by the authors (maybe they clarify it in the future), but allowing this reading seems rather useful because it permits append-only modification of existing ZIP files. (The wording /'A ZIP file MUST have only one "end of central directory record"'/ suggests somehow that the authors didn't intend this, but again one could argue that this is not necessarily to state, that there is only one EOCDR by definition, and that any previous ones are just garbage data that is to be ignored.)
      - cxr 2y ago
        > Unfortunately 4.3.6 does not have MUST/MAY wording so we don't really know if this reading was intended by the authors It's not explicit from the APPNOTE, but that's not the same as saying "we don't really know". We do know—islands of opaque data are allowed and that's the entire reason the format is designed the way it is. Katz designed ZIP for use on machines with floppy drives, and append-only modification of archives spanning multiple media was therefore baked into the scheme from the very roots. He also an produced an implementation that we happen to be able to check, and we know it works that way. The only way to arrive at another interpretation is to look at APPNOTE in isolation and devoid of context. > one could argue that this is not necessarily to state, that there is only one EOCDR by definition, and that any previous ones are just garbage data that is to be ignored That is the correct interpretation; the wording doesn't suggest otherwise.
cxr 2y ago
The article says that WinRAR "displays both ZIP structures", so, no, it doesn't do it right. Of the three, only Windows Explorer is close to the correct behavior (showing the contents of the ZIP that was appended closest to the end, and ignoring everything else). The exception to its correctness is that they report it may fail to process the ZIP and identify it as being corrupt, which shouldn't happen so long as the endmost ZIP is well-formed.
- nh2 2y ago
  > The article says that WinRAR "displays both ZIP structures", so, no, it doesn't do it right. That would be true, but the bleepingcomputer article seems to be misquoting that fact. The original research they reference (which I also find to be a more sensible article), https://perception-point.io/blog/evasive-concatenated-zip-trojan-targets-windows-users/ https://perception-point.io/blog/evasive-concatenated-zip-tr... says that WinRAR only shows they last archive contents, and shows a screenshot of that: > WinRAR, on the other hand, reads the second central directory and displays the contents of the second archive
  - cxr 2y ago
    Thanks, that's a much better article (and published on a site that, while slightly annoying on its own, is infested with fewer rudely intrusive and resource-consuming ads). Reading both, it's clear that (a) you are correct, and (b) the submitted link, besides being materially inaccurate, is shameless reblog spam and should be changed.

by nh2 2y ago

leni536 2y ago

This contradicts the specification, which explicitly supports stream-processing zip files, which necessarily can't happen if your source of truth is the central directory record. Unless if you can wrap your stream processing in some kind of transaction that you can drop once you discover foul play.

nh2 2y ago
Source what you're referring to / explanation? In ZIP, later info wins. I don't see how that isn't always streamable.
- leni536 2y ago
  Hmm, it appears that you are right. I vaguely remembered that zip was streamable, but it appears that it only means that it's stream writable, as in you can write zip file contents from a datastream of unknown size, and append the checksum and file size later in the zip file stream. However such a zip file is definitely not stream readable, as the local file header no longer contains the size of the following file data, so you can't know where to end a given file. So for reading you definitely have to locate the central directory record. In my defense the spec says[1]. > 4.3.2 Each file placed into a ZIP file MUST be preceded by a "local file header" record for that file. Each "local file header" MUST be accompanied by a corresponding "central directory header" record within the central directory section of the ZIP file. Then in 4.3.6 it describes the file format, which seems to be fundamentally incompatible to altering zip files by appending data, as the resulting file would not conform to this format. So basically some implementations (maybe opportunistically, relying on compressed sizes being available in the local file headers) stream read from the zip file, assuming that it's a valid zip file, but not validating. Some other implementations only use the central directory record at the end, but don't validate the file format either. A validating zip file parser should be possible, by locating the central directory record and checking that the referred files with their metadata fully cover the file contents of the zip files, without gaps and overlaps. But this probably won't win any benchmarks. [1] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
  - nh2 2y ago
    Good observations! Indeed: ZIP files are stream-writable, and some ZIP files are stream-readable, but not both: ZIP files that were stream-written are not steam-readable. Also, steaming unzipping always requires that by the time you arrive at the central directory, you delete so-far-unzipped files that don't have entries in it, as those were "deleted". > Then in 4.3.6 it describes the file format, which seems to be fundamentally incompatible to altering zip files by appending data, as the resulting file would not conform to this format. My interpretation of the spec and specifically of 4.3.6 is that it is informational for how ZIP files usually look, and that you may store arbitrary data in between files; such data then doesn't count as "files". This reading then does allow appending and concatenation. Unfortunately 4.3.6 does not have MUST/MAY wording so we don't really know if this reading was intended by the authors (maybe they clarify it in the future), but allowing this reading seems rather useful because it permits append-only modification of existing ZIP files. (The wording /'A ZIP file MUST have only one "end of central directory record"'/ suggests somehow that the authors didn't intend this, but again one could argue that this is not necessarily to state, that there is only one EOCDR by definition, and that any previous ones are just garbage data that is to be ignored.)
    - cxr 2y ago
      > Unfortunately 4.3.6 does not have MUST/MAY wording so we don't really know if this reading was intended by the authors It's not explicit from the APPNOTE, but that's not the same as saying "we don't really know". We do know—islands of opaque data are allowed and that's the entire reason the format is designed the way it is. Katz designed ZIP for use on machines with floppy drives, and append-only modification of archives spanning multiple media was therefore baked into the scheme from the very roots. He also an produced an implementation that we happen to be able to check, and we know it works that way. The only way to arrive at another interpretation is to look at APPNOTE in isolation and devoid of context. > one could argue that this is not necessarily to state, that there is only one EOCDR by definition, and that any previous ones are just garbage data that is to be ignored That is the correct interpretation; the wording doesn't suggest otherwise.

cxr 2y ago

The article says that WinRAR "displays both ZIP structures", so, no, it doesn't do it right. Of the three, only Windows Explorer is close to the correct behavior (showing the contents of the ZIP that was appended closest to the end, and ignoring everything else). The exception to its correctness is that they report it may fail to process the ZIP and identify it as being corrupt, which shouldn't happen so long as the endmost ZIP is well-formed.

nh2 2y ago
> The article says that WinRAR "displays both ZIP structures", so, no, it doesn't do it right. That would be true, but the bleepingcomputer article seems to be misquoting that fact. The original research they reference (which I also find to be a more sensible article), https://perception-point.io/blog/evasive-concatenated-zip-trojan-targets-windows-users/ https://perception-point.io/blog/evasive-concatenated-zip-tr... says that WinRAR only shows they last archive contents, and shows a screenshot of that: > WinRAR, on the other hand, reads the second central directory and displays the contents of the second archive
- cxr 2y ago
  Thanks, that's a much better article (and published on a site that, while slightly annoying on its own, is infested with fewer rudely intrusive and resource-consuming ads). Reading both, it's clear that (a) you are correct, and (b) the submitted link, besides being materially inaccurate, is shameless reblog spam and should be changed.