4 ms·

It is not a schizophrenic zip file to have inline headers that are not referenced in the TOC. the TOC is the only source of truth in a zip file. It was designed

It is not a schizophrenic zip file to have inline headers that are not referenced in the TOC. the TOC is the only source of truth in a zip file. It was designed this way in purpose so that you can add new versions of files on your 20 disk zip without having to re-write all 20 disks. Pkzip would read the TOC from disk 20, append your new file to disk 20 (or 21 if there was not enough space) and then write a new TOC at the end that does not reference the old file still in the zip. That is by design. Reading anything other than the files in the TOC is an invalid zip reader

masklinn 1y ago
While that is true (even if some people really want to stream read zips and will thus trigger tar-style conflicts) that is not what the article is about. The article is really more about software using the "size of central directory" field to find the central directory (a sin Python's standard library apparently commits) versus the "offset of central directory". A commenter on an other post of that article noted that technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect. In fact zip was designed such that the central directory could be split across multiple disks (and later files), so it was not possible to guarantee a simple backwards jump from the EOCD to the start of CD.
- cxr 1y ago
  > technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect Yes, and in fact APPNOTE.TXT expects implementations to deal with discontiguous CD/EOCD records; it actually mandates that some records appear between them. (It's where ZIP64-specific records go, for example.) The whole thing seems to stem from a belief by the author of this post that Info-ZIP is the canonical implementation. (This is of course wrong. That would be PKZIP.) If Info-ZIP is exhibiting the behavior described, then the Info-ZIP implementation is flat out wrong. (This has happened before. See <https://news.ycombinator.com/item?id=27925393 https://news.ycombinator.com/item?id=27925393> where the author of that post provides pushback to Mark Adler, receives pushback from Adler in response, and then just stops pushing and defers to Adler's reading. Adler is an expert in his domain, and he's made valuable contributions to free software, but his expertise is in compression, and he's not the authority on the spec or the format; Adler didn't create ZIP or write APPNOTE.TXT—Phil Katz did.)
  - gynvael 1y ago
    Author here :) > a belief by the author of this post that Info-ZIP is the canonical implementation ↑ Where did you get this from? I don't think nowadays there's such a thing as a canonical implementation — ZIP implementation world is too fragmented and implementations are too widespread. One more note is that the article isn't about who's right or wrong in terms of interpreting APPNOTE.TXT — this is besides the point. The point there is that a parser discrepancy can cause security issues, and the article documents just another discrepancy found in the real world implementations (and there are A LOT of these with regards to the ZIP format).
    - cxr 1y ago
      > this makes sense—technically both the offset of the Central Directory and the size of the Central Directory are redundant when you consider that the End of Central Directory Record should be, well, at the end of the Central Directory. In a proper ZIP file the following equation should be true: (position_of_EoCDR - size_of_CD == offset_of_CD) This is a pretty clear statement on normativity. You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
      - gynvael 1y ago
        > This is a pretty clear statement on normativity. I disagree – it's a technical remark on redundancy of fields (this is related to my hypothesis in the article that redundancy in formats is the primary cause of parser discrepancy). It doesn't acknowledge InfoZIP being canonical in any way. > You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".) Fair, though my point still stands.
iforgotpassword 1y ago
This is not what the article is talking about. It's talking about how the EOCD contains both the size of the central directory and the offset of the start of the central directory, which is redundant. So we end up with some tools honoring the offset, while some subtract the size from the EOCD.
- the_arun 1y ago
  And how bad actors could take advantage of that.
  - m3kw9 1y ago
    I can think they would test which parser you have using some social engineering and create the zip appropriately, but they would also need to know the accountings parser. So I think they would just do the first step and leave it up to chance.
- cxr 1y ago
  It's true that the other comment is unrelated to the content of the article, but it's not true that the offset and size in the EOCD are redundant (for the reasons given in the earlier sibling comment).

masklinn 1y ago

While that is true (even if some people really want to stream read zips and will thus trigger tar-style conflicts) that is not what the article is about. The article is really more about software using the "size of central directory" field to find the central directory (a sin Python's standard library apparently commits) versus the "offset of central directory". A commenter on an other post of that article noted that technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect. In fact zip was designed such that the central directory could be split across multiple disks (and later files), so it was not possible to guarantee a simple backwards jump from the EOCD to the start of CD.

cxr 1y ago
> technically there's no reason for the central directory to be right before the EOCD so seeking backwards from the EOCD by the size of CD is just incorrect Yes, and in fact APPNOTE.TXT expects implementations to deal with discontiguous CD/EOCD records; it actually mandates that some records appear between them. (It's where ZIP64-specific records go, for example.) The whole thing seems to stem from a belief by the author of this post that Info-ZIP is the canonical implementation. (This is of course wrong. That would be PKZIP.) If Info-ZIP is exhibiting the behavior described, then the Info-ZIP implementation is flat out wrong. (This has happened before. See <https://news.ycombinator.com/item?id=27925393 https://news.ycombinator.com/item?id=27925393> where the author of that post provides pushback to Mark Adler, receives pushback from Adler in response, and then just stops pushing and defers to Adler's reading. Adler is an expert in his domain, and he's made valuable contributions to free software, but his expertise is in compression, and he's not the authority on the spec or the format; Adler didn't create ZIP or write APPNOTE.TXT—Phil Katz did.)
- gynvael 1y ago
  Author here :) > a belief by the author of this post that Info-ZIP is the canonical implementation ↑ Where did you get this from? I don't think nowadays there's such a thing as a canonical implementation — ZIP implementation world is too fragmented and implementations are too widespread. One more note is that the article isn't about who's right or wrong in terms of interpreting APPNOTE.TXT — this is besides the point. The point there is that a parser discrepancy can cause security issues, and the article documents just another discrepancy found in the real world implementations (and there are A LOT of these with regards to the ZIP format).
  - cxr 1y ago
    > this makes sense—technically both the offset of the Central Directory and the size of the Central Directory are redundant when you consider that the End of Central Directory Record should be, well, at the end of the Central Directory. In a proper ZIP file the following equation should be true: (position_of_EoCDR - size_of_CD == offset_of_CD) This is a pretty clear statement on normativity. You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".)
    - gynvael 1y ago
      > This is a pretty clear statement on normativity. I disagree – it's a technical remark on redundancy of fields (this is related to my hypothesis in the article that redundancy in formats is the primary cause of parser discrepancy). It doesn't acknowledge InfoZIP being canonical in any way. > You also left out an important part of my comment when you quoted it. (The operative word at the front of the sentence is "seems".) Fair, though my point still stands.

iforgotpassword 1y ago

This is not what the article is talking about. It's talking about how the EOCD contains both the size of the central directory and the offset of the start of the central directory, which is redundant. So we end up with some tools honoring the offset, while some subtract the size from the EOCD.

the_arun 1y ago
And how bad actors could take advantage of that.
- m3kw9 1y ago
  I can think they would test which parser you have using some social engineering and create the zip appropriately, but they would also need to know the accountings parser. So I think they would just do the first step and leave it up to chance.
cxr 1y ago
It's true that the other comment is unrelated to the content of the article, but it's not true that the offset and size in the EOCD are redundant (for the reasons given in the earlier sibling comment).