4 ms·

> There's another non-trivial thing that PKZIP had going for it - it put the directory at the end, which meant you could see the list of files in the archive wi

by creshal 10y ago

> There's another non-trivial thing that PKZIP had going for it - it put the directory at the end, which meant you could see the list of files in the archive without reading the entire archive! Beginning and the end, which continues to bite us in the ass until today, where we regularly stumble over bugs when one part of a toolchain uses one entry and the rest the other.

acqq 10y ago
The directory is only at the end in the PKZIP, there is additional info before each compressed file, like in TAR. It's actually good to have both, as it allows recovering individual files even when there's a corruption in other files or in the directory.
- deleted 10y ago
  [deleted]
- dalke 10y ago
  Yes, it's possible to use the information to recover a corruption. The problem is, what happens if the two are different? This can be through accident or maliciously. If one tool uses the first and another uses the second, then you'll end up with different results, and it can be hard to figure out why they are different. It's like a CD, where audio players interpret the CD different than CD-ROM players. Some anti-copying techniques tried to take advantage of this to produce un-rippable CDs. A problem, however, was that car CD players used CD-ROM drives, so these CDs weren't playable in those cars. (https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_protection https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_prot... ) Or some HTTP header attacks based on duplicate headers. Let's say there's a firewall which allows only access requests for "Content-Type: text/plain". If the attacker includes the header twice, once as text/plain and the other as image/gif, then the firewall might only check the first while the back-end web server interprets the second. This is a silly example; I couldn't remember the real attacks which take advantage of the same mechanism.
  - acqq 10y ago
    I have news for you: you'll find the same "problem" (cache consistency) everywhere in computing. Still having the cache is a good thing. Maybe even the most important thing. "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen The directory at the end was a fantastic cache as the seek times were unacceptably high. It's still worth maintaining for a lot of use cases. If you have a different use case where its nonexistence would make your life easier, fine, but that doesn't mean nobody needs it or that it's "wrong" to have it.
    - dalke 10y ago
      I didn't argue that was right or wrong. I explained why creshal's comment about it biting them in the ass made sense. Your first argument was about the benefits of fixing data corruption, not about latency. Duplication is a poor way to handle data corruption. If you really wanted to recover data robustly, there are better methods for error correction. Yes, you are free make a new argument that there are other benefits to data duplication. Do be aware that the HTTP header duplication example I gave is not one of cache consistency, but the more general problem of data consistency.
      - acqq 10y ago
        > If you really wanted to recover data robustly, there are better methods for error correction. Please tell me which format is better for recovering most of the files from the multifile archive if the corruption occurs somewhere (e.g. a small part of the whole archive file gets corrupted): the tar.gz or the Zip? Which archive format would you use to be "better"? Is it widely used? What are its disadvantages?
        dalke 10y ago
        Sometimes the corruption is in the metadata, in which case you are lucky and the duplication can help. But neither method you mentioned lets you recover your data if the data itself is corrupted. If you really want to recover data, use something with error-correction, like Parchive. RAR 5.0's recovery record is another solution, also based on Reed-Solomon error correction codes. Archive tools usually don't use those methods because performance and compression size are the prime reasons people use an archive system, not robustness to error. In large part because the storage and network protocols are so good that such errors rarely occur. Because they use error-correction codes.

by creshal 10y ago

acqq 10y ago

The directory is only at the end in the PKZIP, there is additional info before each compressed file, like in TAR. It's actually good to have both, as it allows recovering individual files even when there's a corruption in other files or in the directory.

deleted 10y ago
[deleted]
dalke 10y ago
Yes, it's possible to use the information to recover a corruption. The problem is, what happens if the two are different? This can be through accident or maliciously. If one tool uses the first and another uses the second, then you'll end up with different results, and it can be hard to figure out why they are different. It's like a CD, where audio players interpret the CD different than CD-ROM players. Some anti-copying techniques tried to take advantage of this to produce un-rippable CDs. A problem, however, was that car CD players used CD-ROM drives, so these CDs weren't playable in those cars. (https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_protection https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_prot... ) Or some HTTP header attacks based on duplicate headers. Let's say there's a firewall which allows only access requests for "Content-Type: text/plain". If the attacker includes the header twice, once as text/plain and the other as image/gif, then the firewall might only check the first while the back-end web server interprets the second. This is a silly example; I couldn't remember the real attacks which take advantage of the same mechanism.
- acqq 10y ago
  I have news for you: you'll find the same "problem" (cache consistency) everywhere in computing. Still having the cache is a good thing. Maybe even the most important thing. "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen The directory at the end was a fantastic cache as the seek times were unacceptably high. It's still worth maintaining for a lot of use cases. If you have a different use case where its nonexistence would make your life easier, fine, but that doesn't mean nobody needs it or that it's "wrong" to have it.
  - dalke 10y ago
    I didn't argue that was right or wrong. I explained why creshal's comment about it biting them in the ass made sense. Your first argument was about the benefits of fixing data corruption, not about latency. Duplication is a poor way to handle data corruption. If you really wanted to recover data robustly, there are better methods for error correction. Yes, you are free make a new argument that there are other benefits to data duplication. Do be aware that the HTTP header duplication example I gave is not one of cache consistency, but the more general problem of data consistency.
    - acqq 10y ago
      > If you really wanted to recover data robustly, there are better methods for error correction. Please tell me which format is better for recovering most of the files from the multifile archive if the corruption occurs somewhere (e.g. a small part of the whole archive file gets corrupted): the tar.gz or the Zip? Which archive format would you use to be "better"? Is it widely used? What are its disadvantages?
      - dalke 10y ago
        Sometimes the corruption is in the metadata, in which case you are lucky and the duplication can help. But neither method you mentioned lets you recover your data if the data itself is corrupted. If you really want to recover data, use something with error-correction, like Parchive. RAR 5.0's recovery record is another solution, also based on Reed-Solomon error correction codes. Archive tools usually don't use those methods because performance and compression size are the prime reasons people use an archive system, not robustness to error. In large part because the storage and network protocols are so good that such errors rarely occur. Because they use error-correction codes.