7 ms·

Yes, it's possible to use the information to recover a corruption. The problem is, what happens if the two are different? This can be through accident or malic

by dalke 10y ago

Yes, it's possible to use the information to recover a corruption. The problem is, what happens if the two are different? This can be through accident or maliciously. If one tool uses the first and another uses the second, then you'll end up with different results, and it can be hard to figure out why they are different. It's like a CD, where audio players interpret the CD different than CD-ROM players. Some anti-copying techniques tried to take advantage of this to produce un-rippable CDs. A problem, however, was that car CD players used CD-ROM drives, so these CDs weren't playable in those cars. (https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_protection https://en.wikipedia.org/wiki/Compact_Disc_and_DVD_copy_prot... ) Or some HTTP header attacks based on duplicate headers. Let's say there's a firewall which allows only access requests for "Content-Type: text/plain". If the attacker includes the header twice, once as text/plain and the other as image/gif, then the firewall might only check the first while the back-end web server interprets the second. This is a silly example; I couldn't remember the real attacks which take advantage of the same mechanism.

acqq 10y ago
I have news for you: you'll find the same "problem" (cache consistency) everywhere in computing. Still having the cache is a good thing. Maybe even the most important thing. "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen The directory at the end was a fantastic cache as the seek times were unacceptably high. It's still worth maintaining for a lot of use cases. If you have a different use case where its nonexistence would make your life easier, fine, but that doesn't mean nobody needs it or that it's "wrong" to have it.
- dalke 10y ago
  I didn't argue that was right or wrong. I explained why creshal's comment about it biting them in the ass made sense. Your first argument was about the benefits of fixing data corruption, not about latency. Duplication is a poor way to handle data corruption. If you really wanted to recover data robustly, there are better methods for error correction. Yes, you are free make a new argument that there are other benefits to data duplication. Do be aware that the HTTP header duplication example I gave is not one of cache consistency, but the more general problem of data consistency.
  - acqq 10y ago
    > If you really wanted to recover data robustly, there are better methods for error correction. Please tell me which format is better for recovering most of the files from the multifile archive if the corruption occurs somewhere (e.g. a small part of the whole archive file gets corrupted): the tar.gz or the Zip? Which archive format would you use to be "better"? Is it widely used? What are its disadvantages?
    - dalke 10y ago
      Sometimes the corruption is in the metadata, in which case you are lucky and the duplication can help. But neither method you mentioned lets you recover your data if the data itself is corrupted. If you really want to recover data, use something with error-correction, like Parchive. RAR 5.0's recovery record is another solution, also based on Reed-Solomon error correction codes. Archive tools usually don't use those methods because performance and compression size are the prime reasons people use an archive system, not robustness to error. In large part because the storage and network protocols are so good that such errors rarely occur. Because they use error-correction codes.
      - acqq 10y ago
        > But neither method you mentioned lets you recover your data if the data itself is corrupted. I have another experience: If I've made a zip archive of a 1000 relatively same files, even if it's a few GB big, is if I'm left with a third of the archive I can easily recover first third of the files, if I have half of the archive, around the half of the files. Exactly because the common implementations of zip archivers would recognize that the directory at the end is missing but still allow me to extract everything that it can read, as they would use the data in front of every file, even with the fully missing "central directory". If the corruption affects the file 3 of a 1000 files I can extract both the first two and all 4-1000. If the corruption is in the first 0.3 part of the tar.gz, most probably I won't be able to extract anything but the files 1 and 2. All the remaining 998 are then lost. And listing the files inside of the non-corrupted ZIP archive, no matter how big, is effectively instantaneous, thanks to the same central directory. Try to list the files inside of a tar.gz which is a few GB big, then compare. ZIP is a very convenient format for fast access to the list of the files and to the single file in the archive, and still very convenient when the part of the archive is missing. Edit, this is the response to your response: I've never ever mentioned Reed-Solomon, nowhere in the discussion. I've supported in my opinion good designing decision of Zip archives having the central directory and the metadata in front of the every compressed file and I just give some real-life examples of the corrupted files I've encountered and where I've had the best recovery ratio with the ZIP format. And I've never experienced the kind of "data mismatch" myself that you refer to. You are welcome to describe it more, I'd like to read how exactly you experienced it. If the answer is that you yourself wrote a program that ignored the full format of the Zip, or used such a bad library, then there is the real problem. Once again: "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen
xori 10y ago
Garbage in, Garbage out.
creshal 10y ago
Zip inconsistency vulnerability: https://nakedsecurity.sophos.com/2013/08/09/android-master-key-vulnerability-more-malware-found-exploiting-code-verification-bypass/ https://nakedsecurity.sophos.com/2013/08/09/android-master-k... Duplicate HTTP header vulnerabilities: • https://bugzilla.mozilla.org/show_bug.cgi?id=376756 https://bugzilla.mozilla.org/show_bug.cgi?id=376756 • https://splash.riverbed.com/thread/7772 https://splash.riverbed.com/thread/7772 If a spec is open for interpretation, you can be sure that some software gets it wrong.
- acqq 10y ago
  The "zip inconsistency" linked is not a problem of the Zip file format specification. The goal of the zip format was never to provide a tamper-proof cryptographically secure binary package. Whoever uses this structure for some more security-wise complex demands is responsible to make sure that the security assumptions he needs are respected by the code he implements. One simple method were signing the resulting archive, then no modification is possible, and therefore also isn't possible adding a second entry with the same name by the attacker. I really don't care about HTTP headers or Reed-Salomon, as they are completely irrelevant to the Zip format designed in eighties for very specific purposes then (being fast on the floppy disk and 4 MHz 8088 processor, allowing to recover as much as possible from the floppy even after the floppy disk sector failure), I just claim, through the whole conversation here, that the Zip format is not a bad format for having a central directory and the metadata outside of it too, and I gave my arguments for that. NTFS also has some metadata on more than one place and it's an intentional design decision solving real problems: in Zip case allowing for fast access of the list of the files in the archive but also easy recovery of the single files when some part of the archive gets corrupted. I also claim that the Zip format is not in any way "bad" for not making impossible per the archive format design and specification of having the two identical file names in the archive. I can even imagine use cases where exactly allowing that is/was beneficial.

by dalke 10y ago

acqq 10y ago

I have news for you: you'll find the same "problem" (cache consistency) everywhere in computing. Still having the cache is a good thing. Maybe even the most important thing. "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen The directory at the end was a fantastic cache as the seek times were unacceptably high. It's still worth maintaining for a lot of use cases. If you have a different use case where its nonexistence would make your life easier, fine, but that doesn't mean nobody needs it or that it's "wrong" to have it.

dalke 10y ago
I didn't argue that was right or wrong. I explained why creshal's comment about it biting them in the ass made sense. Your first argument was about the benefits of fixing data corruption, not about latency. Duplication is a poor way to handle data corruption. If you really wanted to recover data robustly, there are better methods for error correction. Yes, you are free make a new argument that there are other benefits to data duplication. Do be aware that the HTTP header duplication example I gave is not one of cache consistency, but the more general problem of data consistency.
- acqq 10y ago
  > If you really wanted to recover data robustly, there are better methods for error correction. Please tell me which format is better for recovering most of the files from the multifile archive if the corruption occurs somewhere (e.g. a small part of the whole archive file gets corrupted): the tar.gz or the Zip? Which archive format would you use to be "better"? Is it widely used? What are its disadvantages?
  - dalke 10y ago
    Sometimes the corruption is in the metadata, in which case you are lucky and the duplication can help. But neither method you mentioned lets you recover your data if the data itself is corrupted. If you really want to recover data, use something with error-correction, like Parchive. RAR 5.0's recovery record is another solution, also based on Reed-Solomon error correction codes. Archive tools usually don't use those methods because performance and compression size are the prime reasons people use an archive system, not robustness to error. In large part because the storage and network protocols are so good that such errors rarely occur. Because they use error-correction codes.
    - acqq 10y ago
      > But neither method you mentioned lets you recover your data if the data itself is corrupted. I have another experience: If I've made a zip archive of a 1000 relatively same files, even if it's a few GB big, is if I'm left with a third of the archive I can easily recover first third of the files, if I have half of the archive, around the half of the files. Exactly because the common implementations of zip archivers would recognize that the directory at the end is missing but still allow me to extract everything that it can read, as they would use the data in front of every file, even with the fully missing "central directory". If the corruption affects the file 3 of a 1000 files I can extract both the first two and all 4-1000. If the corruption is in the first 0.3 part of the tar.gz, most probably I won't be able to extract anything but the files 1 and 2. All the remaining 998 are then lost. And listing the files inside of the non-corrupted ZIP archive, no matter how big, is effectively instantaneous, thanks to the same central directory. Try to list the files inside of a tar.gz which is a few GB big, then compare. ZIP is a very convenient format for fast access to the list of the files and to the single file in the archive, and still very convenient when the part of the archive is missing. Edit, this is the response to your response: I've never ever mentioned Reed-Solomon, nowhere in the discussion. I've supported in my opinion good designing decision of Zip archives having the central directory and the metadata in front of the every compressed file and I just give some real-life examples of the corrupted files I've encountered and where I've had the best recovery ratio with the ZIP format. And I've never experienced the kind of "data mismatch" myself that you refer to. You are welcome to describe it more, I'd like to read how exactly you experienced it. If the answer is that you yourself wrote a program that ignored the full format of the Zip, or used such a bad library, then there is the real problem. Once again: "Almost all programming can be viewed as an exercise in caching." —Terje Mathisen

xori 10y ago

Garbage in, Garbage out.

creshal 10y ago

Zip inconsistency vulnerability: https://nakedsecurity.sophos.com/2013/08/09/android-master-key-vulnerability-more-malware-found-exploiting-code-verification-bypass/ https://nakedsecurity.sophos.com/2013/08/09/android-master-k... Duplicate HTTP header vulnerabilities: • https://bugzilla.mozilla.org/show_bug.cgi?id=376756 https://bugzilla.mozilla.org/show_bug.cgi?id=376756 • https://splash.riverbed.com/thread/7772 https://splash.riverbed.com/thread/7772 If a spec is open for interpretation, you can be sure that some software gets it wrong.

acqq 10y ago
The "zip inconsistency" linked is not a problem of the Zip file format specification. The goal of the zip format was never to provide a tamper-proof cryptographically secure binary package. Whoever uses this structure for some more security-wise complex demands is responsible to make sure that the security assumptions he needs are respected by the code he implements. One simple method were signing the resulting archive, then no modification is possible, and therefore also isn't possible adding a second entry with the same name by the attacker. I really don't care about HTTP headers or Reed-Salomon, as they are completely irrelevant to the Zip format designed in eighties for very specific purposes then (being fast on the floppy disk and 4 MHz 8088 processor, allowing to recover as much as possible from the floppy even after the floppy disk sector failure), I just claim, through the whole conversation here, that the Zip format is not a bad format for having a central directory and the metadata outside of it too, and I gave my arguments for that. NTFS also has some metadata on more than one place and it's an intentional design decision solving real problems: in Zip case allowing for fast access of the list of the files in the archive but also easy recovery of the single files when some part of the archive gets corrupted. I also claim that the Zip format is not in any way "bad" for not making impossible per the archive format design and specification of having the two identical file names in the archive. I can even imagine use cases where exactly allowing that is/was beneficial.