3 ms·

And as someone else has probably said, if the second file is a zip file.. the file will open successfully in most zip programs without any modification.

13of40 4y ago
That doesn't make sense to me. Zip was designed so it could be written to a non-seekable stream, meaning the Central Directory structure that points to all of the file records is at the end of the archive. Its location is found at the very end of the archive in the End Of Central Directory record. Problem is that it's a pointer to the offset of the Central Directory relative to the beginning of the archive, and so are the pointers to the file records in the directory. So, if you just append a ZIP file to another file, it will screw up all the internal pointers in the ZIP file. If you overlay a file on the beginning of the ZIP, most of it should be recoverable, though. Another clever hack is that you can put a batch file in the beginning of the ZIP with the compression level set to "none", rename it to .CMD or .BAT, and run it like a batch file. If said batch file locates and unzips itself, you've got a self-extracting archive.
- quasarj 4y ago
  Yeah, I don't quite know the details of how it works. Just that it does $ file some.* some.jpg: JPEG image data, JFIF standard 1.02, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 64x736, components 1 some.zip: Zip archive data, at least v1.0 to extract $ cat some.* > new.jpg $ file new.jpg new.jpg: JPEG image data, JFIF standard 1.02, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 64x736, components 1 $ unzip -l new.jpg Archive: new.jpg warning [new.jpg]: 1557 extra bytes at beginning or within zipfile (attempting to process anyway) Length Date Time Name --------- ---------- ----- ---- 0 2022-04-28 21:53 stuff/ 5 2022-04-28 21:53 stuff/a 5 2022-04-28 21:53 stuff/c 5 2022-04-28 21:53 stuff/b --------- ------- 15 4 files $
- shakna 4y ago
  It works because most zip tools today will ignore all bytes before zip's magic number sequence, because concatenating a zip onto the end of something else has been common since DOS days. So almost every tool can find the correct offset - it's easy, as you pointed out. Now, it can break if your "other" file happens to include the magic bytes. But that's rare enough that you only have to be aware that it can happen.
  - FartyMcFarter 4y ago
    The magic bytes sequence in zip files is 4 bytes long, one can probably use this to compute the probability that a file of size N randomly contains these bytes.
    - jstanley 4y ago
      A file of N bytes has N-3 different places for a 4-byte sequence to start. A 4-byte sequence has 2^32 possible values. For the file not to contain the zip magic, all N-3 positions would have to have one of the 2^32-1 non-zip magic values. The probability that it doesn't contain a zip magic is then P(nonzip) = ((2^32-1)/(2^32)) ^ (N-3) And the probability that it does have a zip magic is P(zip) = 1-P(nonzip). A 1 gigabyte file has about a 20% chance to contain a zip header. A 10 gigabyte file has about a 90% chance. (Maybe?)
      - shakna 4y ago
        The chances are a little more likely, as there are two different 4-byte sequences that can identify a zip file. (Empty and non-empty).
      - frumiousirc 4y ago
        I was curious and yes, I reproduce your numbers. A robust zip decoder can beat those odds. It can bail on a nominal start position by considering a bit more than the local file header signature magic word. The next two bytes is a minimum version needed to extract and its legal values are sparse. A bit further comes 2 bytes giving the compression method which also has sparse values. I'd guess sparse values (small integers) are more likely to be found in a "random" non-zip stream but would still reduce the collision probability. Then there are CRCs. These can greatly reduce the collision probability at the cost of the decoder doing speculative work which gets thrown away if a CRC fails to replicate.

13of40 4y ago

That doesn't make sense to me. Zip was designed so it could be written to a non-seekable stream, meaning the Central Directory structure that points to all of the file records is at the end of the archive. Its location is found at the very end of the archive in the End Of Central Directory record. Problem is that it's a pointer to the offset of the Central Directory relative to the beginning of the archive, and so are the pointers to the file records in the directory. So, if you just append a ZIP file to another file, it will screw up all the internal pointers in the ZIP file. If you overlay a file on the beginning of the ZIP, most of it should be recoverable, though. Another clever hack is that you can put a batch file in the beginning of the ZIP with the compression level set to "none", rename it to .CMD or .BAT, and run it like a batch file. If said batch file locates and unzips itself, you've got a self-extracting archive.

quasarj 4y ago
Yeah, I don't quite know the details of how it works. Just that it does $ file some.* some.jpg: JPEG image data, JFIF standard 1.02, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 64x736, components 1 some.zip: Zip archive data, at least v1.0 to extract $ cat some.* > new.jpg $ file new.jpg new.jpg: JPEG image data, JFIF standard 1.02, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 64x736, components 1 $ unzip -l new.jpg Archive: new.jpg warning [new.jpg]: 1557 extra bytes at beginning or within zipfile (attempting to process anyway) Length Date Time Name --------- ---------- ----- ---- 0 2022-04-28 21:53 stuff/ 5 2022-04-28 21:53 stuff/a 5 2022-04-28 21:53 stuff/c 5 2022-04-28 21:53 stuff/b --------- ------- 15 4 files $
shakna 4y ago
It works because most zip tools today will ignore all bytes before zip's magic number sequence, because concatenating a zip onto the end of something else has been common since DOS days. So almost every tool can find the correct offset - it's easy, as you pointed out. Now, it can break if your "other" file happens to include the magic bytes. But that's rare enough that you only have to be aware that it can happen.
- FartyMcFarter 4y ago
  The magic bytes sequence in zip files is 4 bytes long, one can probably use this to compute the probability that a file of size N randomly contains these bytes.
  - jstanley 4y ago
    A file of N bytes has N-3 different places for a 4-byte sequence to start. A 4-byte sequence has 2^32 possible values. For the file not to contain the zip magic, all N-3 positions would have to have one of the 2^32-1 non-zip magic values. The probability that it doesn't contain a zip magic is then P(nonzip) = ((2^32-1)/(2^32)) ^ (N-3) And the probability that it does have a zip magic is P(zip) = 1-P(nonzip). A 1 gigabyte file has about a 20% chance to contain a zip header. A 10 gigabyte file has about a 90% chance. (Maybe?)
    - shakna 4y ago
      The chances are a little more likely, as there are two different 4-byte sequences that can identify a zip file. (Empty and non-empty).
    - frumiousirc 4y ago
      I was curious and yes, I reproduce your numbers. A robust zip decoder can beat those odds. It can bail on a nominal start position by considering a bit more than the local file header signature magic word. The next two bytes is a minimum version needed to extract and its legal values are sparse. A bit further comes 2 bytes giving the compression method which also has sparse values. I'd guess sparse values (small integers) are more likely to be found in a "random" non-zip stream but would still reduce the collision probability. Then there are CRCs. These can greatly reduce the collision probability at the cost of the decoder doing speculative work which gets thrown away if a CRC fails to replicate.