5 ms·

JPEG of Shakespeare is also a zip file containing his complete works (2018)

182 points by metadat 4y ago

MaxikCZ 4y ago
https://news.ycombinator.com/item?id=18342042 https://news.ycombinator.com/item?id=18342042 discussion from 2018
cowtools 4y ago
I am surprised that it survives twitter's image compression. I'm not familiar with JPEG format, but I'm familiar with PNG. I guess it's using some zlib-compressed EXIF-like metadata field, seeing as ZIP also uses DEFLATE/zlib/whatever?
- Retr0id 4y ago
  It's embedded within an ICC colour profile, and therefore doesn't get touched by any image compression/resizing logic.
  - Y-bar 4y ago
    Some eager image compression techniques will strip metadata including profiles (e. g. ImageOptim: https://imageoptim.com/color-profiles.html https://imageoptim.com/color-profiles.html) But generally you are correct.
    - Retr0id 4y ago
      any image compression/resizing logic [in twitter.com] [in 2018]
  - csdvrx 4y ago
    How would do that nicely and automatically, to do funny things like that in a way that would be resistant to the accidental optimizations/resizing that will be applied to the JPG?
- charonn0 4y ago
  It's a polyglot file. JPEG decoders will ignore data that comes after the end-of-data marker, while zip decoders ignore data that comes before the first archive entry. Essentially, they ran `cat pic.jpg arch.zip > output.jpg`.
Retr0id 4y ago
In the years since I posted that, twitter has changed some of their image processing pipeline, so that precise technique no longer works if you want to upload new files (last time I checked - and I didn't investigate further or look for workarounds). However, it's much easier to embed data into PNGs, and the best twitter-compatible implementation of this I'm aware of can be found here: https://github.com/CleasbyCode/pdvzip https://github.com/CleasbyCode/pdvzip Edit: also, the standard `unzip` utility has become a bit stricter about the validity of input files - in ways which would have been possible to work around, but of course not retroactively.
- vibragiel 4y ago
  I had to run it like this UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip DqteCf6WsAAhqwV.jpg
siraben 4y ago
Using binwalk, you can extract the zip file which in turn contains various rar files which when uncompressed give a 6.8M .html file containing the works of Shakespeare. $ nix run nixpkgs#binwalk DqteCf6WsAAhqwV.jpg DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 JPEG image data, JFIF standard 1.01 182 0xB6 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part001.rar 65575 0x10027 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part002.rar 131112 0x20028 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part003.rar ...
- tyingq 4y ago
  You shouldn't really need binwalk given how zipfiles work with the directory at the end of the file. You should just be able to unzip the file as-is. Though I suppose that could vary based on what you're unzipping it with. They should support it, though, as that's how a self-extracting zip is supported...ignoring any extra bytes at the top. The zip on Linux has "zip -A file.zip" functionality that will strip the non-zip preamble too.
tyingq 4y ago
Ah, I had played around with this the first time the story came out: $ curl -s https://pbs.twimg.com/media/Dq2sPGNU0AEKyyC.jpg | dd status=none bs=1 skip=599 count=40| sh
unnouinceput 4y ago
I'm on Firefox so I used: Right click image -> Open in new window -> Right click -> Save as. The resulted .jpg file I've rename it to .zip then used 7Zip to extract it to a folder. Within that folder I had multiple .rar files so I used 7Zip again on 1st file to extract it to another folder. That one resulted shakespeare.html file which I could indeed see that is a Project Gutenberg collection of all his work. Awesome.
mgrund 4y ago
If you are looking to learn more about these techniques, the field is named steganography: https://en.wikipedia.org/wiki/Steganography https://en.wikipedia.org/wiki/Steganography.
- renonce 4y ago
  I think that's a little bit different from steganography... it looks more like a trick where the author attaches a zip file to the end of a picture and do some work so twitter keeps that file, while steganography allows you to embed that file in the image data so it's just in front of your eyes even if you didn't see it.
qwerty456127 4y ago
7zip unzips it just fine if you rename it to a zip, but Total Commander fails to enter it.
bmacho 4y ago
On Ubuntu I can't unzip it with unzip but 7z works for me, wget https://pbs.twimg.com/media/DqteCf6WsAAhqwV.jpg ; 7z e DqteCf6WsAAhqwV.jpg should fill the directory with a lot of rars.
bobsmooth 4y ago
Steganography is always fun. https://en.wikipedia.org/wiki/Steganography https://en.wikipedia.org/wiki/Steganography
IndySun 4y ago
Or put it in/on/embed via x-ray... https://www.npr.org/2016/01/09/462289635/bones-and-grooves-weird-secret-history-of-soviet-x-ray-music https://www.npr.org/2016/01/09/462289635/bones-and-grooves-w...
nullc 4y ago
"Back in the day" we had some issues on wikipedia with people quietly uploading book cover images where the jpg was also a rar containing the book...
evh 4y ago
For more X is also a Y-style fun, check out PoC||GTFO - "This PDF is also an MBR boot sector" is just the beginning.
RektBoy 4y ago
Nothing new, it took Steam to patch this in steam user's artworks. People shared game hacks this way, hh.
DerekBickerton 4y ago
https://nitter.net/david3141593/status/1057042085029822464 https://nitter.net/david3141593/status/1057042085029822464
jhbadger 4y ago
A similar method is used by Pico-8 and other "fantasy consoles" to create "cartridges" that contain cover art but also the code of the game.
ecolonsmak 4y ago
Is there a way to use this method to stash a crypto key inside the file that could be used to authenticate a nft against a Blockchain?
- TeMPOraL 4y ago
  Yes, but then the key can also be trivially removed, leaving you with the same image, sans the key. Artificial digital scarcity and general-purpose computers don't mix. (Which is why we've been fighting and losing the war on the latter for a good decade now.)

182 points by metadat 4y ago

https://news.ycombinator.com/item?id=18342042 https://news.ycombinator.com/item?id=18342042 discussion from 2018

I am surprised that it survives twitter's image compression. I'm not familiar with JPEG format, but I'm familiar with PNG. I guess it's using some zlib-compressed EXIF-like metadata field, seeing as ZIP also uses DEFLATE/zlib/whatever?

Retr0id 4y ago
It's embedded within an ICC colour profile, and therefore doesn't get touched by any image compression/resizing logic.
- Y-bar 4y ago
  Some eager image compression techniques will strip metadata including profiles (e. g. ImageOptim: https://imageoptim.com/color-profiles.html https://imageoptim.com/color-profiles.html) But generally you are correct.
  - Retr0id 4y ago
    any image compression/resizing logic [in twitter.com] [in 2018]
- csdvrx 4y ago
  How would do that nicely and automatically, to do funny things like that in a way that would be resistant to the accidental optimizations/resizing that will be applied to the JPG?
charonn0 4y ago
It's a polyglot file. JPEG decoders will ignore data that comes after the end-of-data marker, while zip decoders ignore data that comes before the first archive entry. Essentially, they ran `cat pic.jpg arch.zip > output.jpg`.

Retr0id 4y ago

In the years since I posted that, twitter has changed some of their image processing pipeline, so that precise technique no longer works if you want to upload new files (last time I checked - and I didn't investigate further or look for workarounds). However, it's much easier to embed data into PNGs, and the best twitter-compatible implementation of this I'm aware of can be found here: https://github.com/CleasbyCode/pdvzip https://github.com/CleasbyCode/pdvzip Edit: also, the standard `unzip` utility has become a bit stricter about the validity of input files - in ways which would have been possible to work around, but of course not retroactively.

vibragiel 4y ago
I had to run it like this UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip DqteCf6WsAAhqwV.jpg

siraben 4y ago

Using binwalk, you can extract the zip file which in turn contains various rar files which when uncompressed give a 6.8M .html file containing the works of Shakespeare. $ nix run nixpkgs#binwalk DqteCf6WsAAhqwV.jpg DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 JPEG image data, JFIF standard 1.01 182 0xB6 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part001.rar 65575 0x10027 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part002.rar 131112 0x20028 Zip archive data, at least v1.0 to extract, compressed size: 64512, uncompressed size: 64512, name: shakespeare.part003.rar ...

tyingq 4y ago
You shouldn't really need binwalk given how zipfiles work with the directory at the end of the file. You should just be able to unzip the file as-is. Though I suppose that could vary based on what you're unzipping it with. They should support it, though, as that's how a self-extracting zip is supported...ignoring any extra bytes at the top. The zip on Linux has "zip -A file.zip" functionality that will strip the non-zip preamble too.

tyingq 4y ago

Ah, I had played around with this the first time the story came out: $ curl -s https://pbs.twimg.com/media/Dq2sPGNU0AEKyyC.jpg | dd status=none bs=1 skip=599 count=40| sh

unnouinceput 4y ago

I'm on Firefox so I used: Right click image -> Open in new window -> Right click -> Save as. The resulted .jpg file I've rename it to .zip then used 7Zip to extract it to a folder. Within that folder I had multiple .rar files so I used 7Zip again on 1st file to extract it to another folder. That one resulted shakespeare.html file which I could indeed see that is a Project Gutenberg collection of all his work. Awesome.

mgrund 4y ago

If you are looking to learn more about these techniques, the field is named steganography: https://en.wikipedia.org/wiki/Steganography https://en.wikipedia.org/wiki/Steganography.

renonce 4y ago
I think that's a little bit different from steganography... it looks more like a trick where the author attaches a zip file to the end of a picture and do some work so twitter keeps that file, while steganography allows you to embed that file in the image data so it's just in front of your eyes even if you didn't see it.

qwerty456127 4y ago

7zip unzips it just fine if you rename it to a zip, but Total Commander fails to enter it.

bmacho 4y ago

On Ubuntu I can't unzip it with unzip but 7z works for me, wget https://pbs.twimg.com/media/DqteCf6WsAAhqwV.jpg ; 7z e DqteCf6WsAAhqwV.jpg should fill the directory with a lot of rars.

bobsmooth 4y ago

Steganography is always fun. https://en.wikipedia.org/wiki/Steganography https://en.wikipedia.org/wiki/Steganography

IndySun 4y ago

Or put it in/on/embed via x-ray... https://www.npr.org/2016/01/09/462289635/bones-and-grooves-weird-secret-history-of-soviet-x-ray-music https://www.npr.org/2016/01/09/462289635/bones-and-grooves-w...

nullc 4y ago

"Back in the day" we had some issues on wikipedia with people quietly uploading book cover images where the jpg was also a rar containing the book...

evh 4y ago

For more X is also a Y-style fun, check out PoC||GTFO - "This PDF is also an MBR boot sector" is just the beginning.

RektBoy 4y ago

Nothing new, it took Steam to patch this in steam user's artworks. People shared game hacks this way, hh.

DerekBickerton 4y ago

https://nitter.net/david3141593/status/1057042085029822464 https://nitter.net/david3141593/status/1057042085029822464

jhbadger 4y ago

A similar method is used by Pico-8 and other "fantasy consoles" to create "cartridges" that contain cover art but also the code of the game.

ecolonsmak 4y ago

Is there a way to use this method to stash a crypto key inside the file that could be used to authenticate a nft against a Blockchain?

TeMPOraL 4y ago
Yes, but then the key can also be trivially removed, leaving you with the same image, sans the key. Artificial digital scarcity and general-purpose computers don't mix. (Which is why we've been fighting and losing the war on the latter for a good decade now.)