13 ms·
JPEG image of Shakespeare which is also a zip file containing his complete works
- DerekL 8y agoTitle should say “JPEG”, not “JPG”. “JPG” is just a file extension, not the name of the compression method.
- deleted 8y ago[deleted]
- jjoonathan 8y agoI know that the real reason is a requirement for length 3 combined with a slight preference for vowel elision, but I just realized that if one were so inclined it could be interpreted as a snub. Joint Photographic Experts Group -> Joint Photographic Group No experts to be found here! :P
- rothron 8y agoThe extension is somewhat of a misnomer. It's really a jif. JPEG Interchange Format. You can imagine the confusion if the jif extension caught on.
- laumars 8y agoI never really understood why people still adhere to the 3 character file extension limit when that is a throwback to FAT32 on pre-Windows 95 (and some very old mainframes which shouldn't be internet connected anyway). Yet some people still favor the 3 character extension despite it looking objectively uglier (as a dyslexic I do find them harder to read) and dropping a solitary vowel in a file extension saves you nothing in terms of development time / file size / etc. So I honestly don't get why people still use them. Extensions that particularly annoy me are: yml instead of yaml Why do people do this when even the earliest YAML spec is much newer than the last of the systems that had a 3 character extension limit. jpg instead of jpeg Nearly all of the .jpg's in the last 10 years would be too large to load on any systems that couldn't handle a 4 character extension anyway. So backwards compatibility isn't even an argument here. htm instead of html Similar argument as JPG except this time it's more to do with the HTML specification and browsers capable of rendering them.
- wild_preference 8y ago> objectively uglier Not sure why'd you'd say "objectively" when it's clearly not, the hint being the word "ugly." I can certainly appreciate the limitation of "extensions will always be three characters". In fact, I wasn't exposed to extensions habitually longer than that until I did iOS dev. "Main.storyboard" "Model.xcdatamodeld" So if we're going to talk about ugly, I'd start there. ;)
- thx4allthestuff 8y agoBeauty is in the eye of the beholder, so yes, in other words completely subjective.
- laumars 8y agoSorry, as another poster commented, I did mean "subjectively" (doh!) xcdatamodeld is definitely ugly (though main.storyboard is ok in my personal opinion) and I'm sure there will be other examples where file extensions have been taken too far. I think if one wants to use long extensions then the extension needs to at least be readable at a glance (eg xcdatamodeld is ugly to read - even if you switched it the other way around, xcdatamodeld.model would still be unpleasant). So I'd argue that xcdatamodeld is more of an example of a poor naming convention than a problem with longer extensions (I say this with no experience of iOS development, however I see the same problem quite as acutely in the other languages and platforms I've developed on). It should also be noted that my argument was really more focused on people who shorten already short 4 character acronyms - just for the purpose of making it 3 letters (eg the yaml and html examples I gave).
- jlv2 8y ago> 3 character file extension limit when that is a throwback to FAT32 on pre-Windows 95 The 3 character file extension is much older than FAT32. It was part of the original FAT that was created nearly 20 years earlier, and later part of FAT12 and FAT16. But it predated all that in various other microcomputer DOS, as well as minicomputer systems going back into the 60s.
- 8y ago
- glibgil 8y agoHere you go ".JPG image of Shakespeare which is also a .zip file containing his complete works"
- scarcely 8y agoThis is actually a really good point. It ought to strike us as anomalous as hearing someone say "the notes are stored in HTM". Thanks.
- quickthrower2 8y agoThere's an xkcd for that: https://xkcd.com/1301/ https://xkcd.com/1301/
- diimdeep 8y agobinwalk DqteCf6WsAAhqwV.jpg DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 JPEG image data, JFIF standard 1.01 182 0xB6 Zip archive data, at least v1.0 to extract, ..., name: shakespeare.part001.rar ... 1971177 0x1E13E9 End of Zip archive
- BeefySwain 8y agoJust tried it out and it does indeed work. curl 'https://pbs.twimg.com/media/DqteCf6WsAAhqwV.jpg' > shakespeare.zip unzip shakespeare.zip unrar e shakespeare.part001.rar I'm excited for the inevitable curl | unzip | bash method of installing software directly from twitter that is sure to follow.
- fireattack 8y agoInteresting that Winrar seems to have trouble to uncompress the first layer of it (zip to rars). (7-zip works fine.)
- jethro_tell 8y agoWell, if someone out there had paid for winrar, perhaps the support would be better XD
- ty_a 8y agoUnderrated.
- deleted 8y ago[deleted]
- tobyhinloopen 8y ago“I paid for Winrar, AMA”
- rzzzt 8y agohttps://www.reddit.com/r/paidforwinrar https://www.reddit.com/r/paidforwinrar
- HeavyStorm 8y agoWHY????
- 8y ago
- nan0 8y agoDr. Mike Pound from Computerphile did a video on this "Secrets Hidden in Images (Steganography)" [Youtube video] https://youtu.be/TWEXCYQKyDc https://youtu.be/TWEXCYQKyDc
- chaoticmass 8y agoGood video, but what was done here is not Steganography. No ZIP data was embedded into the pixels of the image.
- deleted 8y ago[deleted]
- judge2020 8y agoThere was a similar technique used in the tv show Mr. Robot, season 3 episode 9, to hide some secret information (trying not to spoil anything). Ryan Kazanciyan, the Technical Consultant, wrote a blog piece on this technique. https://link.medium.com/x7YFdksWrR https://link.medium.com/x7YFdksWrR
- tyingq 8y agoThis is also somewhat interesting... "It survives all of twitters scaling, compression and thumbnailing. /how/ is left as an exercise to the reader :P"
- stupidbird 8y agoGuessing Twitter doesn't scale/thumbnail/compress already tiny images... which is probably an oversight in this case because the file size is disproportionately large.
- Derek_MK 8y agoYeah, I guess it's possible that it's only checking dimensions instead of actual filesize.
- martincmartin 8y agoNo, the rars are in the image metadata, not in the actual pixels at all.
- stupidbird 8y agoThey still contribute to filesize.
- daveFNbuck 8y agoThey don't address the metadata in compression, which is probably what the bug report [1] was about. This thumbnail is over 2MB. With the metadata removed, it's only about 2KB. [1] https://twitter.com/David3141593/status/1057042727945322496 https://twitter.com/David3141593/status/1057042727945322496
- yreg 8y agoThat's less exciting
- bepvte 8y agothe answer: ICC profiles (i think)
- AdmiralAsshat 8y agoThis is hardly new. I remember collecting ebooks on 4chan circa 2005 via this method. The fact that it was a jpg allowed you to post it on the imageboard with the zip/RAR file therein containing a txt/doc/rtf of the ebook.
- DavidBuchanan 8y agoAnd yet, nobody has done the same for Twitter until now. The difference is that twitter applies a series of operations to all uploaded image, stripping EXIF data, recompressing, etc., which would normally be difficult to work around.
- awirth 8y agoDid people do this back in the day? 4chan used to be totally fine with just uploading a jpeg concatenated with a zip, but I haven't seen this ICC profile trick before today.
- palunon 8y agoWell, 4chan served you the original file, so you didn't need to evade image processing at all
- lordcorvin1 8y ago4Chan actually strips EXIF data as well
- goatsi 8y agoThere is actually a greasemonkey script that lets you embed encrypted messages into pictures uploaded to imageboards, letting you carry on a side conversation this way. https://github.com/desudesutalk/desudesutalk/wiki/How-to-use-this-script https://github.com/desudesutalk/desudesutalk/wiki/How-to-use...
- Fundlab 8y ago
- thomastjeffery 8y agoIs there any reason the zip contains .rar files?
- bigdubs 8y agoTo get around JPEG block size limits.
- DavidBuchanan 8y agoNot quite, ICC profile chunk size limits.
- fireattack 8y agoIs it possible to have split files with zip?
- romed 8y agoYes, it is. I feel like this hack is a wry nod to the old warez scene where multipart archives within archives were, for some reason, de rigueur.
- oorza 8y agoNo way to resume BBS and later FTP uploads back in the day.
- PhasmaFelis 8y agoI thought it was assumed that the files, once downloaded, would be distributed locally on floppies. The sub-archives were usually 1.44MB to fit on a 3.5". When did resumable BBS download protocols become a thing? ZMODEM at least was widely used by the early '90s.
- Theodores 8y agoYou cram these words into mine ears against the stomach of my sense... There are no paintings of Shakespeare that are definitively him. Nobody knows what he looks like. Sure we have images but we have images of Jesus Christ too, again not too definitive. This looks like the effort of someone wanting to get a few extra Twitter followers. It does not look like the effort of someone who has taken the time to study the smallest of sonnets, never mind try and get through the complete works. Studying Shakespeare is hard, it requires understanding big words and dedication. It is in a totally different realm of intellectual pursuit to Googling a few hacky python scripts together. What next, putting Dickens in a colostomy bag?
- Retr0spectrum 8y agoI take more pleasure from well-crafted polyglot files (e.g. PoC||GTFO) than I ever have from classic literature.
- Theodores 8y agoMy auntie takes more pleasure from watching the news to see what colour tie the newsreader is wearing than from watching the news to find out what is happening in the world. I personally believe that she is missing out on something. I also think that she could watch something on the other side if it is fashion that really interests her, particularly when her comments on ties prevent others in the room from actually hearing the news. I had a school friend that used 7" records as frisbees and made 12" records into plant pot holders. This was fine until he started doing this with my prized records. It is the same here, there is a disconnect going on, a lack of appreciation for Shakespeare which is more than mere 'classic literature', it is an education. PoC||GTFO indeed. TLAs are not helpful.
- detaro 8y agoI don't think you have any basis for claiming intellectual superiority when you started this by judging the education and interests of a person from a single thing they've done.
- willnewman 8y agoOh I did this too a few years ago. Except I used gifs instead of jpegs, and then wrote a Ruby wrapper for backing up files to Flickr. Blog post about it here: https://namwen.svbtle.com/hoardr https://namwen.svbtle.com/hoardr
- ricardobeat 8y agoDid the same a few months before with PNGs! https://github.com/ricardobeat/filr https://github.com/ricardobeat/filr
- deleted 8y ago[deleted]
- catpolice 8y agoAnybody have any insight into what kind of trickery goes into this? I know a decent amount about both of the compression algorithms, but I've forgotten the structure of the file formats so I can't tell if this required any really funny business.
- DavidBuchanan 8y agoIt isn't massively complex, but it did require some custom encoding/decoding logic. I plan to write a blog post on this some time soon...
- danielvf 8y agoI've written my own zip and unzipping code. Basically the "top level" zip file section lives at the end of the zip file and contains pointers back into the file for the actual data. This is so that you can keep adding to a zip file without having to rewrite anything you have done previously - just append a few more files and then your updated directory at the end. JPEG segments contain two special opening bytes, maybe a length, and then the actual data. So it would be trivial to have the last jpeg section in a jpeg file end with zip directory bytes. This directory could point back to data in other sections of the jpeg. ..my guess, anyway.
- plaguuuuuu 8y agoI can't exactly remember what JPG and ZIP look like, but yeah pretty much, I've done it before. What's surprising is that it works on Twitter. I tried the same thing with image hosts and the data got scrubbed. I guess they don't re-encode small images but I dunno.
- Laforet 8y agoIn other words twitter's backend did not sanitize image uploads properly.
- ThrustVectoring 8y agoThe only trickery involved is making sure it survives twitter's compression and image cleaning algorithms. Other than that, programs that read images will ignore garbage data after what should be there for a jpg or whatever, and programs that open archives will ignore anything that comes before the archive header, so it's literally as straightforward as appending a .rar file at the end of a .jpg file.
- alanwong 8y agoCan the technique be used to bypass content censorship (e.g. in China)? Can it be made resistant to detection?
- zuppy 8y agonot really, the data is still there, there are just 2 different file contents saved into one and playing with the zip and jpg format so both of them are still readable.
- woodruffw 8y ago> Can the technique be used to bypass content censorship (e.g. in China)? Steganography can be used to hide data in clever ways, but it isn't a substitute for encryption -- anybody who knows whatever trick you used can extract the payload. You could always encrypt the payload, but all you're doing is giving your adversary another (trivial) hoop to jump through when you could've just encrypted the message to begin with. > Can it be made resistant to detection? Yes and no. There are some clever steganographic techniques that take advantage of PNG and JPEG implementation details to foil basic entropy checks, but anybody who knows the algorithm can trivially extract the payload. In other words, it's security by obscurity, not by any sort of strong cryptographic property.
- waterhouse 8y ago> You could always encrypt the payload, but all you're doing is giving your adversary another (trivial) hoop to jump through when you could've just encrypted the message to begin with. Wouldn't "an encrypted message that no one is sure you sent in the first place" sometimes be more useful than "an encrypted message that any eavesdropper knows you sent" in oppressive-surveillance-state scenarios? (It seems that, if you find some subset of bits in a JPEG/PNG that normally have random distribution and don't affect the image that much, putting an encrypted message into those bits might be indistinguishable from a "completely normal" image even to a well-informed attacker.)
- 0-_-0 8y ago
- llIIllIIllIIl 8y agoраржпег? this trick was used on anonymous forums for years.
- mikorym 8y agoIs there a recent repo?
- mygo 8y agoCould the future of open source and P2P be tweeting apps to one another?
- rgrieselhuber 8y agoWould be interesting if Renaissance paintings were actually similar repositories.
- adamson 8y agoI feel like that'd be a lot more than just interesting
- emiliobumachar 8y agoQuite a what-if. "The Library of Alexandria was not lost, merely hidden"
- Avamander 8y agoThat would be a great scifi plot.
- deleted 8y ago[deleted]
- angel_j 8y agoare the complete works actually encoded in the pixels?
- gm3dmo 8y agoI do extend him, sir, within himself; Crush him together rather than unfold
- Symbiote 8y agoWell found! https://www.opensourceshakespeare.org/views/plays/play_view.php?WorkID=cymbeline&Act=1&Scene=1&Scope=scene&LineHighlight=30#30 https://www.opensourceshakespeare.org/views/plays/play_view....
- rootsudo 8y agoSimple Steganography. I wrote a similar article to use Amazon Photo storage as a free way to store files if anyone is curious how this works: https://medium.com/@eirurueta/free-unlimited-cloud-storage-on-amazon-w-this-one-weird-trick-6f50d35b1159 https://medium.com/@eirurueta/free-unlimited-cloud-storage-o...
- comboy 8y agoWhile technically it would probably fit the definition, usually what's meant by steganography regarding images is hiding the data within the image itself (using differences in pixel values that are not easy to spot). As far as I understand it, this just makes use of the file structure. It's beyond me though how it survives twitter thumbnailer. I would assume that one of its purposes is to strip any unnecessary metadata from the image, making it as small as possible.
- logfromblammo 8y agoThe easiest implementation would be to use the 2 least significant bits in the blue channel, and the least significant bit of the red channel, to encode your data payload. Just sticking the data in the metadata portions of an image file seems like hiding behind the only bush in an otherwise featureless field. ("Reginald Maudlin, will you stand up, please?")
- lewich 8y agowow. old as fuck trick reinvented. people are excited.
- lewich 8y agotry to google rarjpeg....
- deleted 8y ago[deleted]
- bayindirh 8y agoPoC||GTFO's [0] most issues are polyglot files. Taken from 0x16 [1]'s introduction: This file, pocorgtfo16.pdf , is a polyglot that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf
- TeMPOraL 8y ago0x15 is a laser-projectable PDF that's also a ZIP containing, among other things, another PDF that is also a Git repo of its own source code. 0x17 is a PDF that's also a valid ZIP and valid firmware for Apollo Guidance Computer. And that's just the tip of an iceberg. This project is absolutely crazy, in the most positive possible way.
- PhasmaFelis 8y ago> 0x17 is a PDF that's also a valid ZIP and valid firmware for Apollo Guidance Computer. Wait, what? Edit: 0x15 has the additional feature that viewing it in Chrome makes my work machine throw a "Threat detected: A threat has been blocked and quarantined" message, so I think I'm going to avoid this site now. Not to say that I don't trust this random PDF hacker (...that sounded less sarcastic in my head), but I don't need IT asking me pointed questions about what sites I've been visiting.
- TeMPOraL 8y agoYes, that. PoC||GTFO is a well known project, and has a tradition of turning their PDFs into absurdly polyglot files. Your Chrome is giving you a false positive here.
- JayXon 8y agoIf you are interested in this sort of thing, make sure to checkout Funky File Formats by Ange Albertini https://events.ccc.de/congress/2014/Fahrplan/events/5930.html https://events.ccc.de/congress/2014/Fahrplan/events/5930.htm...
- theoh 8y agoSomething else interesting from a few years ago: a compression algorithm that's bijective. http://www3.sympatico.ca/mt0000/bicom/bicom.html http://www3.sympatico.ca/mt0000/bicom/bicom.html It's similar in the sense that there's more than one possible operation on any file (decompress it, or compress it), and different in that the file type doesn't change.
- expertentipp 8y agoDo this for mp3.
- tyingq 8y agoProbably easier. The MP3 standard didn't include any standard for tags, so the tags standards evolved separately. Basically, you can append any random crap you want at the end of an mp3. Since a zip file has it's headers at the bottom, it wouldn't be hard to do. I don't know if popular mp3 sites scrub/transform mp3 files in the way image sites do to images though. Edit: Tried it, and got it to work. The zip command on Linux has a -A option that will fix the offset headers after you cat an mp3 and a zip file together. Just "cat file.mp3 file.zip > newfile.mp3;zip -A newfile.mp3" and you're done. I did notice that some mp3 file upload sites reprocess the mp3 file and strip out the extra bits. Some don't though. Here's a Bach mp3 that you can play online: https://instaud.io/2RLM https://instaud.io/2RLM But, if you download it, via the buttons on the page, you'll see it's also a valid zip file containing a jpg meme of bach.
- barking 8y agoVery thought provoking. The file size is 1.93 MB. I wonder what dimensions would a disk just big enough to store that much have? Not long ago I read of a suggestion to archive information as DNA (for its durability iirc). But in terms of physical space required to store stuff, is DNA more or less efficient than today's disks?
- TeMPOraL 8y agoA good question. Here are the answers from a HN comment thread couple of months ago: https://news.ycombinator.com/item?id=17422950 https://news.ycombinator.com/item?id=17422950
- avian 8y agoThere are 512 GB micro SD cards on the market today. So at that density roughly: 2 MB / 512 GB * 100 mm2 ~= 400 um2 400 square micrometers is an area of a square with 20 micrometer sides. Approx. 1/100th of the cross-section of a human hair.
- mojomark 8y agoShakespeare: "Alas! Mine own quill is too bawbling f'r mine own parchment"
- barking 8y agoThanks, I like that comparison. Now I wish I could think of a way to mesh this with the infinite monkey theorem!
- sonofgod 8y agoSo Borges' Library of Babel -- every possible 3200 character book from a 25-character alphabet; essentially the 'space' equivalent of the infinite monkey theorem's 'time' domain -- is sufficiently large that if we took this universe and shrunk it down to the Planck length (the smallest length there is) and filled another universe with these femtouniverses, and then scaled that one down likewise, and repeated this process EIGHTY times, we'd be able to fit all the books in. And Hamlet would be split amongst dozens of these books. 25^3200 is a large number.
- ainiriand 8y agoI f*ing love technology...
- JetSpiegel 8y agoGreat What If question: what are the odds of taking a picture of a Shakerpeare play and get that play's text in the resulting JPG?.
- candiodari 8y agoFor those who don't know the trick: ZIP files have their index at the end of the file. So you can add a zip file to anything else and have it unzippable. This was done because it allows adding files to zip archive files without rewriting the whole file, instead just start writing where the index starts and add an updated index at the end. Please note that over the years there's been multiple methods of doing this, including partial indexes which don't even rewrite the index. If the file you're adding things to is also tolerant of wrong file sizes and extra data at the end (like JPG and many others), you can just: cat someJPG.jpg someZIP.zip > wth.jpg feh wth.jpg => shows image mv wth.jpg wth.zip; unzip wth.zip => works
- AnabeeKnox 8y agoIt's astonishing that it's as simple as CATting a zip file to the end of a jpg. I feel there are consequences here for any website that accepts image uploads.
- MaxBarraclough 8y ago> I feel there are consequences here for any website that accepts image uploads Steganography can be done even without file-format hacks; all that's special about this hack is its simplicity. It could easily be defeated -- I'm sure Twitter would have no trouble sanitizing uploaded image files if they wanted to.
- martincmartin 8y agoThat's not how this works though.
- candiodari 8y agoJep you're right. It's doing something with 64k JFIF application segments ... wtf. Well, it's still using the trick I pointed out, placing the ZIP file index at the end. So from one viewpoint it's a JFIF("jpg") file with large application segments containing the zipfile data for the shakespeare.part0xx.rar files. From another viewpoint, from the back of the file. It's a incremental zip file (not compressed in one go), with the garbage bytes (the "overwritten") bytes in the zipfile updates forming a valid JFIF file.
- spacelama 8y agoLove it
- tyingq 8y agoI tried doing something similar with an mp3 first, and that worked. You can't upload mp3 files to twitter, but I have one here: https://instaud.io/2RLM https://instaud.io/2RLM Play it on the page, and it's an mp3, download it via the buttons, and it's also a valid zip file with a jpg in it. So then I tried the same thing with MP4 video on twitter. Twitter didn't like that at all :) https://imgur.com/a/O0QFZC9 https://imgur.com/a/O0QFZC9
- dexterdog 8y agoI dealt with this problem on a photo site 15 years ago where people were embedding RAR pieces in their JPGs to use us as a distribution repository. The bandwidth increase was a giveaway so I just wrote an incoming processor that re-compressed everything that had more than X bytes per pixel. I can't believe Twitter doesn't have similar logic given its scale.
- nickspacek 8y agoMaybe they will just deal with it as you did: fix it when it becomes a problem.
- Sargos 8y agoTwitter does re-compress everything. This method survives re-compression.
- dexterdog 8y agoThen it's not recompression. Recompression is pulling (let's say) a JPG into memory so it becomes a bitmap and writing it back out to a JPG. If you want to keep metadata you only keep a specific set and you cap the value lengths.
- enimodas 8y agoSimilarly, here is one with music. Winamp necessary. https://a.doko.moe/uigihv.jpg https://a.doko.moe/uigihv.jpg
- jhabdas 8y agoThis is what a real dcma takedown looks like: https://github.com/CanadianJeff/fuckr https://github.com/CanadianJeff/fuckr
- germs12 8y agoI'm pretty sure Ender's Game has this in it. It was how one of the characters snuck info out.
- hamam 8y agoحامام
- deleted 8y ago[deleted]