9 ms·

Hijacking HTML canvas and PNG images to store arbitrary text data

70 points by ikromin 8y ago

zawerf 8y ago
Sigh safari is the new IE. Making everyone else suffer from nonsensical hacks. But anyway, if the problem is that the json file is opened in a new tab instead of downloading, can't you just encode it with a non-text media type in the data-uri so the browser doesn't understand it? E.g, data:application/octet-stream instead of data:image/png which is what you claim works right now
- ZenPsycho 8y ago
  data uris do not support setting something equivalent to http’s disposition header, which is what prompted the addition of the download html attribute. data uris also do not permit setting a sensible filename, the other feature the download attribute patches over. safari supports neither, and mobile safari is also in general incapable of sensibly dealing with an anonymous blob like that. (what could it even actually do? what app would you open it with? what would it be named?)
  - zawerf 8y ago
    Uh you can definitely set the mime type in data uris. It's like right there as the first part of a data uri... Download attribute let's you choose a filename. But giving up on that for safari sounds like a decent tradeoff to me (with the current approach you end up with a nonsense png extension in the save file anyway). Testing with a quick demo: https://codepen.io/anon/pen/wYpBry?editors=1010 https://codepen.io/anon/pen/wYpBry?editors=1010, it seems like mobile safari will show it as unknown.txt but have options to save to files/dropbox etc. Kind of complicated but probably still less confusing than a png file.
    - ikromin 8y ago
      The download attribute is not supported on mobile Safari unfortunately - https://caniuse.com/download/embed/description https://caniuse.com/download/embed/description
    - murkle 8y ago
      https://codepen.io/anon/pen/wYpBry?editors=1010 https://codepen.io/anon/pen/wYpBry?editors=1010 is very cool, thanks!
    - ZenPsycho 8y ago
      I didn't say anything about mime type. I said disposition (which is what you would use to force a file to download, instead of display in browser, using the http protocol), and filename. If you set mimetype to octet, mobile safari, since it does not support the download attribute, cannot name it anything sensible or choose an appropriate file extension. I always felt like data: urls could obtain a lot more utility if disposition and filename attributes were added to it, but... uhm, browser makers don't seem to agree, and the syntax would make it difficult to add features like that whilst staying backwards compatible.
- realusername 8y ago
  > Sigh safari is the new IE. Making everyone else suffer from nonsensical hacks. I can confirm that, most of my time spent for browser support is on IE11 and Safari. Everything in Edge, Firefox and Chrome generally works fine.
egypturnash 8y ago
Similar, prior work: the video game "Spore" let you export a PNG of the creatures (and other things) you made with its many, many editors. It'd output a PNG of the critter, with all the data needed to build it from Spore's parts library encoded in the least significant bits of the pixels; you could easily share it in email/chat/whatever, then drag it into Spore to get the critter. Lots more appealing than a square of grey noise, lots easier to understand when you run across it a year later... edit: some more details on this, with examples https://nedbatchelder.com/blog/200806/spore_creature_creator_and_steganography.html https://nedbatchelder.com/blog/200806/spore_creature_creator...
- ZenPsycho 8y ago
  Pico8 does the same thing http://pico-8.wikia.com/wiki/P8PNGFileFormat http://pico-8.wikia.com/wiki/P8PNGFileFormat
- saagarjha 8y ago
  A clever use of steganography, it seems, but one that would probably be ruined by image "optimizers" that are all too common today.
  - syn0byte 8y ago
    I suspect that is exactly why they encoded it into the pixel data rather than extra tEXt blocks. Extra blocks and metadata like EXIF and geotags get reaped by optimizers but pixel data is pixel data. (unless you resize or alter color depth.)
    - TeMPOraL 8y ago
      Or your favourite social media platform indiscriminately converts PNGs to JPGs.
- TazeTSchnitzel 8y ago
  A video game I contributed to, Gang Garrison 2, stores maps as PNG files where the image is the actual level background, and other data is hidden in a text PNG chunk. Hides data in the file format, not the image proper, but a similar idea.
mnutt 8y ago
Unfortunately, MobileSafari is making it harder and harder to actually save the image. If you gently press and hold, it'll give you the option to Save to Camera Roll. But with the advent of force touch, pressing too hard means that you'll end up navigating to a page with the image on it, losing your original page's state in the process. It is possible to catch the force touch event, but as far as I can tell it overrides the regular touch event so all you can do is prevent the event and print "you pressed too hard, please release and try again, gently".
- eridius 8y ago
  When was the last time you actually tried this? Because the reality doesn't match your description at all. First off, you have to put a pretty decent amount of pressure on it to trigger the 3D touch behavior instead of the long press behavior. Secondly, when you do trigger 3D touch, it goes into "peek" mode, and you have to keep pressing, and in fact press even harder, before it will "pop" and actually navigate to the image. In the "peek" mode simply swiping up will give you the Save Image action. And if you're the Hulk and every single touch is a force touch, well, even gripping as hard as I can, it still forces the "peek" mode for a second or two before "popping".
  - mnutt 8y ago
    Thank you; peek mode + swipe up does prompt to save the image and I did not know that. Based on user feedback, the "press-and-hold" method is the only one people commonly know about, and the major issue is that if you accidentally peek (or even imperceptibly drift into peek mode for a few ms) it won't work, and you won't get any feedback as to why. It's handy to have a second way to trigger save, but having to include the explanation in order to teach users how to execute the maneuver is not great UX. "Press gently and hold to save. Or, if your phone supports Force Touch and has it enabled, press pretty hard (but not _too_ hard) and swipe up to save. If it doesn't work, try it again."
    - saagarjha 8y ago
      Force Touch in general has difficulty in exposing actions and discoverability in general.
perilunar 8y ago
Depending on the amount of data, you could store it in the URL itself as a fragment identifier
- ikromin 8y ago
  Agreed, but you would not want to use a URL that is larger than around 2k characters, in my case that wasn't enough.
ZenPsycho 8y ago
What I would be interested in is if there would be a way to encode data in an image that is robust to various kinds of jpeg/back/shrinking/enlarging transforms. This is really cool and useful but there's all kinds of normal processes that images go through that would destroy data hidden in an image this way. Perhaps something akin to parity or error correction data hidden somewhere else?
- m1el 8y ago
  Why, yes, there are ways to do this. One of the interesting ways is adding is adding noise generated by inverse Fourier transforms. http://repro.grf.unizg.hr/media/Ante/Radovi/033008_1_FINAL.pdf http://repro.grf.unizg.hr/media/Ante/Radovi/033008_1_FINAL.p...
byteface 8y ago
I did similar several years ago but built my own index and rendered a character per channel... https://github.com/byteface/GTP/blob/master/sentiment/canvas/js/app/GTP.js https://github.com/byteface/GTP/blob/master/sentiment/canvas... meaning 3or4 chars per pixel. then tried to do boyer moore on a dynamic shader. At the time I felt I had the fastest string searching tool on the web, in the world... https://github.com/byteface/GTP/blob/master/play/simplified/js/app/shaders/TEST4.js https://github.com/byteface/GTP/blob/master/play/simplified/... it could all the 'the' in moby dick in a split second. which at the time would take textmate a few seconds. showed a mate who then did his own storage only implementation, closer to what you are doing: https://github.com/claus/PNGDrive https://github.com/claus/PNGDrive years after saw someone did one that stored data in flckr. Think it was posted here. To be honest, IMO the idea is senseless as pure storage as theres other better options. like zip. But if you are going to do something with it on GPU that's when it has new potential.
grenoire 8y ago
If you're using PNGs, why skip the fourth alpha channel?
- byteface 8y ago
  because he used ascii codes which uses 3 numbers and he stored a number on each channel. method requires no table to encode/de-encode
  - ikromin 8y ago
    Nope, and I explain it in the article, it's because of the weird behaviour you get with canvas and alpha channel. If somehow a value of 0 made it to the alpha channel, all of red, green and blue channels were reset to zero as well. It was a bit annoying to deal with and a total pain to change my code. Originally I was using all 4 channels.
    - byteface 8y ago
      ah i see now. ur using ascii code as the color value. i used to gen them with a php lib don't recall experiencing this error on alpha. browser thing?
jcmeyrignac 8y ago
This is not really new. One of my friends built a DAWG into a PNG. https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_state_automaton https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_s... The DAWG contained all french words, in a convenient small picture.
- slig 8y ago
  Did s/he publish somewhere? I'm interested in learning more, thanks!
z3t4 8y ago
Maybe you could store the image in the browser cache. Then users wouldn't need to manually save and upload the image. Most browsers will keep the image cached until the user manually empty the cache (eg forever).
ourcat 8y ago
I did something vaguely similar storing text in QR codes then converting the black and white squares to audio 'noise', which could be saved as a wav then read somewhere else, by reading the audio 'noise', reconstructing the QRCode and reading that ;) - It's not a million miles away from the 'chirp' app idea. demo at https://qrdio.com https://qrdio.com
mnutt 8y ago
A while back I built a site for converting torrents into banner-sized images, with the idea that people could share them on message boards and such, and there were browser extensions that would detect them and let you open them in your bittorrent app.
- jameskegel 8y ago
  What site is it
tuxidomasx 8y ago
This reminded me of a steganography blog post I wrote back when I was doing computer security. It should be possible to binarily concatenate compressed (RAR'd or ZIP'd) text data directly to a JPG. Then you can save it as an image on the client, or uncompress it in the client browser and read the data. The differences are 1) the saved image won't look like random pixels-- it'll look like whatever base image you choose and 2) you don't have to worry about writing the encoding/decoding data as bitmap stuff.
bibbitybobbity 8y ago
Sounds like image steganography. If anyone wants a python example here ya go https://github.com/jermainezhimin/culpriteCTF/blob/master/LSB/LSB.py https://github.com/jermainezhimin/culpriteCTF/blob/master/LS...
notjustanymike 8y ago
Elite did this way back to store the entire galaxy.
tomatotomato37 8y ago
Reminds me of that 4chan spam attack which was an image that told users to save it and change the extension to an executable, which would then execute and make more posts with the same image, continuing the cycle. I believe that they started scanning images for random appended data after that

70 points by ikromin 8y ago

Sigh safari is the new IE. Making everyone else suffer from nonsensical hacks. But anyway, if the problem is that the json file is opened in a new tab instead of downloading, can't you just encode it with a non-text media type in the data-uri so the browser doesn't understand it? E.g, data:application/octet-stream instead of data:image/png which is what you claim works right now

ZenPsycho 8y ago
data uris do not support setting something equivalent to http’s disposition header, which is what prompted the addition of the download html attribute. data uris also do not permit setting a sensible filename, the other feature the download attribute patches over. safari supports neither, and mobile safari is also in general incapable of sensibly dealing with an anonymous blob like that. (what could it even actually do? what app would you open it with? what would it be named?)
- zawerf 8y ago
  Uh you can definitely set the mime type in data uris. It's like right there as the first part of a data uri... Download attribute let's you choose a filename. But giving up on that for safari sounds like a decent tradeoff to me (with the current approach you end up with a nonsense png extension in the save file anyway). Testing with a quick demo: https://codepen.io/anon/pen/wYpBry?editors=1010 https://codepen.io/anon/pen/wYpBry?editors=1010, it seems like mobile safari will show it as unknown.txt but have options to save to files/dropbox etc. Kind of complicated but probably still less confusing than a png file.
  - ikromin 8y ago
    The download attribute is not supported on mobile Safari unfortunately - https://caniuse.com/download/embed/description https://caniuse.com/download/embed/description
  - murkle 8y ago
    https://codepen.io/anon/pen/wYpBry?editors=1010 https://codepen.io/anon/pen/wYpBry?editors=1010 is very cool, thanks!
  - ZenPsycho 8y ago
    I didn't say anything about mime type. I said disposition (which is what you would use to force a file to download, instead of display in browser, using the http protocol), and filename. If you set mimetype to octet, mobile safari, since it does not support the download attribute, cannot name it anything sensible or choose an appropriate file extension. I always felt like data: urls could obtain a lot more utility if disposition and filename attributes were added to it, but... uhm, browser makers don't seem to agree, and the syntax would make it difficult to add features like that whilst staying backwards compatible.
realusername 8y ago
> Sigh safari is the new IE. Making everyone else suffer from nonsensical hacks. I can confirm that, most of my time spent for browser support is on IE11 and Safari. Everything in Edge, Firefox and Chrome generally works fine.

egypturnash 8y ago

Similar, prior work: the video game "Spore" let you export a PNG of the creatures (and other things) you made with its many, many editors. It'd output a PNG of the critter, with all the data needed to build it from Spore's parts library encoded in the least significant bits of the pixels; you could easily share it in email/chat/whatever, then drag it into Spore to get the critter. Lots more appealing than a square of grey noise, lots easier to understand when you run across it a year later... edit: some more details on this, with examples https://nedbatchelder.com/blog/200806/spore_creature_creator_and_steganography.html https://nedbatchelder.com/blog/200806/spore_creature_creator...

ZenPsycho 8y ago
Pico8 does the same thing http://pico-8.wikia.com/wiki/P8PNGFileFormat http://pico-8.wikia.com/wiki/P8PNGFileFormat
saagarjha 8y ago
A clever use of steganography, it seems, but one that would probably be ruined by image "optimizers" that are all too common today.
- syn0byte 8y ago
  I suspect that is exactly why they encoded it into the pixel data rather than extra tEXt blocks. Extra blocks and metadata like EXIF and geotags get reaped by optimizers but pixel data is pixel data. (unless you resize or alter color depth.)
  - TeMPOraL 8y ago
    Or your favourite social media platform indiscriminately converts PNGs to JPGs.
TazeTSchnitzel 8y ago
A video game I contributed to, Gang Garrison 2, stores maps as PNG files where the image is the actual level background, and other data is hidden in a text PNG chunk. Hides data in the file format, not the image proper, but a similar idea.

mnutt 8y ago

Unfortunately, MobileSafari is making it harder and harder to actually save the image. If you gently press and hold, it'll give you the option to Save to Camera Roll. But with the advent of force touch, pressing too hard means that you'll end up navigating to a page with the image on it, losing your original page's state in the process. It is possible to catch the force touch event, but as far as I can tell it overrides the regular touch event so all you can do is prevent the event and print "you pressed too hard, please release and try again, gently".

eridius 8y ago
When was the last time you actually tried this? Because the reality doesn't match your description at all. First off, you have to put a pretty decent amount of pressure on it to trigger the 3D touch behavior instead of the long press behavior. Secondly, when you do trigger 3D touch, it goes into "peek" mode, and you have to keep pressing, and in fact press even harder, before it will "pop" and actually navigate to the image. In the "peek" mode simply swiping up will give you the Save Image action. And if you're the Hulk and every single touch is a force touch, well, even gripping as hard as I can, it still forces the "peek" mode for a second or two before "popping".
- mnutt 8y ago
  Thank you; peek mode + swipe up does prompt to save the image and I did not know that. Based on user feedback, the "press-and-hold" method is the only one people commonly know about, and the major issue is that if you accidentally peek (or even imperceptibly drift into peek mode for a few ms) it won't work, and you won't get any feedback as to why. It's handy to have a second way to trigger save, but having to include the explanation in order to teach users how to execute the maneuver is not great UX. "Press gently and hold to save. Or, if your phone supports Force Touch and has it enabled, press pretty hard (but not _too_ hard) and swipe up to save. If it doesn't work, try it again."
  - saagarjha 8y ago
    Force Touch in general has difficulty in exposing actions and discoverability in general.

perilunar 8y ago

Depending on the amount of data, you could store it in the URL itself as a fragment identifier

ikromin 8y ago
Agreed, but you would not want to use a URL that is larger than around 2k characters, in my case that wasn't enough.

ZenPsycho 8y ago

What I would be interested in is if there would be a way to encode data in an image that is robust to various kinds of jpeg/back/shrinking/enlarging transforms. This is really cool and useful but there's all kinds of normal processes that images go through that would destroy data hidden in an image this way. Perhaps something akin to parity or error correction data hidden somewhere else?

m1el 8y ago
Why, yes, there are ways to do this. One of the interesting ways is adding is adding noise generated by inverse Fourier transforms. http://repro.grf.unizg.hr/media/Ante/Radovi/033008_1_FINAL.pdf http://repro.grf.unizg.hr/media/Ante/Radovi/033008_1_FINAL.p...

byteface 8y ago

I did similar several years ago but built my own index and rendered a character per channel... https://github.com/byteface/GTP/blob/master/sentiment/canvas/js/app/GTP.js https://github.com/byteface/GTP/blob/master/sentiment/canvas... meaning 3or4 chars per pixel. then tried to do boyer moore on a dynamic shader. At the time I felt I had the fastest string searching tool on the web, in the world... https://github.com/byteface/GTP/blob/master/play/simplified/js/app/shaders/TEST4.js https://github.com/byteface/GTP/blob/master/play/simplified/... it could all the 'the' in moby dick in a split second. which at the time would take textmate a few seconds. showed a mate who then did his own storage only implementation, closer to what you are doing: https://github.com/claus/PNGDrive https://github.com/claus/PNGDrive years after saw someone did one that stored data in flckr. Think it was posted here. To be honest, IMO the idea is senseless as pure storage as theres other better options. like zip. But if you are going to do something with it on GPU that's when it has new potential.

grenoire 8y ago

If you're using PNGs, why skip the fourth alpha channel?

byteface 8y ago
because he used ascii codes which uses 3 numbers and he stored a number on each channel. method requires no table to encode/de-encode
- ikromin 8y ago
  Nope, and I explain it in the article, it's because of the weird behaviour you get with canvas and alpha channel. If somehow a value of 0 made it to the alpha channel, all of red, green and blue channels were reset to zero as well. It was a bit annoying to deal with and a total pain to change my code. Originally I was using all 4 channels.
  - byteface 8y ago
    ah i see now. ur using ascii code as the color value. i used to gen them with a php lib don't recall experiencing this error on alpha. browser thing?

jcmeyrignac 8y ago

This is not really new. One of my friends built a DAWG into a PNG. https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_state_automaton https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_s... The DAWG contained all french words, in a convenient small picture.

slig 8y ago
Did s/he publish somewhere? I'm interested in learning more, thanks!

z3t4 8y ago

Maybe you could store the image in the browser cache. Then users wouldn't need to manually save and upload the image. Most browsers will keep the image cached until the user manually empty the cache (eg forever).

ourcat 8y ago

I did something vaguely similar storing text in QR codes then converting the black and white squares to audio 'noise', which could be saved as a wav then read somewhere else, by reading the audio 'noise', reconstructing the QRCode and reading that ;) - It's not a million miles away from the 'chirp' app idea. demo at https://qrdio.com https://qrdio.com

mnutt 8y ago

A while back I built a site for converting torrents into banner-sized images, with the idea that people could share them on message boards and such, and there were browser extensions that would detect them and let you open them in your bittorrent app.

jameskegel 8y ago
What site is it

tuxidomasx 8y ago

This reminded me of a steganography blog post I wrote back when I was doing computer security. It should be possible to binarily concatenate compressed (RAR'd or ZIP'd) text data directly to a JPG. Then you can save it as an image on the client, or uncompress it in the client browser and read the data. The differences are 1) the saved image won't look like random pixels-- it'll look like whatever base image you choose and 2) you don't have to worry about writing the encoding/decoding data as bitmap stuff.

bibbitybobbity 8y ago

Sounds like image steganography. If anyone wants a python example here ya go https://github.com/jermainezhimin/culpriteCTF/blob/master/LSB/LSB.py https://github.com/jermainezhimin/culpriteCTF/blob/master/LS...

notjustanymike 8y ago

Elite did this way back to store the entire galaxy.

tomatotomato37 8y ago

Reminds me of that 4chan spam attack which was an image that told users to save it and change the extension to an executable, which would then execute and make more posts with the same image, continuing the cycle. I believe that they started scanning images for random appended data after that