5 ms·

Abusing Amazon Images

159 points by chaosmachine 17y ago

chaosmachine 17y ago
See also: http://www.gertler.com/nat/abusedimages.html http://www.gertler.com/nat/abusedimages.html
albemuth 17y ago
a ruby dsl library should pop out any moment on github
mcantor 17y ago
Amazon should be paying this guy for documenting their codebase for them.
- martey 17y ago
  Because Amazon has no internal documentation about how to create product images? Or because Amazon's terms of services explicitly prevent doing anything useful with these images?
  - chrischen 17y ago
    Because this guy was very thorough in detailing how to manipulate amazon's dynamic image generator.
stephenjudkins 17y ago
This approach to generating images generates a lot of benefits when you're working at scale. Before, when we wanted an image that's been thumbnailed, cropped, or modified a certain way, we would generate the image (or check if it exists) at page render time, and pass on the URL. This meant that page rendering blocked either checking if the image exists or generating a new image. Further, when our flaky file store starting freaking out it meant the entire site went down. By encoding all the relevant information in the URL, like Amazon does, we offload such work asynchronously to different web requests. If there's info in this URL we want hidden from users (spam-protected email addresses) we encrypt it using Base64-encoded AES. If the file store were to go down--which hasn't happened since we switched to using S3--users would only see broken images instead of the entire site crashing. Further, we store the images on S3 using the exact same URL as we fetch it from, and have our static web server check there first. That means if the image is cached it never hits our application stack. Now, when we generate a page with one of these dynamic images, our app stack only has to generate a URL instead of relying on the vagaries of our file store and image processing libraries. It's made our site more reliable, easier to manage, and faster. If we suddenly start serving up many more images, we could easily replace our image generation service with something higher-performance or more reliable, since it's just a web service. Right now, it works great. Clearly Amazon has seen the benefits of the same approach.
- mrkurt 17y ago
  We did something similar when we were running our own home grown publishing system, though it was only for sizing purposes (no fund stuff like discount badges). If you go this route, be very careful making the URLs too human readable. When it's obvious that you can hammer on URLs to use server resources, it becomes a natural vector for a denial of service. If we went this route again, I'd go one step further than encrypting sensitive data, and encrypt anything that can somehow result in a new image. Unfortunately, doing this safely sorta kills the user friendliness of it. It was fantastic to be able to tell authors "if you want a 100x100 image, add /100w/100h/ before the file name". I think they'd be less inclined to respond to "if you want a 100x100 image, generate a hash from the width x height x secret key and add /100w/100h/the-hash/ before the image".
  - jacquesm 17y ago
    We included a simple list of permitted sizes to protect against that. The velocity controls will block out a bot in a heartbeat anyway, but it seemed like a good thing to have.
- jacquesm 17y ago
  I use a really really dirty hack for this exact problem, I handle the rescaling in the 404 handler. It's the laziest possible approach and it has one huge hidden benefit, which is that if a bot hits your pages you're not going to end up pre-generating a bunch of scaled images that nobody is ever going to use. A simple list of allowed sizes in the handler makes sure that someone can not use URL injection to generate a large series of nonsense images for sizes that we'd normally not use. Typically a url looks like http://host/resourceid_widthxheight.jpg http://host/resourceid_widthxheight.jpg , where width and height are replaced by the desired size. I'll be the first to admit this is a nasty bit of code but I haven't found anything else that comes close in terms of efficiency and flexibility.
  - thwarted 17y ago
    I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get really good efficiency in the common case of the document existing because it is in the filesystem and execution never leaves apache core to run a CGI except in the exceptional case of having to generate it (depending on your usage patterns, of course). This is even documented in the apache docs, albeit with rewrite rules, which is more heavyweight and less efficient. http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advanced.html#on-the-fly-content http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advan... I believe that using ErrorDocument for this purpose is explicitly mentioned somewhere in the apache docs, but I can't find it right now. I'm pretty sure that' where I first learned of it (maybe back in the 1.3 days). I have a feeling that if ErrorDocument had been named (or aliased to) GenerateMissingContent, then this would be a widely accepted method of dynamic on the fly generation with filesystem caching. I've even heard a rationale that having a script generate the content on error and write it into the filesystem where apache can then read it directly on later requests is reimplementing a reverse caching proxy, and you should just stick squid in front of your web servers (with the, ahem, additional overhead of having to run and mange two services). This seems like significant effort of trying to avoid something that has "error" in the name for non-error cases.
deleted 17y ago
[deleted]
joshwa 17y ago
See also: http://www.scene7.com/solutions/dynamic_imaging.asp http://www.scene7.com/solutions/dynamic_imaging.asp - used by many huge e-commerce sites (including my >$1B employer).
liuliu 17y ago
I don't understand why Amazon has to make image cropping/resizing thing on fly. Good image resizing algorithm(cubic/seam curving) is time consuming. Even good algorithms(poisson?) for image modification(add logo) is computational expensive. Just don't get the logic why storing image is expensive than computing one.
- yan 17y ago
  "efficiency" is a vector, not a scalar. I think rolling out deals, new products, and promotions is made a lot cleaner by having that be generative. There's definitely a lot of value to having images assigned to products and not to products+transient properties like discounts.
- erydo 17y ago
  I'm sure their servers are caching the images after generation (or perhaps after 'n' requests or some other optimization in that vein). Just because the URL describes the operations performed on the image doesn't necessarily mean they always cause those operations to be performed each request.
- dangoldin 17y ago
  I imagine that because they have such a large volume of images and different combinations (20% off in one country, 30% off in another, ..) it makes sense for them to create images only when requested. They probably cache the generated images though. In addition, I think that any given day a small percentage of Amazon's products are viewed so it may not make sense to generate an image for every possible combination. They may also have a longer term view and plan on using this graphic manipulation in another product.
- tlrobinson 17y ago
  Surely they cache the results? It seems pretty smart to me. Rather than having some external process that tries to figure out every permutation of image that might be necessary they just have a simple HTTP API that always gives you exactly the image you need. As long as they're caching the image after the first request I don't see any problem.
- 17y ago
releasedatez 17y ago
Thanks for sharing this. It'll help me a lot.
bcl 17y ago
For http://www.movielandmarks.com http://www.movielandmarks.com I ended up caching the DVD images locally and serving them up myself. Some of the URL's served up by AWS go away and I don't re-request them after the initial add of a movie. I'm going to have to read this article in more detail and see if it can clean that up for me and move image serving back to Amazon. If you are going to use Amazon's images, why not sign up for an associate account and provide links back to the product pages so you can generate some income from it.
- mrkurt 17y ago
  You should be careful caching images and other data, since it's (or was) a TOS violation. :)
aarongough 17y ago
IMDB uses a similar system for retrieving/resizing their images (not wholly surprising as they are owned by Amazon)... IMDB's format for resizing images: [server hostname]/images/M/[image identifier]._V1._SX[width]_SY[HEIGHT]_.[format] eg: http://ia.media-imdb.com/images/M/MV5BNzEwNzQ1NjczM15BMl5BanBnXkFtZTcwNTk3MTE1Mg@@._V1._SX1000_SY400_.jpg http://ia.media-imdb.com/images/M/MV5BNzEwNzQ1NjczM15BMl5Ban... Unfortunately I didn't have any luck wringing meaning out of their image identification strings... I also found out rather late in the game that they use an interesting cookie based scheme for defeating people trying to link to their images from external sites...
windsurfer 17y ago
Blurring a large image is really CPU intensive. Someone could DOS amazon using a blur URL.
- NathanKP 17y ago
  It would probably cache the image. So a real DOS would require blurring multiple images at multiple levels.
  - windsurfer 17y ago
    You could just increment the blur amount. Or set a random amount on distributed clients.
qeorge 17y ago
phpThumb[1] + mod_rewrite is a simple way to get up and running with a similar cache of your own. It supports ImageMagick so you've got a lot of options. [1] http://phpthumb.sourceforge.net/ http://phpthumb.sourceforge.net/
patio11 17y ago
This approach might be totally obvious to many people here, but its quick to implement, it performs well, and it scales impressively: 1) The web server checks to see if the image exists in a file store and, if it does, serves it directly. 2) If not, the request goes to your web application stack (Rails, etc) which fires off an ImageMagick process to do whatever magick the image requests, operating on either a base image or something which you magick up at request time. 3) Your application twiddles its thumbs for a second or two. (This has potentially negative consequences for other requests coming into the same mongrel, at least in Rails. Consider doing load balancing which is aware of the mongrel being tied up -- I think Pound does this fairly easily.) 4) After you've got the image, save it to the file store and tell the web server "Send the user the file at this path". 5) The next request for the same URL will hit the webserver, find the file on disk, and stream it automatically. In my case I use a cron script every day to blow away all the images that haven't been accessed in the last X days, but depending on your needs it might be easier to just persist them indefinitely. (I use GIFs to give users a live preview of the PDF they're essentially editing, and generate one every second, so if I didn't do this I would drown in half-written bingo cards. My users go through a couple of gigs a week 20kb at a time.)
- JacobAldridge 17y ago
  Patrick, your observations, awareness, and ability / willingness to communicate them clearly continue to impress me. Cheers.
  - patio11 17y ago
    Thanks! Writing in English helps me take refreshing little breaks from, e.g., trying to figure out what the single comment left in a 3,000 line source file written in Japanese by our Korean outsourcing team means. (Answer: it means today will be a long, long day.)
- emmett 17y ago
  Re point 3: Don't use Pound. Use HAProxy, it's superior along pretty much every dimension. The idea is right though :-).
BillRossum 17y ago
Learn how to put text on a website.

159 points by chaosmachine 17y ago

chaosmachine 17y ago

See also: http://www.gertler.com/nat/abusedimages.html http://www.gertler.com/nat/abusedimages.html

albemuth 17y ago

a ruby dsl library should pop out any moment on github

mcantor 17y ago

Amazon should be paying this guy for documenting their codebase for them.

martey 17y ago
Because Amazon has no internal documentation about how to create product images? Or because Amazon's terms of services explicitly prevent doing anything useful with these images?
- chrischen 17y ago
  Because this guy was very thorough in detailing how to manipulate amazon's dynamic image generator.

stephenjudkins 17y ago

This approach to generating images generates a lot of benefits when you're working at scale. Before, when we wanted an image that's been thumbnailed, cropped, or modified a certain way, we would generate the image (or check if it exists) at page render time, and pass on the URL. This meant that page rendering blocked either checking if the image exists or generating a new image. Further, when our flaky file store starting freaking out it meant the entire site went down. By encoding all the relevant information in the URL, like Amazon does, we offload such work asynchronously to different web requests. If there's info in this URL we want hidden from users (spam-protected email addresses) we encrypt it using Base64-encoded AES. If the file store were to go down--which hasn't happened since we switched to using S3--users would only see broken images instead of the entire site crashing. Further, we store the images on S3 using the exact same URL as we fetch it from, and have our static web server check there first. That means if the image is cached it never hits our application stack. Now, when we generate a page with one of these dynamic images, our app stack only has to generate a URL instead of relying on the vagaries of our file store and image processing libraries. It's made our site more reliable, easier to manage, and faster. If we suddenly start serving up many more images, we could easily replace our image generation service with something higher-performance or more reliable, since it's just a web service. Right now, it works great. Clearly Amazon has seen the benefits of the same approach.

mrkurt 17y ago
We did something similar when we were running our own home grown publishing system, though it was only for sizing purposes (no fund stuff like discount badges). If you go this route, be very careful making the URLs too human readable. When it's obvious that you can hammer on URLs to use server resources, it becomes a natural vector for a denial of service. If we went this route again, I'd go one step further than encrypting sensitive data, and encrypt anything that can somehow result in a new image. Unfortunately, doing this safely sorta kills the user friendliness of it. It was fantastic to be able to tell authors "if you want a 100x100 image, add /100w/100h/ before the file name". I think they'd be less inclined to respond to "if you want a 100x100 image, generate a hash from the width x height x secret key and add /100w/100h/the-hash/ before the image".
- jacquesm 17y ago
  We included a simple list of permitted sizes to protect against that. The velocity controls will block out a bot in a heartbeat anyway, but it seemed like a good thing to have.
jacquesm 17y ago
I use a really really dirty hack for this exact problem, I handle the rescaling in the 404 handler. It's the laziest possible approach and it has one huge hidden benefit, which is that if a bot hits your pages you're not going to end up pre-generating a bunch of scaled images that nobody is ever going to use. A simple list of allowed sizes in the handler makes sure that someone can not use URL injection to generate a large series of nonsense images for sizes that we'd normally not use. Typically a url looks like http://host/resourceid_widthxheight.jpg http://host/resourceid_widthxheight.jpg , where width and height are replaced by the desired size. I'll be the first to admit this is a nasty bit of code but I haven't found anything else that comes close in terms of efficiency and flexibility.
- thwarted 17y ago
  I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get really good efficiency in the common case of the document existing because it is in the filesystem and execution never leaves apache core to run a CGI except in the exceptional case of having to generate it (depending on your usage patterns, of course). This is even documented in the apache docs, albeit with rewrite rules, which is more heavyweight and less efficient. http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advanced.html#on-the-fly-content http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advan... I believe that using ErrorDocument for this purpose is explicitly mentioned somewhere in the apache docs, but I can't find it right now. I'm pretty sure that' where I first learned of it (maybe back in the 1.3 days). I have a feeling that if ErrorDocument had been named (or aliased to) GenerateMissingContent, then this would be a widely accepted method of dynamic on the fly generation with filesystem caching. I've even heard a rationale that having a script generate the content on error and write it into the filesystem where apache can then read it directly on later requests is reimplementing a reverse caching proxy, and you should just stick squid in front of your web servers (with the, ahem, additional overhead of having to run and mange two services). This seems like significant effort of trying to avoid something that has "error" in the name for non-error cases.

deleted 17y ago

[deleted]

joshwa 17y ago

See also: http://www.scene7.com/solutions/dynamic_imaging.asp http://www.scene7.com/solutions/dynamic_imaging.asp - used by many huge e-commerce sites (including my >$1B employer).

liuliu 17y ago

I don't understand why Amazon has to make image cropping/resizing thing on fly. Good image resizing algorithm(cubic/seam curving) is time consuming. Even good algorithms(poisson?) for image modification(add logo) is computational expensive. Just don't get the logic why storing image is expensive than computing one.

yan 17y ago
"efficiency" is a vector, not a scalar. I think rolling out deals, new products, and promotions is made a lot cleaner by having that be generative. There's definitely a lot of value to having images assigned to products and not to products+transient properties like discounts.
erydo 17y ago
I'm sure their servers are caching the images after generation (or perhaps after 'n' requests or some other optimization in that vein). Just because the URL describes the operations performed on the image doesn't necessarily mean they always cause those operations to be performed each request.
dangoldin 17y ago
I imagine that because they have such a large volume of images and different combinations (20% off in one country, 30% off in another, ..) it makes sense for them to create images only when requested. They probably cache the generated images though. In addition, I think that any given day a small percentage of Amazon's products are viewed so it may not make sense to generate an image for every possible combination. They may also have a longer term view and plan on using this graphic manipulation in another product.
tlrobinson 17y ago
Surely they cache the results? It seems pretty smart to me. Rather than having some external process that tries to figure out every permutation of image that might be necessary they just have a simple HTTP API that always gives you exactly the image you need. As long as they're caching the image after the first request I don't see any problem.
17y ago

releasedatez 17y ago

Thanks for sharing this. It'll help me a lot.

bcl 17y ago

For http://www.movielandmarks.com http://www.movielandmarks.com I ended up caching the DVD images locally and serving them up myself. Some of the URL's served up by AWS go away and I don't re-request them after the initial add of a movie. I'm going to have to read this article in more detail and see if it can clean that up for me and move image serving back to Amazon. If you are going to use Amazon's images, why not sign up for an associate account and provide links back to the product pages so you can generate some income from it.

mrkurt 17y ago
You should be careful caching images and other data, since it's (or was) a TOS violation. :)

aarongough 17y ago

IMDB uses a similar system for retrieving/resizing their images (not wholly surprising as they are owned by Amazon)... IMDB's format for resizing images: [server hostname]/images/M/[image identifier]._V1._SX[width]_SY[HEIGHT]_.[format] eg: http://ia.media-imdb.com/images/M/MV5BNzEwNzQ1NjczM15BMl5BanBnXkFtZTcwNTk3MTE1Mg@@._V1._SX1000_SY400_.jpg http://ia.media-imdb.com/images/M/MV5BNzEwNzQ1NjczM15BMl5Ban... Unfortunately I didn't have any luck wringing meaning out of their image identification strings... I also found out rather late in the game that they use an interesting cookie based scheme for defeating people trying to link to their images from external sites...

windsurfer 17y ago

Blurring a large image is really CPU intensive. Someone could DOS amazon using a blur URL.

NathanKP 17y ago
It would probably cache the image. So a real DOS would require blurring multiple images at multiple levels.
- windsurfer 17y ago
  You could just increment the blur amount. Or set a random amount on distributed clients.

qeorge 17y ago

phpThumb[1] + mod_rewrite is a simple way to get up and running with a similar cache of your own. It supports ImageMagick so you've got a lot of options. [1] http://phpthumb.sourceforge.net/ http://phpthumb.sourceforge.net/

patio11 17y ago

This approach might be totally obvious to many people here, but its quick to implement, it performs well, and it scales impressively: 1) The web server checks to see if the image exists in a file store and, if it does, serves it directly. 2) If not, the request goes to your web application stack (Rails, etc) which fires off an ImageMagick process to do whatever magick the image requests, operating on either a base image or something which you magick up at request time. 3) Your application twiddles its thumbs for a second or two. (This has potentially negative consequences for other requests coming into the same mongrel, at least in Rails. Consider doing load balancing which is aware of the mongrel being tied up -- I think Pound does this fairly easily.) 4) After you've got the image, save it to the file store and tell the web server "Send the user the file at this path". 5) The next request for the same URL will hit the webserver, find the file on disk, and stream it automatically. In my case I use a cron script every day to blow away all the images that haven't been accessed in the last X days, but depending on your needs it might be easier to just persist them indefinitely. (I use GIFs to give users a live preview of the PDF they're essentially editing, and generate one every second, so if I didn't do this I would drown in half-written bingo cards. My users go through a couple of gigs a week 20kb at a time.)

JacobAldridge 17y ago
Patrick, your observations, awareness, and ability / willingness to communicate them clearly continue to impress me. Cheers.
- patio11 17y ago
  Thanks! Writing in English helps me take refreshing little breaks from, e.g., trying to figure out what the single comment left in a 3,000 line source file written in Japanese by our Korean outsourcing team means. (Answer: it means today will be a long, long day.)
emmett 17y ago
Re point 3: Don't use Pound. Use HAProxy, it's superior along pretty much every dimension. The idea is right though :-).

BillRossum 17y ago

Learn how to put text on a website.