4 ms·

This approach to generating images generates a lot of benefits when you're working at scale. Before, when we wanted an image that's been thumbnailed, cropped,

This approach to generating images generates a lot of benefits when you're working at scale. Before, when we wanted an image that's been thumbnailed, cropped, or modified a certain way, we would generate the image (or check if it exists) at page render time, and pass on the URL. This meant that page rendering blocked either checking if the image exists or generating a new image. Further, when our flaky file store starting freaking out it meant the entire site went down. By encoding all the relevant information in the URL, like Amazon does, we offload such work asynchronously to different web requests. If there's info in this URL we want hidden from users (spam-protected email addresses) we encrypt it using Base64-encoded AES. If the file store were to go down--which hasn't happened since we switched to using S3--users would only see broken images instead of the entire site crashing. Further, we store the images on S3 using the exact same URL as we fetch it from, and have our static web server check there first. That means if the image is cached it never hits our application stack. Now, when we generate a page with one of these dynamic images, our app stack only has to generate a URL instead of relying on the vagaries of our file store and image processing libraries. It's made our site more reliable, easier to manage, and faster. If we suddenly start serving up many more images, we could easily replace our image generation service with something higher-performance or more reliable, since it's just a web service. Right now, it works great. Clearly Amazon has seen the benefits of the same approach.

mrkurt 17y ago
We did something similar when we were running our own home grown publishing system, though it was only for sizing purposes (no fund stuff like discount badges). If you go this route, be very careful making the URLs too human readable. When it's obvious that you can hammer on URLs to use server resources, it becomes a natural vector for a denial of service. If we went this route again, I'd go one step further than encrypting sensitive data, and encrypt anything that can somehow result in a new image. Unfortunately, doing this safely sorta kills the user friendliness of it. It was fantastic to be able to tell authors "if you want a 100x100 image, add /100w/100h/ before the file name". I think they'd be less inclined to respond to "if you want a 100x100 image, generate a hash from the width x height x secret key and add /100w/100h/the-hash/ before the image".
- jacquesm 17y ago
  We included a simple list of permitted sizes to protect against that. The velocity controls will block out a bot in a heartbeat anyway, but it seemed like a good thing to have.
jacquesm 17y ago
I use a really really dirty hack for this exact problem, I handle the rescaling in the 404 handler. It's the laziest possible approach and it has one huge hidden benefit, which is that if a bot hits your pages you're not going to end up pre-generating a bunch of scaled images that nobody is ever going to use. A simple list of allowed sizes in the handler makes sure that someone can not use URL injection to generate a large series of nonsense images for sizes that we'd normally not use. Typically a url looks like http://host/resourceid_widthxheight.jpg http://host/resourceid_widthxheight.jpg , where width and height are replaced by the desired size. I'll be the first to admit this is a nasty bit of code but I haven't found anything else that comes close in terms of efficiency and flexibility.
- thwarted 17y ago
  I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get really good efficiency in the common case of the document existing because it is in the filesystem and execution never leaves apache core to run a CGI except in the exceptional case of having to generate it (depending on your usage patterns, of course). This is even documented in the apache docs, albeit with rewrite rules, which is more heavyweight and less efficient. http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advanced.html#on-the-fly-content http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advan... I believe that using ErrorDocument for this purpose is explicitly mentioned somewhere in the apache docs, but I can't find it right now. I'm pretty sure that' where I first learned of it (maybe back in the 1.3 days). I have a feeling that if ErrorDocument had been named (or aliased to) GenerateMissingContent, then this would be a widely accepted method of dynamic on the fly generation with filesystem caching. I've even heard a rationale that having a script generate the content on error and write it into the filesystem where apache can then read it directly on later requests is reimplementing a reverse caching proxy, and you should just stick squid in front of your web servers (with the, ahem, additional overhead of having to run and mange two services). This seems like significant effort of trying to avoid something that has "error" in the name for non-error cases.
  - jacquesm 17y ago

mrkurt 17y ago

We did something similar when we were running our own home grown publishing system, though it was only for sizing purposes (no fund stuff like discount badges). If you go this route, be very careful making the URLs too human readable. When it's obvious that you can hammer on URLs to use server resources, it becomes a natural vector for a denial of service. If we went this route again, I'd go one step further than encrypting sensitive data, and encrypt anything that can somehow result in a new image. Unfortunately, doing this safely sorta kills the user friendliness of it. It was fantastic to be able to tell authors "if you want a 100x100 image, add /100w/100h/ before the file name". I think they'd be less inclined to respond to "if you want a 100x100 image, generate a hash from the width x height x secret key and add /100w/100h/the-hash/ before the image".

jacquesm 17y ago
We included a simple list of permitted sizes to protect against that. The velocity controls will block out a bot in a heartbeat anyway, but it seemed like a good thing to have.

jacquesm 17y ago

I use a really really dirty hack for this exact problem, I handle the rescaling in the 404 handler. It's the laziest possible approach and it has one huge hidden benefit, which is that if a bot hits your pages you're not going to end up pre-generating a bunch of scaled images that nobody is ever going to use. A simple list of allowed sizes in the handler makes sure that someone can not use URL injection to generate a large series of nonsense images for sizes that we'd normally not use. Typically a url looks like http://host/resourceid_widthxheight.jpg http://host/resourceid_widthxheight.jpg , where width and height are replaced by the desired size. I'll be the first to admit this is a nasty bit of code but I haven't found anything else that comes close in terms of efficiency and flexibility.

thwarted 17y ago
I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get really good efficiency in the common case of the document existing because it is in the filesystem and execution never leaves apache core to run a CGI except in the exceptional case of having to generate it (depending on your usage patterns, of course). This is even documented in the apache docs, albeit with rewrite rules, which is more heavyweight and less efficient. http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advanced.html#on-the-fly-content http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advan... I believe that using ErrorDocument for this purpose is explicitly mentioned somewhere in the apache docs, but I can't find it right now. I'm pretty sure that' where I first learned of it (maybe back in the 1.3 days). I have a feeling that if ErrorDocument had been named (or aliased to) GenerateMissingContent, then this would be a widely accepted method of dynamic on the fly generation with filesystem caching. I've even heard a rationale that having a script generate the content on error and write it into the filesystem where apache can then read it directly on later requests is reimplementing a reverse caching proxy, and you should just stick squid in front of your web servers (with the, ahem, additional overhead of having to run and mange two services). This seems like significant effort of trying to avoid something that has "error" in the name for non-error cases.
- jacquesm 17y ago