3 ms·

I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get r

I don't know why so many people consider this to be a "dirty hack". It makes total sense to overload ErrorDocument and generate the missing content. You get really good efficiency in the common case of the document existing because it is in the filesystem and execution never leaves apache core to run a CGI except in the exceptional case of having to generate it (depending on your usage patterns, of course). This is even documented in the apache docs, albeit with rewrite rules, which is more heavyweight and less efficient. http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advanced.html#on-the-fly-content http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide_advan... I believe that using ErrorDocument for this purpose is explicitly mentioned somewhere in the apache docs, but I can't find it right now. I'm pretty sure that' where I first learned of it (maybe back in the 1.3 days). I have a feeling that if ErrorDocument had been named (or aliased to) GenerateMissingContent, then this would be a widely accepted method of dynamic on the fly generation with filesystem caching. I've even heard a rationale that having a script generate the content on error and write it into the filesystem where apache can then read it directly on later requests is reimplementing a reverse caching proxy, and you should just stick squid in front of your web servers (with the, ahem, additional overhead of having to run and mange two services). This seems like significant effort of trying to avoid something that has "error" in the name for non-error cases.

jacquesm 17y ago
If there would have been a more appropriately named hook such as 'about_to_404_last_chance_to_make_content' I'm sure I would have used that instead :) The resulting image is indeed written to the filesystem, right next to the non-scaled one (so when we remove we can remove all of them in one go without having to visit another directory). The really nasty trick is that when all is done I redirect the browser to the same url and this time I'm sure it will find it. And if the source file is missing it really does 404, of course. The only tricky situation is when you have a whole pile of people requesting the same image at the same time and it hasn't been saved yet, the first party to do the 'fopen' gets dibs on doing the transform, everybody else simply redirects if the file exists, by the time the browsers have processed the 3xx the file is ready for consumption. On very rare occasions the race is so close that the image gets rescaled and saved twice, I've logged the occurence of this over many millions of rescales and it only happened a handful of times, not worth improving on.
- thwarted 17y ago
  The really nasty trick is that when all is done I redirect the browser to the same url and this time I'm sure it will find it. This is a pretty nasty trick (seems like a risk of a redirect loop if the file can't be written), but unnecessary. You can emit content out of the script used for ErrorDocument the same as you can with a CGI. So you generate the content, write it to the filesystem, emit a "Status: 200" header, which replaces the 404 status apache was going to generate, then send a content-type and the generated content in the response body. Your way may be easier if you have complex cache control or expires headers that apache is configured to ad to static files but not dynamically generated content, since the client is only ever given the file by apache directly from the filesystem. I've never used that though.
  - drusenko 17y ago
    we use this exact approach to generate all of our blog pages. incredibly efficient. the twist we use to handle race conditions is to create a lock file. if the 404 handler is called and the lock file exists, someone else is generating that page as we speak -- it goes into a timed loop to check for the file being available. when found, it serves it directly with the 200 header, like you mentioned. we have a maximum loop value (a second, i think) -- if you hit that, it means the other process died for some reason, and never deleted the lock or the page. in that case, generate and write the page yourself, and remove the lock. all future requests are served off of the file system. one nice part of this caching mechanism is that to remove the cache, you just rm -rf the blog directory. tada!
  - jacquesm 17y ago
    No risk of a loop, the file is guaranteed to exist when it's done, the redirect happens just after the fclose on the 'write' (and only if the file is written successfully). That way the browser will see the exact same headers it would on a request where the file did exist.

jacquesm 17y ago

If there would have been a more appropriately named hook such as 'about_to_404_last_chance_to_make_content' I'm sure I would have used that instead :) The resulting image is indeed written to the filesystem, right next to the non-scaled one (so when we remove we can remove all of them in one go without having to visit another directory). The really nasty trick is that when all is done I redirect the browser to the same url and this time I'm sure it will find it. And if the source file is missing it really does 404, of course. The only tricky situation is when you have a whole pile of people requesting the same image at the same time and it hasn't been saved yet, the first party to do the 'fopen' gets dibs on doing the transform, everybody else simply redirects if the file exists, by the time the browsers have processed the 3xx the file is ready for consumption. On very rare occasions the race is so close that the image gets rescaled and saved twice, I've logged the occurence of this over many millions of rescales and it only happened a handful of times, not worth improving on.

thwarted 17y ago
The really nasty trick is that when all is done I redirect the browser to the same url and this time I'm sure it will find it. This is a pretty nasty trick (seems like a risk of a redirect loop if the file can't be written), but unnecessary. You can emit content out of the script used for ErrorDocument the same as you can with a CGI. So you generate the content, write it to the filesystem, emit a "Status: 200" header, which replaces the 404 status apache was going to generate, then send a content-type and the generated content in the response body. Your way may be easier if you have complex cache control or expires headers that apache is configured to ad to static files but not dynamically generated content, since the client is only ever given the file by apache directly from the filesystem. I've never used that though.
- drusenko 17y ago
  we use this exact approach to generate all of our blog pages. incredibly efficient. the twist we use to handle race conditions is to create a lock file. if the 404 handler is called and the lock file exists, someone else is generating that page as we speak -- it goes into a timed loop to check for the file being available. when found, it serves it directly with the 200 header, like you mentioned. we have a maximum loop value (a second, i think) -- if you hit that, it means the other process died for some reason, and never deleted the lock or the page. in that case, generate and write the page yourself, and remove the lock. all future requests are served off of the file system. one nice part of this caching mechanism is that to remove the cache, you just rm -rf the blog directory. tada!
- jacquesm 17y ago
  No risk of a loop, the file is guaranteed to exist when it's done, the redirect happens just after the fclose on the 'write' (and only if the file is written successfully). That way the browser will see the exact same headers it would on a request where the file did exist.