4 ms·

More generally: once you adopt any of the various schemes for having a inbound proxy/front-end cache (Fastly, CloudFlare, CloudFront, or an in-house varnish/squ

by gojomo 14y ago

More generally: once you adopt any of the various schemes for having a inbound proxy/front-end cache (Fastly, CloudFlare, CloudFront, or an in-house varnish/squid/etc), are all the optimizing habits of moving static assets to a dedicated server now superfluous? I think those optimizing habits are now obsolete: best practice is to have a front-end cache. A corollary is that we usually needn't worry about a dynamic framework serving large static assets: the front-end cache ensures it happens rarely. Unfortunately it's still the doctrine of some projects that a production project will always offload static-serving. So for example, the Django docs are filled with much traditional discouragement around using the staticfiles serving app in production, including vague 'this is insecure' intimations. In fact, once you're using a front-end cache, there's little speed/efficiency reason to avoid that practice. And, if it is specifically supected to be insecure, any such insecurity should to be fixed rather than overlooked simply because "it's not used in production".

ra 14y ago
I think in Django's case, it's not so much "this is insecure", but more that runserver is simple/stupid and hasn't been built or tested for serving multiple concurrent users. Nor should it be. There are plenty of WSGI capable web servers that can be installed into a Django project as apps. If you want to run just Django behind a frontside cache and server static media from the same connection, simply install an appropriate app. Gunicorn and cherrypy both 'drop in replacements' for the built in runserver and both well up to the task.
- gojomo 14y ago
  Yes, but I wasn't talking about the admonitions against runserver -- I was talking about the admonitions against the staticfiles app. I propose that staticfiles under gunicorn is a reasonable choice. Further, if fronted by a inbound proxy cache, it could perhaps even be the reference/recommended setup for high-volume production sites, rather than the current doctrine that such sites should have some extra static-collect/export to a helper server.
slurgfest 14y ago
Tools like Apache and nginx are not ONLY faster at serving files with less load on the system than a script. They are also more thoroughly audited and battle-tested. And their declarative configs won't go wrong just because the person writing them missed an unbelievably subtle corner case introduced by using a Turing-complete language. It's important because there are so many opportunities for error in serving arbitrary files out of a filesystem with some rough and ready script. For example, if you are serving files out of the same filesystem that holds your configs and secret keys then you should be a bit nervous. You have to get the permissions right and make sure you don't have anything improper under a directory which you are publishing as a whole. If your users are uploading files to the same place you should feel really nervous. There are too many easy ways for people to be negligent and screw this up. In the context of designing an opinionated framework, you accept a lot of social liability and you are really dropping the ball if you are setting up tired and ignorant users to screw up this badly, without even a warning in the docs to think about what you are doing. With n script languages and m static file serving implementations per language, there are now (n*m) obscure packages to audit. Not counting their combinations... Your idea to just "fix the insecurity" and remove any warning from the docs means to do things which you merely believe to fix the insecurity, and then overlook the underlying risks of the approach. I am also not sure you are right when you suggest that there cannot be any performance (or reliability) impact of pushing static serving into some script library. Just as these are not audited they are also not nearly as likely to be benchmarked and tuned. If there is a reason to serve static files out of script, that reason will be because of some positive reason (like convenience or the need for some particular flexibility) rather than some vague sense that using Apache is "obsolete".
- gojomo 14y ago
  Here's the thing: Django makes the standard staticfiles app available. It's a great convenience, eliminating extra steps (collecting/moving static assets) and processes (another nginx/etc). And many projects' 'dev'/prototype incarnations are already open to the world, in one way or another. So if this is a security sin, they've already encouraged its widespread commission. A bit of "don't do this" or "don't do this in production" hand-waving in the docs don't resolve a security problem, if there's a real vulnerability in the current implementation. On the other hand, committing to the idea that the bundled staticfiles app may be used this way -- that in fact it's a good and modern way to operate, in production, once you have an inbound proxy cache -- would mean accepting deeper responsibility. It would give up the hedge, "if there's a security bug, we warned you!". It's not taking on n*m obligations: it's taking on 1 language, 1 module. And it's not even a new module or an obscure need... it's exactly the sort of thing an opinionated framework can solve for people. The old opinion -- "take this risk in development, but by the time you get to production use the 'best practice' of a separate static server" -- should be updated to a new opinion -- "the 'best practice' is now a front-end proxy cache, which makes the performance benefits of an extra static server negligible, so we're no longer going to assume everyone will do that in production". An admonition against using other less-tested code to achieve the same effect would still be appropriate. But not nonspecific FUD about the framework's own code -- that it is "probably insecure". Anything that's truly "probably insecure" ought to be fixed.
peterbe 14y ago
Regarding Django and staticfiles; Rightly so, because they're not ready for production and being taken over my the CDN. You need to sprinkle some django_compressor on it first but still that doesn't get the cache headers perfectly right. Or the gzipping.
- gojomo 14y ago
  Wouldn't it be great if a framework got the headers/compression right without sprinking extra options in?

by gojomo 14y ago

ra 14y ago

I think in Django's case, it's not so much "this is insecure", but more that runserver is simple/stupid and hasn't been built or tested for serving multiple concurrent users. Nor should it be. There are plenty of WSGI capable web servers that can be installed into a Django project as apps. If you want to run just Django behind a frontside cache and server static media from the same connection, simply install an appropriate app. Gunicorn and cherrypy both 'drop in replacements' for the built in runserver and both well up to the task.

gojomo 14y ago
Yes, but I wasn't talking about the admonitions against runserver -- I was talking about the admonitions against the staticfiles app. I propose that staticfiles under gunicorn is a reasonable choice. Further, if fronted by a inbound proxy cache, it could perhaps even be the reference/recommended setup for high-volume production sites, rather than the current doctrine that such sites should have some extra static-collect/export to a helper server.

slurgfest 14y ago

Tools like Apache and nginx are not ONLY faster at serving files with less load on the system than a script. They are also more thoroughly audited and battle-tested. And their declarative configs won't go wrong just because the person writing them missed an unbelievably subtle corner case introduced by using a Turing-complete language. It's important because there are so many opportunities for error in serving arbitrary files out of a filesystem with some rough and ready script. For example, if you are serving files out of the same filesystem that holds your configs and secret keys then you should be a bit nervous. You have to get the permissions right and make sure you don't have anything improper under a directory which you are publishing as a whole. If your users are uploading files to the same place you should feel really nervous. There are too many easy ways for people to be negligent and screw this up. In the context of designing an opinionated framework, you accept a lot of social liability and you are really dropping the ball if you are setting up tired and ignorant users to screw up this badly, without even a warning in the docs to think about what you are doing. With n script languages and m static file serving implementations per language, there are now (n*m) obscure packages to audit. Not counting their combinations... Your idea to just "fix the insecurity" and remove any warning from the docs means to do things which you merely believe to fix the insecurity, and then overlook the underlying risks of the approach. I am also not sure you are right when you suggest that there cannot be any performance (or reliability) impact of pushing static serving into some script library. Just as these are not audited they are also not nearly as likely to be benchmarked and tuned. If there is a reason to serve static files out of script, that reason will be because of some positive reason (like convenience or the need for some particular flexibility) rather than some vague sense that using Apache is "obsolete".

gojomo 14y ago
Here's the thing: Django makes the standard staticfiles app available. It's a great convenience, eliminating extra steps (collecting/moving static assets) and processes (another nginx/etc). And many projects' 'dev'/prototype incarnations are already open to the world, in one way or another. So if this is a security sin, they've already encouraged its widespread commission. A bit of "don't do this" or "don't do this in production" hand-waving in the docs don't resolve a security problem, if there's a real vulnerability in the current implementation. On the other hand, committing to the idea that the bundled staticfiles app may be used this way -- that in fact it's a good and modern way to operate, in production, once you have an inbound proxy cache -- would mean accepting deeper responsibility. It would give up the hedge, "if there's a security bug, we warned you!". It's not taking on n*m obligations: it's taking on 1 language, 1 module. And it's not even a new module or an obscure need... it's exactly the sort of thing an opinionated framework can solve for people. The old opinion -- "take this risk in development, but by the time you get to production use the 'best practice' of a separate static server" -- should be updated to a new opinion -- "the 'best practice' is now a front-end proxy cache, which makes the performance benefits of an extra static server negligible, so we're no longer going to assume everyone will do that in production". An admonition against using other less-tested code to achieve the same effect would still be appropriate. But not nonspecific FUD about the framework's own code -- that it is "probably insecure". Anything that's truly "probably insecure" ought to be fixed.

peterbe 14y ago

Regarding Django and staticfiles; Rightly so, because they're not ready for production and being taken over my the CDN. You need to sprinkle some django_compressor on it first but still that doesn't get the cache headers perfectly right. Or the gzipping.

gojomo 14y ago
Wouldn't it be great if a framework got the headers/compression right without sprinking extra options in?