5 ms·

How to serve Django Statics (and not go insane)

53 points by ashrust 14y ago

ashray 14y ago
Nice post, tons of great advice. However, I have to say that the title is wrong. There's nothing django specific about any of those approaches. I've tried django-compress and it was a nightmare, the old style synccompress was actually easier to setup and get working for some reason.. Was hoping for a better rolled solution. Also, the part about cloudfront isn't written very clearly. I had to stop for a moment and think about what it meant. Great idea nonetheless. S3's gzip support sucks. For some odd reason (I don't support IE6..) the gzip from S3 was breaking on IE9. Worked fine on 8 and 10. Broke on 9. =/
- rdpfeffer 14y ago
  You're right, most of this stuff is not django specific. But since we're using it, we thought we'd let everyone know since its likely the most popular python web framework.
- rdpfeffer 14y ago
  Also, I can't tell you how much I hate that "cloudflare" sounds so much like "cloudfront". I've been in at least 6 convos where people were mixing one for the other. Esp, since they solve some of the same problems.
  - ashray 14y ago
    Actually that's exactly what happened to me! :P I thought oh so cloudflare can fix this, eh. I should just reverse proxy my stuff through cloudflare. Then I realized, oh they said cloudfront. So you could still technically keep your stuff in your S3 bucket, though you'd end up paying more for S3 originating transfer as well. In this case, cloudFRONT (the non amazon one!!) may be the better choice. Hate that they sound the same. I even get my OWN THOUGHTS mixed up sometimes :(
    - rdpfeffer 14y ago
      Hilarious, I wonder how much they gain from devs getting mixed up on recommendations. Probably better for Cloudflare if you ask me. Personally, I don't think it makes sense to use CloudFront if you are already using cloudflare. Its easier to use if your just talking web statics.
  - rdl 14y ago
    Just don't get either confused with "stormfront".
jaytaylor 14y ago
You know, I've always found django-compressor to be adequate for my django statics needs. Was there a real need to reinvent the wheel in this case? Were you simply unable to configure django-compressor (or any other pre-existing django statics library) properly?
- rdpfeffer 14y ago
  We got it working but there we're a TON of drawbacks that made it not worth the effort. Here are the top three. 1. The config was overly complex and not flexible enough for our needs. 2. It added extra, and unnecessary deploy steps which slowed down our deployments. 3. Configuring GZIP on S3 and Cloudfront is prohibitively complex. CloudFlare on the other hand, solves many of these problems for us.
  - jaytaylor 14y ago
    In the article you referenced CloudFront, but now you're saying CloudFlare.. so which is it?
    - rdpfeffer 14y ago
      The article has been modified to clarify which service we are using as a reverse proxy and why.
  - ashray 14y ago
    Hang on! So your article says "CDN to cache them (which CloudFront supports)" but you're actually using CloudFLARE ? :O We're all so confused by this! Haha :D
    - cdoxsey 14y ago
      Both CloudFlare and CloudFront can do this. http://aws.typepad.com/aws/2010/11/amazon-cloudfront-support-for-custom-origins.html http://aws.typepad.com/aws/2010/11/amazon-cloudfront-support...
mikeknoop 14y ago
I've gotten https://github.com/cyberdelia/django-pipeline https://github.com/cyberdelia/django-pipeline working recently. It's a young project but it's supporting a heavy frontend asset pipeline for us now and working great.
- jaytaylor 14y ago
  I've tried django-pipeline and it seemed great when I started, but ultimately I had to abandon it because I discovered it would've taken an inordinate amount of work to get it compressing the CSS/JS assets efficiently. Out of the box (just following the documentations suggestions) the django-pipeline compressed JS was 5-10X the size that I get after django-compressor is done with it (and with compressor I didn't have to do any tuning or optimization beyond basic configuration). At any rate, I'm happy that you were able to get it working, Mike! That is really neat. Do you think you'll write a blog post about how you did it?
deleted 14y ago
[deleted]
IgorPartola 14y ago
Anyone know what proxies still don't like foo.js?v=1 style URLs? I thought we were past this. Also, a static build process (a la require.js) solves a lot of these for you in a way that's not specific to Django or any other framework. You also don't end up bloating your web app code with the concern for how static files are to be processed, which I personally like a lot. Another way to go is to use Google's mod_pagespeed [1], which once again is not framework-specific. Lastly, you can try another trick where you pre-generate .gz versions of all the files too, to really speed things up. It's nice not to have to do things on the fly and web servers like nginx can take quite a lot of traffic serving static files, so you can hold off on going the CDN route, unless of course geography matters more to you than offloading server resources. [1] https://developers.google.com/speed/pagespeed/mod https://developers.google.com/speed/pagespeed/mod
- cdoxsey 14y ago
  "Most proxies, most notably Squid up through version 3.0, do not cache resources with a "?" in their URL even if a Cache-control: public header is present in the response. To enable proxy caching for these resources, remove query strings from references to static resources, and instead encode the parameters into the file names themselves." https://developers.google.com/speed/docs/best-practices/caching#LeverageProxyCaching https://developers.google.com/speed/docs/best-practices/cach...
erikcw 14y ago
I've been using CloudFlare on a project and have been mostly very pleased with it. The only persistant problem that I haven't been able to iron out is that some non-US users of my application can't get any of our javascript assets to load in Google Chrome correctly. VPNing into a US IP seems to fix the problem. Has anyone else had any issues like this?
- jaytaylor 14y ago
  That is very interesting. I am very keen to know if this is a widespread problem. Have you filed a ticket with CloudFlare or contacted them about this problem?
  - erikcw 14y ago
    Yes, I opened a ticket with them a few weeks ago. Still working with them on a resolution.
stefantalpalaru 14y ago
How often do you hash the static files? Each time you generate the URL sounds like too much overhead.
- cdoxsey 14y ago
  In production we only compute the hash on deploy.
citricsquid 14y ago
Please label your chart at the top of the post, not sure what the lines are.
csense 14y ago
> breaking anyone who doesn’t have support for GZIP What client doesn't have support for gzip?
- dadams74 14y ago
  Should not matter. The server should only send compressed data if the clients sends a request with the header "Accept-Encoding: gzip, deflate", which indicates that there is support for compressed responses.
amccloud 14y ago
I've done something similar to the hash in the static url. Used the git commit hash to break cache. Nginx would see the hash in the url and just ignore it. Cloudfront would see the hash in the user and thinks it's a new file. https://gist.github.com/4129151 https://gist.github.com/4129151
antihero 14y ago
I've been using django-require in place of compressor and it's fantastic. Also the AMD pattern of loading js seems to be the future.
antihero 14y ago
I might be completely wrong, but due to HTTP/1.1 and Pipelining, surely it's quicker to serve [small] static files from the same server as your page, because you don't have the overhead of re-establishing a connection?

53 points by ashrust 14y ago

Nice post, tons of great advice. However, I have to say that the title is wrong. There's nothing django specific about any of those approaches. I've tried django-compress and it was a nightmare, the old style synccompress was actually easier to setup and get working for some reason.. Was hoping for a better rolled solution. Also, the part about cloudfront isn't written very clearly. I had to stop for a moment and think about what it meant. Great idea nonetheless. S3's gzip support sucks. For some odd reason (I don't support IE6..) the gzip from S3 was breaking on IE9. Worked fine on 8 and 10. Broke on 9. =/

rdpfeffer 14y ago
You're right, most of this stuff is not django specific. But since we're using it, we thought we'd let everyone know since its likely the most popular python web framework.
rdpfeffer 14y ago
Also, I can't tell you how much I hate that "cloudflare" sounds so much like "cloudfront". I've been in at least 6 convos where people were mixing one for the other. Esp, since they solve some of the same problems.
- ashray 14y ago
  Actually that's exactly what happened to me! :P I thought oh so cloudflare can fix this, eh. I should just reverse proxy my stuff through cloudflare. Then I realized, oh they said cloudfront. So you could still technically keep your stuff in your S3 bucket, though you'd end up paying more for S3 originating transfer as well. In this case, cloudFRONT (the non amazon one!!) may be the better choice. Hate that they sound the same. I even get my OWN THOUGHTS mixed up sometimes :(
  - rdpfeffer 14y ago
    Hilarious, I wonder how much they gain from devs getting mixed up on recommendations. Probably better for Cloudflare if you ask me. Personally, I don't think it makes sense to use CloudFront if you are already using cloudflare. Its easier to use if your just talking web statics.
- rdl 14y ago
  Just don't get either confused with "stormfront".

jaytaylor 14y ago

You know, I've always found django-compressor to be adequate for my django statics needs. Was there a real need to reinvent the wheel in this case? Were you simply unable to configure django-compressor (or any other pre-existing django statics library) properly?

rdpfeffer 14y ago
We got it working but there we're a TON of drawbacks that made it not worth the effort. Here are the top three. 1. The config was overly complex and not flexible enough for our needs. 2. It added extra, and unnecessary deploy steps which slowed down our deployments. 3. Configuring GZIP on S3 and Cloudfront is prohibitively complex. CloudFlare on the other hand, solves many of these problems for us.
- jaytaylor 14y ago
  In the article you referenced CloudFront, but now you're saying CloudFlare.. so which is it?
  - rdpfeffer 14y ago
    The article has been modified to clarify which service we are using as a reverse proxy and why.
- ashray 14y ago
  Hang on! So your article says "CDN to cache them (which CloudFront supports)" but you're actually using CloudFLARE ? :O We're all so confused by this! Haha :D
  - cdoxsey 14y ago
    Both CloudFlare and CloudFront can do this. http://aws.typepad.com/aws/2010/11/amazon-cloudfront-support-for-custom-origins.html http://aws.typepad.com/aws/2010/11/amazon-cloudfront-support...

mikeknoop 14y ago

I've gotten https://github.com/cyberdelia/django-pipeline https://github.com/cyberdelia/django-pipeline working recently. It's a young project but it's supporting a heavy frontend asset pipeline for us now and working great.

jaytaylor 14y ago
I've tried django-pipeline and it seemed great when I started, but ultimately I had to abandon it because I discovered it would've taken an inordinate amount of work to get it compressing the CSS/JS assets efficiently. Out of the box (just following the documentations suggestions) the django-pipeline compressed JS was 5-10X the size that I get after django-compressor is done with it (and with compressor I didn't have to do any tuning or optimization beyond basic configuration). At any rate, I'm happy that you were able to get it working, Mike! That is really neat. Do you think you'll write a blog post about how you did it?

deleted 14y ago

[deleted]

IgorPartola 14y ago

Anyone know what proxies still don't like foo.js?v=1 style URLs? I thought we were past this. Also, a static build process (a la require.js) solves a lot of these for you in a way that's not specific to Django or any other framework. You also don't end up bloating your web app code with the concern for how static files are to be processed, which I personally like a lot. Another way to go is to use Google's mod_pagespeed [1], which once again is not framework-specific. Lastly, you can try another trick where you pre-generate .gz versions of all the files too, to really speed things up. It's nice not to have to do things on the fly and web servers like nginx can take quite a lot of traffic serving static files, so you can hold off on going the CDN route, unless of course geography matters more to you than offloading server resources. [1] https://developers.google.com/speed/pagespeed/mod https://developers.google.com/speed/pagespeed/mod

cdoxsey 14y ago
"Most proxies, most notably Squid up through version 3.0, do not cache resources with a "?" in their URL even if a Cache-control: public header is present in the response. To enable proxy caching for these resources, remove query strings from references to static resources, and instead encode the parameters into the file names themselves." https://developers.google.com/speed/docs/best-practices/caching#LeverageProxyCaching https://developers.google.com/speed/docs/best-practices/cach...

erikcw 14y ago

I've been using CloudFlare on a project and have been mostly very pleased with it. The only persistant problem that I haven't been able to iron out is that some non-US users of my application can't get any of our javascript assets to load in Google Chrome correctly. VPNing into a US IP seems to fix the problem. Has anyone else had any issues like this?

jaytaylor 14y ago
That is very interesting. I am very keen to know if this is a widespread problem. Have you filed a ticket with CloudFlare or contacted them about this problem?
- erikcw 14y ago
  Yes, I opened a ticket with them a few weeks ago. Still working with them on a resolution.

stefantalpalaru 14y ago

How often do you hash the static files? Each time you generate the URL sounds like too much overhead.

cdoxsey 14y ago
In production we only compute the hash on deploy.

citricsquid 14y ago

Please label your chart at the top of the post, not sure what the lines are.

csense 14y ago

> breaking anyone who doesn’t have support for GZIP What client doesn't have support for gzip?

dadams74 14y ago
Should not matter. The server should only send compressed data if the clients sends a request with the header "Accept-Encoding: gzip, deflate", which indicates that there is support for compressed responses.

amccloud 14y ago

I've done something similar to the hash in the static url. Used the git commit hash to break cache. Nginx would see the hash in the url and just ignore it. Cloudfront would see the hash in the user and thinks it's a new file. https://gist.github.com/4129151 https://gist.github.com/4129151

antihero 14y ago

I've been using django-require in place of compressor and it's fantastic. Also the AMD pattern of loading js seems to be the future.

antihero 14y ago

I might be completely wrong, but due to HTTP/1.1 and Pipelining, surely it's quicker to serve [small] static files from the same server as your page, because you don't have the overhead of re-establishing a connection?