12 ms·

Scaling Django to 8 Billion Page Views

316 points by mattrobenolt 13y ago

ddorian43 13y ago
Scaling varnish to 30K/sec. Scaling Django to 10K-15K/sec. I remember long time ago:"How i scaled drupal(large number of queries/page) to 3K/pages/sec". It was really Varnish that scaled.
- bliti 13y ago
  Still, Django at 15K is good.
- mattrobenolt 13y ago
  Sure, but the idea is knowing how and when to use your tools properly and what they're good for. It allows us to continue using Django.
- eli 13y ago
  Out of the box Drupal didn't used to play well with Varnish, so that was not a trivial task even if Varnish did the heavy lifting. For serving a cached asset from memory it's going to be really hard to beat Varnish on speed.
bliti 13y ago
15K requests with Django is a nice number to hit. What are you using as a server? Gunicorn? Not a bad setup. load balancers --> Vanquish: if !cache: --> Django Still makes me wonder how faster this would be in Go or Java. I've never been able to get Python to be very efficient over 5K requests.
- mattrobenolt 13y ago
  uwsgi
  - bliti 13y ago
    Gevent is a bit funny sometimes. How are you dealing with it?
    - mattrobenolt 13y ago
      We don't run gevent for disqus.com. We do some some background workers and whatnot, but our main app is single threaded with a large number of processes on each machine.
      - bliti 13y ago
        Interesting. I'd love to know about the specs of your hardware. Maybe a future blog post about that?
    - est 13y ago
      If you really need gevent that badly http://uwsgi-docs.readthedocs.org/en/latest/Gevent.html http://uwsgi-docs.readthedocs.org/en/latest/Gevent.html you can spawn greenthread to do some background task just before return response to client.
mburst 13y ago
Awesome post Matt. I've been meaning to check out Varnish. I'm sad I missed your talk at Djangocon but hopefully it'll make it's way to the web soon.
hardwaresofton 13y ago
"Slowness is likely a result of the fact that your request is communicating with other services across your network. In our case, these other services are PostgreSQL, Redis, Cassandra, and Memcached, just to name a few. Slow database queries and network latency generally outweigh the performance overhead of a robust framework such as Django." This seems to fly in the face of everything I've experienced with frameworks... Is this true? The bottleneck for me is usually never the database backend, unless you've written horrible queries... Maybe it's just because I'm not doing queries complex enough on the scale that disqus is?
- mattrobenolt 13y ago
  It's definitely not a blanket statement. For us, it is. Considering most requests aren't just 1 simple query either. If you even do 5 simple queries, each at 20ms each, that's 100ms spend just waiting for the db. Plus other systems in place. And honestly for us, we'd be happy if all queries were ~20ms. :) Lots are unfortunately slower than that. Especially if all 45k/s were doing the queries.
  - bliti 13y ago
    How many database servers do you have? How do you have them handle the queue?
    - mattrobenolt 13y ago
      1 primary master, with a few read slaves and a few shards.
      - bliti 13y ago
        Thanks for answering my questions. (:
  - beambot 13y ago
    With asynchronous (go-like) channels becoming so popular, I'd love to see some non-blocking Django ORM database calling features. That way, you could fire off a few non-blocking db requests and keep executing on the app server (eg. rendering?) until you hit a point where you needed the database data. I know you could roll your own implementation... but building this into the ORM would still be really slick.
byroot 13y ago
About: > The common pattern for application level caching data = cache.get('stuff') if data is None: data = list(Stuff.objects.all()) cache.set('stuff', data) return data I'm wondering if you simplified the example or if you are just not preventing cache regen race conditions ? Some other frameworks just put back the expired data for a few seconds while the new one is being regenerated to avoid having multiple workers building the same thing. e.g. rails: https://github.com/rails/rails/blob/3182295ce2fa01b02cb9af0b977a9cf83cc5d9aa/activesupport/lib/active_support/cache.rb#L539 https://github.com/rails/rails/blob/3182295ce2fa01b02cb9af0b...
- mattrobenolt 13y ago
  This is a very simplified example. We do something similar to avoid a stampede.
bhauer 13y ago
I applaud Disqus for scaling Django to this tier of sustained load. I applaud them for sharing a clearly-written and approachable explanation of how that was achieved. I also applaud them for their product in general. I think Disqus is a quite excellent embeddable comment tool. I do have some reservations with a few points made by this article. (Below I am speaking generally, and not about Disqus in particular. I don't mean anything below to imply they are doing it wrong. On the contrary, I think they're doing it very right given their circumstances.) Repeated is the conventional wisdom that the performance of your application logic is negligible versus external systems such as your database server or your back-end cache. For low-performance frameworks and platforms that is indeed commonly the case, hence the conventional wisdom. However, there are important caveats: first, do not confuse time spent in your database driver and ORM as waiting for your database server. Your database server vendor will find that hurtful and offensive. Most database servers will be able to retrieve rows from well-indexed tables at far greater rates than low-performance application platforms' ORMs can translate those rows into usable objects. Modern database servers fetching rows from well-indexed tables can keep up with the query demands of the very highest-performance frameworks without saturating a database server's CPUs (with throughput measured in the tens to hundreds of thousands of queries per second per server). Yes, at scale your database server may need attention. But it's not necessarily the pain point you might think it is. Bottom line: profile your application and watch your database server's performance metrics. You may not be waiting on your database despite conventional wisdom. The same is true for other third-party systems such as a back-end cache. Coupling the above with application logic and in-application composition of content into client-digestable markup ("server side templates") will compound the impact of a low-performance platform. While high-performance platforms can execute application logic and compose a server-side template tens of thousands of times per second on modest hardware, low-performance platforms may suffer a ten-times or greater performance penalty by comparison. It is not necessarily true that high-performance frameworks and platforms are lower-productivity if you are starting with a green-field scenario where your development team is free of incumbent preferences. That last bit is crucial, of course. Most teams do have preferences, past experience that can be leveraged, and "know-how" with legacy frameworks. Do not confuse this institutional knowledge with an objective measure of developer efficiency. Developers who are unfamiliar with both Django and a modern high-performance framework may see roughly equal productivity. Measuring your Django-experienced teams' productivity versus their productivity with (for the sake of argument) a Go framework or a modern JVM framework is a biased assessment because of the alternative's learning curve. If we continue to judge net productivity as a combination of learning curve and the resulting and ongoing effort level past the learning curve, little with a learning curve will be honestly evaluated. Yes, reverse proxy caching such as that provided by Varnish is an excellent idea when your application is a public-facing system without a great deal of personalization. But not all systems are public-facing embeddable comments or blogs or news sites (I don't mean this to be critical!). In many systems, a majority of responses are tailored to the specific user and other entities making them unavailable for caching (as the article mentions these requests will typically use a cookie to identify the session and are therefore not cached by Varnish). In these cases, if it weren't already clear from the above, I recommend seriously considering a higher-performance platform and framework that gives you the headroom to deliver responses under high load without necessarily resorting to crutches like a reverse proxy. Yes, leverage caching where-ever and when-ever possible. But when you cannot cache, respond as quickly as possible. Performance is actually an important concern. It's not the concern, but don't keep throwing it under the bus. Further, performance is not only a scale and concurrency concern. It's also a user-experience matter. In addition to reducing the system complexity for high-load and high-concurrency, a high-performance platform means that even without load and concurrency, you are able to respond to user requests more quickly (reduced latency). This leads to user happiness, and in some circumstances better search engine positioning and similar fringe benefits. Again, I want to be clear that I think Disqus is great and this article is a valuable contribution, especially for those who are invested in a similar technology stack with similar usage characteristics.
Trezoid 13y ago
One thing I'd also recommend for speeding up django is swapping in djinja. Straight drop in replacement for almost the entire template engine, but much faster.
- est 13y ago
  This trick will speed-up django template significantly https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.loaders.cached.Loader https://docs.djangoproject.com/en/dev/ref/templates/api/#dja...
- rufugee 13y ago
  What do you lose by using djinja instead of the builtin template engine?
  - Trezoid 13y ago
    You primarily lose a few of the django filters, though they can all be implemented into the jinja filters if you need to
    - antihero 13y ago
      Wouldn't this also mean you have to re-implement any filters that come from third-party apps, too?
- acdha 13y ago
  Benchmarks, please. I've yet to find a case where that was true in a real-world project outside of … dubious … designs where someone was using a dozen nested if/for blocks to avoid writing a templatetag. In one memorable case, someone switched an entire site over to Jinja before doing any profiling – I reverted it and added select_related() to avoid doing 8,000 queries while generating the page.
gojomo 13y ago
If you want highly-tuned geo-distributed Varnish-as-a-service, check out Fastly.com: http://fastly.com http://fastly.com (Disqus is actually listed as a Fastly client.)
- mattrobenolt 13y ago
  Fastly is used for some of our traffic yes, but not all. Our main app is not behind Fastly FWIW.
iknight 13y ago
so going to this link with ghostery enabled is a bad idea. the constant attempts to load resources that are blocked will crash chrome
gojomo 13y ago
Front-side HTTP caches can be essential at giant scale. First they help with naturally-static content, then you can design to maximize the amount that can be HTTP-cached. I would like to see this better recognized as a usual and desirable production setup in the Django community, so as to eventually change the traditional doctrine around static resources. Currently, there's the assumption that every serious production deployment moves static file-serving out of python/Django, to a dedicated side server. So, the interaction of the 'staticfiles' component changes awkwardly around the DEBUG setting, and the docs contain hand-wavey warnings about how using Django to serve static resources is "grossly inefficient and probably insecure". Well, once you've committed to having a front-side HTTP cache, it's pretty damn efficient - one request per resource for an arbitrarily long cache-lifetime period. It requires fewer deployment steps and standalone processes than the assumed ('collectstatic'-and-upload-elsewhere) model. And if the staticfiles app is truly "insecure", that needs fixing: many people run their dev/prototype code in an internet-accessible way, so any known security risks here should get the same attention they get elsewhere. (Disabling the code entirely when DEBUG is true is a dodge.) I'd love a future version of Django to embrace the idea: "staticfiles is a wonderful way to serve static resources, if you run a front-side HTTP cache, which most large projects will".
- eli 13y ago
  I'm a huge fan of Varnish as a caching proxy. But for just serving static assets, I'd suggest Amazon CloudFront. It's easy to set up, you only pay for what you use and you get all the benefits of a global CDN.
  - mattrobenolt 13y ago
    CloudFront is slow. :) Check out Fastly.
    - eli 13y ago
      I demoed it a while ago, maybe I'll take another look. My real point was that you don't need to install any unfamiliar software. I love Varnish, but if the default VCL file doesn't work for you, there's a learning curve.
stefantalpalaru 13y ago
Scaling Django by using it less... It's Varnish that deserves the spotlight, not Django.
- acjohnson55 13y ago
  Well, sort of. To me, it's more like, use Django for what it's best at, which is making your app logic really organized, readable, and maintainable. I think it's a great story because it makes the point that just because Django isn't super high performance doesn't mean you need to go with something super stripped down or in an unreadable language to get good results.
  - iends 13y ago
    But couldn't they save a ton of money in server costs by going with something with higher performance?
    - mattrobenolt 13y ago
      Servers are much cheaper than humans. First of all, the amount of time that'd be required to rewrite things in said "higher performance" language with the hopes of getting some improvements would be very costly. It's not something that'd be done in even a month by one person. It'd be a whole company effort. Learning curve of new language, etc. With that aside, I guarantee, for our use case at least, I'd rather invest my time into tuning a Varnish config and save the same # of servers without all the man power wasted. Our server costs are much much cheaper than our human costs.
      - iends 13y ago
        This is certainly the case now. You have too much technical debt. For future projects should you continue to use Django/Python, or something with better performance? Should this be a lesson to me as a Django developer, that I'm better off starting a company on a different stack if I want to keep server costs low?
tomlu 13y ago
Front-side HTTP caching is all well and good, but what do you guys do in the case when the returned content is (at least partially) user-contextual? Caching isn't really going to help you in these cases.
- mattrobenolt 13y ago
  Obviously you can't cache everything. Hence why we still have 15k/s that are hitting out backends. There are lots of more complicated things we can do with ESI if we really wanted to get into it.
- adamauckland 13y ago
  I'm guessing they cache as much as they can here too https://docs.djangoproject.com/en/1.6/topics/cache/#template-fragment-caching https://docs.djangoproject.com/en/1.6/topics/cache/#template...
  - mattrobenolt 13y ago
    Template fragment caching is not a good thing. We don't do this anymore. If you have to cache your template fragments, I generally think you're doing something wrong. Gonna have a bad time.
jjoe 13y ago
Deploying Varnish is like cheating but in a good way. Varnish isn't a set-and-forget cache proxy. It requires a well thought out VCL and a good deal of attention and maintenance. So the saying not all VCLs are created equal applies in this case. I challenged myself to build the most generic VCL in the sense that I want it to work with the majority of "scripts" in a semi "shared" server deployment. I also wanted to make it as easy to deploy as possible while exposing some of its advanced features. The end result is an array of software "plugin" products that one can download and deploy on cPanel and DirectAdmin (two leading control panels). Shameful plug (wait for it...): http://www.unixy.net/advanced-hosting/varnish-nginx-cpanel/ http://www.unixy.net/advanced-hosting/varnish-nginx-cpanel/ http://www.unixy.net/varnish/ http://www.unixy.net/varnish/ There's a free 14-day trial (no payment or CC required) for those who want to give it a spin.
callesgg 13y ago
I dont get why varnish should be any faster than any other raw static http server. Ok a bit faster i can understand. But like 300 times faster than without it that i don't get.
- mattrobenolt 13y ago
  Varnish caches dynamic requests.
- hrjet 13y ago
  I guess they are using it for caching non-static content.
ksec 13y ago
Ok, Varnish 4.0 were mentioned, by I dont find anything concrete of specific with Google Search. When will that be coming? And I would love if High Scalability do an Interview with Disqus. 8 Billion PV, would love to see their Stack, Backend, and Machines that handles it.
- amarsahinovic 13y ago
  Scaling Realtime at DISQUS: https://speakerdeck.com/northisup/scaling-realtime-at-disqus https://speakerdeck.com/northisup/scaling-realtime-at-disqus
  - mattrobenolt 13y ago
    http://blog.disqus.com/post/51155103801/trying-out-this-go-thing http://blog.disqus.com/post/51155103801/trying-out-this-go-t... This has replaced that. ;)
csense 13y ago
What does Varnish do that nginx doesn't?

316 points by mattrobenolt 13y ago

Scaling varnish to 30K/sec. Scaling Django to 10K-15K/sec. I remember long time ago:"How i scaled drupal(large number of queries/page) to 3K/pages/sec". It was really Varnish that scaled.

bliti 13y ago
Still, Django at 15K is good.
mattrobenolt 13y ago
Sure, but the idea is knowing how and when to use your tools properly and what they're good for. It allows us to continue using Django.
eli 13y ago
Out of the box Drupal didn't used to play well with Varnish, so that was not a trivial task even if Varnish did the heavy lifting. For serving a cached asset from memory it's going to be really hard to beat Varnish on speed.

bliti 13y ago

15K requests with Django is a nice number to hit. What are you using as a server? Gunicorn? Not a bad setup. load balancers --> Vanquish: if !cache: --> Django Still makes me wonder how faster this would be in Go or Java. I've never been able to get Python to be very efficient over 5K requests.

mattrobenolt 13y ago
uwsgi
- bliti 13y ago
  Gevent is a bit funny sometimes. How are you dealing with it?
  - mattrobenolt 13y ago
    We don't run gevent for disqus.com. We do some some background workers and whatnot, but our main app is single threaded with a large number of processes on each machine.
    - bliti 13y ago
      Interesting. I'd love to know about the specs of your hardware. Maybe a future blog post about that?
  - est 13y ago
    If you really need gevent that badly http://uwsgi-docs.readthedocs.org/en/latest/Gevent.html http://uwsgi-docs.readthedocs.org/en/latest/Gevent.html you can spawn greenthread to do some background task just before return response to client.

mburst 13y ago

Awesome post Matt. I've been meaning to check out Varnish. I'm sad I missed your talk at Djangocon but hopefully it'll make it's way to the web soon.

hardwaresofton 13y ago

"Slowness is likely a result of the fact that your request is communicating with other services across your network. In our case, these other services are PostgreSQL, Redis, Cassandra, and Memcached, just to name a few. Slow database queries and network latency generally outweigh the performance overhead of a robust framework such as Django." This seems to fly in the face of everything I've experienced with frameworks... Is this true? The bottleneck for me is usually never the database backend, unless you've written horrible queries... Maybe it's just because I'm not doing queries complex enough on the scale that disqus is?

mattrobenolt 13y ago
It's definitely not a blanket statement. For us, it is. Considering most requests aren't just 1 simple query either. If you even do 5 simple queries, each at 20ms each, that's 100ms spend just waiting for the db. Plus other systems in place. And honestly for us, we'd be happy if all queries were ~20ms. :) Lots are unfortunately slower than that. Especially if all 45k/s were doing the queries.
- bliti 13y ago
  How many database servers do you have? How do you have them handle the queue?
  - mattrobenolt 13y ago
    1 primary master, with a few read slaves and a few shards.
    - bliti 13y ago
      Thanks for answering my questions. (:
- beambot 13y ago
  With asynchronous (go-like) channels becoming so popular, I'd love to see some non-blocking Django ORM database calling features. That way, you could fire off a few non-blocking db requests and keep executing on the app server (eg. rendering?) until you hit a point where you needed the database data. I know you could roll your own implementation... but building this into the ORM would still be really slick.

byroot 13y ago

About: > The common pattern for application level caching data = cache.get('stuff') if data is None: data = list(Stuff.objects.all()) cache.set('stuff', data) return data I'm wondering if you simplified the example or if you are just not preventing cache regen race conditions ? Some other frameworks just put back the expired data for a few seconds while the new one is being regenerated to avoid having multiple workers building the same thing. e.g. rails: https://github.com/rails/rails/blob/3182295ce2fa01b02cb9af0b977a9cf83cc5d9aa/activesupport/lib/active_support/cache.rb#L539 https://github.com/rails/rails/blob/3182295ce2fa01b02cb9af0b...

mattrobenolt 13y ago
This is a very simplified example. We do something similar to avoid a stampede.

bhauer 13y ago

I applaud Disqus for scaling Django to this tier of sustained load. I applaud them for sharing a clearly-written and approachable explanation of how that was achieved. I also applaud them for their product in general. I think Disqus is a quite excellent embeddable comment tool. I do have some reservations with a few points made by this article. (Below I am speaking generally, and not about Disqus in particular. I don't mean anything below to imply they are doing it wrong. On the contrary, I think they're doing it very right given their circumstances.) Repeated is the conventional wisdom that the performance of your application logic is negligible versus external systems such as your database server or your back-end cache. For low-performance frameworks and platforms that is indeed commonly the case, hence the conventional wisdom. However, there are important caveats: first, do not confuse time spent in your database driver and ORM as waiting for your database server. Your database server vendor will find that hurtful and offensive. Most database servers will be able to retrieve rows from well-indexed tables at far greater rates than low-performance application platforms' ORMs can translate those rows into usable objects. Modern database servers fetching rows from well-indexed tables can keep up with the query demands of the very highest-performance frameworks without saturating a database server's CPUs (with throughput measured in the tens to hundreds of thousands of queries per second per server). Yes, at scale your database server may need attention. But it's not necessarily the pain point you might think it is. Bottom line: profile your application and watch your database server's performance metrics. You may not be waiting on your database despite conventional wisdom. The same is true for other third-party systems such as a back-end cache. Coupling the above with application logic and in-application composition of content into client-digestable markup ("server side templates") will compound the impact of a low-performance platform. While high-performance platforms can execute application logic and compose a server-side template tens of thousands of times per second on modest hardware, low-performance platforms may suffer a ten-times or greater performance penalty by comparison. It is not necessarily true that high-performance frameworks and platforms are lower-productivity if you are starting with a green-field scenario where your development team is free of incumbent preferences. That last bit is crucial, of course. Most teams do have preferences, past experience that can be leveraged, and "know-how" with legacy frameworks. Do not confuse this institutional knowledge with an objective measure of developer efficiency. Developers who are unfamiliar with both Django and a modern high-performance framework may see roughly equal productivity. Measuring your Django-experienced teams' productivity versus their productivity with (for the sake of argument) a Go framework or a modern JVM framework is a biased assessment because of the alternative's learning curve. If we continue to judge net productivity as a combination of learning curve and the resulting and ongoing effort level past the learning curve, little with a learning curve will be honestly evaluated. Yes, reverse proxy caching such as that provided by Varnish is an excellent idea when your application is a public-facing system without a great deal of personalization. But not all systems are public-facing embeddable comments or blogs or news sites (I don't mean this to be critical!). In many systems, a majority of responses are tailored to the specific user and other entities making them unavailable for caching (as the article mentions these requests will typically use a cookie to identify the session and are therefore not cached by Varnish). In these cases, if it weren't already clear from the above, I recommend seriously considering a higher-performance platform and framework that gives you the headroom to deliver responses under high load without necessarily resorting to crutches like a reverse proxy. Yes, leverage caching where-ever and when-ever possible. But when you cannot cache, respond as quickly as possible. Performance is actually an important concern. It's not the concern, but don't keep throwing it under the bus. Further, performance is not only a scale and concurrency concern. It's also a user-experience matter. In addition to reducing the system complexity for high-load and high-concurrency, a high-performance platform means that even without load and concurrency, you are able to respond to user requests more quickly (reduced latency). This leads to user happiness, and in some circumstances better search engine positioning and similar fringe benefits. Again, I want to be clear that I think Disqus is great and this article is a valuable contribution, especially for those who are invested in a similar technology stack with similar usage characteristics.

Trezoid 13y ago

One thing I'd also recommend for speeding up django is swapping in djinja. Straight drop in replacement for almost the entire template engine, but much faster.

est 13y ago
This trick will speed-up django template significantly https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.loaders.cached.Loader https://docs.djangoproject.com/en/dev/ref/templates/api/#dja...
rufugee 13y ago
What do you lose by using djinja instead of the builtin template engine?
- Trezoid 13y ago
  You primarily lose a few of the django filters, though they can all be implemented into the jinja filters if you need to
  - antihero 13y ago
    Wouldn't this also mean you have to re-implement any filters that come from third-party apps, too?
acdha 13y ago
Benchmarks, please. I've yet to find a case where that was true in a real-world project outside of … dubious … designs where someone was using a dozen nested if/for blocks to avoid writing a templatetag. In one memorable case, someone switched an entire site over to Jinja before doing any profiling – I reverted it and added select_related() to avoid doing 8,000 queries while generating the page.

gojomo 13y ago

If you want highly-tuned geo-distributed Varnish-as-a-service, check out Fastly.com: http://fastly.com http://fastly.com (Disqus is actually listed as a Fastly client.)

mattrobenolt 13y ago
Fastly is used for some of our traffic yes, but not all. Our main app is not behind Fastly FWIW.

iknight 13y ago

so going to this link with ghostery enabled is a bad idea. the constant attempts to load resources that are blocked will crash chrome

gojomo 13y ago

Front-side HTTP caches can be essential at giant scale. First they help with naturally-static content, then you can design to maximize the amount that can be HTTP-cached. I would like to see this better recognized as a usual and desirable production setup in the Django community, so as to eventually change the traditional doctrine around static resources. Currently, there's the assumption that every serious production deployment moves static file-serving out of python/Django, to a dedicated side server. So, the interaction of the 'staticfiles' component changes awkwardly around the DEBUG setting, and the docs contain hand-wavey warnings about how using Django to serve static resources is "grossly inefficient and probably insecure". Well, once you've committed to having a front-side HTTP cache, it's pretty damn efficient - one request per resource for an arbitrarily long cache-lifetime period. It requires fewer deployment steps and standalone processes than the assumed ('collectstatic'-and-upload-elsewhere) model. And if the staticfiles app is truly "insecure", that needs fixing: many people run their dev/prototype code in an internet-accessible way, so any known security risks here should get the same attention they get elsewhere. (Disabling the code entirely when DEBUG is true is a dodge.) I'd love a future version of Django to embrace the idea: "staticfiles is a wonderful way to serve static resources, if you run a front-side HTTP cache, which most large projects will".

eli 13y ago
I'm a huge fan of Varnish as a caching proxy. But for just serving static assets, I'd suggest Amazon CloudFront. It's easy to set up, you only pay for what you use and you get all the benefits of a global CDN.
- mattrobenolt 13y ago
  CloudFront is slow. :) Check out Fastly.
  - eli 13y ago
    I demoed it a while ago, maybe I'll take another look. My real point was that you don't need to install any unfamiliar software. I love Varnish, but if the default VCL file doesn't work for you, there's a learning curve.

stefantalpalaru 13y ago

Scaling Django by using it less... It's Varnish that deserves the spotlight, not Django.

acjohnson55 13y ago
Well, sort of. To me, it's more like, use Django for what it's best at, which is making your app logic really organized, readable, and maintainable. I think it's a great story because it makes the point that just because Django isn't super high performance doesn't mean you need to go with something super stripped down or in an unreadable language to get good results.
- iends 13y ago
  But couldn't they save a ton of money in server costs by going with something with higher performance?
  - mattrobenolt 13y ago
    Servers are much cheaper than humans. First of all, the amount of time that'd be required to rewrite things in said "higher performance" language with the hopes of getting some improvements would be very costly. It's not something that'd be done in even a month by one person. It'd be a whole company effort. Learning curve of new language, etc. With that aside, I guarantee, for our use case at least, I'd rather invest my time into tuning a Varnish config and save the same # of servers without all the man power wasted. Our server costs are much much cheaper than our human costs.
    - iends 13y ago
      This is certainly the case now. You have too much technical debt. For future projects should you continue to use Django/Python, or something with better performance? Should this be a lesson to me as a Django developer, that I'm better off starting a company on a different stack if I want to keep server costs low?

tomlu 13y ago

Front-side HTTP caching is all well and good, but what do you guys do in the case when the returned content is (at least partially) user-contextual? Caching isn't really going to help you in these cases.

mattrobenolt 13y ago
Obviously you can't cache everything. Hence why we still have 15k/s that are hitting out backends. There are lots of more complicated things we can do with ESI if we really wanted to get into it.
adamauckland 13y ago
I'm guessing they cache as much as they can here too https://docs.djangoproject.com/en/1.6/topics/cache/#template-fragment-caching https://docs.djangoproject.com/en/1.6/topics/cache/#template...
- mattrobenolt 13y ago
  Template fragment caching is not a good thing. We don't do this anymore. If you have to cache your template fragments, I generally think you're doing something wrong. Gonna have a bad time.

jjoe 13y ago

Deploying Varnish is like cheating but in a good way. Varnish isn't a set-and-forget cache proxy. It requires a well thought out VCL and a good deal of attention and maintenance. So the saying not all VCLs are created equal applies in this case. I challenged myself to build the most generic VCL in the sense that I want it to work with the majority of "scripts" in a semi "shared" server deployment. I also wanted to make it as easy to deploy as possible while exposing some of its advanced features. The end result is an array of software "plugin" products that one can download and deploy on cPanel and DirectAdmin (two leading control panels). Shameful plug (wait for it...): http://www.unixy.net/advanced-hosting/varnish-nginx-cpanel/ http://www.unixy.net/advanced-hosting/varnish-nginx-cpanel/ http://www.unixy.net/varnish/ http://www.unixy.net/varnish/ There's a free 14-day trial (no payment or CC required) for those who want to give it a spin.

callesgg 13y ago

I dont get why varnish should be any faster than any other raw static http server. Ok a bit faster i can understand. But like 300 times faster than without it that i don't get.

mattrobenolt 13y ago
Varnish caches dynamic requests.
hrjet 13y ago
I guess they are using it for caching non-static content.

ksec 13y ago

Ok, Varnish 4.0 were mentioned, by I dont find anything concrete of specific with Google Search. When will that be coming? And I would love if High Scalability do an Interview with Disqus. 8 Billion PV, would love to see their Stack, Backend, and Machines that handles it.

amarsahinovic 13y ago
Scaling Realtime at DISQUS: https://speakerdeck.com/northisup/scaling-realtime-at-disqus https://speakerdeck.com/northisup/scaling-realtime-at-disqus
- mattrobenolt 13y ago
  http://blog.disqus.com/post/51155103801/trying-out-this-go-thing http://blog.disqus.com/post/51155103801/trying-out-this-go-t... This has replaced that. ;)

csense 13y ago

What does Varnish do that nginx doesn't?