4 ms·

I was responsible for a 10^6 views/day (due to most traffic being daytime, comes out to 10-20 req/sec) website running on Django. I went through hell arriving

I was responsible for a 10^6 views/day (due to most traffic being daytime, comes out to 10-20 req/sec) website running on Django. I went through hell arriving at this solution because the system was in disarray when I took over and deployed on an expensive cloud solution (so resources were limited). Finally I just put my foot down and demanded we buy servers instead, we would have more bare metal by a factor of 4 than we needed and we would pay only a fraction of what we did for the previous "solution". I could provision all the VMs I wanted, provided I didn't break the host. I arrived at this procedure: Spin up a dyno, worker, caches and warm everything up. Run smoketests to validate everything from the app-server onwards is lit and firing on all cylinders. Redirect nginx traffic to the new setup. If everything turns to shit in a second, reverse traffic change and re-evaluate. If everything works and nothing is broken after an hour, take the old dynos/workers/caches offline. The trick was if there were migrations that broke the older version that you did it in 2 or more steps to isolate the changed model interactions. And nobody should be using gunicorn, it's way less performant than uWSGI (citing:http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node-benchmarks.html http://blog.kgriffs.com/2012/12/18/uwsgi-vs-gunicorn-vs-node...).

rch 11y ago
Good approach, and thank you for mentioning uWSGI. Don't forget to warm up Solr or ES as well.
- Beltiras 11y ago
  I gave up on ES and never had use for Solr. I think the problems I had with ES was how the contractor set it up but the only search I have ever needed has been built with bare-naked ORM, even did a fulltext search using Postgres as a backend without an index, articles size from 10^3-10^5 characters. on the order of 10^5 articles, still finding a set of 10-20 articles containing 2-5 search terms within 1 second. This really surprised me since fulltext search in Postgres is supposed to be bad. I tried to find a bad case where the search would take forever since I thought I had the nightmare case on my hands, didn't. uWSGI is a beast to learn but once you got the hang of it, nothing else will compare.
  - stevepike 11y ago
    Postgres text search is really useful, but dedicated search servers still win when you need to start showing faceted results and more tuning of the scoring.
    - Beltiras 11y ago
      I'm sure. In my case the ES integration was botched and I didn't have time to learn a new piece of software to work out the kinks. It's been on my list of "tech-2-familiarize" but somehow never been exigent enough to pull the trigger on.
    - gt565k 11y ago
      I strongly suggest the ES 2 day training course. I think it's called Core Elasticsearch. It dives deep into how ES works under the hood. It also provides a great overview of all the available features. What I loved most about the course is that the presenters were extremely knowledgeable and gave great tips on how to manage your schema with ES and configure the clusters so that you don't encounter performance issues later on. You can do some amazing things with ES, that you wouldn't necessary even know you could. One example is the percolator[0], which can be used to slam new documents against existing queries and essentially classify the document based on search queries. [0] https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
parhamn 11y ago
This has a name -- its called blue/green deploys. The idea is if you're running the 'blue' version of the site, you run a new 'green' version then then route traffic to it once everything is fine (usually at the LB level) and revert if necessary The way you solve the migration problem (much harder than it sounds) is to do only non backwards breaking migrations between a blue green switch. So you end up doing your migrations in multiple deploys (e.g add a field, and start using the new field with no references to the old field in the code, then remove the old field in the next blue/green deploy). I've generally found that it might not be worth the effort though depending on the context (especially if your initial healtcheck is before the migration).
- Beltiras 11y ago
  It depends on the cost you would assign to an event. I would sometimes on small code changes just deploy directly and touch-reload the app but I was very reluctant to deviate from protocol given experience. In the mad days before my protocol I once had the site in burning/extinguishing/reigniting cycle for 4 hours. The reason it took so long to figure out what was wrong was that the problematic code was live for 10 days in a section that had to run on each request without a problem. To this day I couldn't tell you why it suddenly started to act up. 4 hours of "House is on FIRE!" will make you cautious.
CoffeeDregs 11y ago
And nobody should be using gunicorn, it's way less performant than uWSGI uWSGI may be different now, but 4 years ago I built a browser toolbar C&C API server that peaked at about 2k requests per second. We used uWSGI to front our Django application with SoftLayer HTTP load balancers in front of uWSGI. uWSGI was speaking HTTP and translating to WSGI, but did so only in one thread/process. So while 8-16 Django threads were tootling along, the single uWSGI HTTP->WSGI thread was pegged. We switched to gunicorn and had dramatically better performance.
- Beltiras 11y ago
  Sounds more like a crappy setup than a problem with uWSGI itself. I know it has a steep learning curve. I know that it takes time and effort to change out software pieces that have a long history within an organization (I have some experience in that department) even when a clear and technically superior alternative is available. I have a drawing of my favorite infrastructure that solves a ton of problems before I code one line of anything. nginx -> uWSGI -> Python(Django) -> Celery(via RabbitMQ) -> Postgres It's a bit more complicated than this and redis is missing but it's a nice linear representation of how I like to structure things. The only thing there not lightning-fast is Django but it doesn't need to be. I'm spending most of my time in the other pieces of software.
  - CoffeeDregs 11y ago
    Sounds more like a crappy setup than a problem with uWSGI itself. No. As I detailed, it was a deficiency with uWSGI itself. Complicating the setup as you suggest is a crappy workaround for a uWSGI deficiency. We simplified the setup by switching to gunicorn. I would still use uWSGI, but, as with all software/tools, I would be aware that it has a deficiency. uWSGI docs on production usage of HTTP functionality don't mention this issue so I wanted to let others know. https://uwsgi-docs.readthedocs.org/en/latest/HTTP.html?highlight=http#can-i-use-uwsgi-s-http-capabilities-in-production https://uwsgi-docs.readthedocs.org/en/latest/HTTP.html?highl... I have a drawing of my favorite infrastructure Wish I had the luxury. We're running 20-30 projects/applications (~3000 tasks) across our clusters. Some are Django fronted by gunicorn, some are directly fronted by flask, but 80% are non-HTTP services.
squeaky-clean 11y ago
I'm in a similar situation. A service getting about that many hits per day, with the majority of the traffic being during the day. We have a few servers behind a load balancer to help with the peak traffic. Luckily, traffic peaks during European daytime, so when I'm in the office (USA), it's the low period. I can pull one of our servers out of the load balancer, push the update, put it back in and watch the responses. If things don't blow up, rinse and repeat. It's feels pretty hacky to me though, so I've been recently reading up on Fabric and deployment patterns for Django, great timing with this article being posted!
fmueller 11y ago
This only works if database changes are backwards compatible which is not always the case. In all other cases you're procedure seems only reasonable. I don't understand why anyone would do it differently.
- tonyarkles 11y ago
  That's the trick: transform all database changes to be backward compatible between version n and n+1. It takes a little more time, and you end up with multiple deploys, but it's really quite nice.

rch 11y ago

Good approach, and thank you for mentioning uWSGI. Don't forget to warm up Solr or ES as well.

Beltiras 11y ago
I gave up on ES and never had use for Solr. I think the problems I had with ES was how the contractor set it up but the only search I have ever needed has been built with bare-naked ORM, even did a fulltext search using Postgres as a backend without an index, articles size from 10^3-10^5 characters. on the order of 10^5 articles, still finding a set of 10-20 articles containing 2-5 search terms within 1 second. This really surprised me since fulltext search in Postgres is supposed to be bad. I tried to find a bad case where the search would take forever since I thought I had the nightmare case on my hands, didn't. uWSGI is a beast to learn but once you got the hang of it, nothing else will compare.
- stevepike 11y ago
  Postgres text search is really useful, but dedicated search servers still win when you need to start showing faceted results and more tuning of the scoring.
  - Beltiras 11y ago
    I'm sure. In my case the ES integration was botched and I didn't have time to learn a new piece of software to work out the kinks. It's been on my list of "tech-2-familiarize" but somehow never been exigent enough to pull the trigger on.
  - gt565k 11y ago
    I strongly suggest the ES 2 day training course. I think it's called Core Elasticsearch. It dives deep into how ES works under the hood. It also provides a great overview of all the available features. What I loved most about the course is that the presenters were extremely knowledgeable and gave great tips on how to manage your schema with ES and configure the clusters so that you don't encounter performance issues later on. You can do some amazing things with ES, that you wouldn't necessary even know you could. One example is the percolator[0], which can be used to slam new documents against existing queries and essentially classify the document based on search queries. [0] https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...

parhamn 11y ago

This has a name -- its called blue/green deploys. The idea is if you're running the 'blue' version of the site, you run a new 'green' version then then route traffic to it once everything is fine (usually at the LB level) and revert if necessary The way you solve the migration problem (much harder than it sounds) is to do only non backwards breaking migrations between a blue green switch. So you end up doing your migrations in multiple deploys (e.g add a field, and start using the new field with no references to the old field in the code, then remove the old field in the next blue/green deploy). I've generally found that it might not be worth the effort though depending on the context (especially if your initial healtcheck is before the migration).

Beltiras 11y ago
It depends on the cost you would assign to an event. I would sometimes on small code changes just deploy directly and touch-reload the app but I was very reluctant to deviate from protocol given experience. In the mad days before my protocol I once had the site in burning/extinguishing/reigniting cycle for 4 hours. The reason it took so long to figure out what was wrong was that the problematic code was live for 10 days in a section that had to run on each request without a problem. To this day I couldn't tell you why it suddenly started to act up. 4 hours of "House is on FIRE!" will make you cautious.

CoffeeDregs 11y ago

And nobody should be using gunicorn, it's way less performant than uWSGI uWSGI may be different now, but 4 years ago I built a browser toolbar C&C API server that peaked at about 2k requests per second. We used uWSGI to front our Django application with SoftLayer HTTP load balancers in front of uWSGI. uWSGI was speaking HTTP and translating to WSGI, but did so only in one thread/process. So while 8-16 Django threads were tootling along, the single uWSGI HTTP->WSGI thread was pegged. We switched to gunicorn and had dramatically better performance.

Beltiras 11y ago
Sounds more like a crappy setup than a problem with uWSGI itself. I know it has a steep learning curve. I know that it takes time and effort to change out software pieces that have a long history within an organization (I have some experience in that department) even when a clear and technically superior alternative is available. I have a drawing of my favorite infrastructure that solves a ton of problems before I code one line of anything. nginx -> uWSGI -> Python(Django) -> Celery(via RabbitMQ) -> Postgres It's a bit more complicated than this and redis is missing but it's a nice linear representation of how I like to structure things. The only thing there not lightning-fast is Django but it doesn't need to be. I'm spending most of my time in the other pieces of software.
- CoffeeDregs 11y ago
  Sounds more like a crappy setup than a problem with uWSGI itself. No. As I detailed, it was a deficiency with uWSGI itself. Complicating the setup as you suggest is a crappy workaround for a uWSGI deficiency. We simplified the setup by switching to gunicorn. I would still use uWSGI, but, as with all software/tools, I would be aware that it has a deficiency. uWSGI docs on production usage of HTTP functionality don't mention this issue so I wanted to let others know. https://uwsgi-docs.readthedocs.org/en/latest/HTTP.html?highlight=http#can-i-use-uwsgi-s-http-capabilities-in-production https://uwsgi-docs.readthedocs.org/en/latest/HTTP.html?highl... I have a drawing of my favorite infrastructure Wish I had the luxury. We're running 20-30 projects/applications (~3000 tasks) across our clusters. Some are Django fronted by gunicorn, some are directly fronted by flask, but 80% are non-HTTP services.

squeaky-clean 11y ago

I'm in a similar situation. A service getting about that many hits per day, with the majority of the traffic being during the day. We have a few servers behind a load balancer to help with the peak traffic. Luckily, traffic peaks during European daytime, so when I'm in the office (USA), it's the low period. I can pull one of our servers out of the load balancer, push the update, put it back in and watch the responses. If things don't blow up, rinse and repeat. It's feels pretty hacky to me though, so I've been recently reading up on Fabric and deployment patterns for Django, great timing with this article being posted!

fmueller 11y ago

This only works if database changes are backwards compatible which is not always the case. In all other cases you're procedure seems only reasonable. I don't understand why anyone would do it differently.

tonyarkles 11y ago
That's the trick: transform all database changes to be backward compatible between version n and n+1. It takes a little more time, and you end up with multiple deploys, but it's really quite nice.