8 ms·

Scalable and resilient Django with Kubernetes

96 points by hnarayanan 11y ago

siliconc0w 11y ago
The problem with these tutorials is they still tell you to run the database yourself to emphasize the 'component' or one solution fits all idea of kube/mesos. However, in practice, if you're using AWS or GCloud this is usually a bad idea - just use the managed database solutions provided. They have things like backups, snapshots, restores, upgrades, HA, monitoring, and alerting baked in. These are non-trivial to do yourself.
- ej_campbell 11y ago
  You likely also don't need to run Django yourself either: https://cloud.google.com/appengine/docs/python/cloud-sql/django https://cloud.google.com/appengine/docs/python/cloud-sql/dja...
  - hnarayanan 11y ago
    If you're running a vanilla Django site with no esoteric libraries with special requirements (like numpy/scipy that need compilers etc. to get installed), this is definitely the correct way to go. But if you do have non-standard requirements that fall out of the rails of your PaaS, you need to consider other solutions.
  - waprin 11y ago
    The built-in Django libraries for Django on App Engine are severely outdated. I have new guides that just vendor in recent version: https://cloud.google.com/python/django/appengine https://cloud.google.com/python/django/appengine However there are a lot of tricky issues with Django on App Engine so I would also look into Djangae: https://github.com/potatolondon/djangae https://github.com/potatolondon/djangae As far as Kubernetes, the OP and I had some independent discovery and started working on similar material at the same time. I also have some Django/Kubernetes content, both CloudSQL (MySQL) and Postgres in Kubernetes: https://cloud.google.com/python/django/container-engine https://cloud.google.com/python/django/container-engine (CloudSQL) https://medium.com/google-cloud/deploying-django-postgres-redis-containers-to-kubernetes-9ee28e7a146 https://medium.com/google-cloud/deploying-django-postgres-re... (Postgres) My biggest reason for running the database in Kubernetes is a) CloudSQL doesn't currently support Postgres and b) it's interesting. Seems like OP had similar motivations. Obvious disclosure: I work for GCP.
    - hnarayanan 11y ago
      Parallel explorer and OP here. :) I cited your work as additional reading because I really enjoyed watching your talk on this recently! While I'd figured out a bulk of this stuff out, there were a few things I learnt from it too.
      - waprin 11y ago
        Oh cool, I missed your citation. I am still figuring some things out as well so I am going to check out what you've done. Definitely happy to have you contribute to the ecosystem. It seemed like a content gap that I wanted to fill in, but at Google we would of course prefer to have a healthy external ecosystem of people writing content like yourself!
- hnarayanan 11y ago
  I'm the author of this piece, and I agree with you in general. I am personally experimenting with this within the context of containers because I'm trying to construct something like Vitess[1] from first principles as an intellectual exercise. [1] https://github.com/youtube/vitess https://github.com/youtube/vitess
  - danpalmer 11y ago
    Thanks for the blog post, it was really informative, will definitely be sharing it at work.
    - hnarayanan 11y ago
      Thank you! And please let me know of ways I can improve it. I already have a couple of ideas that I've indicated in the conclusion, but more the merrier. :)
  - markbnj 11y ago
    You did a nice job on this walk-through, so thanks for the effort! I was interested to see that you ended up in the same place I have in the past when deploying databases to kubernetes clusters. A container mounted on a PD is a bit of a hybrid but it works well in practice. Hopefully something like glusterfs or flocker will end up giving us a better way to manage and deploy data volumes.
- danpalmer 11y ago
  I'm not sure I agree. The managed database services provided by Google and Amazon are both expensive, and lack configuration options. I'm sure they're already relatively well tuned, but we have tuned our Postgres installation to our workload more than they would be able to. Running our own Postgres install costs roughly half what it would on AWS, takes very little engineering time to maintain, and can be tuned as needed.
  - throwaway13337 11y ago
    Agreed. When the costs go down and persistence-as-a-service is provided by more cloud hosts, it will make more sense but right now it often doesn't. I wonder about opportunities to provide some of these services as a white label to hosting providers like digital ocean. Is anyone doing that? What stops them?
    - danpalmer 11y ago
      If we're talking about databases specifically, it's either a really easy problem to solve that it's difficult to charge a lot for (i.e. just a Postgres/etc running, with some way of modifying the config), or it's really difficult and requires a lot of resources to get right (i.e. scaling to many nodes, still not really solved well in Postgres/MySQL).
      - throwaway13337 11y ago
        The selling point to the provider would be that it increases stickiness with their clients - that's big - they don't have to charge more.
        lostcolony 11y ago
        I suspect that - If the service is easy enough to provide (this means not just ease of deployment but also up-time, maintenance, etc) while fitting into their margins without issue, it doesn't provide stickiness. If the service is sufficiently complex to provide stickiness, they either need to charge for it, or it will likely eat heavily into their margins. Neither of us know the specifics of Digital Ocean's business plan, but I strongly suspect that were it as simple as you put it, where it's hardly any cost to them, and a huge gain due to increased stickiness, they'd already be doing it, or be moving toward it. And they might be. However, if they aren't, then it's hardly a mystery as to why they're not providing such services.
- markbnj 11y ago
  I don't know if it is a "really bad" idea. I've run stacks on both AWS and GCP, and used both RDS and CloudSQL. They are good solutions and I definitely agree that looking at them first is sensible. They will fit the bill for a lot of applications. However you can get a pretty high level of fault tolerance using kubernetes services mounted on persistent volumes. You don't get all the advantages of a containerized solution but you do get dependency management and fault tolerance.
- jpgvm 11y ago
  If you are interested in mitigating this take a look at Flynn [1]. It's analogous to Kubernetes but is actually much simpler in this case as it can do the Heroku style deployment but also ships HA database appliances with backups out of the box. [1] https://flynn.io https://flynn.io
throwaway13337 11y ago
I'm really struggling to understand why most developers feel they need all this deployment complexity. I've run django and other web deployments with simple shell scripts and occasionally some python to glue it together. Most recently I'm running a django web server and some other custom stuff (a real time stateful server, database, nginx, etc). A somewhat complex setup (or at least as complex as it needs to be), and I have no need of docker, kubernetes. If I want a new server, I just change a few parameters and run the script to deploy. It's straight forward and without magic. The deployment flavor of the month before were DSLs like ansible, puppet, etc. Did these solve real problems? I found they added complexity without adding much. How is the new generation different? Caveat: I'm not google, and I don't deploy huge server farms. There is a use case but the vast majority of us aren't that.
- hnarayanan 11y ago
  I'm the author of the original piece. I echo exactly what you say in a giant caveat way up top in the piece. :) Most of the first half of the article beyond that point basically tries to motivate why you'd want to try this beyond just using a classical VM approach. But the basic idea is that it raises your level of abstraction from working with machines to working with your application components (on abstracted hardware).
  - daxfohl 11y ago
    Seems like there's an opportunity there. It doesn't need to be "all this deployment complexity". Just currently it is. In fact it shouldn't be; the kubernetes level of abstraction has far less going on than the VM level. My guess is that 2-3 years from now it will be easier to set up a kubernetes cluster than to set up even the single VM at the top of your piece, and a corresponding dev env will run seamlessly on Windows, Linux, OSX. The average user won't even know or care what OS they're running in production; it'll just work (think Heroku++) Of course once it becomes ubiquitous, then containers become the new (language agnostic!) way to deploy "libraries", accessed by HTTP, and DLL hell is just moved up an abstraction level. So what have we really achieved....
  - throwaway13337 11y ago
Mizza 11y ago
It's interesting that there seem to be two parallel and opposite trends in applications deployment going developing right now. One is containerizations, where developers are responsible for maintaining fat application stacks that can easily be redeployed and moved around. The other is towards serverlessness with things like [Django-Zappa](https://github.com/Miserlou/django-zappa https://github.com/Miserlou/django-zappa), where scalability is handled automatically by cloud providers. My bias is quite clear - deploying apps should be easy.
jamespacileo 11y ago
Hey Harish, thanks for your talk at the March London Django Meetup. I'm starting to play around with Google Container service as you suggested, having a good time.
- hnarayanan 11y ago
  Welcome!
smitec 11y ago
Putting the 'should we or shouldn't we' aside I'd just like to say thanks for putting together an article that goes all the way from a 'classic' setup to something running on a host. I often find that posts like this are a bit too meta and in trying to write things in a general way they leave out some critical step which makes replicating their ideas difficult.
- hnarayanan 11y ago
  OP here. That was the plan and I'm glad you appreciate it. :) I wanted to start from something I assumed people knew, and tried to motivate why one might want to improve on that. Only then did I introduce the new solution. A lot of tutorials I found jumped too quickly to the 'how' without spending enough time on the 'why' I should care.

96 points by hnarayanan 11y ago

The problem with these tutorials is they still tell you to run the database yourself to emphasize the 'component' or one solution fits all idea of kube/mesos. However, in practice, if you're using AWS or GCloud this is usually a bad idea - just use the managed database solutions provided. They have things like backups, snapshots, restores, upgrades, HA, monitoring, and alerting baked in. These are non-trivial to do yourself.

ej_campbell 11y ago
You likely also don't need to run Django yourself either: https://cloud.google.com/appengine/docs/python/cloud-sql/django https://cloud.google.com/appengine/docs/python/cloud-sql/dja...
- hnarayanan 11y ago
  If you're running a vanilla Django site with no esoteric libraries with special requirements (like numpy/scipy that need compilers etc. to get installed), this is definitely the correct way to go. But if you do have non-standard requirements that fall out of the rails of your PaaS, you need to consider other solutions.
- waprin 11y ago
  The built-in Django libraries for Django on App Engine are severely outdated. I have new guides that just vendor in recent version: https://cloud.google.com/python/django/appengine https://cloud.google.com/python/django/appengine However there are a lot of tricky issues with Django on App Engine so I would also look into Djangae: https://github.com/potatolondon/djangae https://github.com/potatolondon/djangae As far as Kubernetes, the OP and I had some independent discovery and started working on similar material at the same time. I also have some Django/Kubernetes content, both CloudSQL (MySQL) and Postgres in Kubernetes: https://cloud.google.com/python/django/container-engine https://cloud.google.com/python/django/container-engine (CloudSQL) https://medium.com/google-cloud/deploying-django-postgres-redis-containers-to-kubernetes-9ee28e7a146 https://medium.com/google-cloud/deploying-django-postgres-re... (Postgres) My biggest reason for running the database in Kubernetes is a) CloudSQL doesn't currently support Postgres and b) it's interesting. Seems like OP had similar motivations. Obvious disclosure: I work for GCP.
  - hnarayanan 11y ago
    Parallel explorer and OP here. :) I cited your work as additional reading because I really enjoyed watching your talk on this recently! While I'd figured out a bulk of this stuff out, there were a few things I learnt from it too.
    - waprin 11y ago
      Oh cool, I missed your citation. I am still figuring some things out as well so I am going to check out what you've done. Definitely happy to have you contribute to the ecosystem. It seemed like a content gap that I wanted to fill in, but at Google we would of course prefer to have a healthy external ecosystem of people writing content like yourself!
hnarayanan 11y ago
I'm the author of this piece, and I agree with you in general. I am personally experimenting with this within the context of containers because I'm trying to construct something like Vitess[1] from first principles as an intellectual exercise. [1] https://github.com/youtube/vitess https://github.com/youtube/vitess
- danpalmer 11y ago
  Thanks for the blog post, it was really informative, will definitely be sharing it at work.
  - hnarayanan 11y ago
    Thank you! And please let me know of ways I can improve it. I already have a couple of ideas that I've indicated in the conclusion, but more the merrier. :)
- markbnj 11y ago
  You did a nice job on this walk-through, so thanks for the effort! I was interested to see that you ended up in the same place I have in the past when deploying databases to kubernetes clusters. A container mounted on a PD is a bit of a hybrid but it works well in practice. Hopefully something like glusterfs or flocker will end up giving us a better way to manage and deploy data volumes.
danpalmer 11y ago
I'm not sure I agree. The managed database services provided by Google and Amazon are both expensive, and lack configuration options. I'm sure they're already relatively well tuned, but we have tuned our Postgres installation to our workload more than they would be able to. Running our own Postgres install costs roughly half what it would on AWS, takes very little engineering time to maintain, and can be tuned as needed.
- throwaway13337 11y ago
  Agreed. When the costs go down and persistence-as-a-service is provided by more cloud hosts, it will make more sense but right now it often doesn't. I wonder about opportunities to provide some of these services as a white label to hosting providers like digital ocean. Is anyone doing that? What stops them?
  - danpalmer 11y ago
    If we're talking about databases specifically, it's either a really easy problem to solve that it's difficult to charge a lot for (i.e. just a Postgres/etc running, with some way of modifying the config), or it's really difficult and requires a lot of resources to get right (i.e. scaling to many nodes, still not really solved well in Postgres/MySQL).
    - throwaway13337 11y ago
      The selling point to the provider would be that it increases stickiness with their clients - that's big - they don't have to charge more.
      - lostcolony 11y ago
        I suspect that - If the service is easy enough to provide (this means not just ease of deployment but also up-time, maintenance, etc) while fitting into their margins without issue, it doesn't provide stickiness. If the service is sufficiently complex to provide stickiness, they either need to charge for it, or it will likely eat heavily into their margins. Neither of us know the specifics of Digital Ocean's business plan, but I strongly suspect that were it as simple as you put it, where it's hardly any cost to them, and a huge gain due to increased stickiness, they'd already be doing it, or be moving toward it. And they might be. However, if they aren't, then it's hardly a mystery as to why they're not providing such services.
markbnj 11y ago
I don't know if it is a "really bad" idea. I've run stacks on both AWS and GCP, and used both RDS and CloudSQL. They are good solutions and I definitely agree that looking at them first is sensible. They will fit the bill for a lot of applications. However you can get a pretty high level of fault tolerance using kubernetes services mounted on persistent volumes. You don't get all the advantages of a containerized solution but you do get dependency management and fault tolerance.
jpgvm 11y ago
If you are interested in mitigating this take a look at Flynn [1]. It's analogous to Kubernetes but is actually much simpler in this case as it can do the Heroku style deployment but also ships HA database appliances with backups out of the box. [1] https://flynn.io https://flynn.io

throwaway13337 11y ago

I'm really struggling to understand why most developers feel they need all this deployment complexity. I've run django and other web deployments with simple shell scripts and occasionally some python to glue it together. Most recently I'm running a django web server and some other custom stuff (a real time stateful server, database, nginx, etc). A somewhat complex setup (or at least as complex as it needs to be), and I have no need of docker, kubernetes. If I want a new server, I just change a few parameters and run the script to deploy. It's straight forward and without magic. The deployment flavor of the month before were DSLs like ansible, puppet, etc. Did these solve real problems? I found they added complexity without adding much. How is the new generation different? Caveat: I'm not google, and I don't deploy huge server farms. There is a use case but the vast majority of us aren't that.

hnarayanan 11y ago
I'm the author of the original piece. I echo exactly what you say in a giant caveat way up top in the piece. :) Most of the first half of the article beyond that point basically tries to motivate why you'd want to try this beyond just using a classical VM approach. But the basic idea is that it raises your level of abstraction from working with machines to working with your application components (on abstracted hardware).
- daxfohl 11y ago
  Seems like there's an opportunity there. It doesn't need to be "all this deployment complexity". Just currently it is. In fact it shouldn't be; the kubernetes level of abstraction has far less going on than the VM level. My guess is that 2-3 years from now it will be easier to set up a kubernetes cluster than to set up even the single VM at the top of your piece, and a corresponding dev env will run seamlessly on Windows, Linux, OSX. The average user won't even know or care what OS they're running in production; it'll just work (think Heroku++) Of course once it becomes ubiquitous, then containers become the new (language agnostic!) way to deploy "libraries", accessed by HTTP, and DLL hell is just moved up an abstraction level. So what have we really achieved....
- throwaway13337 11y ago

Mizza 11y ago

It's interesting that there seem to be two parallel and opposite trends in applications deployment going developing right now. One is containerizations, where developers are responsible for maintaining fat application stacks that can easily be redeployed and moved around. The other is towards serverlessness with things like [Django-Zappa](https://github.com/Miserlou/django-zappa https://github.com/Miserlou/django-zappa), where scalability is handled automatically by cloud providers. My bias is quite clear - deploying apps should be easy.

jamespacileo 11y ago

Hey Harish, thanks for your talk at the March London Django Meetup. I'm starting to play around with Google Container service as you suggested, having a good time.

hnarayanan 11y ago
Welcome!

smitec 11y ago

Putting the 'should we or shouldn't we' aside I'd just like to say thanks for putting together an article that goes all the way from a 'classic' setup to something running on a host. I often find that posts like this are a bit too meta and in trying to write things in a general way they leave out some critical step which makes replicating their ideas difficult.

hnarayanan 11y ago
OP here. That was the plan and I'm glad you appreciate it. :) I wanted to start from something I assumed people knew, and tried to motivate why one might want to improve on that. Only then did I introduce the new solution. A lot of tutorials I found jumped too quickly to the 'how' without spending enough time on the 'why' I should care.