6 ms·

I'm really struggling to understand why most developers feel they need all this deployment complexity. I've run django and other web deployments with simple sh

I'm really struggling to understand why most developers feel they need all this deployment complexity. I've run django and other web deployments with simple shell scripts and occasionally some python to glue it together. Most recently I'm running a django web server and some other custom stuff (a real time stateful server, database, nginx, etc). A somewhat complex setup (or at least as complex as it needs to be), and I have no need of docker, kubernetes. If I want a new server, I just change a few parameters and run the script to deploy. It's straight forward and without magic. The deployment flavor of the month before were DSLs like ansible, puppet, etc. Did these solve real problems? I found they added complexity without adding much. How is the new generation different? Caveat: I'm not google, and I don't deploy huge server farms. There is a use case but the vast majority of us aren't that.

hnarayanan 11y ago
I'm the author of the original piece. I echo exactly what you say in a giant caveat way up top in the piece. :) Most of the first half of the article beyond that point basically tries to motivate why you'd want to try this beyond just using a classical VM approach. But the basic idea is that it raises your level of abstraction from working with machines to working with your application components (on abstracted hardware).
- daxfohl 11y ago
  Seems like there's an opportunity there. It doesn't need to be "all this deployment complexity". Just currently it is. In fact it shouldn't be; the kubernetes level of abstraction has far less going on than the VM level. My guess is that 2-3 years from now it will be easier to set up a kubernetes cluster than to set up even the single VM at the top of your piece, and a corresponding dev env will run seamlessly on Windows, Linux, OSX. The average user won't even know or care what OS they're running in production; it'll just work (think Heroku++) Of course once it becomes ubiquitous, then containers become the new (language agnostic!) way to deploy "libraries", accessed by HTTP, and DLL hell is just moved up an abstraction level. So what have we really achieved....
- throwaway13337 11y ago
  Right - I saw that. The problem I have is that most people feel like theyre doing 'serious work' so they need a serious solution when the reality is that it really is just a small percentage of companies - not usually even successful startups, for instance - that require this sort of automation. Mostly, it adds complexity rather than reduces it. I know that sounds like we're getting into the weeds, but if you consider that most companies are taking on additional complexity because of this, it becomes a problem.
  - hnarayanan 11y ago
    Again, I don't generally disagree with you. But as our team is growing at my day job, I am finding more and more that we're getting bitten by some developer's "It works on my machine" or a random npm failure at deploy time. And this makes me believe that having a prebuilt (immutable, hermetically sealed) artefact that I can later deploy is the way to go. Containers solve this problem nicely. And once you're going down the container route, you need something like a Kubernetes to easily schedule and network them. My personal site -- the one you're reading the article on -- is literally just a static nginx server on an old Linux box (with the files copied in by rsync). Even the curmudgeon in me now thinks it makes sense to use solutions of different complexity for different situations.
  - jsmthrowaway 11y ago
    I've run production at a 'successful startup,' a million node corporate fleet, and a personal toy project or six, and consider Kubernetes the single most important work being done in the field right now. Docker, no (rkt is better architected). Kubernetes, yes[0]. Don't think of Kubernetes as a replacement for Puppet. Think of it as distributed init/supervisord with service discovery, which you are clumsily building with Puppet. Your automation ends up writing init, units, monit, supervisor, and so on, right? (You shouldn't be deploying apps with Puppet anyway because CD is not CM, but nothing like Kubernetes or Mesos really existed in the open source world until now so a whole generation learned to do it with Puppet.) Kubernetes exists for when your scripted machine fails or you exceed the capacity of your scripted machine and need to horizontally scale. It is not a deployment tool per se; that's only part of what it does as a scheduler and resource manager. Every project, from personal to Fortune 500, needs to plan for those scenarios. I'm not saying Kubernetes is always the answer, I'm saying it solves a problem that every project universally has, despite your claims. The bonus of working that way is now you have an API to your machines and can treat them as a single unit of resources. Kubernetes is about half of building your own mini Heroku. You've heard snowflakes versus cattle, right? Your way is for snowflakes. Kubernetes is for cattle. The threshold where one becomes more productive than the other is the constant debate, and as an SRE, my opinion bucks must people who talk about this on where that threshold lies. If you self-identify as a sysadmin you'll likely have one opinion, devops another, and then you have my group of crazies that generally want to crush all snowflakes. I can speak with experience that our crazy SRE ways scale pretty well to the hundreds of thousands of nodes case but also work quite well for toy systems. Subscribe to everything CoreOS is doing. There's a spectrum of quality there, but they're running with the programmable infrastructure ideal and have largely bet the company on the Kubernetes ecosystem. [0]: Kubernetes makes some interesting choices versus Borg and I think they're going to have trouble scaling it to that, and Borg really shines at Google because of the global filesystem layer that exists on every machine (they don't realize this and are doing gymnastics around storage in the open source side), but if they can mature Kubernetes it'll be a solid building block for platform development and eliminate a lot of the clumsy scripts and automation that you are defending and push us toward programmable infrastructure.
- jordic 11y ago
  There are a lot more advantages on abstracting the hardware from the app. First of all, you don't need one machine per service for scaling the app. All services run on the cluster, and you can add or remove machines, as needed. (On night you can reduce your cluster size, and grow it during the day). But also, you can use the same cluster to run your preproduction enviroment, or use it for CI/CD. (Check deis) As an example, we have an small cluster with two machines and on it, there are: - the main app, - the preprod environment, - two more feature branches (that had to be reviewed) - And also commercials can deploy playground environments to make demos and trials. All are independent apps, sharing resources, on the cluster. As a summary, on daily, we maintain 7 to 10 independent apps instances, and we do regular updates (as new revisions arrives). As an example, the trials or feature branches, are single pods (all included, db, redis, app and worker). Kubernetes is not for deploying pet projects, but as soon as you start working on a real project is a must. You can manage complex deploys with a lot of services.. keeping the costs as lows as two n1-standard-1 machines. And as a plus on gke, you got monitoring, and logging aggregation.
  - hnarayanan 11y ago
    Excellent points.
  - drdaeman 11y ago
    I think the questions was "is this complexity necessary?" If the same (deployment and upgrades for production, staging and testing environments, for any number of hosts) could be achieved with a small shell script - doesn't this mean that cluster thing isn't really useful? I think a lot of "real" projects (smaller ones) work perfectly well without any complicated cluster management tools, and aren't really hindered by lack of what those offer.
danpalmer 11y ago
As someone who does pretty much this for my personal sites, and has a level slightly above at work, I can say anecdotally that your caveat is important. The way you describe works well for me on personal projects, but at work we're really starting to see the benefit of the automation and reproducibility that things like Docker and Kubernetes provide. Ansible is great, but it's still easy to end up with non-reproducible environments, and that's something we've been bitten by quite a few times.
- clebio 11y ago
  Can you expand a bit on how you still end up with non-reproducible environments? Not that I doubt it! Just would like to here some details, or a specific example.
  - luos 11y ago
    Usually when you ssh into the machine to fix something broken then forget to backport the changes to the ansible script.
    - merb 11y ago
      The same thing could happen with docker, too.
      - vetinari 11y ago
        You usually can't ssh into docker container because: 1) it usually doesn't have sshd 2) you change would be thrown away at nearest container restart or migration. The only way for it to persist is to do the right thing, i.e. apply the change where you should and have the container rebuilt.
        merb 11y ago
        You don't need ssh to get into a docker container.
  - 11y ago
drdaeman 11y ago
I think, the only difference between Puppet/Salt/Chef and shell scripts, is that CM tools already have a collection of common building blocks for stuff like "check that package uwsgi is installed" or "check that uwsgi config for application 'mywebsite' matches this template", plus dependencies between those blocks (including dependencies on config file templates and automatic reloads on changes). I tried both approaches and can say that I feel that Puppet and Salt actually save a bit of time, by not having to inventing grep/sed/awk invocations to manipulate files. They're a bit more verbose than `scp ./files/nginx.conf $HOST:/etc/nginx/` or `ssh $HOST sed -ie 's/.../.../ file.conf' for most basic stuff, but way less verbose for anything even a bit more complicated. I don't use Docker or Kubernetes, so no idea about those. But I think configuration management tools can save time. But their usefulness is debatable if, say, all you need to deploy is to scp and untar a simple archive then poke init system to spawn or reload a daemon.
deleted 11y ago
[deleted]
TheIronYuppie 11y ago
It's an interesting question, but the general problem is that even small projects have problems when bringing things to production. For example, let's use Django: - How do you make sure the version of Django on your machine matches the one in production? - What if you do brew update on your Mac and the package for your repo in production is out of date? - How do you roll out a new version of your app (even allowing for downtime) and make sure it did it "completely"? - How do you make sure your app keeps running even if your VM goes down (let's say you need to do a kernel upgrade)? - How do you use one password for your local development database and one for your production database (and not hard wire it into code)? These are super common problems - you're going to have to solve them no matter what. Lots of folks do this with scripts, but one small change can cause all kinds of annoyances. This is why containers have taken off, and all of these are addressed via the combination of containers & Kubernetes. It's a different way of thinking about development, but I've been able to accelerate my velocity substantially once I adopted them. I'd argue that the complexity that you accept as 'normal' for a small site is an unnecessary tax, but I'm biased :) Disclosure: I work at Google on Kubernetes.
- metafunctor 11y ago
  All of the things you've listed can be done with simple, small, and straightforward scripts. To check for versions of Django, and other libraries, you write simple scripts to test for them. Run them in unit tests, and as a pre-check stage in deployment scripts. Our apps do keep running if any single VM goes down. We don't need Kubernetes or anything to handle that. Just a load balancer and an ElasticSearch cluster. Dev hosts obviously have different passwords read from config files only present in development systems.

hnarayanan 11y ago

I'm the author of the original piece. I echo exactly what you say in a giant caveat way up top in the piece. :) Most of the first half of the article beyond that point basically tries to motivate why you'd want to try this beyond just using a classical VM approach. But the basic idea is that it raises your level of abstraction from working with machines to working with your application components (on abstracted hardware).

daxfohl 11y ago
Seems like there's an opportunity there. It doesn't need to be "all this deployment complexity". Just currently it is. In fact it shouldn't be; the kubernetes level of abstraction has far less going on than the VM level. My guess is that 2-3 years from now it will be easier to set up a kubernetes cluster than to set up even the single VM at the top of your piece, and a corresponding dev env will run seamlessly on Windows, Linux, OSX. The average user won't even know or care what OS they're running in production; it'll just work (think Heroku++) Of course once it becomes ubiquitous, then containers become the new (language agnostic!) way to deploy "libraries", accessed by HTTP, and DLL hell is just moved up an abstraction level. So what have we really achieved....
throwaway13337 11y ago
Right - I saw that. The problem I have is that most people feel like theyre doing 'serious work' so they need a serious solution when the reality is that it really is just a small percentage of companies - not usually even successful startups, for instance - that require this sort of automation. Mostly, it adds complexity rather than reduces it. I know that sounds like we're getting into the weeds, but if you consider that most companies are taking on additional complexity because of this, it becomes a problem.
- hnarayanan 11y ago
  Again, I don't generally disagree with you. But as our team is growing at my day job, I am finding more and more that we're getting bitten by some developer's "It works on my machine" or a random npm failure at deploy time. And this makes me believe that having a prebuilt (immutable, hermetically sealed) artefact that I can later deploy is the way to go. Containers solve this problem nicely. And once you're going down the container route, you need something like a Kubernetes to easily schedule and network them. My personal site -- the one you're reading the article on -- is literally just a static nginx server on an old Linux box (with the files copied in by rsync). Even the curmudgeon in me now thinks it makes sense to use solutions of different complexity for different situations.
- jsmthrowaway 11y ago
  I've run production at a 'successful startup,' a million node corporate fleet, and a personal toy project or six, and consider Kubernetes the single most important work being done in the field right now. Docker, no (rkt is better architected). Kubernetes, yes[0]. Don't think of Kubernetes as a replacement for Puppet. Think of it as distributed init/supervisord with service discovery, which you are clumsily building with Puppet. Your automation ends up writing init, units, monit, supervisor, and so on, right? (You shouldn't be deploying apps with Puppet anyway because CD is not CM, but nothing like Kubernetes or Mesos really existed in the open source world until now so a whole generation learned to do it with Puppet.) Kubernetes exists for when your scripted machine fails or you exceed the capacity of your scripted machine and need to horizontally scale. It is not a deployment tool per se; that's only part of what it does as a scheduler and resource manager. Every project, from personal to Fortune 500, needs to plan for those scenarios. I'm not saying Kubernetes is always the answer, I'm saying it solves a problem that every project universally has, despite your claims. The bonus of working that way is now you have an API to your machines and can treat them as a single unit of resources. Kubernetes is about half of building your own mini Heroku. You've heard snowflakes versus cattle, right? Your way is for snowflakes. Kubernetes is for cattle. The threshold where one becomes more productive than the other is the constant debate, and as an SRE, my opinion bucks must people who talk about this on where that threshold lies. If you self-identify as a sysadmin you'll likely have one opinion, devops another, and then you have my group of crazies that generally want to crush all snowflakes. I can speak with experience that our crazy SRE ways scale pretty well to the hundreds of thousands of nodes case but also work quite well for toy systems. Subscribe to everything CoreOS is doing. There's a spectrum of quality there, but they're running with the programmable infrastructure ideal and have largely bet the company on the Kubernetes ecosystem. [0]: Kubernetes makes some interesting choices versus Borg and I think they're going to have trouble scaling it to that, and Borg really shines at Google because of the global filesystem layer that exists on every machine (they don't realize this and are doing gymnastics around storage in the open source side), but if they can mature Kubernetes it'll be a solid building block for platform development and eliminate a lot of the clumsy scripts and automation that you are defending and push us toward programmable infrastructure.
jordic 11y ago
There are a lot more advantages on abstracting the hardware from the app. First of all, you don't need one machine per service for scaling the app. All services run on the cluster, and you can add or remove machines, as needed. (On night you can reduce your cluster size, and grow it during the day). But also, you can use the same cluster to run your preproduction enviroment, or use it for CI/CD. (Check deis) As an example, we have an small cluster with two machines and on it, there are: - the main app, - the preprod environment, - two more feature branches (that had to be reviewed) - And also commercials can deploy playground environments to make demos and trials. All are independent apps, sharing resources, on the cluster. As a summary, on daily, we maintain 7 to 10 independent apps instances, and we do regular updates (as new revisions arrives). As an example, the trials or feature branches, are single pods (all included, db, redis, app and worker). Kubernetes is not for deploying pet projects, but as soon as you start working on a real project is a must. You can manage complex deploys with a lot of services.. keeping the costs as lows as two n1-standard-1 machines. And as a plus on gke, you got monitoring, and logging aggregation.
- hnarayanan 11y ago
  Excellent points.
- drdaeman 11y ago
  I think the questions was "is this complexity necessary?" If the same (deployment and upgrades for production, staging and testing environments, for any number of hosts) could be achieved with a small shell script - doesn't this mean that cluster thing isn't really useful? I think a lot of "real" projects (smaller ones) work perfectly well without any complicated cluster management tools, and aren't really hindered by lack of what those offer.

danpalmer 11y ago

As someone who does pretty much this for my personal sites, and has a level slightly above at work, I can say anecdotally that your caveat is important. The way you describe works well for me on personal projects, but at work we're really starting to see the benefit of the automation and reproducibility that things like Docker and Kubernetes provide. Ansible is great, but it's still easy to end up with non-reproducible environments, and that's something we've been bitten by quite a few times.

clebio 11y ago
Can you expand a bit on how you still end up with non-reproducible environments? Not that I doubt it! Just would like to here some details, or a specific example.
- luos 11y ago
  Usually when you ssh into the machine to fix something broken then forget to backport the changes to the ansible script.
  - merb 11y ago
    The same thing could happen with docker, too.
    - vetinari 11y ago
      You usually can't ssh into docker container because: 1) it usually doesn't have sshd 2) you change would be thrown away at nearest container restart or migration. The only way for it to persist is to do the right thing, i.e. apply the change where you should and have the container rebuilt.
      - merb 11y ago
        You don't need ssh to get into a docker container.
- 11y ago

drdaeman 11y ago

I think, the only difference between Puppet/Salt/Chef and shell scripts, is that CM tools already have a collection of common building blocks for stuff like "check that package uwsgi is installed" or "check that uwsgi config for application 'mywebsite' matches this template", plus dependencies between those blocks (including dependencies on config file templates and automatic reloads on changes). I tried both approaches and can say that I feel that Puppet and Salt actually save a bit of time, by not having to inventing grep/sed/awk invocations to manipulate files. They're a bit more verbose than `scp ./files/nginx.conf $HOST:/etc/nginx/` or `ssh $HOST sed -ie 's/.../.../ file.conf' for most basic stuff, but way less verbose for anything even a bit more complicated. I don't use Docker or Kubernetes, so no idea about those. But I think configuration management tools can save time. But their usefulness is debatable if, say, all you need to deploy is to scp and untar a simple archive then poke init system to spawn or reload a daemon.

deleted 11y ago

[deleted]

TheIronYuppie 11y ago

It's an interesting question, but the general problem is that even small projects have problems when bringing things to production. For example, let's use Django: - How do you make sure the version of Django on your machine matches the one in production? - What if you do brew update on your Mac and the package for your repo in production is out of date? - How do you roll out a new version of your app (even allowing for downtime) and make sure it did it "completely"? - How do you make sure your app keeps running even if your VM goes down (let's say you need to do a kernel upgrade)? - How do you use one password for your local development database and one for your production database (and not hard wire it into code)? These are super common problems - you're going to have to solve them no matter what. Lots of folks do this with scripts, but one small change can cause all kinds of annoyances. This is why containers have taken off, and all of these are addressed via the combination of containers & Kubernetes. It's a different way of thinking about development, but I've been able to accelerate my velocity substantially once I adopted them. I'd argue that the complexity that you accept as 'normal' for a small site is an unnecessary tax, but I'm biased :) Disclosure: I work at Google on Kubernetes.

metafunctor 11y ago
All of the things you've listed can be done with simple, small, and straightforward scripts. To check for versions of Django, and other libraries, you write simple scripts to test for them. Run them in unit tests, and as a pre-check stage in deployment scripts. Our apps do keep running if any single VM goes down. We don't need Kubernetes or anything to handle that. Just a load balancer and an ElasticSearch cluster. Dev hosts obviously have different passwords read from config files only present in development systems.