5 ms·

I currently use Ansible to provision and configure about 50 instances with various home grown services deployed on them, and I'm quite happy with that. I don't

I currently use Ansible to provision and configure about 50 instances with various home grown services deployed on them, and I'm quite happy with that. I don't use Docker, or Kubernetes. What am I missing here? This is an honest question. I don't particularly see the point of docker in production. I guess it can help resolve clashes between dependencies, but I'm doing ok with Ansible. One guy in another team just tells his application developers to throw everything into a docker container and then deploys whatever they give him. I guess that prevents dependency clashes, sure, but to me that seems like it's just inviting different problems though. (Containers with a lot of disorganized junk in them). And we don't need to grow the cluster for now, so I don't see the point of k8. We already have log aggregation, and service restarting, service monitring/metrics. To me, Kubernetes feels like a brand new alternative OS with crappy documentation. On the other hand I've been managing UNIX machines for 20+ years, and that I do know how to fix (mostly, since they keep changing linux so much these days, it's almost as hard to keep up with as javascript). If somebody could tell me what I'm missing by using this approach, I'd be grateful.

auslander 8y ago
Because it's cool, new and good for your blog/CV. I don't see the need too, if stuff is in cloud.
originalvichy 8y ago
You aren't "missing" anything, you are just doing things differently. Containers are a way of storing code and configurations in a single deployable entity. Let's say I want to change a services version from 1.2 to 1.3. If I did this in Ansible, I might do it by commanding all of my services to pull the code of version 1.3 from a git repo and restarting to make the change. If I did this with containers, I'd tell them to pull the image version 1.3 from docker registry and after it's pulled to shut down v.1.2 and start v.1.3. Those things just have some smaller differences that are good in terms of A and worse in terms of B. Ansible helps you by keeping it closer to the traditional way of just pushing code changes to a machine and that's that, and Docker opens up this world where the infrastructure can scale up or down (elasticity) quicker horizontally. Kubernetes is just an extrapolation of that elasticity. Ansible is "good enough" if you don't need elasticity or you have programmed another custom solution to handle that. Docker/Kube and Ansible both have Git at their hearts, so if your app components have all they need as code, and you don't worry about elasticity, you won't gain a lot from wrapping it up in a Dockerfile (as opposed to just git cloning).
ngrilly 8y ago
What do you use for log aggregation and monitoring/metrics? And how do you deploy new versions with zero downtime? And how do you manage kernel/OS upgrades?
- gerbilly 8y ago
  > log aggregation Graylog, with applications using various GELF libraries to send logs to it. > metrics graphite/grafana > And how do you deploy new versions with zero downtime? I don't deploy with zero downtime, but within say ~5min. This is acceptable to us. We use jenkins with a lot of tests to ensure components are in good shape, then we mostly manually deploy them, but with a script. Sometimes we deploy directly from jenkins. > Kernel os upgrades All our stuff is internally hosted, so we handle those infrequently. This stuff isn't exposed the the open internet, so we upgrade when we get around to it, or when we encounter a bug.
  - ngrilly 8y ago
    Thanks for following up! I'm not surprised because I noticed on HN and in other online discussions that almost all the people that advocate against Kubernetes and PaaS (Heroku, App Engine, Clever Cloud, Scalingo, etc.) are willing to tolerate downtime during deploys and during OS upgrades.
    - gerbilly 8y ago
      > are willing to tolerate downtime during deploys and during OS upgrades. Yeah, that's not a big deal for us. We don't have customers, and this environment is for internal use at the company. You could say it's a 'semi production' environment.
      - ngrilly 8y ago
        I like the "semi-production" concept :-)
- therealdrag0 8y ago
ps 8y ago
I use Ansible as well and it does not allow me to scale my app containers easily, especially because of dependencies. I think I either have to manage _everything_ by Ansible including firewalls, IRB/SVI on switches, Openstack security zones, users, roles etc. or I am asking for problems and they will definitely come. NB I do not like to manage my switches, routers and firewalls with Ansible. But my infrastructure is fairly complicated, some bare metal services, many VMs, several docker hosts with dozens of various app containers, private Openstack and some public cloud services (VPNs), Ceph cluster, nagios, ELK etc. Ansible does not watch and maintain the infrastructure and services state - it is a passive tool (I am not sure if Tower is different). You definitely could configure your monitoring to invoke Ansible script if your app server goes down and reschedule services that ran on that host to a different one and reconfigure other parts of infrastructure as needed (dependencies, VXLANs, security rules, service discovery db update, load balancing update etc.) But you would essentially copy the Kubernetes controller functionality. Kubernetes can take care of the service discovery, load balancing, deployment, configuration and networking and other parts of your infrastructure and it does it pretty well in my experience. It maintains the declared state and reacts to its changes. Somebody posted here that it takes just several hours to learn it. YMMV but deploying a resilient cluster in my environment took me much more time. And I have to agree that documentation is pretty weak. E.g. provisioning Cinder volumes and connecting them to the VMs running Kubernetes nodes was a real horror. Btw Ansible generates many of my k8s YAMLs and deploys them.
- gerbilly 8y ago
  > Ansible does not watch and maintain the infrastructure and services state Yeah, but what I've done is that i've deployed a tiny go binary on each cloud instance that is run every few minutes. Instances take turns checking on each other using a round robin sort of approach (no complicated leader election algorithms etc...) The script knows how to check the health of the other instances/services and to restart them or alarm if they get stuck. For fifty hosts, it works fairly well. I didn't say in the original post, but we are not a product group, so our stuff is 'semi production'. We don't have customers to worry about. >Somebody posted here that it takes just several hours to learn it. Sure I can probably learn it quickly, but what I don't want is to now have complicated and mysterious kubernetes problems to solve on a deadline. I understand linux pretty well, been using it for 20+years, so I'm not intimidated by OS level troubleshooting. Sure without containers, you have to be more careful to keep your dependencies from overlapping, but it hasn't been a problem we couldn't handle till now. I don't particularly want to trade problems I'm familiar with solving for a whole new set of unfamiliar problems unless there is a clear benefit.
shlant 8y ago
as someone in almost the same situation (about 50 instances), but chose to go with docker, I just want to say you made the right choice. Of course a few years ago, being young and a one man ops team, I wanted to use the cool new thing and so did everyone else. I went with Ansible after being a Chef guy for a while due to ease of getting started. That was the easy part. Love Ansible, but it was slow to get new Docker functionality for a while, and not having the built in functionality of having Chef runs happening periodically without paying for Tower means I have to be more mindful and deliberate with keeping my infra in sync and up to date. Enter Docker and months and months of headaches just to get something usable on the local workstation level. So much wasted time dealing with breaking changes and figuring out exactly which version to use and very rarely upgrading. Debugging and troubleshooting becomes an unintuitive nightmare. Even after all that, our site still ended up running really slow locally in Docker. This is because with apps (such as CMS) that handle many, many files, I/O slows wayyy down. Ended up discovering Dinghy (shout out to codekitchen) and got it to a decent state (but make sure developers don't accidentally install Docker for Mac as it is and always has been a CPU gobbling mess). Then on top of that there is the container orchestration (consul-template), monitoring (Prometheus is cool, but takes a bit to understand what is needed to get the metrics you want), logging (fluentd is again, cool, but oh man is parsing logs a PITA to understand), debugging tools etc. I can't even imagine what adding k8 on top of the many quirks of Docker would mean. I luckily took one look at needing a whole zookeeper cluster just to get started, and immediately gave up on that.

auslander 8y ago

Because it's cool, new and good for your blog/CV. I don't see the need too, if stuff is in cloud.

originalvichy 8y ago

You aren't "missing" anything, you are just doing things differently. Containers are a way of storing code and configurations in a single deployable entity. Let's say I want to change a services version from 1.2 to 1.3. If I did this in Ansible, I might do it by commanding all of my services to pull the code of version 1.3 from a git repo and restarting to make the change. If I did this with containers, I'd tell them to pull the image version 1.3 from docker registry and after it's pulled to shut down v.1.2 and start v.1.3. Those things just have some smaller differences that are good in terms of A and worse in terms of B. Ansible helps you by keeping it closer to the traditional way of just pushing code changes to a machine and that's that, and Docker opens up this world where the infrastructure can scale up or down (elasticity) quicker horizontally. Kubernetes is just an extrapolation of that elasticity. Ansible is "good enough" if you don't need elasticity or you have programmed another custom solution to handle that. Docker/Kube and Ansible both have Git at their hearts, so if your app components have all they need as code, and you don't worry about elasticity, you won't gain a lot from wrapping it up in a Dockerfile (as opposed to just git cloning).

ngrilly 8y ago

What do you use for log aggregation and monitoring/metrics? And how do you deploy new versions with zero downtime? And how do you manage kernel/OS upgrades?

gerbilly 8y ago
> log aggregation Graylog, with applications using various GELF libraries to send logs to it. > metrics graphite/grafana > And how do you deploy new versions with zero downtime? I don't deploy with zero downtime, but within say ~5min. This is acceptable to us. We use jenkins with a lot of tests to ensure components are in good shape, then we mostly manually deploy them, but with a script. Sometimes we deploy directly from jenkins. > Kernel os upgrades All our stuff is internally hosted, so we handle those infrequently. This stuff isn't exposed the the open internet, so we upgrade when we get around to it, or when we encounter a bug.
- ngrilly 8y ago
  Thanks for following up! I'm not surprised because I noticed on HN and in other online discussions that almost all the people that advocate against Kubernetes and PaaS (Heroku, App Engine, Clever Cloud, Scalingo, etc.) are willing to tolerate downtime during deploys and during OS upgrades.
  - gerbilly 8y ago
    > are willing to tolerate downtime during deploys and during OS upgrades. Yeah, that's not a big deal for us. We don't have customers, and this environment is for internal use at the company. You could say it's a 'semi production' environment.
    - ngrilly 8y ago
      I like the "semi-production" concept :-)
therealdrag0 8y ago

ps 8y ago

I use Ansible as well and it does not allow me to scale my app containers easily, especially because of dependencies. I think I either have to manage _everything_ by Ansible including firewalls, IRB/SVI on switches, Openstack security zones, users, roles etc. or I am asking for problems and they will definitely come. NB I do not like to manage my switches, routers and firewalls with Ansible. But my infrastructure is fairly complicated, some bare metal services, many VMs, several docker hosts with dozens of various app containers, private Openstack and some public cloud services (VPNs), Ceph cluster, nagios, ELK etc. Ansible does not watch and maintain the infrastructure and services state - it is a passive tool (I am not sure if Tower is different). You definitely could configure your monitoring to invoke Ansible script if your app server goes down and reschedule services that ran on that host to a different one and reconfigure other parts of infrastructure as needed (dependencies, VXLANs, security rules, service discovery db update, load balancing update etc.) But you would essentially copy the Kubernetes controller functionality. Kubernetes can take care of the service discovery, load balancing, deployment, configuration and networking and other parts of your infrastructure and it does it pretty well in my experience. It maintains the declared state and reacts to its changes. Somebody posted here that it takes just several hours to learn it. YMMV but deploying a resilient cluster in my environment took me much more time. And I have to agree that documentation is pretty weak. E.g. provisioning Cinder volumes and connecting them to the VMs running Kubernetes nodes was a real horror. Btw Ansible generates many of my k8s YAMLs and deploys them.

gerbilly 8y ago
> Ansible does not watch and maintain the infrastructure and services state Yeah, but what I've done is that i've deployed a tiny go binary on each cloud instance that is run every few minutes. Instances take turns checking on each other using a round robin sort of approach (no complicated leader election algorithms etc...) The script knows how to check the health of the other instances/services and to restart them or alarm if they get stuck. For fifty hosts, it works fairly well. I didn't say in the original post, but we are not a product group, so our stuff is 'semi production'. We don't have customers to worry about. >Somebody posted here that it takes just several hours to learn it. Sure I can probably learn it quickly, but what I don't want is to now have complicated and mysterious kubernetes problems to solve on a deadline. I understand linux pretty well, been using it for 20+years, so I'm not intimidated by OS level troubleshooting. Sure without containers, you have to be more careful to keep your dependencies from overlapping, but it hasn't been a problem we couldn't handle till now. I don't particularly want to trade problems I'm familiar with solving for a whole new set of unfamiliar problems unless there is a clear benefit.

shlant 8y ago

as someone in almost the same situation (about 50 instances), but chose to go with docker, I just want to say you made the right choice. Of course a few years ago, being young and a one man ops team, I wanted to use the cool new thing and so did everyone else. I went with Ansible after being a Chef guy for a while due to ease of getting started. That was the easy part. Love Ansible, but it was slow to get new Docker functionality for a while, and not having the built in functionality of having Chef runs happening periodically without paying for Tower means I have to be more mindful and deliberate with keeping my infra in sync and up to date. Enter Docker and months and months of headaches just to get something usable on the local workstation level. So much wasted time dealing with breaking changes and figuring out exactly which version to use and very rarely upgrading. Debugging and troubleshooting becomes an unintuitive nightmare. Even after all that, our site still ended up running really slow locally in Docker. This is because with apps (such as CMS) that handle many, many files, I/O slows wayyy down. Ended up discovering Dinghy (shout out to codekitchen) and got it to a decent state (but make sure developers don't accidentally install Docker for Mac as it is and always has been a CPU gobbling mess). Then on top of that there is the container orchestration (consul-template), monitoring (Prometheus is cool, but takes a bit to understand what is needed to get the metrics you want), logging (fluentd is again, cool, but oh man is parsing logs a PITA to understand), debugging tools etc. I can't even imagine what adding k8 on top of the many quirks of Docker would mean. I luckily took one look at needing a whole zookeeper cluster just to get started, and immediately gave up on that.