3 ms·

Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad? Of the two Nomad seems much more sane because it does one thing only an

by aprdm 6y ago

Do people that aren't cloudflare scale really see the need for kubernetes and/or Nomad? Of the two Nomad seems much more sane because it does one thing only and is much simpler to manage and deploy. That said, having have used it, we are mostly moving away from it. Consul + Docker/Docker-compose with systemd in "a service per vm" model has proved much easier to administrate to our scale (couple of datacenters, ~1k VM mark). It is actually so simple that is boring. Instead of fiddling with infrastructure the developers spend time solving business problems... Our stack is really boring, developers can find their way around ansible and share galaxy roles to configure / provision infrastructure... Other cloud native projects like Prometheus and Fluentd give us all the visibility we need in a very straight forward and boring way. p.s: Great article!

shanemhansen 6y ago
Imo kubernetes isn't about "scale". Folks like cloudflare often run a very limited number of services (proxy plus control plane), which can make features like quotas/affinities/limits less useful because 100% of the machine installed in the pop is dedicated to one process. I see kubernetes as more useful when organizationally you have more heterogeneous services and separate dev/sre groups. This allows them to divide up responsibilities more easily.
- halbritt 6y ago
  Increases velocity as well. I have a set of developers that can focus on their code and not have to worry about the infrastructure, TLS certs, DNS, storage or whatever. That all gets abstracted away. For SRE, that gives one a common set of orchestration tooling one can develop against.
stitched2gethr 6y ago
From my personal experience, yes, but not just for scale. I've used Kubernetes in a 300k person organization and now in a 35 person organization. Kubernetes isn't the simplest solution, but it's just reliability in a box. You can control the network, and the servers, and the load balancing all through the API which means for most things I can install a fully clustered solution with a handful of commands. Set the number of instances I want to run and forget the whole thing exists. Suddenly I'm cloud agnostic by default and the underlying K8s management is handled by the cloud provider. In the 300k person org it was for scale and reliability. In the 35 person org it's so that 1 guy can manage everything without trying.
- aprdm 6y ago
  I see. My last work experiences have been introducing devops workflows into companies that already had a decent tech body (~200 people) and the traditional divide between IT/Dev. I think the approach I mention ends up being a good compromise from both sides. If I were starting a company from scratch I would certainly consider it differently.
dmwilcox 6y ago
Interesting, I went the other direction recently, from systemd to Nomad. I was motivated by a move away from config management in favor of commands wrapping Nomad API calls. For a "devs on-call" model this was preferable to being gatekeepers of PRs against config management. To glue the whole thing together I got Consul Connect going in the Nomad jobs, so service config complexity was comparable to docker-compose. Saying this out loud makes me realize it was the organizational model I was pursuing that led me here (so called "production ownership"). And I'm not a big config management fan ;). I take it you have dedicated Ops teams or Devs willing to learn Ansible well?
- aprdm 6y ago
  Our devs to their own ops using our blessed ansible galaxy roles + creating their own when necessary. It is all pretty standardized, is rare that someone needs to create a new galaxy role for example. We expect our teams to know their way around Ansible ( which hasn’t been a problem ), we have internal docs and training

by aprdm 6y ago

shanemhansen 6y ago

Imo kubernetes isn't about "scale". Folks like cloudflare often run a very limited number of services (proxy plus control plane), which can make features like quotas/affinities/limits less useful because 100% of the machine installed in the pop is dedicated to one process. I see kubernetes as more useful when organizationally you have more heterogeneous services and separate dev/sre groups. This allows them to divide up responsibilities more easily.

halbritt 6y ago
Increases velocity as well. I have a set of developers that can focus on their code and not have to worry about the infrastructure, TLS certs, DNS, storage or whatever. That all gets abstracted away. For SRE, that gives one a common set of orchestration tooling one can develop against.

stitched2gethr 6y ago

From my personal experience, yes, but not just for scale. I've used Kubernetes in a 300k person organization and now in a 35 person organization. Kubernetes isn't the simplest solution, but it's just reliability in a box. You can control the network, and the servers, and the load balancing all through the API which means for most things I can install a fully clustered solution with a handful of commands. Set the number of instances I want to run and forget the whole thing exists. Suddenly I'm cloud agnostic by default and the underlying K8s management is handled by the cloud provider. In the 300k person org it was for scale and reliability. In the 35 person org it's so that 1 guy can manage everything without trying.

aprdm 6y ago
I see. My last work experiences have been introducing devops workflows into companies that already had a decent tech body (~200 people) and the traditional divide between IT/Dev. I think the approach I mention ends up being a good compromise from both sides. If I were starting a company from scratch I would certainly consider it differently.

dmwilcox 6y ago

Interesting, I went the other direction recently, from systemd to Nomad. I was motivated by a move away from config management in favor of commands wrapping Nomad API calls. For a "devs on-call" model this was preferable to being gatekeepers of PRs against config management. To glue the whole thing together I got Consul Connect going in the Nomad jobs, so service config complexity was comparable to docker-compose. Saying this out loud makes me realize it was the organizational model I was pursuing that led me here (so called "production ownership"). And I'm not a big config management fan ;). I take it you have dedicated Ops teams or Devs willing to learn Ansible well?

aprdm 6y ago
Our devs to their own ops using our blessed ansible galaxy roles + creating their own when necessary. It is all pretty standardized, is rare that someone needs to create a new galaxy role for example. We expect our teams to know their way around Ansible ( which hasn’t been a problem ), we have internal docs and training