4 ms·

what are some of the downsides of K8? any horror stories?

dilatedmind 8y ago
They recently added namespaces to specs, and I think they should add context as well. Not really a horror story, but we ran into some mysterious behavior when one of our services appeared to alternate between two difference instances of an app. It was amusing to find someone had deployed prod specs to both prod and dev clusters, The only difference being their dns. The result was a fight between the two clusters' dns controllers to sync the A record to their respective endpoints.
fear91 8y ago
You need to hire people who are competent with a new, quickly evolving technology. That means it's hard to hire someone good and it's expensive. The upside is that you can replace multiple "normal" ops people with just a few K8 folks but that also requires a bit of culture change of how you do development.
- AndyNemmity 8y ago
  I started a year ago on a team, and there are so many different technologies, things to know, gotchas everywhere, that even competent people can struggle outside of taking a single domain and owning it. It is not remotely easy to actually use and understand all parts of a massive system, at least ours.
zie 8y ago
like @fear91 almost said, but didn't quite get there.. it's operationally VERY complex. It has it's reasons for being so complex, but jeez! doesn't help that tooling and docs are all over the map. Right now I'd say it's useful only if you can afford to hire a handful of very talented k8s knowledgeable people to babysit it, because it def. needs babysitting. Maybe in a few years it will be nicely stable. Otherwise you make someone like Google, Microsoft, etc run it for you. Or skip the complexities and use something operationally simple and sane like Nomad or FreeBSD Jails.
markbnj 8y ago
After three years of running it in production I don't really have any horror stories, but we have definitely run into odd and puzzling things. For example, when we were experimenting with auto-scaling node-pools on GKE, and the cluster would scale a node-pool up then fail to scale it back down due to the fact that there are daemonsets on every node of a GKE cluster, and this left it in a state where it flopped back and forth between two nodes every ten minutes, evicting the entire workload, trying to delete the node, failing, and resetting. Thing is, we have also run into really bad things in our puppeted infrastructure that we run on vms and manage with terraform. Like the time I accidentally ran a command in the wrong folder and deleted 120 vms :). Well maybe that was just me and a deficiency of our tooling, but anyway one thing is for sure: k8s makes it far easier to recover from situations that arise, once you understand what the problem is and update the cluster with the right state, the controllers will set things right.
nvarsj 8y ago
(I've been using k8s for several years now, and have a couple dozen contributions so far) Two big drawbacks. First, with k8s itself is _stability_. K8s was a design-by-committee attempt to make Borg 2.0 with open source. It had massive feature development (which continues), but little actual real usage of those features. So it has taken a long time for things to get into a good shape. Things are much much better now, because of the popularity. "Given enough eyeballs, all bugs are shallow". But expect to debug things, especially if you use a feature not used by GKE. The other big downside is it takes a lot of domain knowledge to manage it, and keep up to date (there is no LTS version). There isn't a standard way of deploying it into on-prem either. So that stuff needs to be built.

They recently added namespaces to specs, and I think they should add context as well. Not really a horror story, but we ran into some mysterious behavior when one of our services appeared to alternate between two difference instances of an app. It was amusing to find someone had deployed prod specs to both prod and dev clusters, The only difference being their dns. The result was a fight between the two clusters' dns controllers to sync the A record to their respective endpoints.

fear91 8y ago

You need to hire people who are competent with a new, quickly evolving technology. That means it's hard to hire someone good and it's expensive. The upside is that you can replace multiple "normal" ops people with just a few K8 folks but that also requires a bit of culture change of how you do development.

AndyNemmity 8y ago
I started a year ago on a team, and there are so many different technologies, things to know, gotchas everywhere, that even competent people can struggle outside of taking a single domain and owning it. It is not remotely easy to actually use and understand all parts of a massive system, at least ours.

zie 8y ago

like @fear91 almost said, but didn't quite get there.. it's operationally VERY complex. It has it's reasons for being so complex, but jeez! doesn't help that tooling and docs are all over the map. Right now I'd say it's useful only if you can afford to hire a handful of very talented k8s knowledgeable people to babysit it, because it def. needs babysitting. Maybe in a few years it will be nicely stable. Otherwise you make someone like Google, Microsoft, etc run it for you. Or skip the complexities and use something operationally simple and sane like Nomad or FreeBSD Jails.

markbnj 8y ago

After three years of running it in production I don't really have any horror stories, but we have definitely run into odd and puzzling things. For example, when we were experimenting with auto-scaling node-pools on GKE, and the cluster would scale a node-pool up then fail to scale it back down due to the fact that there are daemonsets on every node of a GKE cluster, and this left it in a state where it flopped back and forth between two nodes every ten minutes, evicting the entire workload, trying to delete the node, failing, and resetting. Thing is, we have also run into really bad things in our puppeted infrastructure that we run on vms and manage with terraform. Like the time I accidentally ran a command in the wrong folder and deleted 120 vms :). Well maybe that was just me and a deficiency of our tooling, but anyway one thing is for sure: k8s makes it far easier to recover from situations that arise, once you understand what the problem is and update the cluster with the right state, the controllers will set things right.

nvarsj 8y ago

(I've been using k8s for several years now, and have a couple dozen contributions so far) Two big drawbacks. First, with k8s itself is _stability_. K8s was a design-by-committee attempt to make Borg 2.0 with open source. It had massive feature development (which continues), but little actual real usage of those features. So it has taken a long time for things to get into a good shape. Things are much much better now, because of the popularity. "Given enough eyeballs, all bugs are shallow". But expect to debug things, especially if you use a feature not used by GKE. The other big downside is it takes a lot of domain knowledge to manage it, and keep up to date (there is no LTS version). There isn't a standard way of deploying it into on-prem either. So that stuff needs to be built.