3 ms·
Can I be honest? As someone who hopped on the K8s bandwagon back in the early days (circa-early 2017), _do not_ go into production with Kubernetes if you're st
by nameless912 7y ago
Can I be honest?
As someone who hopped on the K8s bandwagon back in the early days (circa-early 2017), _do not_ go into production with Kubernetes if you're still asking this question.
Just a few of the issues I've run into over the past 2 1/2 years or so:
- Kubernetes DNS flaking out completely
- Kubernetes DNS flaking out occasionally (for ~5 percent of queries)
- Giving out too many permissions, causing pods to be deleted without a clear reason why, often taking down production traffic or logging with it
- Giving out too few permissions, making our deployment infrastructure depend on a few lynchpins rather than sharing the production burden
- probably a dozen different logging aggregation systems, none of which strike a balance between speed and CPU cost
- probably a half-dozen different service meshes, all of which suck (with the exception of linkerd, which is actually quite good)
- teams with bad santization practices leaking credentials all over the place
- Running Vault in Kubernetes (really, don't ever do this)
- Disks becoming unattached from their pods for no discernable reason, only to be re-attached minutes later again with no explanation
- At least one major production outage on every single Kubernetes-based system I've built that can be directly attributed to Kubernetes
- Etcd failovers
- Etcd replication failures
- Privilege escalation due to an unsecured Jenkins builder causing credential exfiltration (this one was _super_ fun to fix)
Kubernetes is a powerful tool, and I've helped run some massive (1000+ node, 5000+ pod x 3 AZ's) systems based on K8s, but it took me a solid year of experimenting and tinkering to feel even remotely comfortable putting anything based on K8s into production. If you haven't run into any "major" issues, you're going to very soon. I can only wish you good luck.
- nickthemagicman 7y agoWhat about going into prod with a managed K8s? EKS or GKE?
- cocoa19 7y agoThanks for describing your experience. You do suggest not going into production with k8s, but what if the alternative is as difficult to implement or worse?