15 ms·

Kubernetes Failure Stories

320 points by yankcrime 7y ago

dullgiulio 7y ago
Nice, exactly what I needed today, after firefighting a latency spike because of a compute node after a separate ingress node was restarted. I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...) In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn't bother to have capable admins setting things up the right way). What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs. All have failure modes you need to learn: it takes a dedicated team. And once you have that, you'll need another team and cluster to perform an upgrade. But hey, when it works, it is really cool to deploy software on it.
- ben_jones 7y ago
  I feel like their needs to be a sanity check. Realistically what does kubernetes do to "not make software fail"? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?
  - theflyinghorse 7y ago
    Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).
    - y4mi 7y ago
      But swarm only has overlay network iirc? Swarm is great to get simple orchestration going, but it doesn't really do the same thing. With k8s you can configure multi-point routing, storage backends and load balancing among other things. With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service. Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need. We've used both at my current job for toy projects (self hosted). Never in production however. And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
      - hinkley 7y ago
        Possibly edgy opinion: Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator. The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
        AaronFriel 7y ago
        Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what "Ingress" and "Service" resources do is make sure those knobs are turned to the right settings. The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
        darkarmani 7y ago
        Is it hard to integrate a (HA) hardware load-balancer in front instead?
- clhodapp 7y ago
  If anything, this enhances your actual point but... if you are using self-signed certs with Kubernetes, then your cluster isn't secure.... so you shouldn't do that.
  - geggam 7y ago
    Arguably self signed certs eliminates possible state funded middle man attacks making it more secure
    - clhodapp 7y ago
      As a technology, properly-implemented self-signed certs are totally fine. The problem is that k8s does not have the features necessary to use self-signed certs securely. Instead, it expects you to create your own CA (or CAs: you can use separate ones for different kinds of communication if you want extra bulkheads) and then to share out your private CA's cert to all the k8s components. This achieves your goal of cutting out MITM attacks via unscrupulous commercial CAs while also making it possible to trust families of certs for a given purpose, rather than having to whitelist every single consumer's private key.
    - StreamBright 7y ago
      This is great. If a state funded threat is in your network in the position to place a certificate on a server do you think your self signed certificate will protect you?
      - geggam 7y ago
        Think of it as a cost and effort threshold. Prevents the dragnet / fishing methods from eavesdropping. It's trivial to force $Company to let you in with letter. The effort to break encryption is not trivial. You have to be doing something wrong to get specific attention.
        StreamBright 7y ago
        I mean if you use cert pinning with a public CA you get the same results, aka you can easily spot MITM. I am generally not a big fan of private CAs.
- tialaramex 7y ago
  > a sea of self-signed TLS certificates that can expire I would like to know more about what's going on here. Is this just a sloppy description and in fact Kubernetes uses a private PKI, so that the certs you're using aren't in fact self-signed but signed by a private CA?
  - raesene9 7y ago
    There's (IMO) a mix. Kubernetes and cloud native software make a lot of use of TLS for mutual auth. A standard Kubeadm cluster (very vanilla k8s) has 3 distinct Certificate authorities, all with the CA root keys online. On top of that things like Helm, FluentD and Istio will make use of their own distinct TLS certs. One of the most "fun" pieces is that k8s does not support certificate revocation, so if you use client certs for AuthN, then a user leaving/changing job/losing their laptop can lead to a full re-roll of that certificate authority :)
    - zxcmx 7y ago
      If you issue user certs, it is best to do it from software and have them live for 8h max. Also recommended: keep api behind a jumpbox.
      - raesene9 7y ago
        I've seen short-lived certs as a suggested workaround for the lack of revocation and as a user that might be the best option. That said the distributions I've seen that make use of client certs, don't do that (typical lifetime for a client cert is 1 year), so I'm guessing a load of people using k8s will have these certs floating about...
        zxcmx 7y ago
        It’s a mental shift similar to using iam assume role sessions rather than static keys with perms right on em. Powerful creds should be limited in lifespan and machine issued to users in a transaction that involves mfa. We built our own but Gravitational Teleport I think has these patterns in a product. The other issue even if you get certs right is service tokens (an admin could have grabbed a copy of one and they can be used from outside cluster) so want to keep that api wrapped IMHO.
- peterwwillis 7y ago
  Some people, when faced with a problem, think: I know, I'll use distributed systems! Now they have a new career.
- hinkley 7y ago
  > navigate a sea of self-signed TLS certificates that can expire When certificates become a logistics category of their own, you really do need some monitoring software to warn you when certificates are winding down. Years ago I worked on a complex code signing system and one set of users were adamant we have alerts starting at 90 days out. It took some convincing to get me to agree it was a priority, but some of my coworkers were never convinced.
manishsharan 7y ago
A long time back , I had gone down a rabbit hole to get Kubernetes manage my Mongodb. Thankfully , I got bitch slapped into reality within a few days and gave up because that rabbit hole goes deep. Now my mongodb stands outside of the K8 and I have no issues with it that cant be solved with a rebuild of a replica node.
- ravedave5 7y ago
  A very similar story where I work except an additional addendum of a migration to DocumentDB.
- elsonrodriguez 7y ago
  Glad I'm not the only one who almost got white-whaled into running stateful services on K8s. The API is so slick that it makes you feel you can do anything. But there's can, and there's should.
  - pas 7y ago
    The API is there, but is there a stable MongoDB Operator? ( https://github.com/Ultimaker/k8s-mongo-operator https://github.com/Ultimaker/k8s-mongo-operator ? https://github.com/mongodb/mongodb-enterprise-kubernetes https://github.com/mongodb/mongodb-enterprise-kubernetes ? )
    - elsonrodriguez 7y ago
      Honestly I'd only think of running Mongo on Kubernetes if my organization is thoroughly bored with how well they've managed Mongo upgrades, scaling, outages, migrations, and tuning over the last year on a traditional VM setup. Even so, if everyone's bored with the stability, why change it? I'm an advocate for databases in dev/test environments on K8s due to ease of deployment, but there's too many moving pieces around storage for it to be a great idea for production out the gate.
      - pas 7y ago
cookiecaper 7y ago
After a couple of years of massive frustration with the entire direction of the "devops" segment, I think I'm resolved to get out of it and either move further up back to ordinary application development or further down into the actual kernel, preferably working on FreeBSD or some other OS that's more sane and focused than Linux. Kubernetes represents the complete "operationalization" of the devops space. As companies have built out "devops" teams, they've mostly re-used their existing ops people, plus some stragglers from the dev side. These are the people you hear talking about how great Kubernetes is, because for them, they see it as "run a Helm chart and all done!". Which makes sense, since they were, not too long ago, the same guys fired up about all the super-neato buttons to click in the Control Panel. 90% of "devops" people at non-FAANG companies are operations people who just think of it as a new name for their old job. Among this set, there's no recognition of the massive needless complexity that permeates all the way through Kubernetes, no recognition of the tried-and-tested toolkit thrown away and left behind, no recognition of the fact that we're working so hard to get things that we've had as built-in pieces of any decent server OS for decades. No recognition that Kubernetes exists so it can serve as Google's wedge in the Rent-Your-Server-From-Me Wars, and no awareness that just in general, there's no reason it should be this hard. Of course, to them, it's not hard. They have an interface with buttons, they can run `helm install`, they get pretty graphs via their "service mesh". That's what I mean by "operationalized"; Kuberenetes is meant to be consumed, not configured. You don't ask how or why. You run the Minikube VM image locally and you rent GKE or EKS and go on your merry way. The intricacies are for the geniuses who've blursed us with this death trap to worry about! Worst-case, you use something like kops. Start asking questions or putting pieces together beyond this, and you're starting to sound like you don't have very much of that much-coveted "devops mindset" anymore. "What happens if there's a security issue?" Oh, silly, security issues are a thing of the past in the day and age where we blindly `FROM crazy-joes-used-mongodb-emporium:latest-built-2-years-ago`. Containers don't need updates, you goose. They're beautiful, blubber-powered magic, and the Great Googly Kubernetes in the sky is managing "all that" for us. Right on. I'm picking on Kubernetes specifically here because it's the epitome of all this, but really everything in "devops" world has become this way, and combined with the head-over-heels "omg get us into the cloud right now" mentality that's overtaking virtually every company, it's a bad scene. Systems have gotten so much more convoluted and so much dumber over the last 5 years. The industry has a lot to be embarrassed about right now.
tthisk 7y ago
In my opinion many of these issues with Kubernetes arise because it is not the right abstraction for application developers. You are forced to think about too many low level details to get a stable application running. Many companies forget that they are incharge of getting way more intricate details of the application runtime right when they shift their application to Kubernetes. In some of my recent experience I have seen a company shift an application from app engine to Kubernetes. However they forgot that appengine is using an optimized jvm to run your application. When they deployed to Kubernetes their app grinded to a halt. They forgot about all the observability that comes out of the box in app engine, thus resulting in an unobservable badly performing application. Hopefully more domain specific runtimes build on top of Kubernetes can help application developers deploy to a kubernetes cluster in a sane way. I am putting some of my money on knative, however it is still quite hard to convince clients to invest in this area.
- ma2rten 7y ago
  What do you mean by "observability"?
  - FridgeSeal 7y ago
    Generally, things that answer questions along the lines of: * is my app running * how long for? * if there are associated jobs/worker processes, did they start/complete successfully and how long did they run for * what’s the state of the pod(s) + host instanced my application is running on. * are all the necessary sub components up and running and communicating properly (and are they staying up). It’s not quite monitoring/logging, although they are related.
- threeseed 7y ago
  Not sure what any of this means. Kubernetes does not prevent you from using an optimized JVM to run your application. In fact it doesn't even involve itself at that level rather it focuses just on managing containers which can have anything inside them. And Kubernetes actually has far more observability than most distributed applications since you can get plugins which trace/monitor every connection between containers as well as monitor the health of each container. I think what you're after is a PaaS on top of Kubernetes ? Well there are many of those around as well.
miff2000 7y ago
Some really interesting issues that hopefully, as a new Kubernetes Cluster maintainer, I'll now be able to avoid!
- hjacobs 7y ago
  Thanks, feel free to reach out on Twitter (https://twitter.com/try_except_/ https://twitter.com/try_except_/) --- we are always happy to share experiences on running K8s.
cryptica 7y ago
This is just propaganda to drive companies away from Kubernetes so that they use expensive hosted 'lock-in' PaaS solutions instead of free, open source, no lock-in solutions.
- elsonrodriguez 7y ago
  The tone of the repo doesn't really scream propaganda to me. It just looks like a solid list of failures and what people can learn from them.
- eeZah7Ux 7y ago
  No lock-in in Kubernetes? Hah, try forking it.
- hjacobs 7y ago
  As the author of the repo, I made it pretty clear that it's not about driving people away from Kubernetes. Please see my posts and talks about it, e.g. https://kubernetespodcast.com/episode/038-kubernetes-failure-stories/ https://kubernetespodcast.com/episode/038-kubernetes-failure... and https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html
grantlmiller 7y ago
I'm torn on this list. It is important to learn from the mistakes from others, so I like it (postmortems are great for this reason), BUT it feels like folks are using these examples as reasons to stay away from Kubernetes. There could be a significantly larger list of system failures where K8s is not involved. Similarly there could probably be a list of "Encryption Failure Stories" but that doesn't mean we shouldn't encrypt things. As an industry, one of the things we do pretty well is identify the most viable patterns to solve a problem and then develop and adopt the best primitives of those patterns. This is what Kubernetes is for creating reliable, scalable, distributed systems.
- eeZah7Ux 7y ago
  k8s does not solve any problem around making your system distributed, scalable and reliable. You still have to implement your own architecture to implement quorum, leader election, failover etc where needed. k8s will only redeploy containers when a node fail.
  - geezerjay 7y ago
    > k8s will only redeploy containers when a node fail. That's not true at all. Kubernetes monitors containers' health and automatically scales deployments according to the demand. Thus quite obviously kubernetes does help make your system distributable, scalable and reliable. In fact, that was the design goal of kubernetes.
  - pepemon 7y ago
    There are distributed primitives in Kubernetes (such as leader lock), so you won't have to roll your own "architecture". For example, look at Patroni, it can be run natively on Kube, using its resources for all distributed and redundancy goodness.
msoad 7y ago
You know what's worse than Kubernetes complexity? A shallow abstraction of it by some mediocre team to get promotion and leave everyone else struggle with something that you can't get help from anywhere else. This happened in past two unicorns that I worked for. They template the templates and wrapped the `kubectl` with some leaky abstraction. At least raw Kubernetes is Google-able!
- hjacobs 7y ago
  We try to keep it simple at Zalando, i.e. just using Mustache templating and wrapping kubectl to add extra commands. See https://www.slideshare.net/try_except_/developer-experience-at-zalando-cncf-end-user-sigdx https://www.slideshare.net/try_except_/developer-experience-...
erikrothoff 7y ago
I got enough of random 503’s, network errors, slowdowns on mid-to-high traffic and dropped TCP connections yesterday. I setup dedicated VPS’s for our main app servers using Ansible and good ol’ fashion Capistrano. It feels almost old-school, but so much more stable.
odiroot 7y ago
These are all very informative and even amusing. But this is to be expected. Kubernetes is an enormous system (or ecosystem of systems actually). It's really hard to understand all the pieces and components, even the built-ins. I still struggle (after having set up 2 clusters from scratch) with understanding networking, especially Ingresses and exposing things to the public internet.
neya 7y ago
In one of my ex-companies I worked with, we had a newbie coder who was just getting fascinated by Kubernetes. Unfortunately, every project this person touched would be left half-way by him and he'd move on to another project. This company invested around $1 million into a very new Saas product that had a lot of potential. Now, I'm an old school guy. I like systems where I don't touch devops. The cost for such Paas is usually high, but it saves a lot of my time, which I value more than money (and it is valuable, in many cases). I advised the management to use something like AppEngine which is freakin' awesome for such kind of large scale projects. Unfortunately, this newbie coder was management's pet. They bit the bullet and went on with his advice to use Kubernetes instead. The system randomly failed, they spent tons of time doing devops and microservices on what should have been a simplified monolith. The development time elongated to almost 2 years. By this time, there were competing offerings in the market for much cheaper. This costed an entire team's morale, which lead to missed deadlines and product launches. This lead to many missed opportunities. For instance, there was a HUGE client we were demo'ing this Saas product. It failed spectacularly on a relatively light load (I've hosted Rails apps on AppEngine that can handle FAR better) and we lost that product and the client. It was still not too late and I insisted on management identifying the root cause to switch to a managed Paas stack given how resource constrained we were. But this newbie coder turned it into a political situation ('mine vs yours'). I tried various ways to resolve this, but it didn't work out. As a result, I said 'fuck this, I'm out of here' and I quit. In less than 3 months of my departure, about 10 people quit including one of the senior managers. In about 6 months, the company shrunk from a double digit to single digit company. The company almost went into bankruptcy. The company lost its $1 million investment. Everyone left. The product development was put on indefinite hold. Finally, this newbie coder left as well. The founders had to rebuild their companies from scratch. Co-incidentally, I went on to become an AppEngine consultant. I'm able to run unbelievably large monoliths in production, with almost zero downtime for 3 years in a row simply because I chose to avoid devops, and most importantly infrastructure complexity. I pay more, but it easily is worth my ROI. Most of the time, microservices aren't required and we can get away with a simplified monolith. And if you can get away with monoliths, you probably don't need Kubernetes.
- orthecreedence 7y ago
  > if you can get away with monoliths, you probably don't need Kubernetes Right, especially if you're doing microservices the "right way" which means, each service has not only its own app setup but its own data store. I am responsible for (among other things) maintaining and scaling my company's infrastructure, and while we do have some vague concept of microservices (app, api (monolith), queuing system, various ML apis, headless browser, etc) it is always always ALWAYS "what's the simplest way to do this?" Microservices are a ton of moving parts, and adding in their own data stores is another can of worms. We have a need for a scheduler. We are using Nomad internally. I chose this over Kubernetes. Why? I know that Kubernetes has more muscle behind it, can do more "things" we might need in the future, etc. I chose it because of its operational simplicity. It strikes a perfect balance of "scale these three things up" without needing a dedicated ops team to run it. So far it has performed great, and I really think it comes down to limiting the number of moving pieces. Don't get me wrong, microservices are great if you have an ops team and a 50+ dev team all working on different things. But I often see microservices being pushed without warning people of the inherent costs involved. I'm sure kubernetes is great, and I'm continuously re-evaluating where we are and what we need, but for now simpler is always better. If we do go with Kube, it will likely be a managed version so I don't have to touch the internals...not because I don't want to learn them (I definitely would want to know all the grimy details) but because I just don't have the fucking time to deal with it.
emmelaich 7y ago
Awesome. I'd like to see a list for other related products. Such as Openstack. I bet there's a few stories there.
rurban 7y ago
So the most used tags are: OOMKill, CoreDNS, ndots:5, NotReady nodes. I'd seriously consider disabling the OOMKiller. Not sure about the DNS issues.
hjacobs 7y ago
As the owner of the linked GitHub repo (also rendered on https://k8s.af https://k8s.af --- thanks to Joe Beda), I highly encourage everyone to contribute their failure stories (I'm still looking for the first production service mesh failure story..). Also be aware of availability bias: Kubernetes enables us to collect failure stories in a (more or less) consistent way, this was previously not easily possible (think about on-premise failures, other fragmented orchestration frameworks, etc) --- I'm pretty sure there are much more failure stories in total about other things (like enterprise software), but we will never hear about them as they are buried inside orgs.. BTW, I also have a small post on why I think Kubernetes is more than just a "complex scheduler": https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html

320 points by yankcrime 7y ago

Nice, exactly what I needed today, after firefighting a latency spike because of a compute node after a separate ingress node was restarted. I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...) In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn't bother to have capable admins setting things up the right way). What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs. All have failure modes you need to learn: it takes a dedicated team. And once you have that, you'll need another team and cluster to perform an upgrade. But hey, when it works, it is really cool to deploy software on it.

ben_jones 7y ago
I feel like their needs to be a sanity check. Realistically what does kubernetes do to "not make software fail"? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?
- theflyinghorse 7y ago
  Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).
  - y4mi 7y ago
    But swarm only has overlay network iirc? Swarm is great to get simple orchestration going, but it doesn't really do the same thing. With k8s you can configure multi-point routing, storage backends and load balancing among other things. With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service. Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need. We've used both at my current job for toy projects (self hosted). Never in production however. And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
    - hinkley 7y ago
      Possibly edgy opinion: Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator. The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
      - AaronFriel 7y ago
        Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what "Ingress" and "Service" resources do is make sure those knobs are turned to the right settings. The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
        darkarmani 7y ago
        Is it hard to integrate a (HA) hardware load-balancer in front instead?
clhodapp 7y ago
If anything, this enhances your actual point but... if you are using self-signed certs with Kubernetes, then your cluster isn't secure.... so you shouldn't do that.
- geggam 7y ago
  Arguably self signed certs eliminates possible state funded middle man attacks making it more secure
  - clhodapp 7y ago
    As a technology, properly-implemented self-signed certs are totally fine. The problem is that k8s does not have the features necessary to use self-signed certs securely. Instead, it expects you to create your own CA (or CAs: you can use separate ones for different kinds of communication if you want extra bulkheads) and then to share out your private CA's cert to all the k8s components. This achieves your goal of cutting out MITM attacks via unscrupulous commercial CAs while also making it possible to trust families of certs for a given purpose, rather than having to whitelist every single consumer's private key.
  - StreamBright 7y ago
    This is great. If a state funded threat is in your network in the position to place a certificate on a server do you think your self signed certificate will protect you?
    - geggam 7y ago
      Think of it as a cost and effort threshold. Prevents the dragnet / fishing methods from eavesdropping. It's trivial to force $Company to let you in with letter. The effort to break encryption is not trivial. You have to be doing something wrong to get specific attention.
      - StreamBright 7y ago
        I mean if you use cert pinning with a public CA you get the same results, aka you can easily spot MITM. I am generally not a big fan of private CAs.
tialaramex 7y ago
> a sea of self-signed TLS certificates that can expire I would like to know more about what's going on here. Is this just a sloppy description and in fact Kubernetes uses a private PKI, so that the certs you're using aren't in fact self-signed but signed by a private CA?
- raesene9 7y ago
  There's (IMO) a mix. Kubernetes and cloud native software make a lot of use of TLS for mutual auth. A standard Kubeadm cluster (very vanilla k8s) has 3 distinct Certificate authorities, all with the CA root keys online. On top of that things like Helm, FluentD and Istio will make use of their own distinct TLS certs. One of the most "fun" pieces is that k8s does not support certificate revocation, so if you use client certs for AuthN, then a user leaving/changing job/losing their laptop can lead to a full re-roll of that certificate authority :)
  - zxcmx 7y ago
    If you issue user certs, it is best to do it from software and have them live for 8h max. Also recommended: keep api behind a jumpbox.
    - raesene9 7y ago
      I've seen short-lived certs as a suggested workaround for the lack of revocation and as a user that might be the best option. That said the distributions I've seen that make use of client certs, don't do that (typical lifetime for a client cert is 1 year), so I'm guessing a load of people using k8s will have these certs floating about...
      - zxcmx 7y ago
        It’s a mental shift similar to using iam assume role sessions rather than static keys with perms right on em. Powerful creds should be limited in lifespan and machine issued to users in a transaction that involves mfa. We built our own but Gravitational Teleport I think has these patterns in a product. The other issue even if you get certs right is service tokens (an admin could have grabbed a copy of one and they can be used from outside cluster) so want to keep that api wrapped IMHO.
peterwwillis 7y ago
Some people, when faced with a problem, think: I know, I'll use distributed systems! Now they have a new career.
hinkley 7y ago
> navigate a sea of self-signed TLS certificates that can expire When certificates become a logistics category of their own, you really do need some monitoring software to warn you when certificates are winding down. Years ago I worked on a complex code signing system and one set of users were adamant we have alerts starting at 90 days out. It took some convincing to get me to agree it was a priority, but some of my coworkers were never convinced.

manishsharan 7y ago

A long time back , I had gone down a rabbit hole to get Kubernetes manage my Mongodb. Thankfully , I got bitch slapped into reality within a few days and gave up because that rabbit hole goes deep. Now my mongodb stands outside of the K8 and I have no issues with it that cant be solved with a rebuild of a replica node.

ravedave5 7y ago
A very similar story where I work except an additional addendum of a migration to DocumentDB.
elsonrodriguez 7y ago
Glad I'm not the only one who almost got white-whaled into running stateful services on K8s. The API is so slick that it makes you feel you can do anything. But there's can, and there's should.
- pas 7y ago
  The API is there, but is there a stable MongoDB Operator? ( https://github.com/Ultimaker/k8s-mongo-operator https://github.com/Ultimaker/k8s-mongo-operator ? https://github.com/mongodb/mongodb-enterprise-kubernetes https://github.com/mongodb/mongodb-enterprise-kubernetes ? )
  - elsonrodriguez 7y ago
    Honestly I'd only think of running Mongo on Kubernetes if my organization is thoroughly bored with how well they've managed Mongo upgrades, scaling, outages, migrations, and tuning over the last year on a traditional VM setup. Even so, if everyone's bored with the stability, why change it? I'm an advocate for databases in dev/test environments on K8s due to ease of deployment, but there's too many moving pieces around storage for it to be a great idea for production out the gate.
    - pas 7y ago

cookiecaper 7y ago

After a couple of years of massive frustration with the entire direction of the "devops" segment, I think I'm resolved to get out of it and either move further up back to ordinary application development or further down into the actual kernel, preferably working on FreeBSD or some other OS that's more sane and focused than Linux. Kubernetes represents the complete "operationalization" of the devops space. As companies have built out "devops" teams, they've mostly re-used their existing ops people, plus some stragglers from the dev side. These are the people you hear talking about how great Kubernetes is, because for them, they see it as "run a Helm chart and all done!". Which makes sense, since they were, not too long ago, the same guys fired up about all the super-neato buttons to click in the Control Panel. 90% of "devops" people at non-FAANG companies are operations people who just think of it as a new name for their old job. Among this set, there's no recognition of the massive needless complexity that permeates all the way through Kubernetes, no recognition of the tried-and-tested toolkit thrown away and left behind, no recognition of the fact that we're working so hard to get things that we've had as built-in pieces of any decent server OS for decades. No recognition that Kubernetes exists so it can serve as Google's wedge in the Rent-Your-Server-From-Me Wars, and no awareness that just in general, there's no reason it should be this hard. Of course, to them, it's not hard. They have an interface with buttons, they can run `helm install`, they get pretty graphs via their "service mesh". That's what I mean by "operationalized"; Kuberenetes is meant to be consumed, not configured. You don't ask how or why. You run the Minikube VM image locally and you rent GKE or EKS and go on your merry way. The intricacies are for the geniuses who've blursed us with this death trap to worry about! Worst-case, you use something like kops. Start asking questions or putting pieces together beyond this, and you're starting to sound like you don't have very much of that much-coveted "devops mindset" anymore. "What happens if there's a security issue?" Oh, silly, security issues are a thing of the past in the day and age where we blindly `FROM crazy-joes-used-mongodb-emporium:latest-built-2-years-ago`. Containers don't need updates, you goose. They're beautiful, blubber-powered magic, and the Great Googly Kubernetes in the sky is managing "all that" for us. Right on. I'm picking on Kubernetes specifically here because it's the epitome of all this, but really everything in "devops" world has become this way, and combined with the head-over-heels "omg get us into the cloud right now" mentality that's overtaking virtually every company, it's a bad scene. Systems have gotten so much more convoluted and so much dumber over the last 5 years. The industry has a lot to be embarrassed about right now.

tthisk 7y ago

In my opinion many of these issues with Kubernetes arise because it is not the right abstraction for application developers. You are forced to think about too many low level details to get a stable application running. Many companies forget that they are incharge of getting way more intricate details of the application runtime right when they shift their application to Kubernetes. In some of my recent experience I have seen a company shift an application from app engine to Kubernetes. However they forgot that appengine is using an optimized jvm to run your application. When they deployed to Kubernetes their app grinded to a halt. They forgot about all the observability that comes out of the box in app engine, thus resulting in an unobservable badly performing application. Hopefully more domain specific runtimes build on top of Kubernetes can help application developers deploy to a kubernetes cluster in a sane way. I am putting some of my money on knative, however it is still quite hard to convince clients to invest in this area.

ma2rten 7y ago
What do you mean by "observability"?
- FridgeSeal 7y ago
  Generally, things that answer questions along the lines of: * is my app running * how long for? * if there are associated jobs/worker processes, did they start/complete successfully and how long did they run for * what’s the state of the pod(s) + host instanced my application is running on. * are all the necessary sub components up and running and communicating properly (and are they staying up). It’s not quite monitoring/logging, although they are related.
threeseed 7y ago
Not sure what any of this means. Kubernetes does not prevent you from using an optimized JVM to run your application. In fact it doesn't even involve itself at that level rather it focuses just on managing containers which can have anything inside them. And Kubernetes actually has far more observability than most distributed applications since you can get plugins which trace/monitor every connection between containers as well as monitor the health of each container. I think what you're after is a PaaS on top of Kubernetes ? Well there are many of those around as well.

miff2000 7y ago

Some really interesting issues that hopefully, as a new Kubernetes Cluster maintainer, I'll now be able to avoid!

hjacobs 7y ago
Thanks, feel free to reach out on Twitter (https://twitter.com/try_except_/ https://twitter.com/try_except_/) --- we are always happy to share experiences on running K8s.

cryptica 7y ago

This is just propaganda to drive companies away from Kubernetes so that they use expensive hosted 'lock-in' PaaS solutions instead of free, open source, no lock-in solutions.

elsonrodriguez 7y ago
The tone of the repo doesn't really scream propaganda to me. It just looks like a solid list of failures and what people can learn from them.
eeZah7Ux 7y ago
No lock-in in Kubernetes? Hah, try forking it.
hjacobs 7y ago
As the author of the repo, I made it pretty clear that it's not about driving people away from Kubernetes. Please see my posts and talks about it, e.g. https://kubernetespodcast.com/episode/038-kubernetes-failure-stories/ https://kubernetespodcast.com/episode/038-kubernetes-failure... and https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html

grantlmiller 7y ago

I'm torn on this list. It is important to learn from the mistakes from others, so I like it (postmortems are great for this reason), BUT it feels like folks are using these examples as reasons to stay away from Kubernetes. There could be a significantly larger list of system failures where K8s is not involved. Similarly there could probably be a list of "Encryption Failure Stories" but that doesn't mean we shouldn't encrypt things. As an industry, one of the things we do pretty well is identify the most viable patterns to solve a problem and then develop and adopt the best primitives of those patterns. This is what Kubernetes is for creating reliable, scalable, distributed systems.

eeZah7Ux 7y ago
k8s does not solve any problem around making your system distributed, scalable and reliable. You still have to implement your own architecture to implement quorum, leader election, failover etc where needed. k8s will only redeploy containers when a node fail.
- geezerjay 7y ago
  > k8s will only redeploy containers when a node fail. That's not true at all. Kubernetes monitors containers' health and automatically scales deployments according to the demand. Thus quite obviously kubernetes does help make your system distributable, scalable and reliable. In fact, that was the design goal of kubernetes.
- pepemon 7y ago
  There are distributed primitives in Kubernetes (such as leader lock), so you won't have to roll your own "architecture". For example, look at Patroni, it can be run natively on Kube, using its resources for all distributed and redundancy goodness.

msoad 7y ago

You know what's worse than Kubernetes complexity? A shallow abstraction of it by some mediocre team to get promotion and leave everyone else struggle with something that you can't get help from anywhere else. This happened in past two unicorns that I worked for. They template the templates and wrapped the `kubectl` with some leaky abstraction. At least raw Kubernetes is Google-able!

hjacobs 7y ago
We try to keep it simple at Zalando, i.e. just using Mustache templating and wrapping kubectl to add extra commands. See https://www.slideshare.net/try_except_/developer-experience-at-zalando-cncf-end-user-sigdx https://www.slideshare.net/try_except_/developer-experience-...

erikrothoff 7y ago

I got enough of random 503’s, network errors, slowdowns on mid-to-high traffic and dropped TCP connections yesterday. I setup dedicated VPS’s for our main app servers using Ansible and good ol’ fashion Capistrano. It feels almost old-school, but so much more stable.

odiroot 7y ago

These are all very informative and even amusing. But this is to be expected. Kubernetes is an enormous system (or ecosystem of systems actually). It's really hard to understand all the pieces and components, even the built-ins. I still struggle (after having set up 2 clusters from scratch) with understanding networking, especially Ingresses and exposing things to the public internet.

neya 7y ago

In one of my ex-companies I worked with, we had a newbie coder who was just getting fascinated by Kubernetes. Unfortunately, every project this person touched would be left half-way by him and he'd move on to another project. This company invested around $1 million into a very new Saas product that had a lot of potential. Now, I'm an old school guy. I like systems where I don't touch devops. The cost for such Paas is usually high, but it saves a lot of my time, which I value more than money (and it is valuable, in many cases). I advised the management to use something like AppEngine which is freakin' awesome for such kind of large scale projects. Unfortunately, this newbie coder was management's pet. They bit the bullet and went on with his advice to use Kubernetes instead. The system randomly failed, they spent tons of time doing devops and microservices on what should have been a simplified monolith. The development time elongated to almost 2 years. By this time, there were competing offerings in the market for much cheaper. This costed an entire team's morale, which lead to missed deadlines and product launches. This lead to many missed opportunities. For instance, there was a HUGE client we were demo'ing this Saas product. It failed spectacularly on a relatively light load (I've hosted Rails apps on AppEngine that can handle FAR better) and we lost that product and the client. It was still not too late and I insisted on management identifying the root cause to switch to a managed Paas stack given how resource constrained we were. But this newbie coder turned it into a political situation ('mine vs yours'). I tried various ways to resolve this, but it didn't work out. As a result, I said 'fuck this, I'm out of here' and I quit. In less than 3 months of my departure, about 10 people quit including one of the senior managers. In about 6 months, the company shrunk from a double digit to single digit company. The company almost went into bankruptcy. The company lost its $1 million investment. Everyone left. The product development was put on indefinite hold. Finally, this newbie coder left as well. The founders had to rebuild their companies from scratch. Co-incidentally, I went on to become an AppEngine consultant. I'm able to run unbelievably large monoliths in production, with almost zero downtime for 3 years in a row simply because I chose to avoid devops, and most importantly infrastructure complexity. I pay more, but it easily is worth my ROI. Most of the time, microservices aren't required and we can get away with a simplified monolith. And if you can get away with monoliths, you probably don't need Kubernetes.

orthecreedence 7y ago
> if you can get away with monoliths, you probably don't need Kubernetes Right, especially if you're doing microservices the "right way" which means, each service has not only its own app setup but its own data store. I am responsible for (among other things) maintaining and scaling my company's infrastructure, and while we do have some vague concept of microservices (app, api (monolith), queuing system, various ML apis, headless browser, etc) it is always always ALWAYS "what's the simplest way to do this?" Microservices are a ton of moving parts, and adding in their own data stores is another can of worms. We have a need for a scheduler. We are using Nomad internally. I chose this over Kubernetes. Why? I know that Kubernetes has more muscle behind it, can do more "things" we might need in the future, etc. I chose it because of its operational simplicity. It strikes a perfect balance of "scale these three things up" without needing a dedicated ops team to run it. So far it has performed great, and I really think it comes down to limiting the number of moving pieces. Don't get me wrong, microservices are great if you have an ops team and a 50+ dev team all working on different things. But I often see microservices being pushed without warning people of the inherent costs involved. I'm sure kubernetes is great, and I'm continuously re-evaluating where we are and what we need, but for now simpler is always better. If we do go with Kube, it will likely be a managed version so I don't have to touch the internals...not because I don't want to learn them (I definitely would want to know all the grimy details) but because I just don't have the fucking time to deal with it.

emmelaich 7y ago

Awesome. I'd like to see a list for other related products. Such as Openstack. I bet there's a few stories there.

rurban 7y ago

So the most used tags are: OOMKill, CoreDNS, ndots:5, NotReady nodes. I'd seriously consider disabling the OOMKiller. Not sure about the DNS issues.

hjacobs 7y ago

As the owner of the linked GitHub repo (also rendered on https://k8s.af https://k8s.af --- thanks to Joe Beda), I highly encourage everyone to contribute their failure stories (I'm still looking for the first production service mesh failure story..). Also be aware of availability bias: Kubernetes enables us to collect failure stories in a (more or less) consistent way, this was previously not easily possible (think about on-premise failures, other fragmented orchestration frameworks, etc) --- I'm pretty sure there are much more failure stories in total about other things (like enterprise software), but we will never hear about them as they are buried inside orgs.. BTW, I also have a small post on why I think Kubernetes is more than just a "complex scheduler": https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html