20 ms·
Kubernetes Failure Stories
- fuzzythinker 6y agoMeta. Is this a common usage of .af domains?
- pc86 6y agoI only came here to comment that the domain name was fantastic for this usage.
- zwp 6y agoAfghanistan? Air force? Uh, as fuck? Really? I'm not prudish about words but .lol would've been clearer.
- pc86 6y agoMaybe they tried that first but found out it was registered 3 years ago, and for some reason just redirects to this article: https://www.theregister.com/2017/10/17/docker_ee_kubernetes_support/ https://www.theregister.com/2017/10/17/docker_ee_kubernetes_...
- yongjik 6y agoMaybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.
- jrockway 6y agoIs Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Kubernetes would do; create a Pod to provision compute resources and an IP, pull a container to copy your code into the environment, run a health check to see if that all went OK, and then update the Service's Endpoints to start sending it traffic. Then it would drain the old Pod in a reverse of starting a new one, perhaps running your configured cleanup steps along the way. The difference between the example and what Kubernetes does is that Kubernetes is a computer program that can do this thousands of times for you without really making you think about getting the details right every time. Doesn't seem overly complicated to me. A lot of stuff? Sure. But it's pretty close to the right level of complexity. (Some may argue that this is even too simple. What about blue/green testing? What about canaries? When do my database migrations run? Kubernetes has no answer, and if it did, it certainly wouldn't make it simpler. Perhaps the real problem is that Kubernetes isn't complicated enough!) Anyway, "that's Kubernetes as fuck" is kind of a good meme. I'll give you that. But a deep understanding usually makes memes less amusing.
- jjtheblunt 6y ago> Is Kubernetes really overcomplicated, though? I imagine that a million voices read that and silently, immediately, think "yes, it is." and then giggle without commenting.
- yongjik 6y agoI think we can draw parallels between C and Kubernetes - C can be an ideal tool for some stuff, and it's theoretically possible to write a secure program in C. However, there's a reason why people keep saying that it's practically impossible to write C securely. C provides so many gotchas and footguns that it's not even funny. Unless your problem space really demands it, there's not much reason to use C other than "I love to live dangerously." Similarly, Kubernetes can be an ideal choice for some people. It may even be an OK choice for some others. But there are many for whom there's no reason to use k8s other than "another shiny item on my resume."
- xorcist 6y ago> That's basically what a Deployment in Kubernetes would do If that what was Kubernetes did, then we wouldn't be having this discussion. What you describe is more like what something like Ansible does. Provision, health check, flip load balancer.
- C4stor 6y ago
- plange 6y agoDonated by Joe Beda himself.
- jbeda 6y agoHeh - yeah. Henning does all the hard work of curating these. But I had the domain sitting around and figured that it was a good use of it. Sharing and learning through stories like this is how systems get better.
- karlding 6y agoThere's also contained.af [0], Jessie Frazelle's educational container-based CTF. [0] https://contained.af/ https://contained.af/
- walrus01 6y agoRegistering a .af domain is a lot easier and less costly if you're an Afghan citizen and speak Dari or Pashto. There's a way to do it and pay in person in Kabul. Right now, the entire NIC website only answers at http://nic.af http://nic.af , if you prefix it with www it appears to be down. It's not very common yet because the .af registry is not really set up to work with high volume, fully automated third party domain registrars.
- gen220 6y agoThis is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure than from success (whose analogue is a blogpost with a title "look how easy it is to set up k8s in 15 minutes at home!"). If you really want to think about how to deploy nuanced technology like this at scale, operational crash reports such as these are invaluable.
- tempest_ 6y agoWhich is why postmortem blog posts are always infinity more enlightening than the "How we build X" posts
- barrkel 6y ago"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle https://en.wikipedia.org/wiki/Anna_Karenina_principle
- WJW 6y agoNot to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-storage-size-took-20-minutes lessons and probably many more. It's like Churchills explanation of democracy: terrible, but better than all the alternatives. (at scale, at least)
- teach 6y agoI guess it depends on your definition of "at scale" but IMO with Hashicorp's stuff and maybe something like Backstage you get all the benefits k8s but in a package that's much simpler to reason about and manage.
- mitjak 6y agoreally cool resource for learning but a lot of these have nothing to do with k8s, beyond the company in question having k8s as part of their stack (i'm addressing the possible perception of the post title suggesting k8s horror stories).
- jrockway 6y agoA lot of them do have things to do with k8s, though. Admission webhooks, Istio sidecar injection, etc. The CPU limits = weird latency spikes also shows up a lot there, but it's technically a cgroups problem. (Set GOMAXPROCS=16, set cpu limit to 1, wonder why your program is asleep 15/16th of every cgroups throttling interval. I see that happen to people a lot, the key point being that GOMAXPROCS and the throttling interval are not something they ever manually configured, hence it's surprising how they interact. I ship https://github.com/uber-go/automaxprocs https://github.com/uber-go/automaxprocs in all of my open source stuff to avoid bug reports about this particular issue. Fun stuff! :) DNS also makes a regular appearance, and I agree it's not Kubernetes' fault, but on the other hand, people probably just hard-coded service IPs for service discovery before Kubernetes, so DNS issues are a surprise to them. When they type "google.com" into their browser, it works every time, so why wouldn't "service.namespace.svc.cluster.local" work just as well? (I also love the cloud providers' approach to this rough spot -- GKE has a service that exists to scale up kube-dns if you manually scale it down!) Anyway, it's all good reading. If you don't read this, you are bound to have these things happen to you. Many of these things will happen to you even if you don't use Kubernetes!
- hinkley 6y agoI have an environment variable to set process count for a nodejs service because everyone is finger pointing at who should be the one to sort out the fact that cgroup cpu limits and nprocs disagree most of the time. Our stuff is fine, those other guys should fix their stuff. Thanks, everybody. Good effort.
- tutfbhuf 6y agoThe current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to apply cluster wide updates, one cluster at a time.
- redis_mlc 6y ago> For instance, we had several outages when upgrading the kubernetes version in our clusters. Yes, this was a big problem in 2018-2019. Are things smoother now in your experience? What magnified the problems where I was working at was that the India team would silently do the k8s upgrade overnight, and when the US office opened in the morning we'd find problems several hours later. So several hours of downtime.
- superbcarrot 6y agoSounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?
- bo0tzz 6y agohttps://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api
- seneca 6y agoCAPI is for creating clusters only. It doesn't offer any orchestration.
- moondev 6y agoI'm curious what you mean here, since CAPI allows you to manage and scale clusters like any other Kubernetes object. You can literally "kubectl scale --replicas" a cluster control-plane and worker nodes.
- BasedInfra 6y agoThey walked so we could run our k8s services. Anyone with their failures developing, deploying or running software (or anything!) put it out there. It's a great resource.
- kall 6y agoNice to see HN submissions that I also just added to reading list based on another HN submission, Litestream in this case, right?
- deleted 6y ago[deleted]
- An0mammall 6y agoI first thought this was kind of a smear attempt but I realise now that this is a great learning ressource, thanks!
- kureikain 6y agoThe first link in this article is about switching from fluentbit to fluentd. Me too. I have issue with fluenbit. Especially around their JSON processing. As in this issue: https://github.com/fluent/fluent-bit/issues/1588 https://github.com/fluent/fluent-bit/issues/1588 If you want a simple log forwarding, fluentbit is really good. But if you found yourself started to tweak fluentbit config, or write custom plugin in C and (recompile yourself) then it's time to move to fluentd.
- opsunit 6y agoI did a lot of the work to get sub-second timestamps working within the fluentd+ElasticSearch ecosystem and the thing is a tire fire.
- Benlights 6y agoI see you read the article about litestream, I love following the rabbit hole. Good work!
- cbushko 6y agoThe Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.
- seneca 6y agoIndeed. All the flak k8s gets about being overly complex and difficult to manage (maybe rightfully) ought to leveled 5x at Istio.
- JMTQp8lwXL 6y agoShouldn't even small changes be tested in a staging cluster before promoting to the production cluster?
- coolspot 6y ago“You don’t introduce your dirty changes onto perfectly good staging without some testing on production first”.
- cbushko 6y agoWe go through several stages before it hits prod. Our test environment -> the dev environment -> staging -> prod and still we are bitten by istio. A lot of the problems we have seen do not manifest themselves until you get significant traffic in the cluster. Luckily, it is one of our goals this year to make our setup testable. Our setup is: - an ingressgateway running on each node so that our SIEM can get IP addresses - 1000s of virtual service entries for all the services running (40 per namespace x 25+ namespaces) - 100s of deploys a day which causes pilot to update all the time - the clusters are small with like 50 nodes - we have way too many services and consolidation is slow - we haven't been able to upgrade to the Istio operator yet because there is an outstanding bug that breaks everything
- dennis_jeeves 6y agoWe never test, in the rare situations we test, we first test it on production. :)
- 6y ago
- nhoughto 6y agoThese are great, I found myself trying to predict the related areas, “..Slow connections..” ooh that must be conntrack! Been burnt by kube2iam too.. I guess it’s k8s failure bingo.
- genmud 6y agoI absolutely love it when people talk about failures, it is so nice to be able to learn about stuff without having to make the mistake myself.
- kristopolous 6y agoYes, the opposite of it - adjective-laden flowery non-specific hype really irks me. To me it reads like how people talked about car design before ralph nader pointed out how cars with metal dashboards and no seatbelts are a bad idea.
- trixie_ 6y agoCan we do one for micro-services next?
- UweSchmidt 6y agoIt takes a high level of skill and maturity to articulate such failure stories. The thing you missed is obvious to others, the "bug" you found is actually documented somewhere, and you can mitigate and move on from disaster without completely drilling down and reconstructing the problem. So thanks for posting.
- stephen123 6y agoThis is great. I need this for all the tech I use. So much to learn!
- gadrev 6y agoI picked two or three random stories, and learned a lot just from that, be it technology I didn't know about or just some trick for debugging. Recommended.
- olefoo 6y agoThis is engineering. Not art. We learn from our failures and build better than before.
- legohead 6y agoSurprised I don't see something about "context deadline exceeded" in there. This error has been plaguing me from Kubernetes 1.5 to 1.18, and nobody has a solid answer for it other than maybe it's Docker related. Solution? Restart your node(s).
- neop1x 6y agoIt is just a timeout. Golang uses contexts which can have timeout set. Then you can do a request (HTTP/GRPC) under that context and if it times out before getting a response, this error is returned. To find the real reason you are supposed to know the architecture and monitor metrics. It can be network problem or kernel problem (resource exhaustion).
- squibbles 6y agoDefinitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.
- ivan_ah 6y agoWhat about single-host container (i.e. regular docker host, not k8s) with data partition mounted from host?
- squibbles 6y agoFor me, personally -- I cannot think of a sufficient justification to put a production database in a container. A good database server is designed for performance, reliability, scalability, security, etc., without containers. Putting a production database inside a container introduces a world of unnecessary edge cases and complexity.
- ivan_ah 6y agoThanks for your feedback. I might run the DB on the host then, and just use containers for the app server. I'm not at the scale to warrant a separate host for the DB.
- neop1x 6y agoDepends on requirements. Someone needs one big, highly-optimized DB instance. Someone else needs high-availability 3+ instance cluster. Having a cluster of containers brings performance penalty but if your app is read-heavy, you can read from all instances and multiply read throughput...
- biglost 6y agoAnd using delegated, for local development, multiple versions with a few commands is really great.
- 6y ago
- pid_0 6y agoI will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.
- arlk 6y agoI had a disastrous experience with this. Three years ago I joined a startup to help with infrastructure setup. They wanted to use Kubernetes to install their monolithic django/celery AI/web single app using the same script both for their cloud environment and customers servers with single node installation. I didn't last more than a month. Since then, and for every company I worked with, no single time Kubernetes was proposed as a solution for the problem it really solves. It was always though of as the magical solution for whatever problem we had (or didn't even have).
- _asummers 6y agoFeel free, but it's a bad hill. I can write my own config management, secret management, volume mounting, deployments, replica scaling, hardware affinity, antiaffinity for resilience, rollover kicking out e.g. staging if there's not enough room to schedule production workloads, and the list goes on. I can also figure out how to install and upgrade each piece of software in my universe, across a few different languages. Or I can just use kubernetes. From the perspective of my developers, they tell me a few things like: "I need 4GB RAM, 2 cores, I need access to the postgres secret to talk to it, and here is my hostname" and while they can tell me many, many more things, with that little information, they can have a full application in one of several different languages in minutes. If I were to hand roll this, it'd be held together by prayers and duct tape. And I lose out on a huge active community building tools around this. I now have istio and get cross service telemetry. For free. I can set networking rules up setting up QOS between arbitrary services, both inside my network and outside. All applied behind a single interface. There's a lot of stuff I get for free, and as my team discovers things it needs to do, I have a consistent layer to do that all behind. I'm one guy. I don't work for Google. But my team still manages _a lot_ of services, both on application end, and things like queues, databases, etc. You can say we don't need all of that, but you'd be wrong. We arguably could use more. I can reason about my infrastructure in both the small and the large. Then you get into the idea of transferring to other companies. If someone comes into my org, they can look at what I have and see more or less everything there is in any part of my infrastructure. It's all right there, laid out in YAML. If I were to switch companies, or even teams, it'd be the same. Having a lingua franca of common terms and ideas is super critical there. To say nothing of all the work happening in the ecosystem allowing me to help my users ship features faster to our stakeholders. Doing this in VMs would be a literal nightmare. So, respectfully, I believe your opinion is not correct here. You personally may not need it, but it's not very long before you get to reasonable amounts of services for any project that's sufficiently complex, in my experience. And even when I have just 1 or 2 I still need many of the primitives of kubernetes.
- deathanatos 6y agoOh good lord, I'm in the middle of one of these right now. My team runs a fair number of K8s clusters, mostly on Azure AKS. Of the pods in our clusters, a few talk to the AKS API server for their cluster. Those pods that do, will, occasionally, lose contact with the API server. API calls will start timing out. It'll usually resolve on its own after some time, but then come back later. It's somewhat affected by load. (If we reboot a node or nodes, it usually happens after that reboot.) I think SNAT port exhaustion is likely. Azure claims it isn't, and they don't provide monitoring on it, so I'm forced to take their word on it. Azure support has been less than helpful. They think we're putting too much load on the API server. The sum total of all of our custom pods that make k8s API calls is only like ~1/3 of the total load. The rest is core k8s components, like the kubelet. That to me, doesn't smell like a lot of load. Their other suggestion is to buy the "Uptime SLA", a contract that adds an SLA to the cluster. Otherwise, all they offer is an SLO. (Which, empirically … we don't get.) If someone would love to write the rest of this story… I'd love it. We're rolling out the SLA, and it seems promising, but we're hitting what we think is a separate issue. (Sometimes, the API server will refuse connections from particular nodes, usually for good: we've had to reboot the node to unwedge it. This is the second time that's happened; the first time, Azure support told us to they couldn't debug without a live example. Now we have a live example…) (I do, on the whole, like Kubernetes. It does a lot of things well.)
- nyellin 6y agoWould love to discuss in detail and see if I can help. At my last employer I wrote low level solutions for K8s related to security and at the moment I'm working on a stealth mode startup for K8s troubleshooting. Feel free to shoot me an email at aantny at gmail.
- xiwenc 6y agoThis is the downside of “cloud services”. It’s all sunshine and rainbows when it works. When it breaks, start praying and pull the wallet hoping that helps. You are just at their mercy because you cant look under the hood even if you want to. And it takes a long time before they decide to assign competent people on your case. My experience with Azure is pretty bad. (Premier) Support is next to useless. AWS is somewhat better, specially quality of their software/services/documentation. In my opinion, aws services feel more battle tested than azure. Sorry for the rant, back on topic. Do you have monitoring on the cluster? I would start to collect more low level metrics like open files, network traffic, pings Also what could help is to run daemonset of a pod that can send you diagnostics at regular intervals. Perhaps even capability for remote shell so that you can troubleshoot from within the node when the problem surfaces again. I would definitely be interested to know more how this turns out. I also run a fair share of AKS.
- alexellisuk 6y agoDisclaimer, I am a CNCF Ambassador (voluntary) - so it's in my interest to promote CNCF projects like Kubernetes. It seems like a good time to mention my blog post from last year "Then he asked me “Is Kubernetes right for us?" -> https://alexellisuk.medium.com/then-he-asked-me-is-kubernetes-right-for-us-78695ee35289 https://alexellisuk.medium.com/then-he-asked-me-is-kubernete... Some of the feedback I've had so far is that it was refreshing to get "permission" to consider alternatives vs. the current hype. I use K8s and K3s quite broadly myself, but increasingly see consulting prospects and customers who are not comfortable to make the leap, but are very happy on managed services with their chosen vendor - Azure / AWS / GCP. GitLab recently released a bunch of terraform to show you how to run side projects in the free tier of a cloud - https://gitlab.com/gitlab-org/5-minute-production-app/deploy-template#deploy-template-for-5-minute-production-app https://gitlab.com/gitlab-org/5-minute-production-app/deploy... The OpenFaaS project again is very coupled to Kubernetes, making it easier to use and more reliable is important to the community and project's future. However, we created a version called faasd that works more like docker-compose. It's received much more traction than we expected and companies and individuals are putting it into production. It does't have clustering, and supports only 1 replica per function, so it's surprising. https://github.com/openfaas/faasd https://github.com/openfaas/faasd I'll keep doing my bit to promote solutions that make K8s easier to understand like K3s (see also https://k3sup.dev https://k3sup.dev) and to look into alternatives. But as you will see in my blog post - I don't think it's right to assume Kubernetes is the right solution for every team, and every project, without first talking about the problem being solved.
- politelemon 6y ago> I'll keep doing my bit to promote solutions that make K8s easier to understand like K3s (see also https://k3sup.dev https://k3sup.dev) and to look into alternatives. But as you will see in my blog post - I don't think it's right to assume Kubernetes is the right solution for every team, and every project, without first talking about the problem being solved. Thanks for doing your part. Frankly I don't believe the message is being spread enough. At least in the circles I circle, the unspoken expectation to use k8s for "everything" is persistent. The energy that I save by not dealing with it is sometimes spent on explaining why I'm not dealing with it. Yes, entirely an outcome of my circles. I will however share your links in the future, so thanks.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- TruthWillHurt 6y agoHere's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL
- thinkharderdev 6y agoJust curious, but if your old setup was working, why did you decide to move to Azure AKS?
- TruthWillHurt 6y agoCorporate decided to stop trying to compete with cloud vendors and shut down our in-house Cloud Foundry hosting. Also Microsoft sales folks worked client decision makers pretty hard.
- klohto 6y agoHow is that K8s fault?
- hinkley 6y agoBy that reasoning, how is anything ever anybody's fault? Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?
- klohto 6y agoWhat? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
- ThePhysicist 6y agoIt's too bad we don't have more UNIX-style tools in the DevOps space (in the sense of "do one thing and do it well"). Some features of K8s like software-defined networking would be really useful for our infrastructure, but it seems you'll have to fully commit to the rest of the software to use a single feature. Personally I'd prefer to have a loose set of tools for containerization, networking, orchestration, load balancing etc. that I can combine as I see fit and gradually adopt, and not a single monolith that ties me into a given paradigm for all these things.
- deleted 6y ago[deleted]
- crb002 6y agoI don't get k8s when you pay for EC2/Lambda. Why add another layer of complexity?