23 ms·
Google Kubernetes Engine incident spanning 9 days
- edude03 3y agoCreating a new node group works which is super easy on GKE so it's pretty much a non issue. Definitely frustrating but not as bad as it sounds at first brush
- timo-e-aus-e 3y agooh man, you don't wanna be the engineer on-call for that.
- akokanka 3y agoThis shows the astronomical complexity of k8 systems even gods of k8 fail.
- Closi 3y agoTo be fair, running a Kubernetes Cluster is simpler than running a Fully managed Kubernetes As-A-Service Offering for thousands of clients.
- uniformlyrandom 3y agoThe incident impact (nodepool upgrade issue) seems to be matching the speed of mitigation rollout. One does not want the cure to be worse than the disease; roll forwards should be slow unless the impact is high (and even then, it should be a rollback/freeze rather than fast roll forward).
- water-your-self 3y agoHow do you propose rolling back a kubernetes upgrade that users might depend on
- tuananh 3y agosorry what mitigation is that?
- vinni2 3y agoI have been pulling my hair to fix this all week.
- alectroem 3y agoWow, I literally did a full cluster version upgrade last night without knowing about this. I would have delayed the upgrade if I had known GKE was failing for "a small number of customers" I wish cloud providers would just communicate outages to services I use like this to me!
- talonx 3y agoThey do publish an RSS feed for the status page, but there is no direct way to get notified AFAIK. I used to create a Slack notifier using IFTTT.
- captn3m0 3y agoFor infra folks, I always suggest having a Slack channel with RSS feeds from vendor incident sites. Slack let’s you subscribe directly to an RSS feed using /feed.
- talonx 3y agoYes, I used to do that for my teams. Most infra vendors have RSS feeds for their public status pages.
- alectroem 3y agoThats a really good idea!
- sleepybrett 3y agoDon't put this in a special channel just for vendor incidents. Hopefully you have a channel for each vendor tool where you have a vendor representative present. You put that bot in there. It's much more likely to be noticed and much less likely to be a channel that everyone ignores because it's SNR is to low.
- danielovichdk 3y ago"vendor representative" ? Internal or external contact ? Sounds like a good approach nonetheless
- endisneigh 3y agoSucks that there isn’t anything simpler than k8s that’s production grade. Maybe it’s time to yolo with a regular container that just restarts on failures, ha…
- datadeft 3y agoThere is AWS Autoscaling, ECS, EKS, Hashicorp Nomad, Fly.io and the few more. However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity? Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra. If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.
- bsdnoob 3y agoHashicorp nomad comes to mind.
- jarym 3y agoIt does and most people that used Nomad were very positive about it. But with the turn that Hashicorp have taken I'm not sure that it now stands much of a chance.
- jdoss 3y agoSeconded. Hashicorp Nomad has been a breath of fresh air for doing HA deployments for my workloads. Getting a small cluster setup to self host Nomad is so easier than Kubernetes and defining workloads is much easier to understand too IMO. The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
- hughw 3y agoI hope somebody can elaborate on the licensing problem? I see a community edition...
- DishyDev 3y agoNot a great week for managed Kubernetes services as Digital Ocean have been having an ongoing issue since yesterday morning on their service https://status.digitalocean.com/incidents/fsfsv9fj43w7 https://status.digitalocean.com/incidents/fsfsv9fj43w7
- truthwhisperer 3y ago[dead]
- k8svet 3y agoTo be honest, I'm pretty disappointed that this thread doesn't have some attempt at a root cause now that two major managed providers are having extended outtages linked to upgrades. To think, I once was so in love with it. This kind of crap, and imagining the live-site chaos makes me so glad to have noped-out when I did. With my modest lifestyle, I probably passed on life changing money but I also have 1/10th the stress and anxiety I used to.
- mrweasel 3y agoIn Google terminology is a "mitigation" the same as a solution? I read it as "Yeah, we still have no idea how to fix this correctly, but we have applied a temporary work-around".
- peddling-brink 3y agoI can’t speak for google, but in common parlance, mitigation simply means the problem isn’t affecting you any more. Doubtless they will perform a full post-mortem, identify the contributing factors, and fix them over the coming days, weeks, quarters. Edit: “we are assessing its effectiveness” lol, yeah that sounds more like they they are throwing something at the wall to see if it sticks.
- denysvitali 3y agoWhich is arguably better than just waiting for something to stick to the wall by itself
- yegle 3y agoDisclaimer: Google SRE working in Cloud but not related to the GKE product. Mitigation can vary: an additional firewall rule to stop certain traffic, a rollback to known good version, temporarily redirect traffic away from impacted data center. A mitigation is part of an incident response mainly focused on stopping the pain to whoever are impacted, it may not always be a long term solution.
- VirusNewbie 3y agoIt more or less means actions have been taken to reduce or eliminate the problem, but it is not the same as the long term fix. Mitigation might be 'give more CPU' to the system, despite the performance regression still being in the code etc.
- shizcakes 3y agoWe've been stuck in this state for all 9 days. We've filed tickets, etc, but no resolution has come about yet. Just re-tried yesterday, still not able to update nodepool.
- talonx 3y agoThe incident dashboard does mention creating a new node pool with the new version instead of upgrading - as a workaround. Curious to know why this was not an option for you.
- wg0 3y agoIf there are stateful workloads.
- talonx 3y agoEven if there are stateful workloads, one has to be prepared to migrate at some point. Nodes can crash, become unreachable, etc. - workloads have to be able to survive such events. Edit: But I see your point - if their stateful workloads are not "migration/K8S-incident" ready, it's hard to suddenly build that overnight.
- danjc 3y agoIs it a big deal to just build a new cluster and cut traffic over to it via lb or dns?
- wg0 3y agoMay be this a hated take but just wondering - The place that pretty much invented cluster orchestration and reinvented it as k8s is having problems upgrading it. What chance a bunch of poor sys admins stand running bunch of k8s clusters for a mid size company I wonder. Every time I think of deploying (self managed, have done full stack) it for something mission critical, this upgrade scenario simply makes me rethink it altogether. And even managed k8s has no guarantees and if managed is to be the option, nothing beats ECS in simplicity and smooth operation at certain scales. PS: Full stack K8s means ingress controllers, DNS auto registration, GitOps , logging, monitoring, CI/CD and all the bells and wistles including a management UI behind oauth etc.
- talonx 3y agoWhat bugs me most about GKE is that incidents seem to span a whole of datacenters simultaneously (source: see zones affected by previous incidents), and you are often left with no option except wait for a resolution. Often there are issues like networking problems affecting almost the entire global GKE control plane. Whatever happened to resilience by isolation?
- OhSoHumble 3y agoWhat's really unfortunate is that it feels like not choosing k8s is career suicide for SREs/DevOps/platform engineering. I chose Nomad for my current role and it works fantastic. I used ECS in my last role and it was also fantastic. I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at. If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.
- ricc 3y agoQuick tip: You can still put your target keywords where you have zero to minimal experience in your resume/CV, just put it in an honest statement describing how much experience you have. In your case, you could write, for example, "X years of experience in HashiCorp Nomad and AWS ECS (Kubernetes alternatives for container orchestration)". That should put your resume/CV through the keyword filter without lying about it.
- dilippkumar 3y agoNoob here with some meta-questions about developer and operations complexity. From an outsider’s perspective, it looks like in a 2x2 matrix of developer simplicity/complexity and operational simplicity/complexity, the current patterns all seem to be heavily biased for developer simplicity/operational complexity. 1. Is this assumption correct? 2. Does optimizing for another quadrant: developer complexity / operational simplicity make sense? My intuition is that complexity in code can be managed far better than complexity in operations. Developers have abstractions, reusable libraries, unit tests/integration tests, etc. There may also be weird efficiencies that arise from having developers deal with some of these problems right from the design stage. It seems kubernetes takes a problem and pushes it to fully to operations. Is there a solution that takes this problem and turns it into a developer problem?
- Too 3y agoThe difficult part about Kubernetes for newcomers is that it tackles every problem at once. Large scale best-practices you would previously sweep under the rug are now things you can’t avoid addressing. Centralized logging, secret management, certificate rotation, rbac, liveness probes, rolling deployments, etc. A lot of people complaining about the complexity are doing all those things by hand instead - “just ssh into the server with a shared password, remove old logs and restart it lol”. It’s great if you actually need all of this and consider those practices essential complexity. If you don’t - it feels like someone is shoving accidental complexity down your throat. Turning it into a developer problem will not hide all these things or make them more manageable. It means you will reinvent k8s yourself, poorly.
- diarrhea 3y agoInfrastructure is, or at least should be, code as well. And as it is, you can write tests for it all the same! However, writing those tests is incredibly hard. It doesn’t matter if you approach it from a dev or ops angle. The system under test doesn’t only have side effects, it is side effects. You also cannot mock most things (in my opinion…), as that is either also very hard to instrument or straight up removes the test usefulness altogether. Imagine mocking the AWS management API for your integration tests. Not possible. So what Dev calls integration or e2e tests, ops calls the dev environment. Works, but differently to how devs would do it. I don’t see an alternative. Next, as much as knowledge siloes are being heralded as evil, they exist. Undoing siloes altogether isn’t possible. You’ll end up reerecting them elsewhere. Devs have their skill sets, and ops isn’t part of that. The opposite is also true. The intersection can be substantial, but never enough to have dev to it all alone. I don’t think that’s a bad thing either.
- input_sh 3y agoThey also had an issue with creating and deleting persistent volumes on 1.25. It lasted for 15 days, or half a month(!) last month: https://status.cloud.google.com/incidents/EBxyHQgEPnbM3Syag5yL https://status.cloud.google.com/incidents/EBxyHQgEPnbM3Syag5... I'm also incredibly annoyed at them displaying time in PDT. I genuinely don't understand why they decided on that instead of doing something normal like UTC or detecting my timezone. Especially annoying every six months because Europe and the US don't do Daylight Savings Time changes at the same time, so for a week or two there's an additional hour I have to account for.
- awesomebing1 3y agoGoogle Standard Time, as they say :)