5 ms·
Global Google Kubernetes Engine Outage
- bittermandel 3y agoThis doesn't look like an outage at all. > Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures.
- ithkuil 3y agoyeah, as a user of that service you may or may not be affected by this particular problem.
- eVeechu7 3y agoAgree. The word outage isn't used in the notice. They say 'incident' which is more accurate. The title should be changed.
- voytec 3y agoDon't read into announcements like this too much. Status pages and outage notices are often political. Status pages are rarely dynamic and updates require blessing from upstairs. And more often than not complete outages are referred to as "degraded performance affecting some users".
- eddythompson80 3y agoNot sure why you’re being down voted. Status pages for big companies are never hooked up to automation. It’s just bad PR to show red across the bar. If there is a networking outage, everything on a status page should be red but then that looks bad for PR. So you just set “networking outage” but everything else is green even though everything is realistically down.
- aldarisbm 3y agoit's also not only bad PR, but CSPs are subject to SLAs.
- oooyay 3y agoI don't know how status pages work at Google, but I do work in reliability engineering and I sometimes make recommendations to update the status pages. Some context before I go on is that reliability is often measured by mapping critical features to services and degradation. This gets more challenging as a feature starts to map to more than a couple services and those services begin to have dependencies. When your reliability on average can be measured in its number of nines opposed to its significant preceding digits your signal interpretation game has to step up significantly. These two situations make it infinitely more complex to state whether a given service degradation in a chain of services is truly having external customer impact at a given time. That's why a human needs to make the call to update the status page and why status page availability numbers are different from internal numbers. I spend a good portion of nearly every sprint hunting down systemic issues that'll pop up across the ecosystem of services from a birds eye view. Often, knowing whether external customer impact will be felt for this series of errors relies heavily on knowing the current configuration of services in a chain, their graceful failure mechanisms, what failure manifests as client side, and whether that failure is critical to an SLA. I have not, in my history of reliability engineering, seen anyone object to updating the status page for political reasons.
- re-thc 3y ago> I have not, in my history of reliability engineering, seen anyone object to updating the status page for political reasons. The status page is tied to public SLAs = impact on $$$. Internally you can track anything. What's public is the problem.
- oooyay 3y agoNo, not really. SLAs are calculated on a per customer basis and generally have a legal definition in contracts if they're actual, functioning SLAs. The status pages purpose is generally to head off a flood of customer reported issues. This is why you'll usually see issues that affect a broader subset of users on that page.
- re-thc 3y ago
- cdogl 3y agoA big part of the Kubernetes value proposition is “autoscaling”, used in the loosest sense. Pods will come and go over time, in response to events, etc. as part of normal operations for many systems. If I still had to deploy to Kubernetes, I’d consider this an outage.
- cmckn 3y agoI would expect the Venn diagram of: 1) actually using autoscaling in prod 2) using it for a stateful workload (with PV’s) 3) needing to scale that workload during this time window 4) using this specific k8s version To be razor thin. I’m honestly surprised they’ve reported the impact on this dashboard; but good to see they did.
- deleted 3y ago[deleted]
- smif 3y agoIt seems like 1.25 is a month and a half out from retirement, maybe it's related to that. Kubernetes 1.25 (in Danny Glover voice): I'm getting too old for this shit!
- waych 3y agoautoscaling isn't necessary to elicit downtime. Need to deploy a dev copy of your cluster? How about the CI/CD pipelines that are blocked? Go sit on your hands and wait it out. Almost 2 days now.
- cmckn 3y agoWoah, I completely missed that this has been going on since the 13th?! That’s outrageous, surely something can just be rolled back.
- vel0city 3y agoThis isn't preventing pods from moving from node to node, and for a lot of workloads it wouldn't even prevent making new pods. This is if you make a new storage claim (a PVC) the underlying storage filling that claim isn't being created at the moment, for one particular version of Kubernetes. Existing volumes are not impacted. Every other version is not impacted. Loads of pods don't even use PVCs. In the several hundred deployments I routinely manage, there's only a handful of PVCs, and they aren't exactly dynamically created and destroyed. I've got many GKE clusters and no workloads I'm running are affected, and I imagine most existing workloads in GKE clusters aren't affected. I'm then also doubly not affected, because I'm not running 1.25.
- chucky 3y agoCloud providers (and everyone else) are unfortunately always downplaying their incidents though, so I don't trust that information. I have no idea about this particular case though since I'm not a GKE user. Would be interesting to hear from actual users how serious this is.
- dustedcodes 3y agoI run GKE clusters in europe-west2. Completely unaffected. I run on the latest k8s version. I have an uptime monitoring service which runs in AWS and it reports zero downtime over the last 24 hours.
- voytec 3y ago> Completely unaffected. I run on the latest k8s version. The announcement mentions specific version: "issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25"
- vel0city 3y agoWell yeah that's kind of their point. GKE makes it pretty easy to stay on a recent version.
- mwarkentin 3y agoWe're using their "stable" release channel which means most of our clusters are currently on a version of 1.25. We only have a few deployments using PVCs so impact is pretty minimal anyways for us as far as I can tell.
- vel0city 3y agoEven ignoring the fact this only affects one old version and Google makes it very easy to upgrade (and know when you're behind), I can't imagine this is affecting many workloads. The vast majority of the workloads I run (and I think get run, generally) don't rely on PV's. Then from that, most of the time you're not making wholly new volumes all the time you're generally just consuming the already existing volumes.
- 1-6 3y ago[flagged]
- mrjin 3y agoI found one assertion so true: Cloud is just someone-else's computer.
- gchamonlive 3y agoWhich is exactly what you want. You don't want user to access you laptop.
- mrjin 3y agoNot exactly. I don't want to get into the situation that there is nothing I can do when it's down neither.
- gchamonlive 3y agoYou don't have to if you plan your disaster recovery scenarios correctly.
- kynetic 3y agoDepends really. I am the entire ops team for a non-profit volunteer org, I host their services on the cloud specifically so when something fails I don't have to do anything. Would much rather have a cloud computing company's ops team working to fix the issue instead and we'll just pay our monthly dues. The same can be said of many smaller companies where uptime can be maintained cheaper and more consistently by offloading ops work to a cloud computing company.
- verdverm 3y agoYou still have outages when you run your own hardware and systems. What would that look like for your company compared to the cloud?
- intelVISA 3y agoThe cloud is just digital feudalism, except we even tricked people into selling their own hardware willingly thinking it was 'smart'.
- dustedcodes 3y agoAll my services are up and running and I had zero downtime over the last 24 hours, so can't be that "global".
- glitchcrab 3y agoDid you actually read the summary? It affects volumes for customers running 1.25 only - it is global but only for a subset of customers.
- tacticus 3y agoit affects the creation or deletion of volumes. anything that already exists will be fine and continue operating. On the second oldest version of kube currently supported that is also older than the version in every release channel atm. So it's a subset of a subset of a subset of an old version.
- pyrale 3y agoYou seem to miss the point that "global" is a geographic indication. It does not mean "all features" or "all users". > anything that already exists will be fine and continue operating. Any pod that already exists will be fine, but that doesn't extend to services, as they may autoscale volumes along with pods. I'm not sure many users on such an antiquated kube version would have done that, but if no one did, there would probably not be a notice.
- donor20 3y agoDoesn’t seem like a global outage - where is this headline coming from?
- eddythompson80 3y agoThe term global outage in public clouds is used whenever there is an outage that has the same root cause that impacts more than 2 independent regions
- barkingcat 3y agoglobal is about geography.
- glintik 3y agoUsers of dedicated servers: LOL.
- dangoodmanUT 3y agoClickbait title -_-
- mrweasel 3y agoThat has to be annoying to fix. It's one thing when it's "just" your own cluster and something explodes in your face, it's a quite another when it's other people persistent volumes and you can't just assume that they'll be able to redeploy, because you have no idea what is in those storage claims. Glad it's not me having to deal with this, Kubernetes is still a black box of magic in many respects and there's probably more than a few abstraction down to the actual disks.
- BobbyJo 3y agoMost of the abstraction between disk and container is external to K8s. You can use a single box CPU+disk with K8s if you want, that's just not a configurable product offering, so that's not how any of the cloud providers expose disk. K8s itself is entirely unconcerned with most of the implementation details, it's more a framework for those kinds of things
- deleted 3y ago[deleted]