7 ms·
Google Cloud Incident Report – 2025-06-13
- softveda 1y agoSo code that was untested (the code path that failed was never exercised), perhaps there is no test environment, and not even peer reviewed ( it did not have appropriate error handling nor was it feature flag protected.) was pushed to production, what a surprise !!
- JackSlateur 1y agoNo amount of "whatever" can prevent bugs to reach production
- miyuru 1y agoI would not be surprised if the code was AI generated.
- yen223 1y agoI like the faith you have that people weren't making null-pointer mistakes before LLMs.
- miyuru 1y agoIt did happen before LLMs, but there are well documented process to catch them. Google literally wrote the book on SRE best practices. error handling is very basic, the only explanation these kind of bad code to get pushed to prod is LLMs and high trust on LLM automation. they wont admit this publicly anyway, there is too much money invested on LLMs.
- bananapub 1y agoThere absolutely is a test environment, it was absolutely reviewed and Google has absolutely spent Moon-landing money on testing and in particular static analysis.
- polotics 1y agook so what gives then?
- fidotron 1y agoMoon landing money on static analysis that failed to identify the existence of a completely untested code path? Or even to shake this out with random data generation? This is a dumbfounding level of mistake for an organization such as Google.
- Unroasted6154 1y agoWhat makes you think it was completely untested? The condition that triggered the null pointer exception was obviously not tested, but it doesn't mean it didn't have tests or even 100% unit test coverage for the coverage tools. In addition it looks like the code was not ready for production and the mistake was not gating it behind a feature flag. It didn't go through the normal release process.
- piva00 1y agoIf Google spent Moon-landing level of money in their quality/deployment infrastructure I expect a much better coverage checker than "100% unit tested", they are famous for having a whole fuzzing infrastructure, coverage analysers for more complex interplay of logic is something I use daily in a non-Google levels of spending (even though still a big enough corporation) which often reminds me that I forgot to write a functional test to cover a potential orchestration issue. I don't think "completely untested" is correct but tested way below expectations for such structural piece of code is a lesson they should learn, it does look like an amateur-hour mistake.
- Unroasted6154 1y agoThe main issues to me seem to be that the code was not gated by a flag when it was not ready to be used, thus skipping a lot of the testing / release qualification.
- 1y ago
- sebazzz 1y agoContinuous Integration/Continuous Disaster
- Philpax 1y ago> Without the appropriate error handling, the null pointer caused the binary to crash. We must be at the trillion dollar mistake by now, right?
- bix6 1y agoI wonder how many SLAs they just cooked for the year
- tough 1y agoof their own or their customers with theirs?
- Sytten 1y agoIf only we had a language that could prevent those /s
- paulddraper 1y agoWhich one do you have in mind? Haskell?
- zamadatix 1y agoDoes it really matter which specific language?
- paulddraper 1y agoThe previous comment seemed to think the specific language mattered.
- zamadatix 1y agoDo you mean Sytten's comment? I read that as joking we must be waiting for such safer languages to be invented otherwise we'd be using them, not as a push for a specific existing language itself. Humor can be hard to interpret in text though as it inherently requires trying to read into the message alone, but that you could replace "like Haskell?" with dozens of options via the same assumption is the original point/joke.
- bananapub 1y agolol at whoever approved the report not catching the fuckup of “red-button” instead of “big red button”.
- __float 1y agoWith all of the basic mistakes in the content of this report, how is removal of the word "big" worthy of description as a fuckup?
- koakuma-chan 1y agoStill not rewriting in Rust?
- VWWHFSfQ 1y agoThey probably will now
- _zoltan_ 1y agoenough with rust already
- Unroasted6154 1y agoFor all practical purposes impossible at this scale. The issue is not really the bug tbh.
- koakuma-chan 1y agoWhat is the issue?
- Unroasted6154 1y agoThe issue was that a bug got triggered globally within a few seconds and all the things that led to that. The fact that is was a null pointer exception is almost irrelevant. If you are talking about the issue for migrating to rust, well re-writing hundred of million lines of lines of code never makes sense.
- amy214 1y agoThe issue is how a person looks at it. You could say: 1) the cause was a null pointer, so let's fire that programmer and get a good one in there who doesn't make stupid mistakes. Or instead taking a different angle and attitude: 2) Every programmer makes null pointer mistakes at some point, we need to treat human workers, though incredibly gifted, as not 100% perfect. Thus this is a failure of our system and our policies to catch a scenario that was inevitable This type of thinking and parallel applies to any and all high risk situation, least of which is whether google cloud works, but also things like "my airplane crashed" or "the surgeon removed the wrong leg"
- charcircuit 1y ago>We will enforce all changes to critical binaries to be feature flag protected and disabled by default. Not only if this was true it would kill developer productivity, but how can this even be done. When the compiler versions bumps that's a lot changes that need to be gated. Every team that works on a dependency will have to add feature flags for any change and bug fix they make.
- knorker 1y agoAnd yet it's not even close to the productivity hurting rules and procedures followed for aerospace software. Rules in aerospace are written in blood. Rules of internet software are written in inconvenience, so productivity is usually given much higher priority than reducing risk of catastrophic failure.
- rkagerer 1y agoYeah, some of their takeaways seem overly stifling. This sounds like it wasn't a case of a broken process or missing ingredients. All the tools to prevent it were there (feature flags, null handling, basic static analysis tools). Someone just didn't know to use them. This also got a laugh: We posted our first incident report to Cloud Service Health about ~1h after the start of the crashes, due to the Cloud Service Health infrastructure being down due to this outage. For some customers, the monitoring infrastructure they had running on Google Cloud was also failing, leaving them without a signal of the incident or an understanding of the impact to their business and/or infrastructure. You should always have at least some kind of basic monitoring that's on completely separate infrastructure, ideally from a different vendor. (And maybe Google should too)
- mritchie712 1y agothis is the worst GCP outage I can remember > It took up to ~2h 40 mins to fully resolve in us-central-1 this would have cost their customers tens of millions, maybe north of $100M. not surprised they'd have an extreme write up like this.
- porridgeraisin 1y ago
- master_crab 1y agoThe big issue here (other than the feature rollout) is the lack of throttling. Exponential backoff is a fairly standard integration for scaled applications. Most cloud services use it. I’m surprised it wasn’t implemented for something as fundamental as Service Control.
- foota 1y agoMy reading is that this is on start up. E.g., some config needs to be read when tasks come up. It's easy to have backoff in your normal API path, but miss it in all the other places your code talks to services.
- __turbobrew__ 1y agoDoes borg not backoff your task if it crash loops? That is how k8s does it.
- foota 1y agoI'm not sure, but it clearly wasn't sufficient if it does. I guess the issue here is if you're crash looping, once the task comes up it will generate load retrying to get the config, so even if you're no longer crash looping (and hence no longer backing off at borg) you're still causing overload. As long as the initial rate of tasks coming up is enough to cause overload, this will result in persisting the outage even once all tasks are up (assuming that the overload is sufficient to bring goodput of tasks becoming healthy to near zero). Interestingly you can read that one of the mitigations they applied was to fan out config reads to the multiregional mirrors of the database instead of just the regional us-central1 mirror, presumably the multi regional mirrors brought in significantly more capacity than just the regional mirrors, spreading the load. I'd be curious to know how much configuration they're loading that it caused such load.
- deleted 1y ago[deleted]
- Sytten 1y agoTLDR a dev forgot an if err != nil { return 0, err } in some critical service
- VWWHFSfQ 1y agoThat is still the most bizarre error handling pattern that is just completely accepted by the Go community. And after a decade of trying to fix it, the Go team just recently said that they've given up.
- koakuma-chan 1y agoDoes the Go compiler not force you to handle errors
- Thaxll 1y agoNo language forces to handle errors. Even Rust.
- trealira 1y ago> No language forces to handle errors. Even Rust. It does. If a function returns a Result or Option type, you can't just use its data; you have to either automatically propagate the error case upward, pattern match on both the success and error cases, or just use ".unwrap()" and explicitly choose to crash in the case of an error. There's no implicit crash like there is in a language that lets you dereference null pointers.
- asim 1y agoGoogle post-mortems never cease to amaze me. From seeing it inside the company to outside. The level of detail, its amazing. The thing is. They will never make the same mistake again. They learn from it, put in the correct protocols and error handling and then create an even more robust system. The thing is, at the scale of Google there is always something going wrong, the point is, how is it being handled not to affect the customer/user and other systems. Honestly it's an ongoing thing you don't see unless you're inside and even then on a per team basis you might see things no one else is seeing. It is probably the closet we're going to come to the most complex systems of the universe, because we as humans will never do better than this. Maybe AGI does, but we won't.
- usrnm 1y agoAs I understand it the outage was caused by several mistakes: 1) A global feature release that went everywhere at the same time 2) Null pointer dereference 3) Lack of appropriate retry policies that resulted in a thundering herd problem All of these are absolutely standard mistakes that everyone who's worked in the industry for some time has seen numerous times. There is nothing novel here, no weird distibited system logic, no google scale, just rookie mistakes all the way
- Thaxll 1y agoNull pointer dereference has nothing to do with that problem, they have billions of loc you think that kind of issue will never happen? This is 100% a process problem.
- usrnm 1y agoA null pointer crashing your entire service in 2025 is a process problem
- Mond_ 1y agoI mean, I don't want to be a "Keeps bringing up Rust in the comments" sort of guy, but null pointer dereferences are in fact a problem that can be solved essentially completely using strict enough typing and static analysis. Most existing software stacks (including Google's C++, Go, Java) are by no means in a position to solve this problem, but that doesn't change that it is, in fact, a problem that fundamentally can be solved using types. Of course, that'd require a full rewrite of the service.
- blibble 1y agothis is really amateur level stuff: NPEs, no error handling, no exponential backoff, no test coverage, no testing in staging, no gradual rollout, fail deadly I read their SRE books, all of this stuff is in there: https://sre.google/sre-book/table-of-contents/ https://sre.google/sre-book/table-of-contents/ https://google.github.io/building-secure-and-reliable-systems/raw/toc.html https://google.github.io/building-secure-and-reliable-system... have standards slipped? or was the book just marketing
- tough 1y agosomeone vibe coded a push to prod on friday?
- koakuma-chan 1y agoIronically Gemini wouldn’t forget a null check
- marcinzm 1y agoAs an outsider my quick guess is that at some point after enough layoffs and the CEO accusing everyone of being lazy, people focus on speed/perceived output over quality. After a while the culture shifts so if you block such things then you're the problem and will be ostracized.
- gjsman-1000 1y agoAs an outsider, what I perceive is quite different: HN likes to pretend that FAANG is the pinnacle of existence. The best engineers, the best standards, the most “that wouldn’t have happened here,” the yardstick by which all companies should be measured for engineering prowess. Incidents like this repeatedly happening reveal that’s mostly a myth. They aren’t much smarter, their standards are somewhat wishful thinking, their accomplishments are mostly rooted in the problems they needed to solve just like any other company.
- dvfjsdhgfv 1y ago
- Xenoamorphous 1y ago> We will modularize Service Control’s architecture, so the functionality is isolated and fails open. Thus, if a corresponding check fails, Service Control can still serve API requests. If I understood it correctly, this service checks proper authorisation among other things, so isn’t failing open a security risk?
- foota 1y agoI think this was probably specific to the quota checks.
- bostik 1y agoThis was the bit that I spotted as potentially conflicting as well. Having managed (and sanitised!) tech&security policies at a small tech company, the fail-open vs. fail-closed decisions are rarely clear cut. What makes it worse is that a panicked C-suite member can make a blanket policy decision without consulting anyone outside their own circle. The downstream effects tend to be pretty grim, and to make things worse, they start to show up only after 6 months. It's also a coinflip whether a reverse decision will be made after another major outage - itself directly attributable to the decisions made in the aftermath of the previous one. What makes these kinds of issues particularly challenging is that by their very definition, the conditions and rules will be codified deep inside nested error handling paths. As an engineer maintaining these systems, you are outside of the battle tested happy paths and first-level unhappy paths. The conditions to end up in these second/third-level failure modes are not necessarily well understood, let alone reproducible at will. It's like writing code in C, and having all your multi-level error conditions be declared 'volatile' because they may be changed by an external force at any time, behind your back.
- deleted 1y ago[deleted]
- jeffrallen 1y agoWhatever this "red-button" technology is, is pants. If you know you want to turn something off at incident + 10 mins, it should be off within a minute. Not "Preparing a change to trigger the red-button", but "the stop flag was set by an operator in a minute and was synched globally within seconds". I mean, it's not like they don't have that technology: the worldwide sync was exactly what caused the outage. At $WORK we use Consul for this job.
- foota 1y agoGenerally, even these emergency changes are done not entirely immediately to prevent a fix from making things worse. This is an operational choice though, not a technical limitation. My guess being involved in similar issues in the past is the ~15 minute delay preparing the change was either that it wasn't a normally used big red button, so it wasn't clear how to use it, or there was some other friction preparing the change.
- ollien 1y agoWhat is the difference between a red button and a feature flag, anyway? The report says there was no feature flagging, yet they had this "red button". It sounds to me like something needed to be recompiled and redeployed.
- rvz 1y ago> We will improve our static analysis and testing practices to correctly handle errors and if need be fail open. > Without the appropriate error handling, the null pointer caused the binary to crash. Even worse if this was AI generated C or C++ code, wasn't this not tested before deployment? This is why you write tests before the acutal code and why vibe-coding is a scam as well. This would also never have happened if it was in Rust. I expect far better than this from Google and we are still dealing with null pointer crashes to this day.
- Philpax 1y agoI'm as much of a Rust advocate as anyone, but what does vibe-coding have to do with any of this?
- CaptainFever 1y agoSeems to just be the new thing to casually hate on. "If this was vibe coded, this is even worse. This proves that vibe coding is bad."
- montebicyclelo 1y agoTLDR, unexpected blank fields > policy change was inserted into the regional Spanner tables > This policy data contained unintended blank fields > Service Control... pulled in blank fields... hit null pointer causing the binaries to go into a crash loop
- eddd-ddde 1y agoReally whatever null condition caused a crash is mostly irrelevant. The big problem is instantaneous global replication of policy changes . All code can fail unexpectedly, the gradual rollout of the code was pointless since it doesn't take effect until any policy changes. Yet policy changes are near instantaneous.
- darkwater 1y agoUsually Google and FAANG outages in general are due to things that happen only at Google scale but this incident seems from a generic small/medium company with 30 engineers at most.
- mkl95 1y agoI've seen it happen at companies with 100s of engineers, some of them ex-FAANG, including ex-Googlers. The average FAANG engineer has a way more pedestrian work ethic than the FAANG engineers on HN want you to believe.
- Kye 1y agoNo error handling, empty fields no one noticed. Was this change carelessly vibe coded?
- paulddraper 1y ago> We posted our first incident report to Cloud Service Health about ~1h after the start of the crashes, due to the Cloud Service Health infrastructure being down due to this outage. No one wondered hm maybe this isn’t a good idea?
- stackskipton 1y agoGetting approval from marketing/senior leadership to setup this thing that is outside normal infrastructure is very difficult so it was probably decided against.
- mkl95 1y ago> The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. I've been there. The product guy needs the new feature enabled for everyone, and he needs it yesterday. Suggestions of feature flagging are ignored outright. The feature is then shipped for every user, and fun ensues.
- owenthejumper 1y agoIf this wasn’t vibe coded I’ll eat a frog or something.
- deleted 1y ago[deleted]
- jitl 1y ago> This policy data contained unintended blank fields. Service Control, then regionally exercised quota checks on policies in each regional datastore. This pulled in blank fields for this respective policy change and exercised the code path that hit the null pointer causing the binaries to go into a crash loop. Another example of Hoare’s “billion-dollar mistake” in multiple multiple Google systems: - Why is it possible to insert unintended “blank fields” (nulls)? The configuration should have a schema type that doesn’t allow unintended nulls. Unfortunately Spanner itself is SQL-like and so fields must be declared NOT NULL explicitly, the default is nullable fields. - Even so, the program that manages these policies will have its own type system and possibly an application level schema language for the configuration. This is another opportunity to make invalid states unrepresentable. - Then in Service Control, there’s an opportunity to prove “schema on read” as you deserialize policies from the data store into application objects, again either a programming language type or application level schema could be used to validate policy rows have the expected shape before they leave the data layer. Perhaps the null pointer error occurred in this layer, but since this issue occurred in a new code path, it sounds more likely the invalid data escaped the data layer into application code. - Finally, the Service Control application is written in a language that allows for null pointer references. If I were a maintainer of this system, the minimally invasive chance I would be thinking about, is how to introduce an application level schema to the policy writer and the policy reader that uses a “tagged enum type” or “union type” or “sum type” to represent policies that cannot express null. Ideally each new kind of policy could be expressed as a new variant added to the union type. You can add this in app code without rewriting the whole program to a safe language. Unfortunately it seems proto3, google’s usual schema language, doesn’t have this constraint. Example of one that does: https://github.com/stepchowfun/typical https://github.com/stepchowfun/typical
- esprehn 1y agoI work on Cloud, but not this service. In general: - All the code has unit tests and integration tests - Binary and config file changes roll out slowly job by job, region by region, typically over several days. Canary analysis verifies these slow rollouts. - Even panic rollbacks are done relatively slowly to avoid making the situation worse. For example globally overloading databases with job restarts. A 40m outage is better than a 4 hour outage. I have no insider knowledge of this incident, but my read of the PM is: The code was tested, but not this edge case. The quota policy config is not rolled out as a config file, but by updating a database. The database was configured for replication which meant the change appeared in all the databases globally within seconds instead of applying job by job, region by region, like a binary or config file change. I agree on the frustration with null pointers, though if this was a situation the engineers thought was impossible it could have just as likely been an assert() in another language making all the requests fail policy checks as well. Rewriting a critical service like this in another language seems way higher risk than making sure all policy checks are flag guarded, that all quota policy checks fail open, and that db changes roll out slowly region by region. Disclaimer: this is all unofficial and my personal opinions.
- blibble 1y ago> The code was tested, but not this edge case. so... it wasn't tested
- dieortin 1y agoSo you need to write a test for every single possible case to consider your code tested?
- OtherShrezzing 1y agoI don’t think it’s unreasonable to hold GCP & co to a higher standard than run of the mill software. The platform generates around $100mn per day for Google with a 10% operating margin. Testing all inevitable code paths which have the capacity to bring the platform down as soon as it’s deployed doesn’t seem like an unreasonable ask for a corp operating at that scale.
- gdenning 1y agoWhy did it take so long for Google to update their status page at https://www.google.com/appsstatus/dashboard/ https://www.google.com/appsstatus/dashboard/? According to this report, the issue started at 10:49 am PT, but when I checked the Google status page at 11:35 am PT, everything was still green. I think this is something else they need to investigate.
- dieortin 1y agoThis is explained in the link you’re commenting on.
- QuinnyPig 1y agoI saw a lot of third party services (i.e. CloudFlare) go down, but did any non-Cloud Google properties see an impact? It’d say something that core Google products don’t or won’t take a dependency on Google Cloud…
- sadeshmukh 1y agoSearch, YouTube, Meet, Nest, Workspace - I definitely heard from people I know personally or dealt with myself about YouTube, Nest, and Search, but the others are from downdetector.
- mplanchard 1y agoRegardless of the business need for near instantaneous consistency of the data globally (i.e. quota management settings are global), data replication needs to be propagated incrementally with sufficient time to validate and detect issues. This reads to me like someone finally won an argument they’d been having for some time.
- SageThrowaway 1y ago(Throwaway since I was part of a related team a while back) Service Control (Chemist) is a somewhat old service, been around for about a decade, and is critical for a lot of GCP APIs for authn, authz, auditing, quota etc. Almost mandated in Cloud. There's a proxy in the path of most GCP APIs, that calls Chemist before forwarding requests to the backend. (Hence I don't think fail open mitigation mentioned in post-mortem will work) Both Chemist and the proxy are written in C++, and have picked up a ton of legacy cruft over the years. The teams have extensive static analysis & testing, gradual rollouts, feature flags, red buttons and strong monitoring/alerting systems in place. The SREs in particular are pretty amazing. Since Chemist handles a lot of policy checks like IAM, quotas, etc., other teams involved in those areas have contributed to the codebase. Over time, shortcuts have been taken so those teams don’t have to go through Chemist's approval for every change. However, in the past few years, the organization’s seen a lot of churn and a lot of offshoring too. Which has led to a bigger focus on flashy, new projects led by L8/L9s to justify headcount instead of prioritizing quality, maintenance, and reliability. This shift has contributed to a drop in quality standards and increased pressure to ship things out faster (and one of the reasons I ended up leaving Cloud). Also many of the servers/services best practices common at Google are not so common here. That said, in this specific case, it seems like the issue is more about lackluster code and code review. (iirc code was merged despite some failures). And pushing config changes instantly through Spanner made it worse.
- spacemadness 1y agoGuess all that leet code screening only goes so far, huh?
- dehrmann 1y agoI recently started as a GCP SRE. I don't have insider knowledge about this, and my views on it are my own. The most important thing to look at is how much had to go wrong for this to surface. It had to be a bug without test coverage that wasn't covered by staged rollouts or guarded by a feature flag. That essentially means a config-in-db change. Detection was fast, but rolling out the fix was slow out of fear of making things worse. The NPE aspect is less interesting. It could have been any number of similar "this can't happen" errors. It could have been mutually exclusive fields are present in a JSON object, and the handling logic does funny things. Validation during mutation makes sense, but the rollout strategy is more important since it can catch and mitigate things you haven't thought of.
- JaggerFoo 1y agoI would be interested in seeing the elapsed time to recovery for each location up to us-central-1. Is this information available anywhere?
- throwaway250612 1y agoI am an insider, hence the throw away account. The root cause of this incident was leadership driving velocity by cutting corners. It has been going on for years, eventually over the cliff. This specific failure mode is known as query of death. A query triggers an existing bug that causes the server to crash. It is inevitable for C++ servers. Service Control is in C++. It uses a comprehensive set of engineering guidelines to minimize and tolerate query of death and other failure modes. Before this incident, it had no major incident in the previous decade. This incident is related to a new global quota policy. It was built quickly under leadership pressure, cutting corners. Such features should be built in a secondary service, or at least following the established engineering guidelines. Regarding the action items mentioned in the report, the established engineering guidelines far exceed them. The team has been keeping up with their standard as much as they can.
- flaminHotSpeedo 1y agoYou can't pin this entirely on leadership, allowing global blast radius deployments without an extreme level of scrutiny is a failure of engineering culture. At the very least the global policy should have been deployed prior to the regional service control deployments.
- metadat 1y agoEngineering culture needs leadership and executive support to succeed and thrive. Blaming failures stemming from top down mandates and directives is unfair, because if the grunts don't follow orders they'll be given poor performance reviews or expediently managed out (aka Fired).
- flaminHotSpeedo 1y agoTo some extent, yes, strong leadership can squash engineering culture. But if engineers never push back, how is leadership supposed to know they're asking for dangerous things? For most engineers, that means explaining risks and/or engaging more senior engineer. For L8 and up, it is absolutely their job to say no to leadership. And that's all ignoring things that should have been done here which wouldn't affect the timeline. Leadership doesn't care what order you deploy things in.
- junon 1y agoNull pointers strike again.
- okwhateverdude 1y agoMore like yolo deploys without proper scaffolding in place to handle SHTF. Also, if your red-button takes 40 minutes and deploys to mitigate anything, it isn't a red-button.
- tempodox 1y ago> Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag protected, the issue would have been caught in staging. So some combination of both.
- hardwaresofton 1y agoHas there been any confirmation it was golang?
- rfoo 1y agoPretty sure it's C++. inb4 someone yells Rust. No, your `.unwrap()` happily panics in Rust too.
- ofrzeta 1y agoThe incident report is interesting. Fast reaction time by the SRE team (2 minutes), then the "red button" rollout. But then "Within some of our larger regions, such as us-central-1, as Service Control tasks restarted, it created a herd effect on the underlying infrastructure it depends on (i.e. that Spanner table), overloading the infrastructure. Service Control did not have the appropriate randomized exponential backoff implemented to avoid this. It took up to ~2h 40 mins to fully resolve in us-central-1 as we throttled task creation to minimize the impact on the underlying infrastructure and routed traffic to multi-regional databases to reduce the load." In my experience this happens more often than not: In an exceptional situation like a recovery of many nodes quotas that make sense in regular operations get exceeded quickly and you run into another failure scenario. As long as the underlying infrastructure can cope with it, it's good if you can disable quotas temporarily and quickly. Or throttle the recovery operations that naturally take longer in that case.
- throwaway250612 1y agoExponential backoff is misinformation here. During the server startup, it reads a critical data, intentionally no retry, hence no backoff. A better fix is to quickly spread the load to backup databases that already exist. There are other options too.
- rvnx 1y agoSuch absurdity that as a customer we (HN) knew more than the official support
- secondcoming 1y ago> If this had been flag protected, the issue would have been caught in staging. I’m a bit confused by this, it seems the new code was enabled by default and so should have been caught in staging.
- reassess_blind 1y agoEveryone loves to criticise downtime when it happens to others, saying these are “junior level mistakes” and so on. Until it happens to them, and then there’s a convenient excuse as to why it was either unavoidable or unforeseeable. Truth is humans make mistakes and the expectations are too high. When a brick and mortar business has to shut unexpectedly they’ll put a sign on the door, apologise and that’s that. Only in tech do we stress so much about a few hours per year. I wish everyone would relax a bit.
- 486sx33 1y ago[dead]
- cddotdotslash 1y agoIt’s interesting that multi-region is often touted as a mechanism for resilience and availability, but for the most part, large cloud providers seem hopelessly intertwined across regions during outages like these.
- remus 1y agoI dont think you can really make a conclusion without first understanding occasions where multi-region deployments have prevented incidents.
- Balinares 1y agoMind the unknown unknowns there. You are not aware of all the "outages like these" that could have occurred but did not thanks to regional/zonal sharding.
- flaminHotSpeedo 1y agoThis is really only a problem on GCP, who treats regions far differently from other providers. From a resilience perspective, it's best to treat GCP as a single region with a bunch of zones.
- sunrunner 1y agoMaybe it's just my lack of reading comprehension but some of the wording in this report feels off: > this code change came with a red-button to turn off that particular policy serving path. > the root cause was identified and the red-button (to disable the serving path) was being put in place So the red-button was or wasn't in place on May 29th? The first sentence implies it was ready to be used but the second implies it had to be added. A red-button sounds like a thing that's already in place and can be triggered immediately, but this sounds like an additional change had to be deployed? > Without the appropriate error handling, the null pointer caused the binary to crash This is the first mention of null pointer (and _the_ null pointer too, not just _a_ null pointer) this implies the specific null pointer that would have caused a problem was known at this point? And this wasn't an early issue? I don't mean to play armchair architect and genuinely want to understand this from a blameless post-mortem point-of-view given the scale of the incident, but the wording in this report doesn't quite add up. (Edit for formatting)
- HardCodedBias 1y agoThis was a shockingly simple coding error. It should have been caught by the two code reviewers. Turns out asking an LLM for a code review finds the error and an LLM suggests the correct fix. Rarely do you see a major outage caused by such a glaring error. I suspect that policy changes will be required at GCP. The SRE team was very fast, the logs were inspected quickly and the relevant check that was being failed was identified within minutes. That's impressive. But the coding error, that was shocking.
- sn9 1y agoYou have a link to the code in question?
- flaminHotSpeedo 1y agoI think the most damning thing here is the "what's our approach moving forward" section: Every single bullet point there is basic stuff you expect a cloud provider to do correctly damn near every time. If you're not paying GCP for that expertise, or if that expertise is built on layers of cruft where that expertise was not applied... what are you paying them for?
- welder 1y agoGo could learn from this and implement Swift's optional feature: https://wakatime.com/blog/48-go-desperately-needs-nil-safe-types https://wakatime.com/blog/48-go-desperately-needs-nil-safe-t...