9 ms·
Preliminary Incident Statement
- moks 6y agoPreliminary Incident Statement for google outage
- jimmySixDOF 6y agoThe root cause was an issue in our automated quota management system which reduced capacity for Google's central identity management system, causing it to return errors globally. .... so much for limiting Blast Radius
- anshumankmr 6y agoWhat do you mean by blast radius?
- talonx 6y agoThe area (in this case, number of services or regions) impacted by an outage.
- baskire 6y agoThe fact one service had an issue. And not only did it take said service down globally, it also impacted dozens of other services...globally. A good infra architecture has the “blast radius” of any issue confined to only part of your infra fleet. Avoiding the global outage. Think of it like a navy ship. When a mussel breaches the hull, the ship is designed to contain the leak in a single section. Avoiding the entire ship sinking. Similar pattern is desired in your service and compute infrastructure. The outage google just faced is equivalent to a single missel taking down an entire navy fleet.
- herodoturtle 6y agoI love this analogy, thanks for visuals. > The outage google just faced is equivalent to a single missel taking down an entire navy fleet. I really thought you were going to say "is equivalent to a single missile taking down an entire ship" - but your extreme analogy made even more sense in this context.
- oh_sigh 6y agoHow could one design a system that would not fail when something as critical and cross-cutting as capacity or identity management failed?
- ZiiS 6y agoFor example youtube should have defaulted to the logged out experiance. No one forced them to use the same identity platform for completly unrelated services. Gmail dose raise hard issues but people logged in should not have needed to recontact a auth service.
- coddle-hark 6y agoYou can limit the damage though; in theory there’s no reason why authentication needs to be centralised. You could split the backend up into multiple authentication services and “bucket” the auth requests per email address or something. So essentially make a bunch of different authentication services that can fail independently. I’m sure they have good reasons for not doing that though.
- pjc50 6y ago> When a mussel breaches the hull I knew barnacles were a problem, I hadn't realised there were other dangerous bivalves. :)
- baskire 6y agoHaha, that gave me a good laugh on my iPhone keyboard swipe typo.
- joshxyz 6y agoPetition to have Blast Radius Awards per year among top companies pls
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- Ekaros 6y agoIf all of your services make use of one service, maybe that should be treated as very special case...
- baskire 6y agoCurious how Spotify and other enterprise GCP users have mitigated GCPs repeated global outages. This isn’t their first nor will it be their last.
- bobviolier 6y agoThe Google Cloud was almost unaffected, during the outage you could not log into the cloud console or use tools like gcloud - but the services itself (vms, gke, pubsub, etc) kept working throughout the outage.
- wdb 6y agoThat's not correct, you wouldn't be able to connect these services after restart. E.g. you wouldn't be able to auth to Cloud SQL or Datastore. Stopping development
- jrockway 6y agoServices hosted on GCP didn't lose any traffic during this incident. This was a control plane outage, not a data plane outage. That means that if you needed to go in and adjust something to resolve your own outage, you were in trouble, but if your services were healthy, nothing bad happened. As the time of the outage increases, service health probably trends to zero without being able to manage the service... but for a few hours, it's not a disaster. Usually.
- semi-extrinsic 6y agoOn the flipside, isn't this just a case where you are bound by the CAP theorem? If you want your central identity management database to be consistent and available, as I imagine Google do, it can't be parallel?
- simiones 6y agoI assume you meant "it can't be partition-tolerant"?
- semi-extrinsic 6y agoYes, sorry, sloppy of me.
- cowsandmilk 6y agoNote IAM is also global in AWS, perhaps the only global service other than those that are effectively global by requiring you to use us-east-1
- osdotsystem 6y agoEven Google!
- ur-whale 6y ago>Even Google! The time when Google was special is long gone.
- sriram_malhar 6y agoHow so?
- mav3rick 6y agoLong live the flame bait
- cube00 6y agoautomated quota management system At least it's not just us pleb users at the mercy of their automated systems. I wonder if someone internally appealed the decision and got told their review had been carefully considered and subsequently declined.
- byecomputer 6y agoRecord-breaking regulatory fines? Baked into the budget. Picking up the phone? Unfeasible, can't be done.
- mtgx 6y agoHonestly, it should be illegal for any company of significant size not to offer support to its users/paying customers in a reasonable timeframe. Situations like Google or Facebook automatically banning your accounts for life because of an algorithm and then taking months to solve the issue should not exist. Not when these companies pride themselves on increasing their tens of billions in profit with a few billion dollars more.
- herodoturtle 6y agoYeah was thinking the same. "But picking up the phone doesn't scale" Me: looks at size of fines paid in last 10 years.
- byecomputer 6y agoAlso fine with payrolling a call center that schedules haircutting appointments and restaurant reservations when Google Duplex shorts out—but clearly that's a temporary stopgap for a new technology, unlike the quite finished Google Play robot which bans your app and then makes you guess where and what exactly the problem was.
- rebuilder 6y agoThe fines seem to scale just... fine.
- taneq 6y ago
- passerby1 6y ago"Don't split us the google company! We are tightly coupled, things will go wrong, your precious gmail account will fail more"
- nevi-me 6y agoDoes it make sense to put quotas on internal services, that I assume are unlikely to be used externally? Is it a security recommendation? I'm curious, as I might learn something from interesting asnwers to the above questions.
- cesaref 6y agoTo be fair, if they didn't use it due to concerns about availability, then that would suggest they needed more work on the service. I imagine once they knew what the issue was, they would have been able to fix it as i'm pretty sure that service doesn't need to be authorised by itself to be configured ;) The trouble I imagine was getting visiblity of what the issue was...
- marcan_42 6y ago> I imagine once they knew what the issue was, they would have been able to fix it as i'm pretty sure that service doesn't need to be authorised by itself to be configured ;) You'd be surprised at the circular dependencies that creep up at Google scale... (I used to work there and we had last resort secure access mechanisms, but I wouldn't be surprised if the normal front door wound up shut pretty hard for this one)
- clavoie 6y ago(Disclaimer, I worked at Google over a decade ago, for about half a decade; as an SRE for about 2 of those years) Yes, it absolutely makes sense. You never know when a typo in a config file or bug in your job-management code will go bonkers and try to take over all resources. Same think as disk quotas on computers with tons of users: you want to limit the damage that a user can accidentally (or not...) do. Good quota systems saved my team's bacon way back when a few times, when fewer people were at the company; I can only imagine how useful they are at what, 10-20x the size?
- byecomputer 6y agoGood thing quota systems can't go bonkers and take over all the resources, that would be a nightmare!
- cube2222 6y agoThis post would better be named like "Google Outage Preliminary Incident Statement". I had no idea what I'm gonna see before I read the link.
- elaus 6y agoIt's suggested in the HN guidelines to keep the original title whenever possible and not to editorialize. But I agree, without prior knowledge of yesterday's outtake it's hard to guess what's behind that title.
- dang 6y agoIt's ok to use a subtitle or a representative phrase from the article body, if that better expresses what the submission is about. The main thing we're trying to avoid is people cherry-picking some detail to put in the title or using it to put their own spin on a story. Since "Preliminary Incident Statement" is basically a subtitle in the OP, I've put that in the title field above.
- kortilla 6y agoIt’s not really editorializing to add detail of what a vague thing is referring to. Editorializing would be changing it to something like “Breaking the silence after a catastrophic outage caused by our gross incompetence”.
- herodoturtle 6y agoHah yes. Or what was that one we had the other day? "The death of Google"
- lmm 6y agoMod policy is to replace any useful or informative submitter-provided title with the title of the linked page, so the result would have been the same after a few hours.
- SoulMan 6y agoWell we could see its google's website
- InvOfSmallC 6y agoAm I the only one reading: "someone launched a set replica 0 in prod"? :D just kidding.
- lxgr 6y agoI guess at least someone's buying the largest Google One storage plan next year...
- TonyTrapp 6y agoAnd now we know why they are limiting their free space offerings! ;)
- raybb 6y agoWhat does this mean? I don't see any mention of Google One or storage on the page.
- fuzzybear3965 6y agoI guess I'm confused like @raybb. Why would a service outage make it more likely that someone will opt in to a storage plan? I wouldn't want to jump into a service that's been recently demonstrated to be unavailable for extended periods of time.
- remus 6y agoI think the parent is jokingly suggesting that google's auth service is going to be buying a largest google one storage plan so they don't run out of quota.
- rkagerer 6y agoMany of our internal users and tools experienced similar errors
- o- 6y agoThat sounds exactly like the new autopilot system they describe in their USENIX paper. Talk: https://eventsonair.withgoogle.com/events/autopilot-research-talk-series/watch?talk=autopilot https://eventsonair.withgoogle.com/events/autopilot-research... Paper: https://research.google/pubs/pub49174/ https://research.google/pubs/pub49174/
- kyrra 6y agoGoogler, opinions are my own. The paper and the system that caused this incident are different. Google has a ton of different automated systems for maintaining production. The paper you linked is for a system called autopilot, which can scale a specific jobs memory and CPU usage up or down depending on historical load of that job. Think of it as having different instance sizes on GCP or AWS, then you have a tool that monitors how much actual CPU and memory your job uses on the instance, in will upsize or downsize your machine depending on load over time. The system that broke based on the incident report above had to do with quota that a given production system could use as a maximum. Similar resource management automation, but very different systems. So one is about fine-tuning, while the other is about preventing excessive resource usage.
- bondarchuk 6y agoI wonder if we'll ever get to see the post-mortem of specifically the status page saying that everything's fine when not everything's fine. That's the one that interests me most.
- grumple 6y agoWhen AWS had their major outage a few weeks ago, comments indicated that the status pages are updated manually at these companies. It's heavily influenced by contractual and internal political factors.
- piyh 6y agoBuilding incentives to not truthfully report into those contracts with uptime guarantees like the WHO and pandemic bonds.
- grumple 6y agoUnfortunately, much of the world is built like this. Those who draw attention to failures get blamed, not rewarded (even from a young age: "he who smelt it, dealt it").
- jlgaddis 6y agoDid you miss the "Additional Details" section? > Many of our internal users and tools experienced similar errors, which added delays to our outage external communication. Also: > We will publish an analysis of this incident once we have completed our internal investigation.