6 ms·
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
by somewhatgoated 5mo ago
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
- corpoposter 5mo agoIIRC the Paris datacenter flood took down a whole “region” and some data was permanently unrecoverable.
- JoRyGu 5mo agoAWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.
- ajross 5mo ago> AWS goes down catastrophically but are back up in minutes/hours most of the time The outage in the linked article appears to have been resolved in 4-5 hours.
- graemep 5mo agoFailover is supposed to protect you every time, unless something really exceptional happens. While its possible to to isolate the effects, judging by how many things stop working when there is an AWS failure a lot of people fail to do that. I think the shit of responsibility to AWS removes the incentive to put effort into resilience against AWS failure.
- pixl97 5mo agoGCP never goes down because they banned all their customers.
- somewhatgoated 5mo agoA funny meme but just untrue
- plandis 5mo agoGCP has had outages. From a quick search it looks like they had a global outage less than a year ago: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
- devmor 5mo agoThere was a pretty bad one last summer - their IAM system got a bad update and it broke almost all GCP services for an hour or so, since every authenticated API call reaches out to IAM. It had lasting effects for us for a little over 3 hours.
- abofh 5mo agoI've been in AWS for almost twenty years at this point. It's been a long time since I've seen a global outage of the data plane on anything. The control plane, especially the US-east-1 services? Yes - but if you're off of east-1, your outages are measured in missile strikes, not botched deployments.
- andreareina 5mo agoDidn't the latest outage affect people not on us-east-1 because internal aws services depend on us-east-1?
- erikerikson 5mo agoThe impacts are usually partial. For example, scaling is impacted but everything already deployed contributes to work up to capacity. Or, you can't change configuration but the previous configuration works as configured. Often surprisingly not so impactful even if there can be limited work stoppage.
- hasyimibhar 5mo agoThe problem with the us-east-1 outage is that a lot of big companies are there, so even if you try your best not to depend on us-east-1, your third party providers are most likely there. In my previous company, we were completely down during us-east-1 outage because of other dependencies that are beyond our control.
- erikerikson 5mo agoEntirely fair. I have thus far avoided that problem. Not always engineering's choice.
- HighGoldstein 5mo agoConsidering how many AWS and non-AWS services go down at least partially when us-east-1 fails, this reads somewhat like "Don't worry that the steering wheel and pedals aren't working, your engine is still running on cruise control".
- blobbers 5mo agoUnfortunately, if everyone goes down people are understanding. If just _you_ go down, then its oddly less forgiveable.
- shamcleren 5mo ago[dead]
- danesparza 5mo agoYou can read the parent post, right?
- Izikiel43 5mo agoI still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups
- Barbing 5mo agoGoogle Cloud accidentally deletes $125 billion Australian pension fund - May 2024 https://www.business-standard.com/world-news/google-cloud-accidentally-deletes-125-billion-australian-pension-fund-124051800606_1.html https://www.business-standard.com/world-news/google-cloud-ac...
- adamtaylor_13 5mo agoPerhaps you don't notice GCP outages because so few companies rely on them?
- fragmede 5mo agoGCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-google-cloud-for-icloud.html https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot https://cloud.google.com/customers/the-home-depot but most customers don't want to make a big deal about being on GCP as it's none of our business who's hosting them.
- shye 5mo agoApple also uses AWS, and I won't be surprised if they also use Azure. Big companies are multicloud, and not because it's a good idea (it rarely is), but because they inherited multiple environments on different CSPs, and maintaining those where they are is often cheaper than migrating them to a different CSP.
- Alive-in-2025 5mo agoI wonder if big companies can get a special contract with something like you can't delete my service automatically (unless it's an emergency)
- jedberg 5mo agoIf you're big enough you don't need a contract for that, that's just their default method of operation.
- GoblinSlayer 5mo ago
- manyatoms 5mo agoHow is blackhole-ing a customer not considered an outage?
- nemothekid 5mo ago>On the other hand i can’t remember when there was a serious outage on GCP They had a really bad global outage a year ago. At least with AWS outages are contained to a single region.
- onion2k 5mo agoYou can't have 100% uptime. It's unfeasible, especially for a startup. You should be telling your customers that downtime might happen, sometimes for reasons beyond your control, and that if it does then you'll do your best to recover and to compensate them for the inconvenience. You should cultivate a relationship with your early customers that makes them feel bad for you when there's an outage rather than angry about how it impacts them. Maybe even go as far as firing the customers who give you a hard time over it. That way if your cloud provider falls over it's really annoying but not a big deal. Your cloud provider blocking your business from running is far worse.
- mlhpdx 5mo agoNone of the AWS “outages” have impacted us. They have either been regional, in which case we stand down the region (we run multiple hot regions), or didn’t involve things we need to maintain operation. I can’t imagine AWS ever doing such a cascading delete. I mean, they have made deletion protection a difficult thing to ignore even for individual resources.