9 ms·

Google Cloud Partial Outage

90 points by sz4kerto 7y ago

F117-DK 7y ago
Oh my! GCP is on fire?
pritambarhate 7y ago
https://status.cloud.google.com/incident/compute https://status.cloud.google.com/incident/compute It just looks horrific from the reliability standpoint. May be Google should consider adding columns for affected regions! Maybe they are too honest in admitting the failures than other providers, but that page paints a very bad picture.
- breakingcups 7y ago
  I'm glad they do. Compared to AWS this transparancy is a breath of fresh air. And I'm no Google fan either.
tlackemann 7y ago
I like how I found out about this here and not through an email. Tons of customer reports from our site of being logged out randomly (we use Firebase) and we had no idea why.
- deleted 7y ago
  [deleted]
- tlackemann 7y ago
  The outage is going on for almost 12-hours. Quick math says that's 99.86% reliability. On top of this, they want to start charging $70/mo for GKE because they "[...] are also introducing a Service Level Agreement (SLA) that's financially backed with a guaranteed availability of 99.95%" (pulled from pricing[0]) This is unacceptable. [0] https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
  - BilalBudhani 7y ago
    I'm guessing because we are going through a pandemic they might be taking more time than usual to apply fixes.
    - pm90 7y ago
      They’ve been pretty vocal about the fact that this has nothing to do with the pandemic.
      - quicklime 7y ago
        The outage itself might not have been caused by anything related to the pandemic, but the resolution is likely being done by engineers adjusting to working from home.
  - 9nGQluzmnq3M 7y ago
    The current/remaining issue is "Affected customers may experience delayed IAM modifications that surface across multiple Google Cloud Platform services." In simple terms, user login/auth is working fine, it's just modifications (new service accounts, access changes, etc) that are delayed. Obviously not great, but this doesn't really affect the critical path of any systems that are already up and running.
amelius 7y ago
Even Google needs humans to keep its services running.
- naringas 7y ago
  I'll bet it was also humans who caused this outage in the first place.
breakingcups 7y ago
Already looking forward to that post-mortem. It might be morbid, but they are often my favorite things to read.
rainboiboi 7y ago
I guess the number of comments here reflect how many people are using Google Cloud - apparently not much?
- dijit 7y ago
  There's nothing really to say about it. I run about 8,000 CPU cores in GCP and what am I going to say? I like the transparency, in my experience it's more reliable than AWS- although AWS does _not_ report problems until they're absolutely undeniable. Other than that, shit happens, this is upsetting but eh.
higooglebye 7y ago
Google Cloud was the least reliable major cloud provider in the last 2 years. May be they are too busy killing their other services, so they don't have time for other least important things like cloud.
- Kye 7y ago
  Is this based on independent research, or officially-reported downtime? There are enough anecdotes in threads here about cloud reliability to suggest AWS is bad at reporting outages.
pm90 7y ago
From a strictly “reliability” perspective it’s not a good look for Google. Their reputation for developing high uptime systems has been endangered by the recent series of outages and service interruptions. When I signed on to GCP, I assumed that the same people who wrote the SRE handbook were in charge here... how could this stuff happen? However, as an engineer... I can empathize with their situation. GCP has less public cloud experience than AWS. They’re somewhat transparent about outages (so much better than being gaslit by AWS). Shit happens and at least the same problem doesn’t happen twice, which is what I really care about. However my non technical manager won’t think this way and neither will other companies considering Google Cloud. GCP is in a tough spot from a marketing perspective if they get a reputation for being unreliable due to these incidents.
- izacus 7y ago
  Is GCP actually less reliable than AWS or Azure, or is that only HN effect where everything bad about Google gets voted to the top? I'm hearing a lot of talk from my contacts about AWS just simply denying and hiding any outages until they're undeniably proven - so perhaps the issue is just the fact that GCP is too honest?
  - buttersbrian 7y ago
    Curious about this as well. Google hate here can be palpable
    - deleted 7y ago
      [deleted]
    - zepto 7y ago
      Wind back 5 years or more and this place was a Google love fest. Somehow it seems that Google has managed to disappoint a lot of people.
      - lostmsu 7y ago
        I am afraid you would need to rewind further than that.
angristan 7y ago
This is a disaster... > We are currently investigating an issue affecting Dataflow, BigQuery, DialogFlow, Kubernetes Engine, Cloud Firestore, App Engine, Cloud Functions, Cloud Monitoring, Cloud MemoryStore, Cloud Spanner, Cloud Storage, Cloud Composer, Cloud Dataproc, Cloud KMS, Cloud Container Registry, Compute Engine, Cloud IAM, Cloud SQL, Firebase Storage, Cloud Healthcare API, Cloud AI, Firebase Machine Learning, Data Catalog and Cloud Console.
- lonelappde 7y ago
  When you see that, it's almost certainly a low level networking outage (networking hardware failure or a very low level networking config change that isn't possible to canary with the vendor's current technology, possibly combined with something insufficient failover resources due to other issues).
FigmentEngine 7y ago
The bad part here is that its impacting multiple regions. really hard for customers to build high-availability solutions when isolation boundaries can be breached this way. (disclosure ex-aws)
brianwawok 7y ago
What region is this? I have not seen a blip in central.
Jyaif 7y ago
Apparently the outage is caused by a third party router bug: https://twitter.com/uhoelzle/status/1243398255083311105 https://twitter.com/uhoelzle/status/1243398255083311105
jiveturkey 7y ago
resolved as of 2020-03-27 06:54 US/Pacific
- a012 7y ago
  What UTC time is it?

90 points by sz4kerto 7y ago

Oh my! GCP is on fire?

https://status.cloud.google.com/incident/compute https://status.cloud.google.com/incident/compute It just looks horrific from the reliability standpoint. May be Google should consider adding columns for affected regions! Maybe they are too honest in admitting the failures than other providers, but that page paints a very bad picture.

breakingcups 7y ago
I'm glad they do. Compared to AWS this transparancy is a breath of fresh air. And I'm no Google fan either.

tlackemann 7y ago

I like how I found out about this here and not through an email. Tons of customer reports from our site of being logged out randomly (we use Firebase) and we had no idea why.

deleted 7y ago
[deleted]
tlackemann 7y ago
The outage is going on for almost 12-hours. Quick math says that's 99.86% reliability. On top of this, they want to start charging $70/mo for GKE because they "[...] are also introducing a Service Level Agreement (SLA) that's financially backed with a guaranteed availability of 99.95%" (pulled from pricing[0]) This is unacceptable. [0] https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing
- BilalBudhani 7y ago
  I'm guessing because we are going through a pandemic they might be taking more time than usual to apply fixes.
  - pm90 7y ago
    They’ve been pretty vocal about the fact that this has nothing to do with the pandemic.
    - quicklime 7y ago
      The outage itself might not have been caused by anything related to the pandemic, but the resolution is likely being done by engineers adjusting to working from home.
- 9nGQluzmnq3M 7y ago
  The current/remaining issue is "Affected customers may experience delayed IAM modifications that surface across multiple Google Cloud Platform services." In simple terms, user login/auth is working fine, it's just modifications (new service accounts, access changes, etc) that are delayed. Obviously not great, but this doesn't really affect the critical path of any systems that are already up and running.

amelius 7y ago

Even Google needs humans to keep its services running.

naringas 7y ago
I'll bet it was also humans who caused this outage in the first place.

breakingcups 7y ago

Already looking forward to that post-mortem. It might be morbid, but they are often my favorite things to read.

rainboiboi 7y ago

I guess the number of comments here reflect how many people are using Google Cloud - apparently not much?

dijit 7y ago
There's nothing really to say about it. I run about 8,000 CPU cores in GCP and what am I going to say? I like the transparency, in my experience it's more reliable than AWS- although AWS does _not_ report problems until they're absolutely undeniable. Other than that, shit happens, this is upsetting but eh.

higooglebye 7y ago

Google Cloud was the least reliable major cloud provider in the last 2 years. May be they are too busy killing their other services, so they don't have time for other least important things like cloud.

Kye 7y ago
Is this based on independent research, or officially-reported downtime? There are enough anecdotes in threads here about cloud reliability to suggest AWS is bad at reporting outages.

pm90 7y ago

From a strictly “reliability” perspective it’s not a good look for Google. Their reputation for developing high uptime systems has been endangered by the recent series of outages and service interruptions. When I signed on to GCP, I assumed that the same people who wrote the SRE handbook were in charge here... how could this stuff happen? However, as an engineer... I can empathize with their situation. GCP has less public cloud experience than AWS. They’re somewhat transparent about outages (so much better than being gaslit by AWS). Shit happens and at least the same problem doesn’t happen twice, which is what I really care about. However my non technical manager won’t think this way and neither will other companies considering Google Cloud. GCP is in a tough spot from a marketing perspective if they get a reputation for being unreliable due to these incidents.

izacus 7y ago
Is GCP actually less reliable than AWS or Azure, or is that only HN effect where everything bad about Google gets voted to the top? I'm hearing a lot of talk from my contacts about AWS just simply denying and hiding any outages until they're undeniably proven - so perhaps the issue is just the fact that GCP is too honest?
- buttersbrian 7y ago
  Curious about this as well. Google hate here can be palpable
  - deleted 7y ago
    [deleted]
  - zepto 7y ago
    Wind back 5 years or more and this place was a Google love fest. Somehow it seems that Google has managed to disappoint a lot of people.
    - lostmsu 7y ago
      I am afraid you would need to rewind further than that.

angristan 7y ago

This is a disaster... > We are currently investigating an issue affecting Dataflow, BigQuery, DialogFlow, Kubernetes Engine, Cloud Firestore, App Engine, Cloud Functions, Cloud Monitoring, Cloud MemoryStore, Cloud Spanner, Cloud Storage, Cloud Composer, Cloud Dataproc, Cloud KMS, Cloud Container Registry, Compute Engine, Cloud IAM, Cloud SQL, Firebase Storage, Cloud Healthcare API, Cloud AI, Firebase Machine Learning, Data Catalog and Cloud Console.

lonelappde 7y ago
When you see that, it's almost certainly a low level networking outage (networking hardware failure or a very low level networking config change that isn't possible to canary with the vendor's current technology, possibly combined with something insufficient failover resources due to other issues).

FigmentEngine 7y ago

The bad part here is that its impacting multiple regions. really hard for customers to build high-availability solutions when isolation boundaries can be breached this way. (disclosure ex-aws)

brianwawok 7y ago

What region is this? I have not seen a blip in central.

Jyaif 7y ago

Apparently the outage is caused by a third party router bug: https://twitter.com/uhoelzle/status/1243398255083311105 https://twitter.com/uhoelzle/status/1243398255083311105

jiveturkey 7y ago

resolved as of 2020-03-27 06:54 US/Pacific

a012 7y ago
What UTC time is it?