4 ms·

Google Cloud Console Incident

54 points by tonymet 8y ago

simook 8y ago
I'm interested in the root cause. Console operations and architecture for HA is one of my current responsibilities. It's important to me that customers can depend on a functional, responsive, and stable console. I think the console experience heavily influences the trust and overall customer experience of any service.
- WestCoastJustin 8y ago
  100% agree. I wanted to record some screencasts of the interface today and it is a little inconvenient. However, Google has an extremely robust post mortem process and I am sure they will correct things so this specific event does not happen again [1, 2]. They will typically post some type of public boiled down version of the post mortem too. Hopefully that happens in this case! [1] https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-... [2] https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/
  - sethvargo 8y ago
    Sorry for the interruption. I'm not sure how much information we'll be able to share publicly, but there will definitely be an internal PM once the incident is resolved.
    - WestCoastJustin 8y ago
      Yeah, I'm sure there is a healthy omg happening right now :)
lfx 8y ago
Also seems BitQuery having issues too. Wondering if this end up being related with Summer Time change? Happened many times in the past in companies where I worked.
- WestCoastJustin 8y ago
  You might be able to use the older interface still (has been working for me) [1]. I think they are testing a new beta interface that is right inside the console. I still like the older interface myself. [1] https://bigquery.cloud.google.com https://bigquery.cloud.google.com
sethvargo 8y ago
Hey everyone - Seth from Google here. We're aware of this issue and our teams are working to resolve it as quickly as possible. We apologize for any interruption.
- ct0 8y ago
  Thanks Seth!
- dejaime 8y ago
  Thanks for the work and for the update
gouggoug 8y ago
This is the second time in 2 weeks this happens. Last time it happened (Tue, Feb 26, around 12:41 PM), my Kubernetes cluster became unresponsive (and my website). I was never able to figure out if that was purely a coincidence or not, but I do not believe it was.

54 points by tonymet 8y ago

I'm interested in the root cause. Console operations and architecture for HA is one of my current responsibilities. It's important to me that customers can depend on a functional, responsive, and stable console. I think the console experience heavily influences the trust and overall customer experience of any service.

WestCoastJustin 8y ago
100% agree. I wanted to record some screencasts of the interface today and it is a little inconvenient. However, Google has an extremely robust post mortem process and I am sure they will correct things so this specific event does not happen again [1, 2]. They will typically post some type of public boiled down version of the post mortem too. Hopefully that happens in this case! [1] https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-... [2] https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/
- sethvargo 8y ago
  Sorry for the interruption. I'm not sure how much information we'll be able to share publicly, but there will definitely be an internal PM once the incident is resolved.
  - WestCoastJustin 8y ago
    Yeah, I'm sure there is a healthy omg happening right now :)

lfx 8y ago

Also seems BitQuery having issues too. Wondering if this end up being related with Summer Time change? Happened many times in the past in companies where I worked.

WestCoastJustin 8y ago
You might be able to use the older interface still (has been working for me) [1]. I think they are testing a new beta interface that is right inside the console. I still like the older interface myself. [1] https://bigquery.cloud.google.com https://bigquery.cloud.google.com

sethvargo 8y ago

Hey everyone - Seth from Google here. We're aware of this issue and our teams are working to resolve it as quickly as possible. We apologize for any interruption.

ct0 8y ago
Thanks Seth!
dejaime 8y ago
Thanks for the work and for the update

gouggoug 8y ago

This is the second time in 2 weeks this happens. Last time it happened (Tue, Feb 26, around 12:41 PM), my Kubernetes cluster became unresponsive (and my website). I was never able to figure out if that was purely a coincidence or not, but I do not believe it was.