3 ms·

Our app is down. I can't even access any pages in Google Cloud Console. Timing out. Sometimes it completes showing all our clusters gone, then a timeout error.

Our app is down. I can't even access any pages in Google Cloud Console. Timing out. Sometimes it completes showing all our clusters gone, then a timeout error. This is brutal. This isn't just GKE either... Edit: Things are working for us now Edit: Still getting timeouts and service unavailable Edit: I'm getting 503 (service unavailable) from buckets, but nothing on the status page indicating there's any issue. Edit: Seems our Cloud SQL instance was restarted as well Edit: Multiple restarts of our production database Edit: Dashboard finally updated to reflect growing # of services effected Edit: This wasn't an "App Engine" incident. It was a very wide-ranging incident. Just change the title to "Google Cloud Incident" and be done with it Edit: Things have seemed to stabilize for us Was supposed to have today off with my family (Remembrance Day in Canada), but now I have to deal with support issues all day. Thanks Google!

sethvargo 7y ago
I’m sorry this incident is taking time away from your family. Our team is working to mitigate as quickly as possible. Initially the scope of the issue was unclear, but now the title and dashboard are updated, sorry about that.
mritchie712 7y ago
I have been fucking with this all morning thinking it was something we did (we actually were changing some permissions last night / today), how about an email? There was no push of this info, I shouldn't have to opt in to that.
- numbsafari 7y ago
  (Google Customer, not Employee) We have two slack channels here. One is where our internal monitoring agents post, as well as an RSS subscription to the GCP status board. We have a separate channel for less critical things (like GitHub). Unfortunately, it does depend on GCP updating their dashboard. However, to date, we’ve only been impacted by one of these major outages. This morning, I saw the alert in our channel, but all our things were still operating (fortunately!). YMMV, but I’ve found this very helpful.
  - sethvargo 7y ago
    FWIW, we're exploring ways to make this better. We know it's a point of pain for customers. One of the challenges is that the "project owner" isn't always the right person to receive these kinds of alerts, since often that's someone in the finance department or central IT team. There are a few ideas being thrown around about more proactive notification right now.
    - pg_is_a_butt 7y ago
      Allow the project owner to designate a User or email address or endpoint to receive "pre-alert non-binding internal potential-warning notifications" or whatever the google lawyers will let you call it. Someone will surely argue that it will let evil-doers know if their attacks are on the radar... hopefully someone else will argue that at least some of your customers pay with clean money and would appreciate the customer service.

sethvargo 7y ago

I’m sorry this incident is taking time away from your family. Our team is working to mitigate as quickly as possible. Initially the scope of the issue was unclear, but now the title and dashboard are updated, sorry about that.

mritchie712 7y ago

I have been fucking with this all morning thinking it was something we did (we actually were changing some permissions last night / today), how about an email? There was no push of this info, I shouldn't have to opt in to that.

numbsafari 7y ago
(Google Customer, not Employee) We have two slack channels here. One is where our internal monitoring agents post, as well as an RSS subscription to the GCP status board. We have a separate channel for less critical things (like GitHub). Unfortunately, it does depend on GCP updating their dashboard. However, to date, we’ve only been impacted by one of these major outages. This morning, I saw the alert in our channel, but all our things were still operating (fortunately!). YMMV, but I’ve found this very helpful.
- sethvargo 7y ago
  FWIW, we're exploring ways to make this better. We know it's a point of pain for customers. One of the challenges is that the "project owner" isn't always the right person to receive these kinds of alerts, since often that's someone in the finance department or central IT team. There are a few ideas being thrown around about more proactive notification right now.
  - pg_is_a_butt 7y ago
    Allow the project owner to designate a User or email address or endpoint to receive "pre-alert non-binding internal potential-warning notifications" or whatever the google lawyers will let you call it. Someone will surely argue that it will let evil-doers know if their attacks are on the radar... hopefully someone else will argue that at least some of your customers pay with clean money and would appreciate the customer service.