4 ms·

Of course both dashboards show all green: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status https://status.clo

Of course both dashboards show all green: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status https://status.cloud.google.com/ https://status.cloud.google.com/

ncr100 6y ago
I am experiencing intermittent issues with GMail not loading new conversations in a timely fashion, and showing "something went wrong" notification messages.
mmaunder 6y ago
There are definitely issues. GDocs functionality is intermittent. e.g. the ability to move a doc into a new folder or even see what folder it's in using the little folder icon at the top while editing has disappeared. I've suggested to our team they copy/paste locally anything they're currently editing.
cgb223 6y ago
What good is a dashboard if it doesn’t show actual status
- reustle 6y ago
  Are there any large community run status dashboards? Seems like it is something that we can't trust companies to be honest about (due to SLAs).
  - booleandilemma 6y ago
    Just check Hacker News, of course :)
  - gruez 6y ago
    https://downdetector.ca/ https://downdetector.ca/
- Areading314 6y ago
  Most of the time, it does!
  - ooobit2 6y ago
    This is why I can't run a company. "Sometimes, it's probably almost approximately around what we think could be along the spectrum of known values." - Me, as CEO.
    - judge2020 6y ago
      "Our business is not a coherent single entity and we don't have the best operational policies that try to remedy that so I won't make any promises about quality while still sounding like we're worth $1b"
      - ooobit2 6y ago
        "I apologize for having misspoke earlier. The correct quote is: '...while still sounding like we're maybe worth $1b.' We could be worth more than the disclosed amount; however, we aren't sure if or how often, if ever, someone may be counting that number. It's really up in the air on occasion at around this point." - Me, still CEO, now an award-winning CEO.
- deleted 6y ago
  [deleted]
- remus 6y ago
  What does 'actual status' mean though? If I'm running a website from a single server, the power goes out and the server goes down then it's kinda obvious. I don't think it's so obvious when you have hundreds of data centers serving traffic to billions of people. If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service. You need to draw a line in the sand somewhere, but whatever measure you choose is going to be somewhat arbitrary so I think it's reasonable to have a human make the final call (based on some known criteria).
  - onedognight 6y ago
    > If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service. Then put that on the dashboard! It certainly is an outage for the affected users (by your scenario). - 999/1000 Data centers up - 100,000 Users without service - 6 paying customers without service (will get money back)
    - tweenagedream 6y ago
      Except, sometimes a data center goes down and no one notices. So if a data center goes down, and no user around notices, is it really down?
kempbellt 6y ago
Just guessing here, but I imagine that updates for service-wide statuses which are used on the scale that Google operates require manual vetting. It is frustrating that these pages are not updated in real-time, but I do understand wanting to be sure before publishing a message to your entire userbase that you are experiencing disruptions.
- notyourday 6y ago
  > Just guessing here, but I imagine that updates for service-wide statuses which are used on the scale that Google operates require manual vetting. Gee, you mean we just discovered that area where Google admits that either automation or machine learning does not actually work?!
  - MacsHeadroom 6y ago
    It's usually less to do with whether automation works and more to do with having an internal party to hold responsible, when it comes to these things. Google doesn't need an internal party to hold responsible for the concerns of free mail users or their own contractors who are called "content creators." In part because Gmail users and YouTubers don't lead fortune 50 companies and play golf with Google Execs like Google Cloud customers do.
agleason 6y ago
I'm betting it's that something gnarly got checked in rather than service outage
- tyingq 6y ago
  I'll second that with something more specific. A bad update to their software defined routers or switches.
  - tyingq 6y ago
    "A pool of servers that route traffic to application backends crashed" https://mobile.twitter.com/uhoelzle/status/1309313556995661825?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1309313556995661825%7Ctwgr%5Eshare_3&ref_url=https%3A%2F%2Fd-14256395341513389610.ampproject.net%2F2009112034002%2Fframe.html https://mobile.twitter.com/uhoelzle/status/13093135569956618... So, "kind of" confirmed, I think. Assuming a bad update is the most likely culprit for a whole pool of servers to all crash at the same time.
- Thorrez 6y ago
  Can't something gnarly getting checked in cause a service outage?
briffle 6y ago
They updated a few min after you posted: https://status.cloud.google.com/incident/zall/20010 https://status.cloud.google.com/incident/zall/20010
- profmonocle 6y ago
  One of our GKE clusters has been stuck "upgrading" for 12 hours. Totally stalled, no way to cancel, and auto-scaling is non-functional. Support seems pretty stumped by the issue and they haven't updated us since about an hour after I reported, so I wonder if this is part of a bigger issue?
  - judge2020 6y ago
    Doubt it, you're probably just unlucky and have a cursed instance that is misconfigured in a very subtle way. Anecdote: I had an AWS instance that was stuck 'terminating' for 10 days in 2017 and finally got killed - maybe the hypervisor finally timed out or a SRE saw the anomaly.
maximilianroos 6y ago
Why is the Google Status dashboard slower than HackerNews at reporting the downtime? To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.
- memset 6y ago
  The time you mark the dashboard as “red” is when the clock starts ticking against your SLA and error budget.
  - gowld 6y ago
    That makes no sense. The dashboard is for current best-effort knowledge. SLA and budget are for actual outages.
    - londons_explore 6y ago
      Not true. A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality. Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now". Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
- manigandham 6y ago
  Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first. Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.

ncr100 6y ago

I am experiencing intermittent issues with GMail not loading new conversations in a timely fashion, and showing "something went wrong" notification messages.

mmaunder 6y ago

There are definitely issues. GDocs functionality is intermittent. e.g. the ability to move a doc into a new folder or even see what folder it's in using the little folder icon at the top while editing has disappeared. I've suggested to our team they copy/paste locally anything they're currently editing.

cgb223 6y ago

What good is a dashboard if it doesn’t show actual status

reustle 6y ago
Are there any large community run status dashboards? Seems like it is something that we can't trust companies to be honest about (due to SLAs).
- booleandilemma 6y ago
  Just check Hacker News, of course :)
- gruez 6y ago
  https://downdetector.ca/ https://downdetector.ca/
Areading314 6y ago
Most of the time, it does!
- ooobit2 6y ago
  This is why I can't run a company. "Sometimes, it's probably almost approximately around what we think could be along the spectrum of known values." - Me, as CEO.
  - judge2020 6y ago
    "Our business is not a coherent single entity and we don't have the best operational policies that try to remedy that so I won't make any promises about quality while still sounding like we're worth $1b"
    - ooobit2 6y ago
      "I apologize for having misspoke earlier. The correct quote is: '...while still sounding like we're maybe worth $1b.' We could be worth more than the disclosed amount; however, we aren't sure if or how often, if ever, someone may be counting that number. It's really up in the air on occasion at around this point." - Me, still CEO, now an award-winning CEO.
deleted 6y ago
[deleted]
remus 6y ago
What does 'actual status' mean though? If I'm running a website from a single server, the power goes out and the server goes down then it's kinda obvious. I don't think it's so obvious when you have hundreds of data centers serving traffic to billions of people. If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service. You need to draw a line in the sand somewhere, but whatever measure you choose is going to be somewhat arbitrary so I think it's reasonable to have a human make the final call (based on some known criteria).
- onedognight 6y ago
  > If one DC becomes inaccessible does that count as an outage? It'll sure feel like it to the affected users, but the vast majority can still use your service. Then put that on the dashboard! It certainly is an outage for the affected users (by your scenario). - 999/1000 Data centers up - 100,000 Users without service - 6 paying customers without service (will get money back)
  - tweenagedream 6y ago
    Except, sometimes a data center goes down and no one notices. So if a data center goes down, and no user around notices, is it really down?

kempbellt 6y ago

Just guessing here, but I imagine that updates for service-wide statuses which are used on the scale that Google operates require manual vetting. It is frustrating that these pages are not updated in real-time, but I do understand wanting to be sure before publishing a message to your entire userbase that you are experiencing disruptions.

notyourday 6y ago
> Just guessing here, but I imagine that updates for service-wide statuses which are used on the scale that Google operates require manual vetting. Gee, you mean we just discovered that area where Google admits that either automation or machine learning does not actually work?!
- MacsHeadroom 6y ago
  It's usually less to do with whether automation works and more to do with having an internal party to hold responsible, when it comes to these things. Google doesn't need an internal party to hold responsible for the concerns of free mail users or their own contractors who are called "content creators." In part because Gmail users and YouTubers don't lead fortune 50 companies and play golf with Google Execs like Google Cloud customers do.

agleason 6y ago

I'm betting it's that something gnarly got checked in rather than service outage

tyingq 6y ago
I'll second that with something more specific. A bad update to their software defined routers or switches.
- tyingq 6y ago
  "A pool of servers that route traffic to application backends crashed" https://mobile.twitter.com/uhoelzle/status/1309313556995661825?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1309313556995661825%7Ctwgr%5Eshare_3&ref_url=https%3A%2F%2Fd-14256395341513389610.ampproject.net%2F2009112034002%2Fframe.html https://mobile.twitter.com/uhoelzle/status/13093135569956618... So, "kind of" confirmed, I think. Assuming a bad update is the most likely culprit for a whole pool of servers to all crash at the same time.
Thorrez 6y ago
Can't something gnarly getting checked in cause a service outage?

briffle 6y ago

They updated a few min after you posted: https://status.cloud.google.com/incident/zall/20010 https://status.cloud.google.com/incident/zall/20010

profmonocle 6y ago
One of our GKE clusters has been stuck "upgrading" for 12 hours. Totally stalled, no way to cancel, and auto-scaling is non-functional. Support seems pretty stumped by the issue and they haven't updated us since about an hour after I reported, so I wonder if this is part of a bigger issue?
- judge2020 6y ago
  Doubt it, you're probably just unlucky and have a cursed instance that is misconfigured in a very subtle way. Anecdote: I had an AWS instance that was stuck 'terminating' for 10 days in 2017 and finally got killed - maybe the hypervisor finally timed out or a SRE saw the anomaly.

maximilianroos 6y ago

Why is the Google Status dashboard slower than HackerNews at reporting the downtime? To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.

memset 6y ago
The time you mark the dashboard as “red” is when the clock starts ticking against your SLA and error budget.
- gowld 6y ago
  That makes no sense. The dashboard is for current best-effort knowledge. SLA and budget are for actual outages.
  - londons_explore 6y ago
    Not true. A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality. Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now". Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
manigandham 6y ago
Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first. Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.