6 ms·

Why is the Google Status dashboard slower than HackerNews at reporting the downtime? To get ahead of the cynics — it would not serve the least generous of Goog

Why is the Google Status dashboard slower than HackerNews at reporting the downtime? To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.

memset 6y ago
The time you mark the dashboard as “red” is when the clock starts ticking against your SLA and error budget.
- gowld 6y ago
  That makes no sense. The dashboard is for current best-effort knowledge. SLA and budget are for actual outages.
  - londons_explore 6y ago
    Not true. A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality. Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now". Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
manigandham 6y ago
Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first. Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.
- gowld 6y ago
  There's no reason for service credits to tied be to the dashboard and not to determinations made after the investigation is over.
  - manigandham 6y ago
    The SLA guarantees a certain level of no problems. The dashboard officially says there's a problem. This creates liability.
- a012 6y ago
  They should have pro-active monitoring system and update status page automatically when it detected disruption. Especially when you account to it's extremely hard to get human supports from Google because they automated it.
  - mook 6y ago
    I wouldn't be surprised if, at Google's scale, something or other is disrupted all the time. I think I'd prefer a status page that reflected that, or even no status page at all with an explanation, rather than a lying one, though.
  - hknd 6y ago
    Google has multiple internal alerting systems and people are aware of any disruption/delay/anomaly from the minute it happens. The on-caller for the specific service decides on how to produce. The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.
- AnotherTechie 6y ago
  when soemthing this broadly painful happens at a company of Google's scale --typically you have something like a hundred people internally that all got paged.
roomey 6y ago
Honestly, incident management is hard. It is a different skill set to straight up SRE management, and all the training in the world won't get you ready. Believe me it's true in AWS too, true in a lot of places. It's complicated, and the human factor is massive in an unpredicted scenario. I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it. One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same
x86_64Ubuntu 6y ago
They are just doing a dry run for when they finally close shop on Google Cloud...
hknd 6y ago
The status dashboard is something which will be updated by an operations employee manually (yes by hand), who is a couple of layers behind the actual SWE/SRE who gets the page.

memset 6y ago

The time you mark the dashboard as “red” is when the clock starts ticking against your SLA and error budget.

gowld 6y ago
That makes no sense. The dashboard is for current best-effort knowledge. SLA and budget are for actual outages.
- londons_explore 6y ago
  Not true. A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality. Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now". Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...

manigandham 6y ago

Properly investigating takes time. There are constant reports of something not working by customers so they need to confirm that it actually is internal and widespread first. Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.

gowld 6y ago
There's no reason for service credits to tied be to the dashboard and not to determinations made after the investigation is over.
- manigandham 6y ago
  The SLA guarantees a certain level of no problems. The dashboard officially says there's a problem. This creates liability.
a012 6y ago
They should have pro-active monitoring system and update status page automatically when it detected disruption. Especially when you account to it's extremely hard to get human supports from Google because they automated it.
- mook 6y ago
  I wouldn't be surprised if, at Google's scale, something or other is disrupted all the time. I think I'd prefer a status page that reflected that, or even no status page at all with an explanation, rather than a lying one, though.
- hknd 6y ago
  Google has multiple internal alerting systems and people are aware of any disruption/delay/anomaly from the minute it happens. The on-caller for the specific service decides on how to produce. The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.
AnotherTechie 6y ago
when soemthing this broadly painful happens at a company of Google's scale --typically you have something like a hundred people internally that all got paged.

roomey 6y ago

Honestly, incident management is hard. It is a different skill set to straight up SRE management, and all the training in the world won't get you ready. Believe me it's true in AWS too, true in a lot of places. It's complicated, and the human factor is massive in an unpredicted scenario. I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it. One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same

x86_64Ubuntu 6y ago

They are just doing a dry run for when they finally close shop on Google Cloud...

hknd 6y ago

The status dashboard is something which will be updated by an operations employee manually (yes by hand), who is a couple of layers behind the actual SWE/SRE who gets the page.