4 ms·

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of hea

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

notatoad 7y ago
seriously, they've got a text field on the official status page, why not put the text boulos posted here in that instead of the meaningless text they've got there?
- boulos 7y ago
  Can you expand on why you find it “meaningless”? As my other comment says, I’m not in SRE and the real people fixing it are trying their best to remediate the problem. I agree that the text I posted (with blessing from SRE!) gives you some more detail, but you can’t do anything differently with it, right? What about the new text do you prefer? (We’re happy to improve!)
  - toufka 7y ago
    Your, even brief, description is interpretable by your clients and some customers - and is actually really informative. It helps estimate the magnitude of the issue, and the types of downstream problems to expect or avoid. Knowing an astroid took out the entire continent tells you something about the repairability, resources required to fix the problem, and generally provides context for later updates, as opposed to other contexts like a cut fiber line, a burning datacenter or a bad power supply.
  - notatoad 7y ago
    I think the difference between your comment here and the info on the status page is that after reading your comment i feel like i know what's happening. you're right, there's no additional actionable information there, the status page contains everything i actually need to know. but a bit more information makes me feel better. I guess the difference is your comment reassures me that you actually know what's going on. the status page text (prior to the 14:31 update) could equally mean "we've got this under control" or "shit's broken and we don't know why"
  - emerongi 7y ago
    "Can we meet up on Friday?" "No" vs "No, I already have plans with X" First case gives you all the information needed (denial), however in the second case I understand the situation much better. I wouldn't call the text on the status page meaningless though - it's pretty nice and concise already (which is what you want in a "crisis"). Just some brief description of the problem would be good, even though technically unnecessary.
- david-cako 7y ago
  I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.
  - sudosteph 7y ago
    True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
    - ti_ranger 7y ago
      > Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. It has. > Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates? The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard. PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
    - KirinDave 7y ago
boulos 7y ago
The outage information is fairly reasonable. Not everyone cares (nor should they!) about the why only what the situation is, and that people are on it. This is extra detail. I mostly responded because there was confusion downthread (and in the title) about being “down”. During an outage is a tricky time for comms, so short corrections are best until a full postmortem can be done.
NikolaeVarius 7y ago
You seem to have forgotten twitter
- benburleson 7y ago
  We can dream.
deleted 7y ago
[deleted]
hajhatten 7y ago
This reminds me of an incident in Sweden a couple of years ago. We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule. This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacation probably. The websites and apps they've setup were ridiculously underpowered and were basically DDOS'ed by the spike in traffic they were getting. News outlets were also struggling, but did way better. The only thing that withstood the sudden burst in traffic without a hitch was facebook and twitter. The official statement i think was that the alarm was triggered by accident (never happened before, i think). But goes to show how badly our emergency response is setup.
- issati 7y ago
  It goes to show how badly it is set up for a false alarm. In a real emergency all the primary functions would go up (taking over radio broadcasts for example) so there wouldn't be the same problem. It is still bad of course because of the "cry wolf" factor.
  - archy_ 7y ago
    I think a similar situation happened in Hawaii last year and it took awhile to send a false alert message.

notatoad 7y ago

seriously, they've got a text field on the official status page, why not put the text boulos posted here in that instead of the meaningless text they've got there?

boulos 7y ago
Can you expand on why you find it “meaningless”? As my other comment says, I’m not in SRE and the real people fixing it are trying their best to remediate the problem. I agree that the text I posted (with blessing from SRE!) gives you some more detail, but you can’t do anything differently with it, right? What about the new text do you prefer? (We’re happy to improve!)
- toufka 7y ago
  Your, even brief, description is interpretable by your clients and some customers - and is actually really informative. It helps estimate the magnitude of the issue, and the types of downstream problems to expect or avoid. Knowing an astroid took out the entire continent tells you something about the repairability, resources required to fix the problem, and generally provides context for later updates, as opposed to other contexts like a cut fiber line, a burning datacenter or a bad power supply.
- notatoad 7y ago
  I think the difference between your comment here and the info on the status page is that after reading your comment i feel like i know what's happening. you're right, there's no additional actionable information there, the status page contains everything i actually need to know. but a bit more information makes me feel better. I guess the difference is your comment reassures me that you actually know what's going on. the status page text (prior to the 14:31 update) could equally mean "we've got this under control" or "shit's broken and we don't know why"
- emerongi 7y ago
  "Can we meet up on Friday?" "No" vs "No, I already have plans with X" First case gives you all the information needed (denial), however in the second case I understand the situation much better. I wouldn't call the text on the status page meaningless though - it's pretty nice and concise already (which is what you want in a "crisis"). Just some brief description of the problem would be good, even though technically unnecessary.
david-cako 7y ago
I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.
- sudosteph 7y ago
  True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
  - ti_ranger 7y ago
    > Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. It has. > Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates? The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard. PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
  - KirinDave 7y ago

boulos 7y ago

The outage information is fairly reasonable. Not everyone cares (nor should they!) about the why only what the situation is, and that people are on it. This is extra detail. I mostly responded because there was confusion downthread (and in the title) about being “down”. During an outage is a tricky time for comms, so short corrections are best until a full postmortem can be done.

NikolaeVarius 7y ago

You seem to have forgotten twitter

benburleson 7y ago
We can dream.

deleted 7y ago

[deleted]

hajhatten 7y ago

This reminds me of an incident in Sweden a couple of years ago. We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule. This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacation probably. The websites and apps they've setup were ridiculously underpowered and were basically DDOS'ed by the spike in traffic they were getting. News outlets were also struggling, but did way better. The only thing that withstood the sudden burst in traffic without a hitch was facebook and twitter. The official statement i think was that the alarm was triggered by accident (never happened before, i think). But goes to show how badly our emergency response is setup.

issati 7y ago
It goes to show how badly it is set up for a false alarm. In a real emergency all the primary functions would go up (taking over radio broadcasts for example) so there wouldn't be the same problem. It is still bad of course because of the "cry wolf" factor.
- archy_ 7y ago
  I think a similar situation happened in Hawaii last year and it took awhile to send a false alert message.