3 ms·

I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, whic

I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.

sudosteph 7y ago
True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
- ti_ranger 7y ago
  > Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. It has. > Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates? The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard. PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
- KirinDave 7y ago
  I'm late to this party but I just wanna add, boulos could be wrong or inaccurate and it wouldn't be a big deal. Those status updates are communications to customers, and thus tend to be more conservative. Inaccuracy is a much bigger deal there than the informal status here. Sadly, the closer you are to the action of a thing like this (for example, I'm on NetInfra SRE and we were part of the group that put in place the current mitigations you're seeing work now), the less you can say without fear of subtle inaccuracy or releasing non-public information.

sudosteph 7y ago

True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?

ti_ranger 7y ago
> Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. It has. > Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates? The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard. PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
KirinDave 7y ago
I'm late to this party but I just wanna add, boulos could be wrong or inaccurate and it wouldn't be a big deal. Those status updates are communications to customers, and thus tend to be more conservative. Inaccuracy is a much bigger deal there than the informal status here. Sadly, the closer you are to the action of a thing like this (for example, I'm on NetInfra SRE and we were part of the group that put in place the current mitigations you're seeing work now), the less you can say without fear of subtle inaccuracy or releasing non-public information.