3 ms·

Did a quick write-up on this here: http://webdev360.com/users-report-major-amazon-ec2-outage-but-official-dashboard-stays-mute-41419.html http://webdev360.com/u

Did a quick write-up on this here: http://webdev360.com/users-report-major-amazon-ec2-outage-but-official-dashboard-stays-mute-41419.html http://webdev360.com/users-report-major-amazon-ec2-outage-bu... Does anyone know if ops teams generally have to clear these sort of announcements with PR/comms? I assume they would, given how they get reported on.

rdl 15y ago
At a competent company, no. There are policies in place before the outage, but having your PR people in the loop slows things down to the point where you're worthless to your customers. The exception is you loop in legal, PR, etc. if someone is actually injured/dies, or if crimes are involved. A lot of providers try to NDA their "ops to customer" service outage notifications, but most customers flagrantly violate those NDAs. Automated service dashboards are supposed to be automatic; ops teams often put in short statements (especially time to fix and any interim way to mitigate the outage). Definitive statements after the outage are run by PR (and generally announced senior to ops), but service notifications of outages (vs. causes, compensation, and long term corrective actions) are not.
- louisgoddard 15y ago
  Thanks -- that's what I assumed initially, but had second thoughts after seeing how delayed the announcements were.
  - rdl 15y ago
    I suspect there is some human admin level disconnect between their network ops/routing and the AWS team itself. A connectivity outage wouldn't necessarily get detected within AWS, and they probably don't have good monitoring within the AWS product to detect problems like that. The network team presumably doesn't have a good way to push status updates to the AWS dashboard automatically (and it's kind of a grey area what is a "network outage" -- if you lost routes to just Pakistan, that's not really a big deal for most AWS customers. If you lose routes to everyone, yes, that's a big deal.)
EwanToo 15y ago
Somewhat ironically, I can't get to that site.
- louisgoddard 15y ago
  I think our servers were down for a few minutes earlier -- should work now.

rdl 15y ago

At a competent company, no. There are policies in place before the outage, but having your PR people in the loop slows things down to the point where you're worthless to your customers. The exception is you loop in legal, PR, etc. if someone is actually injured/dies, or if crimes are involved. A lot of providers try to NDA their "ops to customer" service outage notifications, but most customers flagrantly violate those NDAs. Automated service dashboards are supposed to be automatic; ops teams often put in short statements (especially time to fix and any interim way to mitigate the outage). Definitive statements after the outage are run by PR (and generally announced senior to ops), but service notifications of outages (vs. causes, compensation, and long term corrective actions) are not.

louisgoddard 15y ago
Thanks -- that's what I assumed initially, but had second thoughts after seeing how delayed the announcements were.
- rdl 15y ago
  I suspect there is some human admin level disconnect between their network ops/routing and the AWS team itself. A connectivity outage wouldn't necessarily get detected within AWS, and they probably don't have good monitoring within the AWS product to detect problems like that. The network team presumably doesn't have a good way to push status updates to the AWS dashboard automatically (and it's kind of a grey area what is a "network outage" -- if you lost routes to just Pakistan, that's not really a big deal for most AWS customers. If you lose routes to everyone, yes, that's a big deal.)

EwanToo 15y ago

Somewhat ironically, I can't get to that site.

louisgoddard 15y ago
I think our servers were down for a few minutes earlier -- should work now.