4 ms·

Reporting status externally is hard. I worked at a cloud provider and while we had very detailed metrics about how our systems were functioning, it was difficu

by efitz 3y ago

Reporting status externally is hard. I worked at a cloud provider and while we had very detailed metrics about how our systems were functioning, it was difficult to distill that information in a way that customers would understand if they were being impacted or not, and what the impact would be. Just reporting raw numbers wouldn’t give customers the context to understand what was actually going on. How do you actually report, for example, that .003% of your customers are having a really bad day but the rest are just fine?

12345hn6789 3y ago
Aren't these exactly what p99 illustrates? Of course you'd have to have your use cases logged well enough to be able to aggregate the 99th percentile accurately to each customer flow
mistrial9 3y ago
because you define thresholds for event classification.. the difference between 10024 customers having failure and 10100 customers having failure is not the question, right? when many hundreds of thousands of customers are failing at once, is that really very difficult to determine? secondly, there are financial, management and information security pressures to NOT REPORT reality to the public. This happens VERY OFTEN in real business. In fact, that is why legal enforcement actions and real consequences are crucial versus Big Business.
- efitz 3y ago
  In a cloud, massive outages are the rare events, and technically easy to report. Small outages happen all the time, and are difficult to report accurately. I think AWS has pivoted to trying to report status in each individual customer's support portal, so that they can give a dashboard that reports the state of the cloud from that customer's perspective. That way a rack down that only affects a few customers is only reported to those customers, and the dashboard doesn't have to always be red for everyone (or green for everyone, even those affected).
bradleyjg 3y ago
If it can’t be done then don’t do it. A dot that’s always green, no matter what, is worse than nothing.
- RajT88 3y ago
  Agree. As a customer, you need to be able to correlate your impact to anything which might be going on. Because those cloud provider internal tools are often wrong about who's impacted and who's not.

by efitz 3y ago

12345hn6789 3y ago

Aren't these exactly what p99 illustrates? Of course you'd have to have your use cases logged well enough to be able to aggregate the 99th percentile accurately to each customer flow

mistrial9 3y ago

because you define thresholds for event classification.. the difference between 10024 customers having failure and 10100 customers having failure is not the question, right? when many hundreds of thousands of customers are failing at once, is that really very difficult to determine? secondly, there are financial, management and information security pressures to NOT REPORT reality to the public. This happens VERY OFTEN in real business. In fact, that is why legal enforcement actions and real consequences are crucial versus Big Business.

efitz 3y ago
In a cloud, massive outages are the rare events, and technically easy to report. Small outages happen all the time, and are difficult to report accurately. I think AWS has pivoted to trying to report status in each individual customer's support portal, so that they can give a dashboard that reports the state of the cloud from that customer's perspective. That way a rack down that only affects a few customers is only reported to those customers, and the dashboard doesn't have to always be red for everyone (or green for everyone, even those affected).

bradleyjg 3y ago

If it can’t be done then don’t do it. A dot that’s always green, no matter what, is worse than nothing.

RajT88 3y ago
Agree. As a customer, you need to be able to correlate your impact to anything which might be going on. Because those cloud provider internal tools are often wrong about who's impacted and who's not.