3 ms·
In my experience I have encountered 3 different types of outages: 1. Multiple availability zones in a single region are experiencing reduced performance/outag
by cjhanks 10y ago
In my experience I have encountered 3 different types of outages:
1. Multiple availability zones in a single region are experiencing reduced performance/outage.
2. A single availability zone is experience reduced performance/outage.
3. A single service (bound to your account) is having reduced performance.
The dashboard displays item 1, and very rarely item 2. It never displays item 3, and that's the one most people actually care about.
Many AWS services appear to be siloed into user accounts, such as S3 or DynamoDB. Sometimes the systems comprising your storage might be severely degraded (due to poor access patterns or simply random statistic degradation). In such cases users have absolutely no way of being notified about degraded performance unless they have built metrics to monitor their services.
For example; I once wrote a program which was making access calls to S3 which was essentially destroying my performance and reliability. I had not instrumented my code to report fairly lowlevel information (number of TCP-RST, authentication negotiation time, packet loss). In fact, most of the low-level metrics are hidden so deep in their SDK's, that instrumenting logging is itself a major engineering effort.
I think 99% of customers would be happy if there was a simple REST endpoint to query how my services are running. Fidelity could even be something like 15 minute accuracy.
GET: /MyStatus
{
"S3": [
{
"BucketName": "...",
"Status": "DEGRADED",
"FailingOperations": [
"ListMultipartUpload",
"CreateMultipartUpload",
]
],
"EC2": []
...
}
- cyberferret 10y agoCloudWatch?
- cjhanks 10y agoCloudWatch only tells you a small part of the story. AWS has much richer statistics internally that they do not expose isolating why issues are happening.