4 ms·

interesting they use blue as the color that indicate an "issue". at a glance, looks like nothing to see here. i have a feeling that if the entire AWS infrastru

interesting they use blue as the color that indicate an "issue". at a glance, looks like nothing to see here. i have a feeling that if the entire AWS infrastructure were to go offline somehow they would refer to that as 'increase error rates'

ashtonkem 6y ago
100% is technically an increase, if you’re rules lawyering.
jacques_chester 6y ago
I suspect it's an accessibility decision. Blue is more distinguishable from green by colour-blind folks. Red is not distinguishable.
Spivak 6y ago
Downtime is kinda hard to grok for huge distributed services like this. If you’re down when anyone gets errors you’ll probably always be down — not really useful. If you’re down when everyone gets errors you’ll probably always be up — again not really useful. From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all that useful except as a probability that any issues you’re seeing are your fault.
- temp-dude-87844 6y ago
  On this topic, a popular previous post on HN is 'Your nines are not my nines' [1] by rachelbythebay. [1] https://news.ycombinator.com/item?id=20451714 https://news.ycombinator.com/item?id=20451714
- obnauticus 6y ago
  Hence why they people use percentile measurements for alerting at places like AMZN. This person worked on Alexa and wrote an article about it: https://medium.com/@djsmith42/how-to-metric-edafaf959fc7 https://medium.com/@djsmith42/how-to-metric-edafaf959fc7
  - deathanatos 6y ago
    Maybe they do, but that's sort of the point? I've had to file numerous <s>bugs</s> support tickets with cloud providers which amounted to something like "99% of requests to S3 are failing and your status page is 'This is fine.' HALP"; and the response comes "oh, it was just your shard/node/backend/puzzle piece", like the fact that it was some private implementation detail really matters to me, a customer, when I'm seeing basically 100% failure. But if you measure something as a percentile, and it's just my "shard" or whatever, I'm like 0.0001% of S3 traffic, so you'll never notice it at that scale. A small constant stream of errors is somewhat acceptable, so long as they're just the randomly distributed noise that is random failures and bit flips. But persistent errors that block forward progress or some customer having 0% success is what causes SWEs to ask themselves "I wonder what GCP is like.", and it becomes very clear that their monitoring doesn't indicate when the service is on fire for someone, only when it is on fire for everyone. Azure had an outage yesterday (that was only visible to customers / wasn't reflected on the public status page): there's apparently no more capacity in West US for certain VMs. And, well, it is sort of a basic supposition that our cloud provider can provide a cloud, when asked. And region hopping is not simple: a whole host of resources just aren't compatible if you're not in the same region. (E.g., a vnet requires the VMs be in the same region.) (And that was "elevated" error rates, or something, again. Where, as a customer, "elevated" would appear to mean "100%".) Not a great start to the week. Someone posted Rachel-by-the-Bay's "Your nines are not my nines" article in a sibling comment, which is basically this. People way discount the TCO of the cloud, IMO. I'd also add I've worked at companies where every 5xx was hunted down. It was easy to get complacent, chalk them up to "oh, probably just a network blip" and we thankfully had a staff eng. who was very good and log diving and showing that no, it wasn't a blip, it was a bug in our code, and we should fix that bug.
TechBro8615 6y ago
The first color I noticed on the page was the orange RSS icons, although that's probably because I was primed by the submission's headline.
cocoa19 6y ago
Interesting, I just noticed today Glassdoor changed the Negative Experience chart color from red to a more neutral bluish color. Maybe it's a recent trend in the industry to distance themselves from negative traits? Amazon doesn't want you to think they're bad, Glassdor doesn't want their customers (corporations) to be seen in a bad light?

ashtonkem 6y ago

100% is technically an increase, if you’re rules lawyering.

jacques_chester 6y ago

I suspect it's an accessibility decision. Blue is more distinguishable from green by colour-blind folks. Red is not distinguishable.

Spivak 6y ago

Downtime is kinda hard to grok for huge distributed services like this. If you’re down when anyone gets errors you’ll probably always be down — not really useful. If you’re down when everyone gets errors you’ll probably always be up — again not really useful. From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all that useful except as a probability that any issues you’re seeing are your fault.

temp-dude-87844 6y ago
On this topic, a popular previous post on HN is 'Your nines are not my nines' [1] by rachelbythebay. [1] https://news.ycombinator.com/item?id=20451714 https://news.ycombinator.com/item?id=20451714
obnauticus 6y ago
Hence why they people use percentile measurements for alerting at places like AMZN. This person worked on Alexa and wrote an article about it: https://medium.com/@djsmith42/how-to-metric-edafaf959fc7 https://medium.com/@djsmith42/how-to-metric-edafaf959fc7
- deathanatos 6y ago
  Maybe they do, but that's sort of the point? I've had to file numerous <s>bugs</s> support tickets with cloud providers which amounted to something like "99% of requests to S3 are failing and your status page is 'This is fine.' HALP"; and the response comes "oh, it was just your shard/node/backend/puzzle piece", like the fact that it was some private implementation detail really matters to me, a customer, when I'm seeing basically 100% failure. But if you measure something as a percentile, and it's just my "shard" or whatever, I'm like 0.0001% of S3 traffic, so you'll never notice it at that scale. A small constant stream of errors is somewhat acceptable, so long as they're just the randomly distributed noise that is random failures and bit flips. But persistent errors that block forward progress or some customer having 0% success is what causes SWEs to ask themselves "I wonder what GCP is like.", and it becomes very clear that their monitoring doesn't indicate when the service is on fire for someone, only when it is on fire for everyone. Azure had an outage yesterday (that was only visible to customers / wasn't reflected on the public status page): there's apparently no more capacity in West US for certain VMs. And, well, it is sort of a basic supposition that our cloud provider can provide a cloud, when asked. And region hopping is not simple: a whole host of resources just aren't compatible if you're not in the same region. (E.g., a vnet requires the VMs be in the same region.) (And that was "elevated" error rates, or something, again. Where, as a customer, "elevated" would appear to mean "100%".) Not a great start to the week. Someone posted Rachel-by-the-Bay's "Your nines are not my nines" article in a sibling comment, which is basically this. People way discount the TCO of the cloud, IMO. I'd also add I've worked at companies where every 5xx was hunted down. It was easy to get complacent, chalk them up to "oh, probably just a network blip" and we thankfully had a staff eng. who was very good and log diving and showing that no, it wasn't a blip, it was a bug in our code, and we should fix that bug.

TechBro8615 6y ago

The first color I noticed on the page was the orange RSS icons, although that's probably because I was primed by the submission's headline.

cocoa19 6y ago

Interesting, I just noticed today Glassdoor changed the Negative Experience chart color from red to a more neutral bluish color. Maybe it's a recent trend in the industry to distance themselves from negative traits? Amazon doesn't want you to think they're bad, Glassdor doesn't want their customers (corporations) to be seen in a bad light?