4 ms·

I'm a principal at AWS and spent years dealing with this on CloudFront and Route 53. A lot right, and a little off, in the post. Yes, we should definitely commu

I'm a principal at AWS and spent years dealing with this on CloudFront and Route 53. A lot right, and a little off, in the post. Yes, we should definitely communicate to the viewer what's happened and appropriate next steps if any. But there are a handful of challenges that Ive not seen reasonably solved. The first problem is that different viewers need different context, which is especially true for service providers like CDNs. Telling an end user "the page is taking too long" makes sense. But what if its the CDN customer (developer), theyre going to need request IDs and other diagnostics just like Fastly did display. Playing the "we need the RID from the headers you didnt know about to capture" game is a losing proposition. The post and it'd "good" examples suggest a way to contact support, but there is ~0% chance support can help without additional information that isnt included. Expanding on that the post is making really dangerous assumptions and carrying those through to the semantics of the text. Exposing authnz failures as Access Denied vs Not Found is problematic in its own right. Going further to tell the user "You need to login first" is asking for pain in the other 10% of cases where it's a credential or authz problem. Again, different users need different context. And there's no way to know what context to provide apriori. Lastly it's somewhere between incredibly hard to impossible to distinguish the cause ("why") on a per request basis. Especially for service providers like CDNs. The CDN has a cache miss, but can't retrieve the content. Is it a 504 Gateway Timeout because the origin is not responding, 502 Bad Gateway because the origin TLS is broken, 502 because the local clock is off, or a 503 Service Unavailable because of an internal service timeout? Even if you can distinguish what does the CDN represent to the end user; a general fault in the CDN? A specific failure of the origin? Or a simplistic "unavailable." Again, different viewers need different context and semantics. Which you're not going to do at thousands or millions or tps.

naikrovek 5y ago
these seem solvable. there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both. show the errors that make sense for that action and that error. no one is saying that there should be a small list of "approved" errors that everyone would see. show what makes sense in that situation, and don't show something that doesn't help. you don't need to have prophetic insight into the actual causes. just show the flipping errors. don't hide things behind friendly messages; show the friendly message and the error. it's insulting when an error message is hidden from me in favor of something like "oopsies! we broke something! so sorry!!" insulting and disrespectful. there is no need for a single solution. do what's right for you, and most of all, start thinking about ways that things can work, instead of going straight to the reasons that they might not. don't talk yourself out of a good decision because it might not work for every last situation.
- judge2020 5y ago
  Showing the error itself only opens the provider to information disclosure - with the various automated infrastructure monitoring systems running on Fastly or CF they know when and where errors are happening with full traces that help them debug. > there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both As the OP stated, there is quite literally nothing support can do for system-wide outages like this, and when you show support information on these pages, the regular users that see them end up asking for support. There are a non-zero amount of posts monthly on the Cloudflare community forum from people asking about error messages they see for sites they don’t own.
  - vladvasiliu 5y ago
    You could do a mix of both: give information useful for support (trace id, error code, etc) but don't explicitly say "send an email to support@bigcorp.com / call 01 23 45 67". The people on the tech side who are the CDN's clients will know who and how to call for support and if it's an end-user they'll be able to transmit this info. Bonus points for the tech guys to figure that there's an issue on their end and that they don't need to bother the CDN support. I also agree with GP's point that extremely vague error messages are aggravating. Maybe I did something stupid / unexpected, and with a usable error message I'll be able to fix it on my own. Like sending the wrong file format, or whatever. I absolutely hate when a system will just say "there's an error". And having cute animals or phrases doesn't change the fact that the error is useless. This is one of the reasons why I absolutely hate working with Windows. "An error occurred. Call your sysadmin". Right. I'm the sysadmin. What now? This also trains users to never say what's wrong when they call for support. "Yeah, there's an issue with [thing]. What issue? It doesn't work. Oh, of course. There's only exactly one failure mode, so I'll get right on it." I work in operations and get those fairly often even from people who should know better (software engineers and such) and I find it frustrating to no end that I always have to pull information from them even when there are clear error codes. So no, hiding this kind of info serves absolutely no purpose.
    - hunter2_ 5y ago
rozenmd 5y ago
Hey, Fair point, I mainly picked the first contrived example that came into mind, rather than thinking long and hard about what the "correct" error message should be. Will update the article to clarify that.
- donavanm 5y ago
  Cheers. Didnt mean to be too contrarian here or pick apart the broader message, which I absolutely agree with. More of an expression of my personal experience struggling with the same problem. As unsatisfying as it is I suspect you settle for getting it right 95% of the time and including enough details that the user can self service, or contact support, in the other cases.
QuinnyPig 5y ago
Very true, at the individual error level. That said, Fastly fixed the entire issue in less time than it would have taken AWS to update the status page to the blue diamond, so there’s that.
- WatchDog 5y ago
  "Service is experiencing increased error rates" - Error rates increased from 0 to 100%
  - slver 5y ago
    No service has 0% error rate, and 100% errors doesn't even mean much without context (you never receive all possible errors at once).
    - jfrunyon 5y ago
      Error rate = (number of requests which returned an error) / (total number of requests) You can choose plenty of time periods to satisfy a 0% error rate for most services - somewhere from milliseconds to days or even years.
  - sharken 5y ago
    Or 50% error rates in case one of two nodes is down and you keep calling the faulty node. But the faulty node should not be called in a well designed system.
    - dijit 5y ago
      But then you're serving 100% faulty. Error rates (as alert thresholds and end user reporting) are a service thing, not a node thing.
- Xorlev 5y ago
  In large organizations, status boards are "external communications" and this must be handled by a team dedicated to that. It works about as well as you might expect.
adreamingsoul 5y ago
Agreed.

naikrovek 5y ago

these seem solvable. there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both. show the errors that make sense for that action and that error. no one is saying that there should be a small list of "approved" errors that everyone would see. show what makes sense in that situation, and don't show something that doesn't help. you don't need to have prophetic insight into the actual causes. just show the flipping errors. don't hide things behind friendly messages; show the friendly message and the error. it's insulting when an error message is hidden from me in favor of something like "oopsies! we broke something! so sorry!!" insulting and disrespectful. there is no need for a single solution. do what's right for you, and most of all, start thinking about ways that things can work, instead of going straight to the reasons that they might not. don't talk yourself out of a good decision because it might not work for every last situation.

judge2020 5y ago
Showing the error itself only opens the provider to information disclosure - with the various automated infrastructure monitoring systems running on Fastly or CF they know when and where errors are happening with full traces that help them debug. > there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both As the OP stated, there is quite literally nothing support can do for system-wide outages like this, and when you show support information on these pages, the regular users that see them end up asking for support. There are a non-zero amount of posts monthly on the Cloudflare community forum from people asking about error messages they see for sites they don’t own.
- vladvasiliu 5y ago
  You could do a mix of both: give information useful for support (trace id, error code, etc) but don't explicitly say "send an email to support@bigcorp.com / call 01 23 45 67". The people on the tech side who are the CDN's clients will know who and how to call for support and if it's an end-user they'll be able to transmit this info. Bonus points for the tech guys to figure that there's an issue on their end and that they don't need to bother the CDN support. I also agree with GP's point that extremely vague error messages are aggravating. Maybe I did something stupid / unexpected, and with a usable error message I'll be able to fix it on my own. Like sending the wrong file format, or whatever. I absolutely hate when a system will just say "there's an error". And having cute animals or phrases doesn't change the fact that the error is useless. This is one of the reasons why I absolutely hate working with Windows. "An error occurred. Call your sysadmin". Right. I'm the sysadmin. What now? This also trains users to never say what's wrong when they call for support. "Yeah, there's an issue with [thing]. What issue? It doesn't work. Oh, of course. There's only exactly one failure mode, so I'll get right on it." I work in operations and get those fairly often even from people who should know better (software engineers and such) and I find it frustrating to no end that I always have to pull information from them even when there are clear error codes. So no, hiding this kind of info serves absolutely no purpose.
  - hunter2_ 5y ago

rozenmd 5y ago

Hey, Fair point, I mainly picked the first contrived example that came into mind, rather than thinking long and hard about what the "correct" error message should be. Will update the article to clarify that.

donavanm 5y ago
Cheers. Didnt mean to be too contrarian here or pick apart the broader message, which I absolutely agree with. More of an expression of my personal experience struggling with the same problem. As unsatisfying as it is I suspect you settle for getting it right 95% of the time and including enough details that the user can self service, or contact support, in the other cases.

QuinnyPig 5y ago

Very true, at the individual error level. That said, Fastly fixed the entire issue in less time than it would have taken AWS to update the status page to the blue diamond, so there’s that.

WatchDog 5y ago
"Service is experiencing increased error rates" - Error rates increased from 0 to 100%
- slver 5y ago
  No service has 0% error rate, and 100% errors doesn't even mean much without context (you never receive all possible errors at once).
  - jfrunyon 5y ago
    Error rate = (number of requests which returned an error) / (total number of requests) You can choose plenty of time periods to satisfy a 0% error rate for most services - somewhere from milliseconds to days or even years.
- sharken 5y ago
  Or 50% error rates in case one of two nodes is down and you keep calling the faulty node. But the faulty node should not be called in a well designed system.
  - dijit 5y ago
    But then you're serving 100% faulty. Error rates (as alert thresholds and end user reporting) are a service thing, not a node thing.
Xorlev 5y ago
In large organizations, status boards are "external communications" and this must be handled by a team dedicated to that. It works about as well as you might expect.

adreamingsoul 5y ago

Agreed.