9 ms·

What the Fastly outage can teach us about writing error messages

266 points by rozenmd 5y ago

RavinduL 5y ago
Have they published a post-mortem for the outage yet? I'm curious as to what happened here
- takeda 5y ago
  They wrote that they had to reinstall Workbench. Isn't that too early for pay mortem? I didn't experience it myself, but I think it happened at most few hours ago.
- unityByFreedom 5y ago
  Here it is https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...
  - maxgashkov 5y ago
    Zero Trust is not really applicable here. ZT is about not implicitly allowing someone elevated access just because they have access to network and requiring explicit grant based on the identity and other rules. Fastly's downtime seems to be caused by an automatically generated config that got deployed in production as a result of change requested by a legitimate customer. This happened to CF in its early days and I really doubt that ZT had anything to do with the fact that they do not have this kind of problem anymore. It's probably some sanity checks before they deploy updated lua scripts to their fleet of nginx's if anything.
  - joshuamorton 5y ago
    Zero trust has little to do with individual clients and more to do with the idea of the internal "corp" network not being trusted. That is, there isn't a conventional network you VPN into at which point all traffic is trusted. The generic topic your looking for is probably something like "customer isolation" ("service isolation" might also be relevant, but is used also in the context of "tenant isolation" which isn't really what you want). See this thread: https://news.ycombinator.com/item?id=25237836 https://news.ycombinator.com/item?id=25237836 for some talk about how AWS does "cellularization" which is a form of workload/service isolation/partitioning. In general I don't think there's much discussion of this issue on the wider web.
hvocode 5y ago
I've had more than one intern or young engineer that I've had to teach to resist the urge to put witty comments and jokes in project docs, comments, or errors. Especially in error messages or things that can be exposed to users/customers. Even if it's an error you think only the dev team will see, you never know if it will make its way out to audiences you didn't intend. Somehow I doubt people faced with this error message really were amused by the reference to the old Amiga OS errors....
- JadeNB 5y ago
  > Somehow I doubt people faced with this error message really were amused by the reference to the old Amiga OS errors.... I'll bet whoever put that in there feels extra silly about misspelling "meditation", too.
  - takeda 5y ago
    It looks like the spelling was fixed a while ago, does that mean fastly is running an old version of varnish?
    - rgj 5y ago
      Fastly changed it on purpose so they could distinguish between error messages from their Varnish and error messages from an origin customer owned Varnish.
- marcosdumay 5y ago
  I'd say that some safe joke on the end of your error message is perfectly fine, as long as you are perfectly clear where the problem is. Just be very sure not to unintentionally blame the user. It is fine even if the user won't get the joke. Software is a highly personal and creative thing, it should have a personality. The everything gray enterprise spaces have gone too far already. Also, I really miss Linux yelling on panics. Now, the same joke on a highly visible position or on the beginning of the message is harmful and will impede people from solving the problem.
xwdv 5y ago
Good error messages are hard. There’s so many things that can go wrong, and each needs its own custom explanation. Never really found a good way to organize this. And since errors should be uncommon, it feels like a waste of time coming up with thoughtful messages.
- SilverRed 5y ago
  The problem is that you have to know exactly what has gone wrong to show a proper message. But if you know what is wrong, most of the time the correct action is to fix the program so it does not go wrong instead of showing a better error. With the main exception of user submitted data validations where it is up to the user to submit correct data. For the average web service, there the error is a stack trace and there is nothing the user can do so a blank screen with a "something went wrong, we have logged this erorr" is the only real option.
  - bombcar 5y ago
    Even if there’s usually nothing the user can do I still like to have all the stack traces, etc visible - because sometimes there’s enough of a hint that you can get a workaround. A number of DNS failures I’ve worked around with the hosts file.
    - gowld 5y ago
      You've worked around server-side DNS failures?
      - bombcar 5y ago
        Yes - often the DNS issue is unrelated to the server itself - and sometimes a quick lookup shows an obvious cname error - or shows a round-robin dns where the response you have is failing but the others work. Or if the host name has .eu.domain or something else indicating a geographic location changing that can get around a localized failure sometimes.
- sachinag 5y ago
gonzo41 5y ago
So that looks like a stock Varnish 503. I'm pretty sure the magic behind fastly is a ton of Varnish cache. What's interesting (VCL can be unforgiving), also that it was a big central stuff up. Which makes me think they have layers and layers of caching in depth and some top level config just blacked out everything. I kind of would have thought they'd have lots of small instances.
- res0nat0r 5y ago
  Fastly is indeed powered by a lot of Varnish magic. My company used(s) them for the CDN for some of our large website frontends, and they have engineers which will help generate the insanely convoluted and large VCL templates which can be setup in your account and will handle all of the magic routing / caching / much more complex things you need for your origin.
  - gonzo41 5y ago
    Yeah I've used varnish before. I like it a lot, but I do think coloring inside the lines as much as possible is probably the safest thing. Some of the VMOD's and more exciting things you can do with varnish become really trick to manage.
- transitorykris 5y ago
  This is not true Edit: trying to reverse engineer the “magic”, architecture, and the failure from a varnish error message is folly, and misleads others. How do I know the comment is patently false? I ran those teams at Fastly for 3 years.
  - ntauthority 5y ago
    Alternately, document your 'magic architecture' in public so that people don't have to offend you or other employees of the firm by making incorrect assumptions based on the little you do show in public. Also helps others pick up if you do go out of business or end up acquired, and before you do it even leads to a more level playing field. However I guess a level playing field wouldn't have given you a NYSE listing... so I guess you're just being selfish?
kajaktum 5y ago
But who's going to host that nice unicorn images? and those nice CSS? I agree with putting up a link tho.
- orf 5y ago
  Embed both in the page.
  - res0nat0r 5y ago
    I suspect most folks, especially ones which operate at GB/s rates serving traffic would not like to serve up a 100kb base64 encoded image every time an error is thrown.
    - orf 5y ago
      Sure, which is why they probably wouldn’t serve up a 100kb base64 encoded image every time an error is thrown. It’s more than possible to embed a small vector image to add some humanity to an error page without breaking the bank, bandwidth-wise
    - Ayesh 5y ago
      Browsers send the Accept headers, and I have configured my CDN to return a plain text page for static assets (that users wouldn't see on the browser), but text/html requests get a useful 10-20 kb response. Should be a trivial thing to setup.
    - Clewza313 5y ago
      Or just cache the image with a major CDN, like Fastly. No wait...
    - ascorbic 5y ago
      Why not? The non-error page was likely much larger.
- kaushikt 5y ago
  We create separate 404 and 500 error pages. Question is - should we do this for most status codes?
aniforprez 5y ago
Reminds me of the quintessential tweet about error messages https://twitter.com/cherrikissu/status/972524442600558594?s=19 https://twitter.com/cherrikissu/status/972524442600558594?s=...
- function_seven 5y ago
  With a small group of Corporate Memphis people frowning or looking bewildered. Give me “Unspecified error” any day over that.
- calvinmorrison 5y ago
  Right up there with Slack etcs fake loading messages "Gearing up the dildonator" "Implicating the fairies" "Hogtying George Bush" Dude - just give me a spinner or a progress bar, and if something errors during the load out give me some sort of stack trace or error ID I can use to help
  - endgame 5y ago
    The one I especially hated: "You look nice today" 1. "Nice" is an adjective with nearly zero meaning. 2. Either you have access to my camera outside of calls and are analysing my appearance (WTF) or you're making stuff up. 3. You're making chat software. Stop trying to butter me up.
    - labster 5y ago
      Nice reply, you’re on fire today! No, really, you should find a fire extinguisher.
    - fastball 5y ago
      You sound British.
  - underwater 5y ago
    ProductBoard likes to tell me: > Without good products, life would be a mistake. > Make products that matter. > Product excellence isn't a point in time. It is a state of mind, it is a way of life. > So many feature requests, so little time. Maybe they're being silly? I hope so. But I've also met product managers and designers who seem to think like that. It makes me internally sigh every time I have to use their product.
donavanm 5y ago
I'm a principal at AWS and spent years dealing with this on CloudFront and Route 53. A lot right, and a little off, in the post. Yes, we should definitely communicate to the viewer what's happened and appropriate next steps if any. But there are a handful of challenges that Ive not seen reasonably solved. The first problem is that different viewers need different context, which is especially true for service providers like CDNs. Telling an end user "the page is taking too long" makes sense. But what if its the CDN customer (developer), theyre going to need request IDs and other diagnostics just like Fastly did display. Playing the "we need the RID from the headers you didnt know about to capture" game is a losing proposition. The post and it'd "good" examples suggest a way to contact support, but there is ~0% chance support can help without additional information that isnt included. Expanding on that the post is making really dangerous assumptions and carrying those through to the semantics of the text. Exposing authnz failures as Access Denied vs Not Found is problematic in its own right. Going further to tell the user "You need to login first" is asking for pain in the other 10% of cases where it's a credential or authz problem. Again, different users need different context. And there's no way to know what context to provide apriori. Lastly it's somewhere between incredibly hard to impossible to distinguish the cause ("why") on a per request basis. Especially for service providers like CDNs. The CDN has a cache miss, but can't retrieve the content. Is it a 504 Gateway Timeout because the origin is not responding, 502 Bad Gateway because the origin TLS is broken, 502 because the local clock is off, or a 503 Service Unavailable because of an internal service timeout? Even if you can distinguish what does the CDN represent to the end user; a general fault in the CDN? A specific failure of the origin? Or a simplistic "unavailable." Again, different viewers need different context and semantics. Which you're not going to do at thousands or millions or tps.
- naikrovek 5y ago
  these seem solvable. there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both. show the errors that make sense for that action and that error. no one is saying that there should be a small list of "approved" errors that everyone would see. show what makes sense in that situation, and don't show something that doesn't help. you don't need to have prophetic insight into the actual causes. just show the flipping errors. don't hide things behind friendly messages; show the friendly message and the error. it's insulting when an error message is hidden from me in favor of something like "oopsies! we broke something! so sorry!!" insulting and disrespectful. there is no need for a single solution. do what's right for you, and most of all, start thinking about ways that things can work, instead of going straight to the reasons that they might not. don't talk yourself out of a good decision because it might not work for every last situation.
anigbrowl 5y ago
I have a particular dislike for Apple over this - I think they set a trend of unhelpful error messages tied to obscure codes back in the 1990s. Sadly they've been joined by Google. 'Something went wrong' could have come from the Sirius Cybernetics Corporation.
- Clewza313 5y ago
  Obscure but unique and googlable error codes are great. At a previous job, we had a large project to go through the codebase and tag every single user facing error with one, then publicly document them with likely causes and solutions. Support volume dipped noticeably.
  - userbinator 5y ago
    Google is unfortunately getting worse for such things. Search for some error code and you get pages of spam and those that don't contain the code you're looking for, but a "very similar" one. Even one digit off makes a huge difference.
- egypturnash 5y ago
  Your plastic pal is busy remembering how to be fun! Come back later, and go stick your head in a pig.
userbinator 5y ago
Unfortunately the majority of internet users aren't trained in the art of reading HTTP status codes I think the majority know what 404 is, and possibly 403, but I agree about the more obscure ones. That said, I don't think it's a bad idea to rely on the "default exception handling behaviour" that the majority of users, even non-computer-literate ones, will have: they'll retry a few times, see that it doesn't work, and go elsewhere for a while.
- ozi 5y ago
  you vastly overestimate the majority.
- sschueller 5y ago
  If we keep dumbing down the world for everyone instead of teaching those who don't know we will destroy ourselves. People are not stupid and if we keep assuming it we are doing a disservice to all. No more www, no more protocol in the address bar and apple is selling iMac colors in it's commercials...
  - denysvitali 5y ago
    ... and blending the search eith the address bar. I spent a good 1 hour to explain the difference to a tech illiterate on why typing "mywebsite.com" was different from typing "mywebsite com" and picking the first result on Google. I'm not sure he understood, but I really have to admit that this trend of dumbing down things is only make them worse, in a way.
    - aembleton 5y ago
      They'll understand when they try and buy airline tickets that end up going through some third party with a 'service charge'. One of my friends has recently learnt that lesson when she went to buy flights on Ryanair; just typing that into Google and clicking the first link. £30 service charge.
  - aembleton 5y ago
    How will it destroy us? Do you think we should use hex or binary instead of their ASCII or UTF-8 equivalents? After-all, ASCII must be dumbing down as it makes it easier for non specialists to read.
u678u 5y ago
Isn't it time we had more error messages? I feel like we got 40x, 50x and the pioneers left lots of space for more. I quite liked the windows 0x800*** hex error numbers, though some were more useful than others.
- DocTomoe 5y ago
  The old, original HTTP error codes are useful because they are standardized in their meaning (even if you have to beat the difference between 404 and 410 into an SEO Consultant's head with a bat). New error codes are only useful if they are generally understood. Maybe instead of using more codes for sub-use-cases, use the permissible error text to express these (HTTP-Status: 503 0x63F0 Data corrupt ?)
bigyikes 5y ago
Ahhhh. I’ve spent countless hours debugging “Guru Meditation” errors from FreeRTOS on my ESP32 dev boards. I always wondered what the heck that meant. Turns out I was just not in on the joke.
- vaylian 5y ago
  > I’ve spent countless hours debugging You are the guru. And you are meditating on the problem.
fouc 5y ago
the Fastly outage can teach us to stop relying entirely on Fastly, AWS, Cloudflare, and so on. Diversify your deployment setup!
- Terretta 5y ago
  This way, if any one of them has problems, you get incident management practice. Win win!
timeinput 5y ago
Just today I had some code that helpfully said "should never get here" and aborted. It left us to figure out what the possible options of how we could end up in that spot "we should never get to". It was maddening because it's ~12 year old code that had some library updates. A little bit more help like the error code or really *anything* would've been helpful.
- amalcon 5y ago
  A couple of years ago, I redid the error code definitions in an internal C++ library. The error returns were of the form: -__LINE__ which, if you know C++, means your error code is just the negation of the current line number. That's super convenient when writing. It's really annoying when someone actually sees such an error code and emails you about it -- because they obviously can't do anything with it in software. The obvious problem is that the semantics of an error code depend on the software revision they built with, but you also need to figure out which source file has an error return at that line that the user could have reached at their context. The new error codes are almost as ergonomic to write (involving a little splash of code generation to make it so, not ideal but worth it). However, they can actually be handled in software and there's a functional perror-equivalent.
_def 5y ago
A great example of content marketing.
bullen 5y ago
I'm a believer in not hiding things and logging the whole problem all the way to the end consumer: http://test.rupy.se/?id=2 http://test.rupy.se/?id=2 Because when you develop things you are the end consumer.
- IshKebab 5y ago
  Also, consumers may not understand backtraces but they generally understand how to Google them or ask other people what they mean, or tell you about them. Technical errors are way better than just "I'm sorry we couldn't process that right now."
  - chakkepolja 5y ago
    This is probably a dumb question, but isn't a backtrace likely to reveal some sensitive information from a security perspective?
    - IshKebab 5y ago
      Possibly, though I would say that is a little paranoid. But anyway I was really using "backtrace" as a synonym for "technical details that users don't understand". Not very clear, sorry!
- karlerss 5y ago
  This is considered a critical security vulnerability. [0] [0] https://owasp.org/www-project-top-ten/2017/A6_2017-Security_Misconfiguration https://owasp.org/www-project-top-ten/2017/A6_2017-Security_...
  - bullen 5y ago
    Not if it's open-source. Security by obfuscation is generally not a good option if you can avoid it.
    - karlerss 5y ago
Hermel 5y ago
My preferred format is simply what & why, which is the second piece of information in the list in the article. For example: Cannot serve website. (What?) Reason: could not connect to database. (Why?) Most of the time, it is very easy to programmatically assemble such messages. It is much harder to automatically figure out who caused it and when it will be fixed.
aliasEli 5y ago
Also try to translate the messages into the user's language when that is feasible. The native language for the majority of internet users is not English.
eecc 5y ago
Wow, the internet. It’s like my meetings where everyone feels entitled to expressing their opinion, splitting hairs on the most tangential aspects of the issue. Thanks god for coffee, and nice views out of the window
- system2 5y ago
  10 of our clients called and asked why they are paying us if the sites are not working today. I think a proper 503 browser message could help us a little?
  - eecc 5y ago
    Uh, really? I bet Prometheus would do a better job
pif 5y ago
Bullsh*t! 99% of users are only interested in accessing the content they requested. It either works, or it doesn't.
- vaylian 5y ago
  I tend to agree. But I also think that bad error messages are what has gotten there in the first place. People just click on OK on error messages without reading them, because they learned that it doesn't really matter if they read them most of the time.
secondcoming 5y ago
I would consider a cutesy rainbow unicorn to be unnecessary.
nottorp 5y ago
The problem with writing user friendly error messages is in a lot of cases you have no idea what context they'll show in. I mean, do you think they could have predicted yesterday's particular problem at Fastly? Better to give some plain debug info than tell the user "have you tried turning it off and on again?" in my opinion.
phkamp 5y ago
I created that particular error-message in varnish cache 15 years ago. Max has many good points about error messages in general, but all of them require access to out of band information. We in the Varnish Cache Project do not have access to that information, we dont know who runs the varnish instance or what kind of information they serve to what kind of clients. This is why the default '503 message only exposes the "XID" nonce: That allows the administrator of this cache instance to find all the details in the log files. Varnish Cache users who want to present something else can do that from VCL, and I'm pretty sure Fastly normally does. But when all else fails, and here it must have, Varnish Cache errs on the side of caution.
- phkamp 5y ago
  I've taken the opportunity to write up the story of the first time Varnish Cache's 503 was heard around the world, or at least in Norway: http://varnish-cache.org/docs/trunk/phk/503aroundtheworld.html http://varnish-cache.org/docs/trunk/phk/503aroundtheworld.ht...
mekster 5y ago
Quite back then in a personal project, when putting error logs, I also mandated to put what the fix is along with the error message in error log functions. I suppose no one does it.
sbadger 5y ago
More in reply to the comments than the original article.... In my experience no developer ever actually imagines that their code could possibly have bugs. Yes yes, they will say that of course all software has bugs, but deep down they know that that doesn't mean their code, just other's code. Turn all that code into products and that means when something goes wrong it isn't the product's fault. Therefore it must be the user's fault. Error messages can't possibly explain the problem, because the product can't know what the dumb user did. So don't bother really trying. This from the perspective of someone doing customer support.
dpedu 5y ago
It seems to me one of these articles pops up every time there's an outage related to Varnish. There's plenty of poor error messages out there but "Guru Mediation" really grinds folk's gears.

266 points by rozenmd 5y ago

RavinduL 5y ago

Have they published a post-mortem for the outage yet? I'm curious as to what happened here

takeda 5y ago
They wrote that they had to reinstall Workbench. Isn't that too early for pay mortem? I didn't experience it myself, but I think it happened at most few hours ago.
unityByFreedom 5y ago
Here it is https://www.fastly.com/blog/summary-of-june-8-outage https://www.fastly.com/blog/summary-of-june-8-outage Posted about 17 hours after the incident. In short, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i...
- maxgashkov 5y ago
  Zero Trust is not really applicable here. ZT is about not implicitly allowing someone elevated access just because they have access to network and requiring explicit grant based on the identity and other rules. Fastly's downtime seems to be caused by an automatically generated config that got deployed in production as a result of change requested by a legitimate customer. This happened to CF in its early days and I really doubt that ZT had anything to do with the fact that they do not have this kind of problem anymore. It's probably some sanity checks before they deploy updated lua scripts to their fleet of nginx's if anything.
- joshuamorton 5y ago
  Zero trust has little to do with individual clients and more to do with the idea of the internal "corp" network not being trusted. That is, there isn't a conventional network you VPN into at which point all traffic is trusted. The generic topic your looking for is probably something like "customer isolation" ("service isolation" might also be relevant, but is used also in the context of "tenant isolation" which isn't really what you want). See this thread: https://news.ycombinator.com/item?id=25237836 https://news.ycombinator.com/item?id=25237836 for some talk about how AWS does "cellularization" which is a form of workload/service isolation/partitioning. In general I don't think there's much discussion of this issue on the wider web.

hvocode 5y ago

I've had more than one intern or young engineer that I've had to teach to resist the urge to put witty comments and jokes in project docs, comments, or errors. Especially in error messages or things that can be exposed to users/customers. Even if it's an error you think only the dev team will see, you never know if it will make its way out to audiences you didn't intend. Somehow I doubt people faced with this error message really were amused by the reference to the old Amiga OS errors....

JadeNB 5y ago
> Somehow I doubt people faced with this error message really were amused by the reference to the old Amiga OS errors.... I'll bet whoever put that in there feels extra silly about misspelling "meditation", too.
- takeda 5y ago
  It looks like the spelling was fixed a while ago, does that mean fastly is running an old version of varnish?
  - rgj 5y ago
    Fastly changed it on purpose so they could distinguish between error messages from their Varnish and error messages from an origin customer owned Varnish.
marcosdumay 5y ago
I'd say that some safe joke on the end of your error message is perfectly fine, as long as you are perfectly clear where the problem is. Just be very sure not to unintentionally blame the user. It is fine even if the user won't get the joke. Software is a highly personal and creative thing, it should have a personality. The everything gray enterprise spaces have gone too far already. Also, I really miss Linux yelling on panics. Now, the same joke on a highly visible position or on the beginning of the message is harmful and will impede people from solving the problem.

xwdv 5y ago

Good error messages are hard. There’s so many things that can go wrong, and each needs its own custom explanation. Never really found a good way to organize this. And since errors should be uncommon, it feels like a waste of time coming up with thoughtful messages.

SilverRed 5y ago
The problem is that you have to know exactly what has gone wrong to show a proper message. But if you know what is wrong, most of the time the correct action is to fix the program so it does not go wrong instead of showing a better error. With the main exception of user submitted data validations where it is up to the user to submit correct data. For the average web service, there the error is a stack trace and there is nothing the user can do so a blank screen with a "something went wrong, we have logged this erorr" is the only real option.
- bombcar 5y ago
  Even if there’s usually nothing the user can do I still like to have all the stack traces, etc visible - because sometimes there’s enough of a hint that you can get a workaround. A number of DNS failures I’ve worked around with the hosts file.
  - gowld 5y ago
    You've worked around server-side DNS failures?
    - bombcar 5y ago
      Yes - often the DNS issue is unrelated to the server itself - and sometimes a quick lookup shows an obvious cname error - or shows a round-robin dns where the response you have is failing but the others work. Or if the host name has .eu.domain or something else indicating a geographic location changing that can get around a localized failure sometimes.
sachinag 5y ago

gonzo41 5y ago

So that looks like a stock Varnish 503. I'm pretty sure the magic behind fastly is a ton of Varnish cache. What's interesting (VCL can be unforgiving), also that it was a big central stuff up. Which makes me think they have layers and layers of caching in depth and some top level config just blacked out everything. I kind of would have thought they'd have lots of small instances.

res0nat0r 5y ago
Fastly is indeed powered by a lot of Varnish magic. My company used(s) them for the CDN for some of our large website frontends, and they have engineers which will help generate the insanely convoluted and large VCL templates which can be setup in your account and will handle all of the magic routing / caching / much more complex things you need for your origin.
- gonzo41 5y ago
  Yeah I've used varnish before. I like it a lot, but I do think coloring inside the lines as much as possible is probably the safest thing. Some of the VMOD's and more exciting things you can do with varnish become really trick to manage.
transitorykris 5y ago
This is not true Edit: trying to reverse engineer the “magic”, architecture, and the failure from a varnish error message is folly, and misleads others. How do I know the comment is patently false? I ran those teams at Fastly for 3 years.
- ntauthority 5y ago
  Alternately, document your 'magic architecture' in public so that people don't have to offend you or other employees of the firm by making incorrect assumptions based on the little you do show in public. Also helps others pick up if you do go out of business or end up acquired, and before you do it even leads to a more level playing field. However I guess a level playing field wouldn't have given you a NYSE listing... so I guess you're just being selfish?

kajaktum 5y ago

But who's going to host that nice unicorn images? and those nice CSS? I agree with putting up a link tho.

orf 5y ago
Embed both in the page.
- res0nat0r 5y ago
  I suspect most folks, especially ones which operate at GB/s rates serving traffic would not like to serve up a 100kb base64 encoded image every time an error is thrown.
  - orf 5y ago
    Sure, which is why they probably wouldn’t serve up a 100kb base64 encoded image every time an error is thrown. It’s more than possible to embed a small vector image to add some humanity to an error page without breaking the bank, bandwidth-wise
  - Ayesh 5y ago
    Browsers send the Accept headers, and I have configured my CDN to return a plain text page for static assets (that users wouldn't see on the browser), but text/html requests get a useful 10-20 kb response. Should be a trivial thing to setup.
  - Clewza313 5y ago
    Or just cache the image with a major CDN, like Fastly. No wait...
  - ascorbic 5y ago
    Why not? The non-error page was likely much larger.
kaushikt 5y ago
We create separate 404 and 500 error pages. Question is - should we do this for most status codes?

aniforprez 5y ago

Reminds me of the quintessential tweet about error messages https://twitter.com/cherrikissu/status/972524442600558594?s=19 https://twitter.com/cherrikissu/status/972524442600558594?s=...

function_seven 5y ago
With a small group of Corporate Memphis people frowning or looking bewildered. Give me “Unspecified error” any day over that.
calvinmorrison 5y ago
Right up there with Slack etcs fake loading messages "Gearing up the dildonator" "Implicating the fairies" "Hogtying George Bush" Dude - just give me a spinner or a progress bar, and if something errors during the load out give me some sort of stack trace or error ID I can use to help
- endgame 5y ago
  The one I especially hated: "You look nice today" 1. "Nice" is an adjective with nearly zero meaning. 2. Either you have access to my camera outside of calls and are analysing my appearance (WTF) or you're making stuff up. 3. You're making chat software. Stop trying to butter me up.
  - labster 5y ago
    Nice reply, you’re on fire today! No, really, you should find a fire extinguisher.
  - fastball 5y ago
    You sound British.
- underwater 5y ago
  ProductBoard likes to tell me: > Without good products, life would be a mistake. > Make products that matter. > Product excellence isn't a point in time. It is a state of mind, it is a way of life. > So many feature requests, so little time. Maybe they're being silly? I hope so. But I've also met product managers and designers who seem to think like that. It makes me internally sigh every time I have to use their product.

donavanm 5y ago

I'm a principal at AWS and spent years dealing with this on CloudFront and Route 53. A lot right, and a little off, in the post. Yes, we should definitely communicate to the viewer what's happened and appropriate next steps if any. But there are a handful of challenges that Ive not seen reasonably solved. The first problem is that different viewers need different context, which is especially true for service providers like CDNs. Telling an end user "the page is taking too long" makes sense. But what if its the CDN customer (developer), theyre going to need request IDs and other diagnostics just like Fastly did display. Playing the "we need the RID from the headers you didnt know about to capture" game is a losing proposition. The post and it'd "good" examples suggest a way to contact support, but there is ~0% chance support can help without additional information that isnt included. Expanding on that the post is making really dangerous assumptions and carrying those through to the semantics of the text. Exposing authnz failures as Access Denied vs Not Found is problematic in its own right. Going further to tell the user "You need to login first" is asking for pain in the other 10% of cases where it's a credential or authz problem. Again, different users need different context. And there's no way to know what context to provide apriori. Lastly it's somewhere between incredibly hard to impossible to distinguish the cause ("why") on a per request basis. Especially for service providers like CDNs. The CDN has a cache miss, but can't retrieve the content. Is it a 504 Gateway Timeout because the origin is not responding, 502 Bad Gateway because the origin TLS is broken, 502 because the local clock is off, or a 503 Service Unavailable because of an internal service timeout? Even if you can distinguish what does the CDN represent to the end user; a general fault in the CDN? A specific failure of the origin? Or a simplistic "unavailable." Again, different viewers need different context and semantics. Which you're not going to do at thousands or millions or tps.

naikrovek 5y ago
these seem solvable. there's no reason that a good error message AND pertinent support information are mutually exclusive. show them both. show the errors that make sense for that action and that error. no one is saying that there should be a small list of "approved" errors that everyone would see. show what makes sense in that situation, and don't show something that doesn't help. you don't need to have prophetic insight into the actual causes. just show the flipping errors. don't hide things behind friendly messages; show the friendly message and the error. it's insulting when an error message is hidden from me in favor of something like "oopsies! we broke something! so sorry!!" insulting and disrespectful. there is no need for a single solution. do what's right for you, and most of all, start thinking about ways that things can work, instead of going straight to the reasons that they might not. don't talk yourself out of a good decision because it might not work for every last situation.

anigbrowl 5y ago

I have a particular dislike for Apple over this - I think they set a trend of unhelpful error messages tied to obscure codes back in the 1990s. Sadly they've been joined by Google. 'Something went wrong' could have come from the Sirius Cybernetics Corporation.

Clewza313 5y ago
Obscure but unique and googlable error codes are great. At a previous job, we had a large project to go through the codebase and tag every single user facing error with one, then publicly document them with likely causes and solutions. Support volume dipped noticeably.
- userbinator 5y ago
  Google is unfortunately getting worse for such things. Search for some error code and you get pages of spam and those that don't contain the code you're looking for, but a "very similar" one. Even one digit off makes a huge difference.
egypturnash 5y ago
Your plastic pal is busy remembering how to be fun! Come back later, and go stick your head in a pig.

userbinator 5y ago

Unfortunately the majority of internet users aren't trained in the art of reading HTTP status codes I think the majority know what 404 is, and possibly 403, but I agree about the more obscure ones. That said, I don't think it's a bad idea to rely on the "default exception handling behaviour" that the majority of users, even non-computer-literate ones, will have: they'll retry a few times, see that it doesn't work, and go elsewhere for a while.

ozi 5y ago
you vastly overestimate the majority.
sschueller 5y ago
If we keep dumbing down the world for everyone instead of teaching those who don't know we will destroy ourselves. People are not stupid and if we keep assuming it we are doing a disservice to all. No more www, no more protocol in the address bar and apple is selling iMac colors in it's commercials...
- denysvitali 5y ago
  ... and blending the search eith the address bar. I spent a good 1 hour to explain the difference to a tech illiterate on why typing "mywebsite.com" was different from typing "mywebsite com" and picking the first result on Google. I'm not sure he understood, but I really have to admit that this trend of dumbing down things is only make them worse, in a way.
  - aembleton 5y ago
    They'll understand when they try and buy airline tickets that end up going through some third party with a 'service charge'. One of my friends has recently learnt that lesson when she went to buy flights on Ryanair; just typing that into Google and clicking the first link. £30 service charge.
- aembleton 5y ago
  How will it destroy us? Do you think we should use hex or binary instead of their ASCII or UTF-8 equivalents? After-all, ASCII must be dumbing down as it makes it easier for non specialists to read.

u678u 5y ago

Isn't it time we had more error messages? I feel like we got 40x, 50x and the pioneers left lots of space for more. I quite liked the windows 0x800*** hex error numbers, though some were more useful than others.

DocTomoe 5y ago
The old, original HTTP error codes are useful because they are standardized in their meaning (even if you have to beat the difference between 404 and 410 into an SEO Consultant's head with a bat). New error codes are only useful if they are generally understood. Maybe instead of using more codes for sub-use-cases, use the permissible error text to express these (HTTP-Status: 503 0x63F0 Data corrupt ?)

bigyikes 5y ago

Ahhhh. I’ve spent countless hours debugging “Guru Meditation” errors from FreeRTOS on my ESP32 dev boards. I always wondered what the heck that meant. Turns out I was just not in on the joke.

vaylian 5y ago
> I’ve spent countless hours debugging You are the guru. And you are meditating on the problem.

fouc 5y ago

the Fastly outage can teach us to stop relying entirely on Fastly, AWS, Cloudflare, and so on. Diversify your deployment setup!

Terretta 5y ago
This way, if any one of them has problems, you get incident management practice. Win win!

timeinput 5y ago

Just today I had some code that helpfully said "should never get here" and aborted. It left us to figure out what the possible options of how we could end up in that spot "we should never get to". It was maddening because it's ~12 year old code that had some library updates. A little bit more help like the error code or really *anything* would've been helpful.

amalcon 5y ago
A couple of years ago, I redid the error code definitions in an internal C++ library. The error returns were of the form: -__LINE__ which, if you know C++, means your error code is just the negation of the current line number. That's super convenient when writing. It's really annoying when someone actually sees such an error code and emails you about it -- because they obviously can't do anything with it in software. The obvious problem is that the semantics of an error code depend on the software revision they built with, but you also need to figure out which source file has an error return at that line that the user could have reached at their context. The new error codes are almost as ergonomic to write (involving a little splash of code generation to make it so, not ideal but worth it). However, they can actually be handled in software and there's a functional perror-equivalent.

_def 5y ago

A great example of content marketing.

bullen 5y ago

I'm a believer in not hiding things and logging the whole problem all the way to the end consumer: http://test.rupy.se/?id=2 http://test.rupy.se/?id=2 Because when you develop things you are the end consumer.

IshKebab 5y ago
Also, consumers may not understand backtraces but they generally understand how to Google them or ask other people what they mean, or tell you about them. Technical errors are way better than just "I'm sorry we couldn't process that right now."
- chakkepolja 5y ago
  This is probably a dumb question, but isn't a backtrace likely to reveal some sensitive information from a security perspective?
  - IshKebab 5y ago
    Possibly, though I would say that is a little paranoid. But anyway I was really using "backtrace" as a synonym for "technical details that users don't understand". Not very clear, sorry!
karlerss 5y ago
This is considered a critical security vulnerability. [0] [0] https://owasp.org/www-project-top-ten/2017/A6_2017-Security_Misconfiguration https://owasp.org/www-project-top-ten/2017/A6_2017-Security_...
- bullen 5y ago
  Not if it's open-source. Security by obfuscation is generally not a good option if you can avoid it.
  - karlerss 5y ago

Hermel 5y ago

My preferred format is simply what & why, which is the second piece of information in the list in the article. For example: Cannot serve website. (What?) Reason: could not connect to database. (Why?) Most of the time, it is very easy to programmatically assemble such messages. It is much harder to automatically figure out who caused it and when it will be fixed.

aliasEli 5y ago

Also try to translate the messages into the user's language when that is feasible. The native language for the majority of internet users is not English.

eecc 5y ago

Wow, the internet. It’s like my meetings where everyone feels entitled to expressing their opinion, splitting hairs on the most tangential aspects of the issue. Thanks god for coffee, and nice views out of the window

system2 5y ago
10 of our clients called and asked why they are paying us if the sites are not working today. I think a proper 503 browser message could help us a little?
- eecc 5y ago
  Uh, really? I bet Prometheus would do a better job

pif 5y ago

Bullsh*t! 99% of users are only interested in accessing the content they requested. It either works, or it doesn't.

vaylian 5y ago
I tend to agree. But I also think that bad error messages are what has gotten there in the first place. People just click on OK on error messages without reading them, because they learned that it doesn't really matter if they read them most of the time.

secondcoming 5y ago

I would consider a cutesy rainbow unicorn to be unnecessary.

nottorp 5y ago

The problem with writing user friendly error messages is in a lot of cases you have no idea what context they'll show in. I mean, do you think they could have predicted yesterday's particular problem at Fastly? Better to give some plain debug info than tell the user "have you tried turning it off and on again?" in my opinion.

phkamp 5y ago

I created that particular error-message in varnish cache 15 years ago. Max has many good points about error messages in general, but all of them require access to out of band information. We in the Varnish Cache Project do not have access to that information, we dont know who runs the varnish instance or what kind of information they serve to what kind of clients. This is why the default '503 message only exposes the "XID" nonce: That allows the administrator of this cache instance to find all the details in the log files. Varnish Cache users who want to present something else can do that from VCL, and I'm pretty sure Fastly normally does. But when all else fails, and here it must have, Varnish Cache errs on the side of caution.

phkamp 5y ago
I've taken the opportunity to write up the story of the first time Varnish Cache's 503 was heard around the world, or at least in Norway: http://varnish-cache.org/docs/trunk/phk/503aroundtheworld.html http://varnish-cache.org/docs/trunk/phk/503aroundtheworld.ht...

mekster 5y ago

Quite back then in a personal project, when putting error logs, I also mandated to put what the fix is along with the error message in error log functions. I suppose no one does it.

sbadger 5y ago

More in reply to the comments than the original article.... In my experience no developer ever actually imagines that their code could possibly have bugs. Yes yes, they will say that of course all software has bugs, but deep down they know that that doesn't mean their code, just other's code. Turn all that code into products and that means when something goes wrong it isn't the product's fault. Therefore it must be the user's fault. Error messages can't possibly explain the problem, because the product can't know what the dumb user did. So don't bother really trying. This from the perspective of someone doing customer support.

dpedu 5y ago

It seems to me one of these articles pops up every time there's an outage related to Varnish. There's plenty of poor error messages out there but "Guru Mediation" really grinds folk's gears.