7 ms·

General question here about upstream outages that can take down your site, e.g. DNS outage, AWS zone outage, etc. What to tell your customers when an upstream

General question here about upstream outages that can take down your site, e.g. DNS outage, AWS zone outage, etc. What to tell your customers when an upstream service provider experiences an outage? I mean, if you're running ifttt.com your users might be savvy enough to understand that a DNS outage isn't your fault; but pinterest.com or whatever (painting with broad strokes here, forgive me) might not have a user base that would understand that events out of your control have made your site inaccessible. How do you reassure your customers? What's the proper tone to take?

EvanAnderson 14y ago
I don't think there's a tone you can take that will make non-technical users understand that it's "not your fault". To the non-technical user your site is just down and, potentially, a service they're paying for is unavailable. I'd argue that such an outage _is_ your fault. If you're worried about Customer perceptions as a result of outages of third-party services your site relies on then, I'd argue, you need to have redundancy in your choice of third-party services. If it's important enough to worry about it should be important enough to spend some money on and do something about. If that drives up your costs then your product's cost, to your Customers, needs to reflect that.
whafro 14y ago
Good question. I believe in explaining as clearly as possible, but ultimately in taking responsibility. You chose the provider, decided on the level of risk you're comfortable with, and that ultimately failed. It was that choice that got in the way of your users. If you can point to ways you'll improve the service in the future as a result of the outage, all the better. When we had a large DoS attack at Posterous, I wrote two posts, one as soon as possible (http://blog.posterous.com/todays-outage-and-changes-for-custom-domains http://blog.posterous.com/todays-outage-and-changes-for-cust...), and the next as a bit of a post-mortem (http://blog.posterous.com/moving-forward http://blog.posterous.com/moving-forward). Both explained that there were many factors beyond our control, but that the responsibility was ultimately ours, and we were working to learn from the event and improve our services as a result. They weren't perfect posts, but I think they went a long way toward being open and honest with our users in the midst of a major negative event.
thaumaturgy 14y ago
At the risk of professing an attitude that I generally dislike: there's no proper tone to take other than full responsibility. Like Steve Jobs said, when you're the janitor, reasons matter. Once you're a service provider -- whether your services in turn rely on other services or not -- reasons stop mattering. As a practical matter, you and I know that there's no way you can build a fully scalable, fully redundant infrastructure from the ground-up in your first week. Hell, if you ever build that kind of infrastructure at all, you'll be way ahead of most companies. But, that's the kind of infrastructure you should be working towards building, all the time. You should have a clear roadmap for ensuring data integrity, then dealing with security, then dealing with redundancy, and finally high availability. If your service falls over for any reason, ultimately it's because you haven't done something on your roadmap yet. There's no way to explain that to your customers that doesn't sound like you're trying to pass the fault on to someone else -- because that's exactly what you're doing. So just 'fess up to your customers: "one of the services that our business relies on had some serious technical problems that affected us today, but we recognize that ultimately it's our responsibility to make sure that our service is always available to you. We're constantly working on our infrastructure to make it more reliable, but we clearly still have more work to do. We will be changing some of our priorities so that this won't be a problem in the future. Thank you for sticking with us." (And then do it, otherwise this will backfire on you the next time you have an outage of similar cause.) As an aside: I generally take a softer stance towards user responsibilities -- of course everyone should have backups, but Joe Schmoe just doesn't have time for that -- but a much harder stance towards businesses. Once you accept money from someone, you put yourself into a position of absolute responsibility for whatever it is that people rely on you for. If you can't guarantee the availability of your service or the safety of their data, then you shouldn't be taking their money.
- quicksilver03 14y ago
  Interesting, at the same time I both agree and disagree with what you wrote. You're saying that you want to offer 100% uptime, but how much does that cost? A client paying 5 USD/month has the same uptime expectancy as another one paying 2000 USD/month?
  - 14y ago

EvanAnderson 14y ago

I don't think there's a tone you can take that will make non-technical users understand that it's "not your fault". To the non-technical user your site is just down and, potentially, a service they're paying for is unavailable. I'd argue that such an outage _is_ your fault. If you're worried about Customer perceptions as a result of outages of third-party services your site relies on then, I'd argue, you need to have redundancy in your choice of third-party services. If it's important enough to worry about it should be important enough to spend some money on and do something about. If that drives up your costs then your product's cost, to your Customers, needs to reflect that.

whafro 14y ago

Good question. I believe in explaining as clearly as possible, but ultimately in taking responsibility. You chose the provider, decided on the level of risk you're comfortable with, and that ultimately failed. It was that choice that got in the way of your users. If you can point to ways you'll improve the service in the future as a result of the outage, all the better. When we had a large DoS attack at Posterous, I wrote two posts, one as soon as possible (http://blog.posterous.com/todays-outage-and-changes-for-custom-domains http://blog.posterous.com/todays-outage-and-changes-for-cust...), and the next as a bit of a post-mortem (http://blog.posterous.com/moving-forward http://blog.posterous.com/moving-forward). Both explained that there were many factors beyond our control, but that the responsibility was ultimately ours, and we were working to learn from the event and improve our services as a result. They weren't perfect posts, but I think they went a long way toward being open and honest with our users in the midst of a major negative event.

thaumaturgy 14y ago

At the risk of professing an attitude that I generally dislike: there's no proper tone to take other than full responsibility. Like Steve Jobs said, when you're the janitor, reasons matter. Once you're a service provider -- whether your services in turn rely on other services or not -- reasons stop mattering. As a practical matter, you and I know that there's no way you can build a fully scalable, fully redundant infrastructure from the ground-up in your first week. Hell, if you ever build that kind of infrastructure at all, you'll be way ahead of most companies. But, that's the kind of infrastructure you should be working towards building, all the time. You should have a clear roadmap for ensuring data integrity, then dealing with security, then dealing with redundancy, and finally high availability. If your service falls over for any reason, ultimately it's because you haven't done something on your roadmap yet. There's no way to explain that to your customers that doesn't sound like you're trying to pass the fault on to someone else -- because that's exactly what you're doing. So just 'fess up to your customers: "one of the services that our business relies on had some serious technical problems that affected us today, but we recognize that ultimately it's our responsibility to make sure that our service is always available to you. We're constantly working on our infrastructure to make it more reliable, but we clearly still have more work to do. We will be changing some of our priorities so that this won't be a problem in the future. Thank you for sticking with us." (And then do it, otherwise this will backfire on you the next time you have an outage of similar cause.) As an aside: I generally take a softer stance towards user responsibilities -- of course everyone should have backups, but Joe Schmoe just doesn't have time for that -- but a much harder stance towards businesses. Once you accept money from someone, you put yourself into a position of absolute responsibility for whatever it is that people rely on you for. If you can't guarantee the availability of your service or the safety of their data, then you shouldn't be taking their money.

quicksilver03 14y ago
Interesting, at the same time I both agree and disagree with what you wrote. You're saying that you want to offer 100% uptime, but how much does that cost? A client paying 5 USD/month has the same uptime expectancy as another one paying 2000 USD/month?
- 14y ago