15 ms·

Dear Heroku: Quit blaming all of us when you fail. Do this instead…

271 points by pardner 14y ago

cpfohl 14y ago
Wow, didn't know that (don't use Heroku). Good article. Good solution.
zbowling 14y ago
It may be difficult from a platform for them to tell where the outage is exactly. Also I don't think they should tell everyone heroku is hosting it so I don't think that is a good solution.
- pardner 14y ago
  re your second point: Yeah I pondered that, and they could make the "ooops" more generic. However, we freely talk about being on Heroku since by and large it seems to engender customer confidence. And it's exactly a secret since the dns will point to proxy.heroku.com after all.
reilly3000 14y ago
They ought to create an interface for serving up a custom 500 error page.
- Snappy 14y ago
  You mean like this? https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
jtarud 14y ago
We know when Heroku is down cause our emails from client's app drown our inboxes, and our clients get pounded by their clients. I think it would be ideal to allow you to customize these messages to make things easier, but I can't imagine the infrastructure they would need to have in place to support this. The option presented by the article is lot simpler.
rsenk330 14y ago
What if the problem only affects a subset of users? Then wouldn't any application errors for unaffected users (e.g. a typo in code) say it is Heroku's fault when it really isn't?
deleted 14y ago
[deleted]
mshafrir 14y ago
Heroku has a mechanism for displaying custom error and maintenance pages, served off of S3. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
- le_isms 14y ago
  The earlier Heroku outage also brought down custom error pages. Our site was only displaying a 500 server error via nginx.
  - Snappy 14y ago
    But was it serving _your_ 500 error page? In which case you could make it say anything you want. Heroku platform errors are not 500s; they're 502s (or 503s, I forget).
michaelw 14y ago
Why do you care? Do you think this helps your customer at all? Perhaps you're worried about your technical reputation. All you're doing is moving the blame to some part of your code to the decision you made to host on Heroku. Down is down. Unavailable is unavailable. To your customers that's all that matters. For what it's worth I think hosting on Heroku makes plenty of sense and I'm actually moving my app (Crisply) to Heroku so that my technical team spends less time writing chef scripts, less time managing database clusters and more time adding value. But when Heroku goes down my customers will be just as screwed.
- deleted 14y ago
  [deleted]
- bcrescimanno 14y ago
  Well said; I'd even take it a step further and say that your customers (unless your customers are technical folks) have absolutely no idea about the distinction anyway. Saying that it's hosted on Heroku means nothing to them. Moreover, there may be companies hosting on Heroku who insist that the service is "white label" for any number of reasons; forcing something like onto them would devalue Heroku to those companies.
  - dkersten 14y ago
    unless your customers are technical folks But Heroku can't know if the customer of their customers are technical folks or not and neither can you. Some people do care, the OP for example seems to care and we can assume thats because his customers do in fact care (only the OP can decide if his customers care or not).
- doktrin 14y ago
  Frankly, I believe it certainly can matter. It may not make any difference to all clients, but some will draw a distinction between avoidable / unavoidable downtime. Rational clients understand they are not hiring demigods or living in some uptime utopia. However, they expect their service providers to adhere to best practices and make logical decisions. In this case it's Heroku, in another setting it might be a NetApp filer. Both are reasonable solutions in their appropriate environment, yet both may fail. There's a significant difference between downtime caused by the failure of a reasonably trusted solution as opposed to that due to design flaws, bleeding edge prototyping and flat out bad decisions. The former wouldn't undermine my faith in a service provider, while the latter certainly might.
Maro 14y ago
I don't think the OP is right in general. What if your customer sees Heroku's name, and gets confused? She starts asking questions like: Who is on the other end? Am I in business with X or with Heroku? Who should i call?
- glesica 14y ago
  This is actually a really good point. I can't find it now, but there's an article floating around somewhere about a government employee in some small town accusing Apache or Debian or some project of being "hackers" because his web server broke and was showing the default "congrats it works!" page instead of the town web site.
  - jsprinkles 14y ago
    That's not an isolated occurrence. Most Web projects get it all the time from randoms, sometimes completely unaffiliated with the site in question.
  - jrockway 14y ago
    That's why Apache's default page is now <h1>It works!</h1>.
  - kingofspain 14y ago
    An ex-boss accused me of hacking and embezzlement(!) when he discovered he was paying £10/month for DNS services a few years back. It was something in place from years before I even started! He also flipped out over firewall issues on his new Macbook (the site is down and taken the internet with it!) and problems with his ISP (some weird adult filter). Point is, some people expect a site to be a single atomic thing with no dependencies or connections between. Through anything "unusual" in there and confusion and chaos reigns. My ex-boss was somewhat extreme I admit, but I've experienced it to a lesser degree many many times.
- 46Bit 14y ago
  Just don't mention Heroku by name. "Webhost" or something is enough. I'm in two minds about the whole thing though.
ultrasaurus 14y ago
If we put our normal-user-hat on for a minute: I don't see how that would make any difference for 90%+ of users. They'll see the website isn't working, but another site is, so your site is broken. End of story. (With my developer hat on, Heroku outages are fun: our internal switchboard at http://www.pagerduty.com http://www.pagerduty.com lights up like a christmas tree)
neilmiddleton 14y ago
The only change I would make, if any, is to remove the sentence about being the application owner. Aside from that, that's all I'm going to tell a customer anyway.
famousactress 14y ago
Bullshit. It's your fault. I'm your user, you took my money. We're done here. Everything is your fault.
- pardner 14y ago
  Actually I do not disagree.... I make the platform choices, so i live with the results. However, IMO an error message that distinguishes between an "hosting" issue and an "app" issue is not only fair, it is in fact, meaningful data to (at least some) customers.
  - famousactress 14y ago
    I'm not sure about that either. If it was my product I think I'd want a polite on-brand 500 page saying my hamsters are working on it. Seeing something that suggests hosting is down doesn't frankly matter to me, even as a technical user... and to someone in between myself and my mom, with enough knowledge to understand what a "hosting company" is... I wonder if they might feel ripped off as well: "Oh great. I gave this company my money and they don't even have their own servers!"
  - neilmiddleton 14y ago
    Why, do your users care?
    - daenz 14y ago
      You might be providing a service to people who would like to know what part of the chain broke. If hosting company A fails all the time, and that's visible to me as a technical user of service A, I would avoid hosting company A for anything that I happen to host. I'm not the average end user, but that information is still valuable to me, and I would think less of hosting company A, instead of service A. By Heroku not listing when its their downtime, they are insulating their reputation as a hosting company from end users, at the expense of the customers already using them. It's a little shady. I agree that the average end user would probably not care, but most not caring does not mean it's not valuable information to some people. So I see where the original poster is coming from.
duck 14y ago
I don't want my users to know I use Heroku, and by using Heroku I understand that if they go down my site goes down. It really is "our" problem at that point with regards to how our users understand it.
fleitz 14y ago
Add a custom 500 error page, problem sovled, you can make it say anything you want.
chao- 14y ago
As someone who was inconvenienced by the outage, and with no mitigation strategy in place, I DON'T blame Heroku. The weight is placed squarely on me (lone tech in our company) for not having researched how to distribute services alongside Heroku, or fall back to something else, or whatever the proper term is. I've been googling like mad since this morning, finding a few mostly-unanswered StackOverflow questions and a smattering of blog posts, but I haven't learned much. The only clear-cut answers I've seen are: 1. Hire a sysadmin who knows more than you do (But whole point is that I want to learn for myself!). 2. Pay for a service that will host in multiple geographic locations for you, and do the switchover (recovery? fallback? I don't know my terms here) for you. 3. A few mentions of "load balancers" and "heartbeat monitors". Sounds self-explanatory, and these are my current terms of googling. Any suggestions on where to start acquiring this sort of skill? I'm prepared to teach myself anything, but the problem is not knowing the terms for what I want to learn. EDIT: Well, just watching this thread is helping a bit.
- gaius 14y ago
  Even vendors that promise that, e.g. Amazon, aren't infallible.
  - chao- 14y ago
    The gist that I'm getting from a few places seems to be: Have separate hosts/service providers, and do the load balancing yourself, or make switch yourself, when one fails. I've yet to find many detailed examples, though, as most similar articles deal with load balancing with in your own locally-managed network, or co-located set of machines. The more generalized "Cloud plus Dedicated" fallback/load balancing seems fairly involved, and raises a lot of other questions, but at least I've got a path to follow now. Also would be more expensive, as a backup server might just be hanging around doing nothing at times. Then again, it would pay for itself in satisfied customers after just a single event.
- rdl 14y ago
  The cost of running your own infrastructure at this level is slower development and ongoing hassle, vs. Heroku. Unless your application absolutely must be up with higher availability than Heroku provides, it's probably not worth the effort. The easiest thing to do is to use something like Cloudflare in front of Heroku, so at least when Heroku is down, you can serve a static page to customers informing them of the problem and estimated time to fix.
arihant 14y ago
You can customize the error page. It is your fault for not reading the docs and serving the default. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
iamleppert 14y ago
It’s your fault for not having a fault tolerant site that runs on another service provider. This is what happens when you put your eggs in one basket and that basket bursts into flames. If reliability is so important, make it a priority instead of just expecting stuff to work or for a more politically correct error message — which leads me to my next point: who cares about the ERROR message? The damage has been done by that point and half the people won't bother to read any further. Queue sounds of people clicking back buttons as fast as they can.
- slurgfest 14y ago
  Question: given that Heroku involves a certain amount of platform lock-in, how do you write a Heroku app that runs on another service provider? It hardly makes sense if Heroku says "well, it's your fault for trusting us."
abscondment 14y ago
This seems specious. Correctly assigning blame won't matter to readers; most people couldn't care less. Sticking a different brand name on the failure is side-stepping the issue: nothing stays up 100% of the time. Create a custom page that treats the situation with a little bit of levity. If legitimate downtime happens often enough that someone would actually internalize the difference between your failures and Heroku's, you have bigger problems than your error page.
halayli 14y ago
Isn't it your fault to have picked Heroku in this case?
- slurgfest 14y ago
  Yes, it is your fault for trusting what Heroku says about its availability. But it would be classy for Heroku to take responsibility. It is not classy for Heroku to say "it's your fault because you trusted us" in front of the users, which seems to be the principal defense of Heroku in these comments.
gaius 14y ago
Does Heroku not have custom ErrorDocumenrs? We had those in the 90's...
ynniv 14y ago
Quite off topic, but I'm always sad to see really poor scalability: To my surprise, this blog post hit the top spot on HN at least briefly. My blog started throwing some app errors. I've had a couple of hit HN stories on my blog without a problem, and it was hosted from my apartment on an old server with 256MB of RAM. Now, it is static pages served through nginx, but I'm pretty sure that a few thousand hits shouldn't require 10 Heroku dynos to not fall over. Kids these days. (the mindset, not the age)
- pardner 14y ago
  I doubt it requires anywhere near 10 dynos, normally I just run 1, and I suspect 2 or 3 dynos would work fine today... but since 10 dynos only costs 45 cents per hour, and it's presumably just for an hour or two, I simply threw two handfuls at the problem and went back to my actual work. Handling a one-time spike didn't seem like something worth optimizing when I could throw the price of a cup of coffee at the problem.
  - ynniv 14y ago
    Ah, my fault for not recognizing pragmatism at work. Rare events are definitely not worth optimizing for.
deleted 14y ago
[deleted]
dllthomas 14y ago
If Heroku is down, and I discover that site X that I want to visit was hosted on Heroku, I'm more likely to hear that Heroku is back up or that site X is back up, than just that site X is back up. I also can skip checking any other sites I know to be using Heroku during the outage. It is therefore mildly useful data to a user.
starrhorne 14y ago
Heroku isn't for apps that can't stand downtime. My experience has been that if you have 2-3 heroku apps, and you monitor them with a 3rd party tool, you'll see random "server not found" behavior every few weeks. (And no, they're not just timing out from dyno spinup). Usually this isn't a system wide outage and never gets mentioned on their status page. So only use heroku if: a) Uptime is non-critical & you just don't want to deal with setting up a server b) Uptime is non-critical & You don't know how to set up a server
sunkencity 14y ago
All this ruckus for 18 mins of downtime? Moved my main app off heroku this monday for various reasons (mainly to get better log access and to run the app in europe).
balloot 14y ago
This is misguided. Nobody cares why your site is down, and for most sites 99% of your users will have no clue what is meant by "This site is hosted by Heroku". And a good chunk of that other 1% isn't even going to bother reading the error text accompanying the whitescreen. In the end, you chose to host your site on a platform that went down. That is just as much your fault as a typo in the code. If you had a setup with a hosted machine at Rackspace and the power goes out, you don't expect a custom error. So why would you expect one from Heroku?
- smokeyj 14y ago
  My only gripe with the heroku app error page is it doesn't show your companies branding. I would like to be able to upload a static fail page with a generic message for my customers. Heck, let me specify a URL to redirect to when Heroku crashes.
  - bapbap 14y ago
    This might help: https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
    - iamgilesbowkett 14y ago
      response from the OP: "Thanks, yes, but when the platform itself fails it seems that doesn’t always work. Hence my suggestion they temporarily change the default error message."
  - Goladus 14y ago
    That was my first through as well. I was thinking "why is this author demanding that Heroku put their brand on his app's error messages?"
    - deleted 14y ago
      [deleted]
- 14y ago
DanaDanger 14y ago
A gentle reminder: http://whoownsmyavailability.com/ http://whoownsmyavailability.com/
seanp2k2 14y ago
"throw more dynos at it" You kids and your lingo these days.... /back in my day/ we used to have servers. REAL, PHYSICAL servers :)
pearkes 14y ago
You can customize your error pages to be whatever you want. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
- slurgfest 14y ago
  So the solution is for you to run a script which monitors Heroku for outages and changes the error page? If you are doing that, you might as well write an app against another platform.
deleted 14y ago
[deleted]
URSpider94 14y ago
Many companies I know would immediately fire a service provider for ever disclosing their existence to an end customer. If anything, Heroku's customers should be able to replace the default error message such that it conforms to the the customer's site branding.
notatoad 14y ago
From a customer's perspective, there are only two parties in their relationship with you: you, and them. When something goes wrong with your application, you either accept the blame, or you make the customers feel like they broke something. To the average user, seeing an error message like "heroku is down" (or any other jargon) leaves the possibility that they might have broken something, and the failure is on their end. The end result of this interaction is that your software has made your user feel bad about themselves. This is not a way to get your users to return to you. Heroku's error message could be friendlier, but it currently contains only words that any user can understand, which reassures your customers that even though the service they are looking for is unavailable, there is nothing they could have done to improve the situation. Your customers might leave with a lowered opinion of your service, but your app doesn't make them feel ashamed of themselves, which is a much better outcome.
peterkelly 14y ago
I think that for completeness, it should display a complete blame derivation graph that explains to the user the full chain of events, right back to the original person who was ultimately responsible. After all, it wouldn't be fair for Heroku to be blamed just because a piece of networking equipment failed - the user should be informed which vendor is at fault, and in turn, which supplier the failed component within said equipment came from.
overworkedasian 14y ago
at the end of the day, if you have a service that other people/businesses/clients rely on, that they need 24/7 up time, then you really need to have a plan B that is not on heroku or aws. a REAL disaster recovery plan needs to be thought out and implemented. if you dont want your users to see the "there is a problem with this app" on heroku, then its your job to figure out that plan B is. If you cant afford it a plan B, then well, tough shits. as someone that has worked in the hosting business for years on the operations side, its also the responsibility of the client to plan that scenario where your primary host is not reachable (regardless if its an application level issue, network or power outage). the hosting company can only build so many N+1 backups (network/power/etc) as they can afford/physically fit. you can buy all the load balancing you want, redundant web servers and database servers. if you arent hosting in a secondary place and your primary host fails, all those redundant servers you are paying for arent going to mean a damn thing.
awicklander 14y ago
Or you could get on EngineYard and stop caring what Heroku does. That's worked wonders for my business.
awicklander 14y ago
Or you could get on Engine Yard and stop caring what Heroku does. That's worked wonders for my business.
Tomis02 14y ago
This is what a culture of pointing fingers leads to. The author should realize the customer does not care why the site is down.
16s 14y ago
I've seen so much blame in IT/systems/coding in the last 15 years that I can't recount it all. Anytime a vendor or service provider or consultant is involved, get ready for finger pointing when things go wrong (from both sides). I think many managers like being able to blame them and see this as a benefit of the relationship. Outside providers should just expect to be blamed for things they did not do and charge for that accordingly.

271 points by pardner 14y ago

Wow, didn't know that (don't use Heroku). Good article. Good solution.

It may be difficult from a platform for them to tell where the outage is exactly. Also I don't think they should tell everyone heroku is hosting it so I don't think that is a good solution.

pardner 14y ago
re your second point: Yeah I pondered that, and they could make the "ooops" more generic. However, we freely talk about being on Heroku since by and large it seems to engender customer confidence. And it's exactly a secret since the dns will point to proxy.heroku.com after all.

reilly3000 14y ago

They ought to create an interface for serving up a custom 500 error page.

Snappy 14y ago
You mean like this? https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...

jtarud 14y ago

We know when Heroku is down cause our emails from client's app drown our inboxes, and our clients get pounded by their clients. I think it would be ideal to allow you to customize these messages to make things easier, but I can't imagine the infrastructure they would need to have in place to support this. The option presented by the article is lot simpler.

rsenk330 14y ago

What if the problem only affects a subset of users? Then wouldn't any application errors for unaffected users (e.g. a typo in code) say it is Heroku's fault when it really isn't?

deleted 14y ago

[deleted]

mshafrir 14y ago

Heroku has a mechanism for displaying custom error and maintenance pages, served off of S3. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...

le_isms 14y ago
The earlier Heroku outage also brought down custom error pages. Our site was only displaying a 500 server error via nginx.
- Snappy 14y ago
  But was it serving _your_ 500 error page? In which case you could make it say anything you want. Heroku platform errors are not 500s; they're 502s (or 503s, I forget).

michaelw 14y ago

Why do you care? Do you think this helps your customer at all? Perhaps you're worried about your technical reputation. All you're doing is moving the blame to some part of your code to the decision you made to host on Heroku. Down is down. Unavailable is unavailable. To your customers that's all that matters. For what it's worth I think hosting on Heroku makes plenty of sense and I'm actually moving my app (Crisply) to Heroku so that my technical team spends less time writing chef scripts, less time managing database clusters and more time adding value. But when Heroku goes down my customers will be just as screwed.

deleted 14y ago
[deleted]
bcrescimanno 14y ago
Well said; I'd even take it a step further and say that your customers (unless your customers are technical folks) have absolutely no idea about the distinction anyway. Saying that it's hosted on Heroku means nothing to them. Moreover, there may be companies hosting on Heroku who insist that the service is "white label" for any number of reasons; forcing something like onto them would devalue Heroku to those companies.
- dkersten 14y ago
  unless your customers are technical folks But Heroku can't know if the customer of their customers are technical folks or not and neither can you. Some people do care, the OP for example seems to care and we can assume thats because his customers do in fact care (only the OP can decide if his customers care or not).
doktrin 14y ago
Frankly, I believe it certainly can matter. It may not make any difference to all clients, but some will draw a distinction between avoidable / unavoidable downtime. Rational clients understand they are not hiring demigods or living in some uptime utopia. However, they expect their service providers to adhere to best practices and make logical decisions. In this case it's Heroku, in another setting it might be a NetApp filer. Both are reasonable solutions in their appropriate environment, yet both may fail. There's a significant difference between downtime caused by the failure of a reasonably trusted solution as opposed to that due to design flaws, bleeding edge prototyping and flat out bad decisions. The former wouldn't undermine my faith in a service provider, while the latter certainly might.

Maro 14y ago

I don't think the OP is right in general. What if your customer sees Heroku's name, and gets confused? She starts asking questions like: Who is on the other end? Am I in business with X or with Heroku? Who should i call?

glesica 14y ago
This is actually a really good point. I can't find it now, but there's an article floating around somewhere about a government employee in some small town accusing Apache or Debian or some project of being "hackers" because his web server broke and was showing the default "congrats it works!" page instead of the town web site.
- jsprinkles 14y ago
  That's not an isolated occurrence. Most Web projects get it all the time from randoms, sometimes completely unaffiliated with the site in question.
- jrockway 14y ago
  That's why Apache's default page is now <h1>It works!</h1>.
- kingofspain 14y ago
  An ex-boss accused me of hacking and embezzlement(!) when he discovered he was paying £10/month for DNS services a few years back. It was something in place from years before I even started! He also flipped out over firewall issues on his new Macbook (the site is down and taken the internet with it!) and problems with his ISP (some weird adult filter). Point is, some people expect a site to be a single atomic thing with no dependencies or connections between. Through anything "unusual" in there and confusion and chaos reigns. My ex-boss was somewhat extreme I admit, but I've experienced it to a lesser degree many many times.
46Bit 14y ago
Just don't mention Heroku by name. "Webhost" or something is enough. I'm in two minds about the whole thing though.

ultrasaurus 14y ago

If we put our normal-user-hat on for a minute: I don't see how that would make any difference for 90%+ of users. They'll see the website isn't working, but another site is, so your site is broken. End of story. (With my developer hat on, Heroku outages are fun: our internal switchboard at http://www.pagerduty.com http://www.pagerduty.com lights up like a christmas tree)

neilmiddleton 14y ago

The only change I would make, if any, is to remove the sentence about being the application owner. Aside from that, that's all I'm going to tell a customer anyway.

famousactress 14y ago

Bullshit. It's your fault. I'm your user, you took my money. We're done here. Everything is your fault.

pardner 14y ago
Actually I do not disagree.... I make the platform choices, so i live with the results. However, IMO an error message that distinguishes between an "hosting" issue and an "app" issue is not only fair, it is in fact, meaningful data to (at least some) customers.
- famousactress 14y ago
  I'm not sure about that either. If it was my product I think I'd want a polite on-brand 500 page saying my hamsters are working on it. Seeing something that suggests hosting is down doesn't frankly matter to me, even as a technical user... and to someone in between myself and my mom, with enough knowledge to understand what a "hosting company" is... I wonder if they might feel ripped off as well: "Oh great. I gave this company my money and they don't even have their own servers!"
- neilmiddleton 14y ago
  Why, do your users care?
  - daenz 14y ago
    You might be providing a service to people who would like to know what part of the chain broke. If hosting company A fails all the time, and that's visible to me as a technical user of service A, I would avoid hosting company A for anything that I happen to host. I'm not the average end user, but that information is still valuable to me, and I would think less of hosting company A, instead of service A. By Heroku not listing when its their downtime, they are insulating their reputation as a hosting company from end users, at the expense of the customers already using them. It's a little shady. I agree that the average end user would probably not care, but most not caring does not mean it's not valuable information to some people. So I see where the original poster is coming from.

duck 14y ago

I don't want my users to know I use Heroku, and by using Heroku I understand that if they go down my site goes down. It really is "our" problem at that point with regards to how our users understand it.

fleitz 14y ago

Add a custom 500 error page, problem sovled, you can make it say anything you want.

chao- 14y ago

As someone who was inconvenienced by the outage, and with no mitigation strategy in place, I DON'T blame Heroku. The weight is placed squarely on me (lone tech in our company) for not having researched how to distribute services alongside Heroku, or fall back to something else, or whatever the proper term is. I've been googling like mad since this morning, finding a few mostly-unanswered StackOverflow questions and a smattering of blog posts, but I haven't learned much. The only clear-cut answers I've seen are: 1. Hire a sysadmin who knows more than you do (But whole point is that I want to learn for myself!). 2. Pay for a service that will host in multiple geographic locations for you, and do the switchover (recovery? fallback? I don't know my terms here) for you. 3. A few mentions of "load balancers" and "heartbeat monitors". Sounds self-explanatory, and these are my current terms of googling. Any suggestions on where to start acquiring this sort of skill? I'm prepared to teach myself anything, but the problem is not knowing the terms for what I want to learn. EDIT: Well, just watching this thread is helping a bit.

gaius 14y ago
Even vendors that promise that, e.g. Amazon, aren't infallible.
- chao- 14y ago
  The gist that I'm getting from a few places seems to be: Have separate hosts/service providers, and do the load balancing yourself, or make switch yourself, when one fails. I've yet to find many detailed examples, though, as most similar articles deal with load balancing with in your own locally-managed network, or co-located set of machines. The more generalized "Cloud plus Dedicated" fallback/load balancing seems fairly involved, and raises a lot of other questions, but at least I've got a path to follow now. Also would be more expensive, as a backup server might just be hanging around doing nothing at times. Then again, it would pay for itself in satisfied customers after just a single event.
rdl 14y ago
The cost of running your own infrastructure at this level is slower development and ongoing hassle, vs. Heroku. Unless your application absolutely must be up with higher availability than Heroku provides, it's probably not worth the effort. The easiest thing to do is to use something like Cloudflare in front of Heroku, so at least when Heroku is down, you can serve a static page to customers informing them of the problem and estimated time to fix.

arihant 14y ago

You can customize the error page. It is your fault for not reading the docs and serving the default. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...

iamleppert 14y ago

It’s your fault for not having a fault tolerant site that runs on another service provider. This is what happens when you put your eggs in one basket and that basket bursts into flames. If reliability is so important, make it a priority instead of just expecting stuff to work or for a more politically correct error message — which leads me to my next point: who cares about the ERROR message? The damage has been done by that point and half the people won't bother to read any further. Queue sounds of people clicking back buttons as fast as they can.

slurgfest 14y ago
Question: given that Heroku involves a certain amount of platform lock-in, how do you write a Heroku app that runs on another service provider? It hardly makes sense if Heroku says "well, it's your fault for trusting us."

abscondment 14y ago

This seems specious. Correctly assigning blame won't matter to readers; most people couldn't care less. Sticking a different brand name on the failure is side-stepping the issue: nothing stays up 100% of the time. Create a custom page that treats the situation with a little bit of levity. If legitimate downtime happens often enough that someone would actually internalize the difference between your failures and Heroku's, you have bigger problems than your error page.

halayli 14y ago

Isn't it your fault to have picked Heroku in this case?

slurgfest 14y ago
Yes, it is your fault for trusting what Heroku says about its availability. But it would be classy for Heroku to take responsibility. It is not classy for Heroku to say "it's your fault because you trusted us" in front of the users, which seems to be the principal defense of Heroku in these comments.

gaius 14y ago

Does Heroku not have custom ErrorDocumenrs? We had those in the 90's...

ynniv 14y ago

Quite off topic, but I'm always sad to see really poor scalability: To my surprise, this blog post hit the top spot on HN at least briefly. My blog started throwing some app errors. I've had a couple of hit HN stories on my blog without a problem, and it was hosted from my apartment on an old server with 256MB of RAM. Now, it is static pages served through nginx, but I'm pretty sure that a few thousand hits shouldn't require 10 Heroku dynos to not fall over. Kids these days. (the mindset, not the age)

pardner 14y ago
I doubt it requires anywhere near 10 dynos, normally I just run 1, and I suspect 2 or 3 dynos would work fine today... but since 10 dynos only costs 45 cents per hour, and it's presumably just for an hour or two, I simply threw two handfuls at the problem and went back to my actual work. Handling a one-time spike didn't seem like something worth optimizing when I could throw the price of a cup of coffee at the problem.
- ynniv 14y ago
  Ah, my fault for not recognizing pragmatism at work. Rare events are definitely not worth optimizing for.

deleted 14y ago

[deleted]

dllthomas 14y ago

If Heroku is down, and I discover that site X that I want to visit was hosted on Heroku, I'm more likely to hear that Heroku is back up or that site X is back up, than just that site X is back up. I also can skip checking any other sites I know to be using Heroku during the outage. It is therefore mildly useful data to a user.

starrhorne 14y ago

Heroku isn't for apps that can't stand downtime. My experience has been that if you have 2-3 heroku apps, and you monitor them with a 3rd party tool, you'll see random "server not found" behavior every few weeks. (And no, they're not just timing out from dyno spinup). Usually this isn't a system wide outage and never gets mentioned on their status page. So only use heroku if: a) Uptime is non-critical & you just don't want to deal with setting up a server b) Uptime is non-critical & You don't know how to set up a server

sunkencity 14y ago

All this ruckus for 18 mins of downtime? Moved my main app off heroku this monday for various reasons (mainly to get better log access and to run the app in europe).

balloot 14y ago

This is misguided. Nobody cares why your site is down, and for most sites 99% of your users will have no clue what is meant by "This site is hosted by Heroku". And a good chunk of that other 1% isn't even going to bother reading the error text accompanying the whitescreen. In the end, you chose to host your site on a platform that went down. That is just as much your fault as a typo in the code. If you had a setup with a hosted machine at Rackspace and the power goes out, you don't expect a custom error. So why would you expect one from Heroku?

smokeyj 14y ago
My only gripe with the heroku app error page is it doesn't show your companies branding. I would like to be able to upload a static fail page with a generic message for my customers. Heck, let me specify a URL to redirect to when Heroku crashes.
- bapbap 14y ago
  This might help: https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...
  - iamgilesbowkett 14y ago
    response from the OP: "Thanks, yes, but when the platform itself fails it seems that doesn’t always work. Hence my suggestion they temporarily change the default error message."
- Goladus 14y ago
  That was my first through as well. I was thinking "why is this author demanding that Heroku put their brand on his app's error messages?"
  - deleted 14y ago
    [deleted]
14y ago

DanaDanger 14y ago

A gentle reminder: http://whoownsmyavailability.com/ http://whoownsmyavailability.com/

seanp2k2 14y ago

"throw more dynos at it" You kids and your lingo these days.... /back in my day/ we used to have servers. REAL, PHYSICAL servers :)

pearkes 14y ago

You can customize your error pages to be whatever you want. https://devcenter.heroku.com/articles/error-pages#customize_pages https://devcenter.heroku.com/articles/error-pages#customize_...

slurgfest 14y ago
So the solution is for you to run a script which monitors Heroku for outages and changes the error page? If you are doing that, you might as well write an app against another platform.

deleted 14y ago

[deleted]

URSpider94 14y ago

Many companies I know would immediately fire a service provider for ever disclosing their existence to an end customer. If anything, Heroku's customers should be able to replace the default error message such that it conforms to the the customer's site branding.

notatoad 14y ago

From a customer's perspective, there are only two parties in their relationship with you: you, and them. When something goes wrong with your application, you either accept the blame, or you make the customers feel like they broke something. To the average user, seeing an error message like "heroku is down" (or any other jargon) leaves the possibility that they might have broken something, and the failure is on their end. The end result of this interaction is that your software has made your user feel bad about themselves. This is not a way to get your users to return to you. Heroku's error message could be friendlier, but it currently contains only words that any user can understand, which reassures your customers that even though the service they are looking for is unavailable, there is nothing they could have done to improve the situation. Your customers might leave with a lowered opinion of your service, but your app doesn't make them feel ashamed of themselves, which is a much better outcome.

peterkelly 14y ago

I think that for completeness, it should display a complete blame derivation graph that explains to the user the full chain of events, right back to the original person who was ultimately responsible. After all, it wouldn't be fair for Heroku to be blamed just because a piece of networking equipment failed - the user should be informed which vendor is at fault, and in turn, which supplier the failed component within said equipment came from.

overworkedasian 14y ago

at the end of the day, if you have a service that other people/businesses/clients rely on, that they need 24/7 up time, then you really need to have a plan B that is not on heroku or aws. a REAL disaster recovery plan needs to be thought out and implemented. if you dont want your users to see the "there is a problem with this app" on heroku, then its your job to figure out that plan B is. If you cant afford it a plan B, then well, tough shits. as someone that has worked in the hosting business for years on the operations side, its also the responsibility of the client to plan that scenario where your primary host is not reachable (regardless if its an application level issue, network or power outage). the hosting company can only build so many N+1 backups (network/power/etc) as they can afford/physically fit. you can buy all the load balancing you want, redundant web servers and database servers. if you arent hosting in a secondary place and your primary host fails, all those redundant servers you are paying for arent going to mean a damn thing.

awicklander 14y ago

Or you could get on EngineYard and stop caring what Heroku does. That's worked wonders for my business.

awicklander 14y ago

Or you could get on Engine Yard and stop caring what Heroku does. That's worked wonders for my business.

Tomis02 14y ago

This is what a culture of pointing fingers leads to. The author should realize the customer does not care why the site is down.

16s 14y ago

I've seen so much blame in IT/systems/coding in the last 15 years that I can't recount it all. Anytime a vendor or service provider or consultant is involved, get ready for finger pointing when things go wrong (from both sides). I think many managers like being able to blame them and see this as a benefit of the relationship. Outside providers should just expect to be blamed for things they did not do and charge for that accordingly.