6 ms·
Heroku is down
- sohn 14y agoThe cloud stinks
- trebuch3t 14y agoI was going to get details from status.heroku.com but that's down too.
- wilfra 14y agoIt is up for me: Potential Platform Issues Jun 7, 2012 15:55 UTC - Update We have confirmed widespread errors on the platform. Our engineers are continuing to investigate. Posted Jun 7, 2012 15:58 UTC Issue Our automated systems have detected potential platform errors. We are investigating. Posted Jun 7, 2012 15:55 UTC
- skram 14y agostatus.heroku.com is up for me. My apps aren't though.
- pilap82 14y agoit's up sometimes :) Potential Platform Issues 5m+ We have confirmed widespread errors on the platform. Our engineers are continuing to investigate. this is the message for Production and Development
- AntonTrollback 14y agohttp://www.nooooooooooooooo.com/ http://www.nooooooooooooooo.com/
- jboggan 14y agoWhat's their usual outage duration? I'm supposed to demo a project in a few hours.
- wilfra 14y agoThey keep a historical archive of incidents at https://status.heroku.com/past https://status.heroku.com/past Last time I can recall this happening to us it was down for several hours.
- jboggan 14y ago502 Bad Gateway . . . is there a page that displays the status of the status page?
- deleted 14y ago[deleted]
- sakai 14y agoQuestion for the more experienced: does that seem like a LOT of issues recently, or are they simply being presented at a more granular level than for other similar services?
- garindra 14y agoStatus update on https://status.heroku.com/incidents/372 https://status.heroku.com/incidents/372 : "We have confirmed widespread errors on the platform. Our engineers are continuing to investigate."
- jt2190 14y agoAlso can follow @herokustatus on twitter: https://twitter.com/#!/herokustatus https://twitter.com/#!/herokustatus Edit: Also IRC #heroku Edit 2: heh... I just noticed the "subscribe to notifiations" link on the incident page: https://status.heroku.com/incidents/372 https://status.heroku.com/incidents/372
- splatcollision 14y agoHeroku: Awesome when it's up, but when it's down, it's really really down...
- andQ 14y agohttps://status-old.heroku.com/ https://status-old.heroku.com/ seems to work
- mikejarema 14y agoInteresting to see that www.heroku.com runs on their own platform, and is just as susceptible to platform downtime. On the other hand its surprising to me that when heroku goes down, it goes down hard, namely that all hosted sites including their own are unavailable.
- mjackson 14y agoTotally agree with this assessment. I don't know how their routing mesh works exactly, but it seems that they're coupled too closely. Either everybody is mostly working fine, or everybody's down.
- asparagui 14y agoand it's back up.... just a glitch in the matrix. it occurs when they switch over local control.
- wilfra 14y agoWe are still down: http://warsocial.com http://warsocial.com Edit: Came back for a second. Went down again. Came back. Went down.
- joshcrews 14y agoI checked 5 sites, all single dyno: 3 down, 2 up
- joshcrews 14y agoupdate: all 5 down
- bad_user 14y agoI have 4 apps on 20 paid dynos. All of them down.
- masonhensley 14y ago2 of 3 up, all single dyno staticish sites; but it could be cloudflare propping them up with their cache.
- bdesimone 14y agoWhen will heroku have a multizone offering? I know most sites hosted on heroku are pet projects with no real need that type of uptime, but this type of downtime makes it impossible to use them as enterprise customers.
- chao- 14y agoAgreed. We are finally scaling from "small scale production" to having actual customers we want to take care of. Worse, we began a huge marketing push this week and had been seeing promising results. Now we'll probably miss a boatload of potential customers. Heroku had been a joy to work with until now. I ask of you all (as most of you are far more experienced in these matters than I), what is the traditional practice to mitigate this sort of risk? Paying for hosting with two separate companies? EDIT: I understand the benefits of cloud hosting over hiring a sysadmin. At the same time, I'm interested to learn about what possible solutions there are. I'm at the point where I don't know what I don't know, and even the name of a topic or technology would be a huge help.
- bad_user 14y agoOn June 1st (this month) they've had routing errors. On June the 5th (the day before yesterday) they again had routing errors. My apps triggered errors left and right and I was bashing my head against the keyboard because I noticed the errors earlier than their status update and I believed it was our fault. Plus this last week the latency of the requests have been incredibly high, while the traffic to my apps have not increased. Again I assumed it was my fault. I'm contemplating moving back to EC2 or Linode.
- kennystone 14y agoHeroku is really good. You will probably have more downtime than them unless you hire a whole team of operations and admins and pay a lot of money for hosting. It may also take a while to build out the deployment and development tools they have.
- wheels 14y ago
- switz 14y agoIf status.heroku goes down due to 500 errors (overloading) use https://status-old.heroku.com/ https://status-old.heroku.com/
- riffraff 14y agothe incident specific page is up often when the main page is not, in this case https://status.heroku.com/incidents/372 https://status.heroku.com/incidents/372
- soulclap 14y agoIt seems like they are quick and post a lot of updates on this status page each time when something's wrong. (I am not using Heroku but had the 'joy' of trying to run a site on Dreamhost for a while, so I have seen the worst.)
- namm 14y agomy demo site is still down.
- RoyceFullerton 14y agoConfirmed. One of my apps is down. I hope they resolve it soon, but I don't pay anything at the moment so I am not too upset.
- instakill 14y agoWhat's crap about these outages is that custom 500 error pages that you've created aren't shown.
- bitsweet 14y agoWhat would really help in these situations is a failover to our maintenance pages or some other static page we could provide instead of showing the Heroku "Application Error". This is probably harder then it seems, especially when the outage is related to their routing infrastructure.
- mcs 14y agoWell, there is an option to have error pages on your heroku domains, but that is probably dependent on at least a minimum level of the routing layer working. https://devcenter.heroku.com/articles/error-pages https://devcenter.heroku.com/articles/error-pages
- bitsweet 14y agoYeah, I have that setup but its not being used at the moment with this outage - if full redundancy with multiple regions is a huge challenge, perhaps some interim basic routing redundancy would go a long way so we could at least display a branded error page during an outage.
- pardner 14y agoI agree. There is a very simple way to do this which I just posted (since it apparently hasn't occurred to Heroku). All they need to do is change the default app error message once they have confirmed they have a platform issue. So within a few minutes of an outage, a better message is displayed to our customers that does NOT make it look like we screwed up. http://blog.pardner.com/2012/06/dear-heroku-dammit-quit-blaming-all-of-us-when-you-fail-do-this-instead/ http://blog.pardner.com/2012/06/dear-heroku-dammit-quit-blam...
- dholowiski 14y agoHow would that help?
- zsherman 14y agoMy apps are finally back up, how's everyone else doing?
- kunalmodi 14y agothings seem to be working for me, but Heroku seems a little off still. I got an error when I tried to view my app logs though, which is a little scary
- Cushman 14y agoThey went down literally the moment our demo thing started. I swear this is because of me.
- loceng 14y agoWhat luck.
- loceng 14y agoWhat luck.
- jszielenski 14y agoSame here! This happened last year too! Annoying.
- g0atbutt 14y agoAs a newbie, I've really enjoyed deploying on Heroku. However these outages really terrify me. Is there an easy way to host a redundant version of your app outside of Heroku?
- philip1209 14y agoIf you use Cloudflare it will serve a static version of the site until it is back online.
- garindra 14y agoI always use Cloudflare in front of Heroku; mainly because of that and the cheap & simple SSL set-up that they have.
- deleted 14y ago[deleted]
- bad_user 14y agoDo note that Cloudflare is throwing mallware/captcha notices to legitimate users, which is extremely annoying. When I get such a screen I immediately close the website I'm visiting. And no, contrary to what they tell me, I do not have "a virus" nor is my computer part of a botnet.
- pbreit 14y agoAlso, careful with Cloudflare SSL. It only secures between the browser and Cloudflare's data center. The connection between your server and Cloudflare remains non-SSL.
- garindra 14y agoYou actually have the option to enable SSL both on (customer->cloudflare) and (cloudflare->your server).
- 14y ago
- freditup 14y agoWhat kind of service guarantees does Heroku have?
- bretthoerner 14y agohttp://www.whoownsmyavailability.com/ http://www.whoownsmyavailability.com/
- lemieux 14y agoWhat is it suppose to do?
- yourcelf 14y agoI think it's similar to http://isitchristmas.com/ http://isitchristmas.com/ . It's a simple answer that's always the same, to drive the point home: you are always the one responsible for your uptime, no matter whether you choose dedicated hosting, the cloud, your own closet, etc. You can't outsource responsibility.
- grandalf 14y agoHeroku is a great platform and has come a really long way -- but it could really use some competition. Some of the main issues that would be resolved quickly with competition are: - it would be possible to have the non-DB part of your heroku app be spread across multiple availability zones. - worker pricing would be much lower or based on actual CPU cycles used. - add-on providers would be vetted more thoroughly before getting a spot in the add-on store. - they'd reopen the #heroku irc channel for informal support
- ricardobeat 14y agoyou mean like AWS, EngineYard, appfog, dotCloud?
- bdesimone 14y agoIaaS != PaaS
- ricardobeat 14y agoAmazon excluded those are PaaS.
- grandalf 14y agoI haven't looked at DotCloud, but the others are not really competitors.
- believeUme 14y agoGoes to show that the five nines is a bunch of crap. More like two nines.
- patrickgzill 14y ago5 nines is 5 minutes of downtime per year. 4 nines is 52 minutes of downtime per year. See: http://en.wikipedia.org/wiki/High_availability#Percentage_calculation http://en.wikipedia.org/wiki/High_availability#Percentage_ca... for a handy chart and follow along with any of your favorite *aaS providers!
- grandalf 14y agoRedHat was timing its blog postings in anticipation of this: https://openshift.redhat.com/community/blogs/new-openshift-release-june-7-2012-instant-apps-new-windows-client-and-more https://openshift.redhat.com/community/blogs/new-openshift-r...
- benmccann 14y agoThere's no big announcement there that I see. They release a blog post like this at least once a week.
- pardner 14y agostill getting sporatic errors
- fragsworth 14y agoEventually, they need to offer multiple independent regions, so if you're inclined (like Netflix does with AWS) you can develop a way to fail-over to another region. At least then, it won't be "Heroku is down", it will be "Heroku West is down" or some such thing.
- cardmagic 14y agohttp://AppFog.com http://AppFog.com has multiple regions today: Ireland, Singapore and US, addin Rackspace and HP soon. All free and backed by CloudFoundry
- kitsune_ 14y agoWe've recently begun to move our entire infrastructure to external providers (vps and cloud) because we felt that we couldn't guarantee a sufficiently high level of quality and had to devote too much time and money to operations. In over four months we've probably already had more downtime than in the past 5 years. Despite paying quite a lot for "redundant", clustered offerings. There was one nasty bug in the hypervisor that killed the entirety of one of our providers' vps infrastructure - for over a day. Our experience with "the cloud" was a little bit better, but we aren't entirely satisfied. I don't know, maybe we should just order services from three different providers mirror our applications ourselves as a fail-over mechanism.
- dholowiski 14y ago> maybe we should just order services from three different providers mirror our applications ourselves as a fail-over mechanism. Yes, yes a thousand times yes. And dedicated servers don't cost much more than VPS's.
- patrickgzill 14y agoYes. I recommend having a look at either ProxMox (proxmox.com) which can simultaneously do OpenVZ and KVM; or, XCP for Xen-based stuff - see http://www.xen.org/products/cloudxen.html http://www.xen.org/products/cloudxen.html
- neilmiddleton 14y agoHeroku status is currently showing production uptime as 99.97%. That's not too shabby, and definitely not as bad as you make out.
- jsprinkles 14y agoHe implied that Heroku is not his sole provider. I'm skeptical of that figure, too, as it's probably something on the Heroku sales site (99.7% to the 30-byte health check!), or it's manually updated, in which case it's probably rounded.
- cardmagic 14y ago
- ralphleon 14y agoIt's so scary to think that our entire startup almost died with this glitch. Our enterprise customers use our product in the mornings... this happened at 9-fucking-am right during our peak hours. Obviously it's our fault for not having a redundant system / for trusting heroku. Though I still can't help but be a little pissed as I email 100 people about how sorry I am for their service disruption. Heroku didn't email me.