17 ms·
Heroku was Down
Incident: https://status.heroku.com/incidents/2402
Update: our apps appear back up after 23 minutes total downtime. Others are reporting applications still down.
Update: it appears most or all services have been restored.
- digerata 5y agoOurs are up. US East. Heroku Shield.
- deleted 5y ago[deleted]
- VWWHFSfQ 5y agoIt is hard-down for me
- deleted 5y ago[deleted]
- emilsedgh 5y agoYes it appears to be down for us as well.
- ericpauley 5y agoOur app is currently 100% down at the router layer, and dashboard requests are mostly failing.
- VWWHFSfQ 5y agoRouting and logs are down for me as well
- btown 5y agoSame here, hard down for us
- pwned1 5y agoIt would be nice if they updated their status page. sigh
- nickrubin 5y agoYes, all apps are down
- greset 5y agoSame here
- teagee 5y agoHN is truly a market leader in status page technology
- deleted 5y ago[deleted]
- duxup 5y agoExcept when it is a false alarm. I've seen a few get to the front page. But when we're right we're right.
- jcuenod 5y ago10/9 times
- for1nner 5y agoHN: All of our amps go to 11.
- dec0dedab0de 5y agoEven when it's a false alarm it's usually something else is having a problem that is affecting many people and manifesting itself as a particular service being down.
- rileytg 5y ago*was
- jonpon 5y agoI came to HN to make sure it was actually down.
- js4ever 5y agoHN was also down for a moment because too much traffic
- Tainnor 5y ago
- typeofhuman 5y agoYes all of my apps are down.
- iamricks 5y agoSo... what are the chances this is related to a cyber attack from Russia? Also our apps are down
- ezekg 5y agoI wasn't gonna say it...
- 2OEH8eoCRo0 5y ago
- brimble 5y agoNow is a good time for everyone to double-check their backups and go over their disaster recovery plans. I know I am. I mean, yesterday was a good day. But today's the best you've got if you didn't do it then.
- driverdan 5y agoPractically zero. Why would Russia care about Heroku?
- adamnemecek 5y agoMaybe not heroku per se but something hosted on heroku?
- dc-programmer 5y agoCollateral damage (but probably DNS lol)
- Trasmatta 5y agoHeroku runs on AWS, so it would be an AWS attack of some sort. But AWS in general seems to be up from what I can tell, so I'm feeling like it's not a cyberattack, and just a Heroku specific problem.
- briandear 5y agohttps://www.cnn.com/2022/02/24/tech/russia-ukraine-us-sanctions-cyberattacks/index.html https://www.cnn.com/2022/02/24/tech/russia-ukraine-us-sancti...
- seannui 5y agoNo update to their own status or their Twitter (@herokustatus) after 10+ minutes. Pro.
- canadianwriter 5y agoGotta love asking in slack "is uh... literally everything down right now?"
- pwned1 5y agoBack up just now.
- ageitgey 5y agoSame here
- ezekg 5y agoI've been with Heroku for 7 years and this is the first time in recent memory that I've seen Heroku completely go down. Nothing at all works.
- digerata 5y agoAhh... were you sleeping in 2021? They went down in September, November, and December. Probably some other times earlier in the year as well. I stopped keeping track at how bad their service is because of how bad AWS's service is.
- ezekg 5y agoI'm well aware of their other outages. But I don't remember another time when they were completely down like this, even down to their status page.
- digerata 5y agoWell, they aren't completely down now either. Heroku Shield is up. Dashboard is up. EU is reported as up. As an aside, I spoke with our AE in early January on where they are going in dealing with AWS unreliability. One would think they would have a good answer. They don't.
- jrochkind1 5y agoMy app didn't go down during any of those times. I recall at least one of them I could not deploy new versions of my app though, which is pretty bad. I don't believe I had any downtime at all due to heroku platform outages in 2021. i believe you if you say you did though! This particular outage definitely seems to be of a rare level of severity.
- heyoni 5y agoTheir authentication system completely broke I think in January...for hours.
- moeadham 5y agoUS down for us, EU is up
- jlangenauer 5y agoAll our apps are down as well. And it seems (and I really have to almost laugh here) that https://status.heroku.com https://status.heroku.com no longer loads.
- dspillett 5y agoAs always, a note to infrastructure providers: HOST YOUR LIVE STATUS DETAILS ON OTHER INFRASTRUCTURE. Of course, they won't. If they host is on someone else's then that might look bad (tacitly saying that a competitor is reliable and might be up when they are down) and if they hive off an extra copy of some of their infrastructure there will still be single points of failure either accidentally, by human error (someone somehow messing up both segments at once), or by design (possibly through management trying to save pennies when they noticed this extra bit of infrastructure on the balance sheet).
- bin_bash 5y agoI am pretty sure Heroku has a Linode box or something for their status page. I may be wrong though. They may also have moved it but that would seem like an incredibly dumb idea. I worked there and I vaguely remember something like this but it's been a long time. If I had to guess this is probably a DNS issue (it's always DNS).
- wuputah 5y agoI agree with your lead statement and argued as such, but was overruled. Last I knew and understood, Heroku Status is static pages pushed out to Fastly, with the internal admin site (that does that work) running in a Heroku Private Space. If you look at the DNS, it still appears to be served by Fastly, and Heroku Private Spaces are generally pretty isolated infra, so I would be curious what the failure mode was here. But ultimately this is the fire you play with when you self-host your status site...
- amotinga 5y agoyes, site is down
- pedroborges 5y agoOur app is down too :(
- reubano 5y agoYep, all 6 of my Heroku apps are down.
- reubano 5y agoInterestingly enough, my 2 Squarespace sites weren't affected. Anyone know who they host with?
- andygcook 5y agoOur app is currently down as well and we host on Heroku. We received a down alert from our monitoring service at 11:32AM EST.
- tempnow987 5y agoStatus Page was running a bit slowly for me connecting to developers.salesforce.com. Link here to latest snapshot: https://postimg.cc/McZHpCWg https://postimg.cc/McZHpCWg No issues noted there.
- perryraskin 5y agoMy Heroku app is completely down as well
- deleted 5y ago[deleted]
- reubano 5y agoAWS is down too, this is likely the cause since Heroku runs on it. https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/
- jrochkind1 5y agoAWS status page is all green.... Just kidding! I know the information content of the AWS status page is literally zero, it's always green!
- somenewaccount1 5y agoNah, it's just on a 12 hour delay between updates.
- jermaustin1 5y agoAWS status page is up, but going VERY slow, and says no issues.
- XzAeRosho 5y agoAs usual whenever there's an AWS outage
- reubano 5y agoI still don't get why they use images <td class="bb pad04 top center" style="width: 32px"> <img src="/images/status0.gif"> </td> Instead of unicode:https://www.htmlsymbols.xyz/unicode/U%2b2705 https://www.htmlsymbols.xyz/unicode/U%2b2705
- jermaustin1 5y agoif the image tag is drawn dynamically then it is actually probably less bandwidth than the unicode character since the image can be cached at multiple locations including the browser.
- mabsboza 5y agoreporting from Nicaragua, our apps are down!
- rlopezc 5y agoDown for us too
- yekurtal 5y agoMine is still up.
- mhale 5y agoHeroku apps are down, but API access is available. Might be a good time to run: heroku pg:backups:download
- jpmoyn 5y agoDown along with the status page right now
- mmarcant 5y ago2/3 of our apps have been timing out of synthetic checks since 8:30a PT
- noodle 5y agoThings just came back up for me
- nordec 5y agoSame here
- vendiddy 5y agoDown for us as well.
- kennysmoothx 5y agoOur Heroku applications are all down, it seems to be that it is a Heroku Router/DNS issue. We can access our logs and the application is running, just no incoming requests.
- jpmw 5y agoLet me bet: it's DNS. It's always DNS.
- ne0n 5y agoIt was, but I don't know why. I'm curious to hear if Heroku releases any information about how this happened. Heroku's DNS was returning a single 100.64.x.x address which is in a reserved range.
- jpmw 5y agoThey typically publish post-mortems, I think? Not 100% sure. They definitely do in-house. We'll have to wait and see I guess.
- rileytg 5y agostatus, dashboard and landing page are down for me. (socal.)
- sharksauce 5y ago% heroku status › Warning: Our terms of service have changed: https://dashboard.heroku.com/terms-of-service https://dashboard.heroku.com/terms-of-service Apps: Yellow Data: No known issues at this time. Tools: No known issues at this time. === Availability of Common Runtime apps 2022-02-24T16:50:47.249Z https://status.heroku.com/incidents/2402 https://status.heroku.com/incidents/2402 investigating 2022-02-24T16:50:47.249Z (1 minute ago) Engineers are looking into reports of connectivity issues to Common Runtime apps in the US and EU regions.
- jrochkind1 5y agooh I didn't know that was a CLI command, nice!
- Justsignedup 5y agois it related to an AWS outage? https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/
- Gaussian 5y agoWe're down
- andycloke 5y agoAll 6 of my apps & dashboard are down too.
- ferajay 5y agoSame here
- ericpauley 5y agoAnyone still seeing outages? Our services are all back up.
- leetrout 5y agoHeroku just sent an email to us confirming issues. Links to a status page but that isn't loading. https://status.heroku.com/incidents/2402 https://status.heroku.com/incidents/2402
- neilobremski 5y agohttps://status.heroku.com/incidents/2402 https://status.heroku.com/incidents/2402 (It took a few minutes to get this link to work for me)
- smoldesu 5y agoRust's crate repository (crates.io) seems to be down too. I wonder if they're connected...
- jrochkind1 5y agoAt first the heroku status page was still all green for me... although took 30 seconds to load. I guess when the status page takes 30 seconds to load, that's an indicator! I did figure, okay, a 30-second-to-load status page probably means my app outage is a heroku platform problem. (Also an indicator the status page is sharing too much platform with the platform it's supposed to be reporting on? Also in this case an indication that the platform problems are pretty deep?) Interestingly, my app logs (via papertrail, which is still up) show that some traffic is getting through continually through this current outage, although I can't (and my monitoring app can't either, which pinged me).
- escot 5y agoMines back up, at least 20min down time
- bisRepetita 5y agoSome of my apps are back up. Others are not.
- forgingahead 5y agoOur apps are all down as well, Heroku Status not loading either (or just incredibly sluggish).
- drstewart 5y agoCypress is also down
- jamil7 5y agoAnd we're back? I think.
- jrochkind1 5y agoMy heroku app was back up again as of about 100 seconds before this comment timestamp. My app was down about 35 minutes total, according to my monitoring.
- craigkerstiens 5y agoIt appears to only be affecting apps that have multiple dynos. While Heroku is busy fixing the issue if you scale your app to a single dyno (assuming it can handle the traffic) it should restore availability.
- ericpauley 5y agoWe also had single-dyno apps affected.
- craigkerstiens 5y agoIt may have been during an earlier part of the instance, but we just scaled multiple affected apps to a single dyno and all recovered. ps restart with multiple dynos and no effect.
- jeremyjh 5y agoI did too. I'm curious if you also have pre-boot enabled? I think that may set the routing configuration the same as if multiple dynos are up.
- bisRepetita 5y agoI had single dynos affected.
- reubano 5y agoMy 6 apps are all single dyno. Not sure if I have pre-boot enabled or not though....
- jrochkind1 5y agoMy single-dyno app was affected. However, while I could not connect to my app, nobody I know could connect to my app, and my monitoring service could not connect to my app for a ping... my app logs showed that some traffic continued to connect throughout the outage. So it was not entirely universal. And was clearly a routing problem of some kind.
- reubano 5y agoBack up now!
- dec0dedab0de 5y agoI've only ever used heroku for free tier level personal projects, but as I understand it they use AWS to do the actual hosting. I can understand an outage affecting their deployment process, but what could cause running servers to go down? As I typed that out I remembered that they handle DNS, load balancing, and databases, so I guess any one of them.
- alpb 5y agoAll it takes is a load balancer misconfiguration on a core service to take a large scale service down.
- jrochkind1 5y agoit indeed behaved like a routing issue of some kind (my app was still UP, and was still logging, just no traffic could get to it), and a heroku incident status line said "Engineers are recovering affected routing components," so, yup.
- henryaj 5y agoEU outage towards the end of last year was similarly bad, but lasted much longer. Asked Heroku for at least a refund for the dyno time when our apps were unavailable and was flatly told no.
- bovage 5y agomy apps are back up now.
- leros 5y agoI was able to get my apps up faster by restarting the dynos
- ab-dm 5y agoWell that was a fun one to wake up to. We were down for ~33 minutes overnight.