6 ms·

Heroku Is Currently Down

175 points by iamricks 5y ago

detaro 5y ago
presumably because of AWS: https://news.ycombinator.com/item?id=29648286 https://news.ycombinator.com/item?id=29648286
- iamricks 5y ago
  Weird, my apps are hosted on us-east but they are up, i just can't login to Heroku nor can i deploy to any app right now.
  - qeternity 5y ago
    It’s just a single AZ by the sound of it
    - jonstaab 5y ago
      Yes, but I think it's elastic compute that's down, which hosts a lot of AWS and Heroku's own infrastructure, hence the large blast radius.
    - jonny_eh 5y ago
      And that single AZ took down all of Heroku? This'll be an interesting post-mortem to read.
  - whoevercares 5y ago
    Your existing apps are likely in the “data plane” path which is more reliable by design of most cloud providers.
nevster 5y ago
Can't login using their id service https://downforeveryoneorjustme.com/id.heroku.com https://downforeveryoneorjustme.com/id.heroku.com
bizzleDawg 5y ago
All 3 heroku hosted sites which I look after appear to be up and running normally, despite the heroku console being unavailable. Not sure if auto-scaling is impacted yet. Edit: all my apps are in the EU region, which is eu-west-1 (Ireland)
- iamricks 5y ago
  What region are your apps hosted in? My apps are still up all on us-east-1
- moeadham 5y ago
  We are also up - hosting in EU. Can't access the dashboard. All our monitor are green, but we did experience a bit of increased latency for about 30 minutes.
aeonflux 5y ago
I just noticed that I cannot upload images on Slack. Tried to use imgur to post a link and that doesn't work either.
clavicat 5y ago
It's affecting Substack.
denghauser 5y ago
Can't log in to their console either. Apps are running. seems the id service is down.
mikece 5y ago
AWS is having issues in us-east-1 again -- are these two services connected?
- michaelbuckbee 5y ago
  Yes, Heroku is a (PaaS) built on top of AWS (IaaS).
deleted 5y ago
[deleted]
michaelbuckbee 5y ago
Status site: https://status.heroku.com/ https://status.heroku.com/
- hn_throwaway_69 5y ago
  I imagine they'll eventually feed us with the tried and tested line, "this outage prevented us from updating the status page". And we will reply with the obvious response, "run your status page on different infrastructure". Edit: status page updated
ItsBob 5y ago
I got an error trying to download the git client for Windows... must be this too.
TheTaytay 5y ago
I created a ticket through Salesforce's "premium" phone support, but the URL of the ticket is hosted at help.heroku.com, which is also down. As of now, Heroku's status page appears green. Our Heroku app is hosted in Amazon East, but appears to be functioning at the moment.
- dbraga 5y ago
  is there a support email? i sent an email via support@heroku.com with title urgent but no reply yet
  - mritzmann 5y ago
    Sorry no offense, but it's obvious you've never worked in support before.... :-) I'm sure they already know that half the infrastructure is down. Every support request brings exactly nothing, except more chaos.
    - deepsun 5y ago
      Auto-responders must still reply "Thank you for your message it's really important to us". At least you'd know the auto-response servers are still up.
    - is_true 5y ago
      People are anxious, if you have a problem in your infrastructure at least set an automatic reply linking the status page or with a short text describing the situation.
3np 5y ago
Noticed this when trying to build an Electron app; atom.io and therefore all related artifacts are down.
KrishnaShripad 5y ago
Trello is down as well (at least the checklists aren't loading anymore). GraphQL requests timing out in the Console. Not sure if it is connected to Heroku going down. Just wanted to add this here in case anyone else is also experiencing the same issue.
iamricks 5y ago
I am able to deploy now but i still cannot access the dashboard
born-jre 5y ago
may be they ran out of dynos
ajag-oai 5y ago
jug 5y ago
Got issues on Imgur too.
nlitened 5y ago
Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their own services? I would think that if somebody had expertise to do it flawlessly, it should have been them. I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.
- samwillis 5y ago
  I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it. Edit: maybe the dashboard is just back up and I don’t see it down.
- stickfigure 5y ago
  Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so? Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.
  - more_corn 5y ago
    RDS has this mostly solved. They silently mirror the master into another AZ. If the primary fails the internal routing redirects to the clone.
    - stickfigure 5y ago
      Tell me, are transactions guaranteed to have replicated before they commit? If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.
cpursley 5y ago
Render.com is such a better option these days (and I've been a huge Heroku fan since the beginning). Render has better support and pricing. In addition to great docker support and free static sites, they support distributed Elixir out of the box as well as autoscaling. Still, they're not immune to these types of outages - I believe they're dependent on Google Could but are moving towards running their own data centers at some point. But so far, uptime has been great for us. Fly.io also looks promising; keeping an eye on that one.
- bberenberg 5y ago
  We're looking at Digital Ocean App Platform for this right now. Seems to be a very similar implementation and similar pricing. Has anyone does a comparison they can share?
- _fat_santa 5y ago
  I've been a Netlify customer for a number of years now and I'm considering switching over to Render. My biggest gripe with Netlify recently is don't make it straightforward to upload site without using their build system, and there are a bunch of little workarounds to implement to make everything work right. How is render for static sites? Do they make you build the site with them or is it a simple "here's an NPM package that github actions can consume".
  - anurag 5y ago
    (I work at Render) We do need a Github/Gitlab repo, but we're not wedded to it. So I'm clear, you want the static site hosted on Render but built via GH Actions?
- yawnxyz 5y ago
  Thanks so much for the share! Fly.io's pricing page seems impenetrable to me... I just checked out Render.com and it looks so easy to understand. I've been wanting to run a Node backend API thing but can't figure out where to put it... been using Cloudflare workers but it kept timing out (going over 10s). Will definitely check out Render's services and $1/mo crons.

175 points by iamricks 5y ago

detaro 5y ago

presumably because of AWS: https://news.ycombinator.com/item?id=29648286 https://news.ycombinator.com/item?id=29648286

iamricks 5y ago
Weird, my apps are hosted on us-east but they are up, i just can't login to Heroku nor can i deploy to any app right now.
- qeternity 5y ago
  It’s just a single AZ by the sound of it
  - jonstaab 5y ago
    Yes, but I think it's elastic compute that's down, which hosts a lot of AWS and Heroku's own infrastructure, hence the large blast radius.
  - jonny_eh 5y ago
    And that single AZ took down all of Heroku? This'll be an interesting post-mortem to read.
- whoevercares 5y ago
  Your existing apps are likely in the “data plane” path which is more reliable by design of most cloud providers.

nevster 5y ago

Can't login using their id service https://downforeveryoneorjustme.com/id.heroku.com https://downforeveryoneorjustme.com/id.heroku.com

bizzleDawg 5y ago

All 3 heroku hosted sites which I look after appear to be up and running normally, despite the heroku console being unavailable. Not sure if auto-scaling is impacted yet. Edit: all my apps are in the EU region, which is eu-west-1 (Ireland)

iamricks 5y ago
What region are your apps hosted in? My apps are still up all on us-east-1
moeadham 5y ago
We are also up - hosting in EU. Can't access the dashboard. All our monitor are green, but we did experience a bit of increased latency for about 30 minutes.

aeonflux 5y ago

I just noticed that I cannot upload images on Slack. Tried to use imgur to post a link and that doesn't work either.

clavicat 5y ago

It's affecting Substack.

denghauser 5y ago

Can't log in to their console either. Apps are running. seems the id service is down.

mikece 5y ago

AWS is having issues in us-east-1 again -- are these two services connected?

michaelbuckbee 5y ago
Yes, Heroku is a (PaaS) built on top of AWS (IaaS).

deleted 5y ago

[deleted]

michaelbuckbee 5y ago

Status site: https://status.heroku.com/ https://status.heroku.com/

hn_throwaway_69 5y ago
I imagine they'll eventually feed us with the tried and tested line, "this outage prevented us from updating the status page". And we will reply with the obvious response, "run your status page on different infrastructure". Edit: status page updated

ItsBob 5y ago

I got an error trying to download the git client for Windows... must be this too.

TheTaytay 5y ago

I created a ticket through Salesforce's "premium" phone support, but the URL of the ticket is hosted at help.heroku.com, which is also down. As of now, Heroku's status page appears green. Our Heroku app is hosted in Amazon East, but appears to be functioning at the moment.

dbraga 5y ago
is there a support email? i sent an email via support@heroku.com with title urgent but no reply yet
- mritzmann 5y ago
  Sorry no offense, but it's obvious you've never worked in support before.... :-) I'm sure they already know that half the infrastructure is down. Every support request brings exactly nothing, except more chaos.
  - deepsun 5y ago
    Auto-responders must still reply "Thank you for your message it's really important to us". At least you'd know the auto-response servers are still up.
  - is_true 5y ago
    People are anxious, if you have a problem in your infrastructure at least set an automatic reply linking the status page or with a short text describing the situation.

3np 5y ago

Noticed this when trying to build an Electron app; atom.io and therefore all related artifacts are down.

KrishnaShripad 5y ago

Trello is down as well (at least the checklists aren't loading anymore). GraphQL requests timing out in the Console. Not sure if it is connected to Heroku going down. Just wanted to add this here in case anyone else is also experiencing the same issue.

iamricks 5y ago

I am able to deploy now but i still cannot access the dashboard

born-jre 5y ago

may be they ran out of dynos

ajag-oai 5y ago

jug 5y ago

Got issues on Imgur too.

nlitened 5y ago

Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their own services? I would think that if somebody had expertise to do it flawlessly, it should have been them. I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.

samwillis 5y ago
I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it. Edit: maybe the dashboard is just back up and I don’t see it down.
stickfigure 5y ago
Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so? Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.
- more_corn 5y ago
  RDS has this mostly solved. They silently mirror the master into another AZ. If the primary fails the internal routing redirects to the clone.
  - stickfigure 5y ago
    Tell me, are transactions guaranteed to have replicated before they commit? If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.

cpursley 5y ago

Render.com is such a better option these days (and I've been a huge Heroku fan since the beginning). Render has better support and pricing. In addition to great docker support and free static sites, they support distributed Elixir out of the box as well as autoscaling. Still, they're not immune to these types of outages - I believe they're dependent on Google Could but are moving towards running their own data centers at some point. But so far, uptime has been great for us. Fly.io also looks promising; keeping an eye on that one.

bberenberg 5y ago
We're looking at Digital Ocean App Platform for this right now. Seems to be a very similar implementation and similar pricing. Has anyone does a comparison they can share?
_fat_santa 5y ago
I've been a Netlify customer for a number of years now and I'm considering switching over to Render. My biggest gripe with Netlify recently is don't make it straightforward to upload site without using their build system, and there are a bunch of little workarounds to implement to make everything work right. How is render for static sites? Do they make you build the site with them or is it a simple "here's an NPM package that github actions can consume".
- anurag 5y ago
  (I work at Render) We do need a Github/Gitlab repo, but we're not wedded to it. So I'm clear, you want the static site hosted on Render but built via GH Actions?
yawnxyz 5y ago
Thanks so much for the share! Fly.io's pricing page seems impenetrable to me... I just checked out Render.com and it looks so easy to understand. I've been wanting to run a Node backend API thing but can't figure out where to put it... been using Cloudflare workers but it kept timing out (going over 10s). Will definitely check out Render's services and $1/mo crons.