3 ms·
For us it was only API down most of the day. We didn't have to deploy anything or fix anything, so we almost got away without downtime. But then, about 7 hours
by jeremyjh 1y ago
For us it was only API down most of the day. We didn't have to deploy anything or fix anything, so we almost got away without downtime.
But then, about 7 hours in (~1:40 PM EDT) for about 30 minutes all web requests were failing with a 503, logged as H99 "Platform error". No requests made it to our dyno during that period.
We can't failover to contingency without making API calls - the read replica has to be forked and promoted. I've always comforted myself by saying "well, the data plane and control plane have always been separate and aren't impacted by the same events". No longer.
My guess is this may have been some kind of trouble-shooting that caused this issue, but it doesn't matter. The point is I can no longer ignore this global single point of failure and really the only answer is to migrate away. Which I am reluctant to do, and I don't know that it will increase reliability.
We have never had application downtime as a result of a Heroku issue in the last 8 years this has been in production. 30 minutes in 8 years isn't bad. But if it were down for 3 days I'd have the same options: fuck and all.