7 ms·
Heroku's Managed DB's have been down for 2+ hours
- kentf 5y agoLuckily we had followers in different zones by chance. Still scary though. What are the best solutions for replicating Heroku in different clouds?
- mappu 5y agoIf by "replicating" you mean replicating the experience, then you're looking for Dokku Easy setup + MIT license and you get the same git push deploys, Heroku-compatible buildpacks or bring your own Dockerfile. I would recommend running it on probably on Linode / DigitalOcean / Vultr.
- strken 5y agoDokku is great for a single host. If you have a more complicated setup you can go a long way with post-receive hooks, although it won't be as magical without buildpacks.
- aswinmohanme 5y agoFly.io comes close. You have docker based builds on the cloud with native multi region support. Postgres is in beta right now.
- mcintyre1994 5y agoThey also have built-in Heroku migration: https://fly.io/docs/app-guides/speed-up-a-heroku-app/ https://fly.io/docs/app-guides/speed-up-a-heroku-app/
- corobo 5y agoDigitalOcean's App platform things plays nicely with Heroku buildpacks from what I've seen
- pgn674 5y agoIt mentions an issue at an upstream service provider. Is it AWS and their Degraded EBS Volume Performance in Northern Virginia? https://status.aws.amazon.com/ https://status.aws.amazon.com/
- hiyer 5y agoLikely - they're also reporting provisioning failures in Virgina, which is consistent with what AWS is reporting as well.
- mcjiggerlog 5y agoDefinitely appears to be a wider issue, circleci is having a major outage too: https://status.circleci.com https://status.circleci.com.
- Daniel_sk 5y agoSignal is down too due to an outage of a service provider (I assume AWS).
- forgingahead 5y agoHeroku has been strangely unreliable the past few weeks. Even their ticket response team has been slow, with their support engineers often talking past the issue to just send a scripted reply. We have the majority of our client apps hosted with them, but most don't require 24/7 availability. This is still concerning though, and we do have one high-availability app hosted on them now that we're trying to plan contingencies for. Open to any suggestions for alternatives! Ideally I'd keep things on Heroku, but it would be nice to have failsafes that could be activated relatively quickly in the event of similar issues.
- deleted 5y ago[deleted]
- lbruder 5y agoSimple dynos can be replicated with Dokku and Ledokku as a GUI. Just get an Ubuntu VM on Digitalocean, Vultr or whatever, install and configure UFW, fail2ban and automatic security updates, install dokku and you're set. For managed databases with replication however, Dokku still leaves much to be desired...
- i386 5y agoI want a birthday cake. But first I'll be growing and milling my own grain, raising chickens and a cow. Water will be manually pumped from a well.
- MikeDelta 5y agoIt is seriously not that bad at all, I would compare it to making your own cake from the flour, water, butter vs buying ready-made batter.
- subsection1h 5y agoHeroku provides many features like pipelines and review apps that would be impossible to implement on a single VPS and very time-consuming to implement on multiple VPSes. Anyone who recommends a single VPS as a hosting solution (as lbruder did) is likely a hobbyist or a student.
- throwdecro 5y agoIs there an "uncanny reliability" range where increasing reliability on the part of a service provider makes things worse, by being so close to 100% reliable that any failure is a shock? Maybe it's better to go with cheaper services that fail more often, thus keeping customers in good practice for how to deal with it.
- remus 5y agoYes. There's a nice example of this in the Google SRE book (I think it may have been their internal paxos service?) If I remember they ended up building in planned downtime so users could learn to degrade gracefully if the service went down.
- GeneralMayhem 5y agoGoogle does this pretty regularly internally. Every system has a published SLO, and for a couple weeks a year major components will respect their SLO and not a single request or millisecond better. If you were relying on something performing 10x better than what it's rated for in order to provide your own guarantees, then that's on you.
- strzibny 5y agoThis is something along the line what I say in the Scaling chapter in my book[0]. If your infra is really simple (like a server or two), you can actually recreate it in a different provider and beat any hard to fix issue (whole AWS region going down or this Heroku's databases problem). Especially with smaller applications, you might be able to beat the provider time to fix the issue, and you never know when it might be critical for you to be able to do that. My book also contains a Bash script to configure you a PostgreSQL cluster in a few minutes with/without attached storage space, with self-signed SSL, SELinux, and more. Great for simple apps and as a start in learning production PostgreSQL. [0] https://deploymentfromscratch.com/ https://deploymentfromscratch.com/ [1] https://gist.github.com/strzibny/4f38345317a4d0866a35ede5aba99a1e https://gist.github.com/strzibny/4f38345317a4d0866a35ede5aba...
- 5y ago
- TedShiller 5y agoHeroku may have been down for 2+ hours, but MongoDB has been unreliable for 10+ years.
- Insalgo 5y agoAnd why are you referring to mongo here?
- supermatt 5y agoWhatever happened to 5-nines uptime? It seems no cloud service provider these days is able to offer what was considered an industry standard. AWS even have documents telling people how to achieve exactly this! https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html https://docs.aws.amazon.com/wellarchitected/latest/reliabili... Why don't "premium" service providers like heroku, etc, do this?
- ranguna 5y ago99.999% uptime still mean around 7 and a half hours of downtime per year.
- rafBM 5y ago99.999% uptime is 5m 15s per year: https://uptime.is/99.999 https://uptime.is/99.999
- deleted 5y ago[deleted]
- bobviolier 5y agoNo, just 5 minutes https://uptime.is/99.999 https://uptime.is/99.999
- ranguna 5y agoUps, quick maths == wrong maths.
- makeitdouble 5y agoTBF there is very few real world services that offer customers and non giant size companies 5-nines of uptime. E.g. my electricity provider doesn't.
- supermatt 5y agoServices providers such as Heroku should be easily able to have five-nines uptime. They ONLY offer fully managed services, which can be backed by the multi-cloud, multi-AZ setup I refer to - but instead a single product outage from a single upstream provider in a single datacenter is affecting all their clients. This is a regular occurrence for Heroku - and they charge a substantial premium for their "service".
- pvsukale3 5y agoI have been trying to deploy fix to a bug we deployed yesterday. I think they have stopped deploys as well. As the deploys are being rejected without any explanation.
- thomaslord 5y agoInterestingly, I have an app using Heroku Postgres that seems to have had zero issues during this outage. I can see data that was stored during this period of time and Rollbar doesn't show any DB connection errors.