18 ms·

Tell HN: Heroku deleted my database with no warning

460 points by fireworks 4y ago

Last December, Heroku nuked the database on one of my active projects. I was travelling at the end of the year and did not catch wind of this until I returned and saw messages about an issue with the app. Sure enough, I checked and noticed that the database was gone and detached on December 9th. Before the hate comes out, yes I know Heroku deprecated free tiers. However, I did not understand this would affect my projects on paid dynos. The real issue here is that I never received a single email or notice of any kind to my email about this. From researching, it appears most people received SEVERAL notices about this. I did not think there was an issue with my setup because I received zero communication. Upon reaching out, Heroku has told me that they cannot recover the database. They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent. So I guess just a warning to all - your database might be nuked at any time. I learned my lesson about not doing an offsite backup regularly. I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days. RIP.

gwbas1c 4y ago
What frustrated me about Heroku nuking free projects was that the paid version was, IMO, too expensive. (Granted, I understand that the free tier was probably a money sink for them.) $5-10 a month is pocket change in some contexts, but for a hobby or a one-off project, it's a huge chunk of change. I really wish they offered a lower-cost tier.
- shortcake27 4y ago
  Or let me share dynos across all my projects. $5-$10 per month for all projects, sounds good! $5 per month for each project adds up very quickly.
  - throwawaykai 4y ago
    That was the new “eco” plan that was introduced alongside the ending of free plans — $5/mo for 1000 hours shared across all eco dynos.
    - gwillen 4y ago
      I definitely wish you guys had advertised that earlier / better. I feel like I got lucky because I put off migrating until just before the sunset, at which point I discovered eco dynos. I swear they either didn't exist or were harder to find when the sunset was first announced. Thanks for making them available at all, though. (Context: I maintain an app for an event that runs about one week a year. The ability to use free dynos the other 51 weeks means we retain the ability to do one-off analytics queries, minor development work, etc. during the off-season, without having to delete and recreate the app or something every year. Eco isn't quite as good as free, but it means we can still have separate staging and production instances during the off-season, without paying extra for staging to be idle, and without having to destroy and recreate staging every year.)
      - shortcake27 4y ago
        Wow, I also don’t recall seeing this either. It must have been marketed poorly. Same for Postgres, looks like there are now some more cost effective options which weren’t available when I migrated my stuff away.
- m00x 4y ago
  That's 2 cups of coffee, or 1 latte. It's objectively not a huge chunk of change unless you live in a very LCOL area. You can also go to any competitors, which have their smallest tier around the same price. Linode has a $5/m tier. You could also buy a Raspberry Pi for $50 and run it in your closet.
  - gwbas1c 4y ago
    > or a hobby or a one-off project I don't want to pay $5 / month indefinitely to host a weekend project. I don't want to deal with the complexities of hosting a weekend project myself. I've done plenty of that. I just don't want to end up spending $100-200 / month to host a lifetime of weekend projects and experiments.
    - m00x 4y ago
      Then you can get a raspberry Pi and host it in your closet.
      - gwbas1c 4y ago
        > I don't want to deal with the complexities of hosting a weekend project myself. I've done plenty of that. That involves port forwarding, figuring out something like dyndns, and whatever I need to share domains on a single IP.
- Towaway69 4y ago
  My personal take is that I'd rather work with heroku than dealing with aws or gcs. Sure there are others out there that provide the same easy of use, I am just too lazy to switch ;) OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on.
  - xp84 4y ago
    > I've never had them wipe a database on me, Give it time, apparently ;) If you're like me and are really into ease of use, I really endorse Digital Ocean. People don't seem to use it in Big Important Companies, but I never had a problem with anything I hosted there when I was my last, medium-sized company, and everything was so easy to manage in the GUI without having to learn proprietary APIs and whole stacks of infrastructure-as-code stuff like in AWS land.
    - Towaway69 4y ago
      DO is also good and less restrictive for a new user than aws (I.e. getting more resources). I remember having to beg to get more than 1(or 2) servers at aws as a new user. I haven't looked recently but does DO have the heroku click to add resources and git push to deploy offerings?
      - xp84 4y ago
        I think this product is the closest: https://www.digitalocean.com/products/app-platform https://www.digitalocean.com/products/app-platform
  - gwbas1c 4y ago
    > OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on. About a year ago they shut down my personal blog based on a vague "violation of our terms." There was no violation, either. (I suspect they had to shut down phishing and other abusive sites, and cast a little too wide of a net.) When they claimed they turned it back on, I had to keep pestering them because they goofed something or other. Eventually I couldn't push updates via Github. (I wrote the blog engine as an exercise to learn NodeJS.) Again, if I could have bought an ultra-cheap tier, I would have been happy to pay.
swader999 4y ago
I'm curious how this affected your projects on paid dynos, can you provide more details around that?
- fireworks 4y ago
  So I did not realize this at first (would have helped if I were, you know, emailed some information about this), but my project was still using a free Postgres addon despite being on a paid dyno setup. So that I suppose was nuked along with the general free tier deprecation.
  - m00x 4y ago
    Did they explain why the notifications didn't work?
    - fireworks 4y ago
      "I discussed this with our backend team and upon deeper investigation, we were able to confirm that there was some issue sending the notifications to your email." Nope.
      - dboreham 4y ago
        Guesses 1, 2 and 3 are : their MTA's IP was black listed by a reputable mail hosting provider's spam filtering system (because they send spam).
      - joemi 4y ago
        That phrasing doesn't rule out an issue on your email server's end, like the email could have bounced. (Though I'm guessing that's unlikely.)
        fireworks 4y ago
        Would be surprised if that was this case - vanilla GSuite setup.
ericpauley 4y ago
This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at night, but clearly that's gone now.
- xp84 4y ago
  I assumed the value prop of being on Heroku in 2023 was purely "not having to do the pain in the butt to migrate your legacy app that you made on Heroku in the pre-2010 era" - in other words, I would be really shocked if Heroku got significant new business or any growth at all, now that it's just expensive AWS with some basic CI integration points. I also assume Salesforce only bought them as a cash generator and has no interest in investing in it. So if they saved bottom line from this move, that's a win for them. (Feel free to correct my assumptions if I'm very wrong)
  - jrochkind1 4y ago
    "Pre-2010"? Heroku only launched in April 2009. So I would assume the vast majority of current customers are not actually from a "pre-2010 era". Heroku didn't even have a postgres add-on until November 2010 -- the salesforce purchase went through in Dec 2010. In fact, the heroku "golden years" were mostly during the early salesforce period. (Heroku put matz, creator of ruby, on staff in July 2011; there was no Heroku API until 2014!). (source for timeline: https://www.heroku.com/about https://www.heroku.com/about) Heroku is still quite a bit of value-added over "AWS with some basic CI integration", people chose it and still choose it because it requires a lot lot lot less in-house expertise and management hours than AWS for most kinds of standard apps. (I'm not totally sure what AWS services/architecutres you are thinking of when you say that; but I'll say: for pretty much all of them.) (Heroku def has more competition in that space than it did 10 years ago, opinions differ on the relative merits) AWS of course already existed when Heroku was launched, so if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then? I think Salesforce thought it would somehow be more "synergistic" with their other offerings than it has turned out to be. That they'd get Salesforce customers on heroku when they needed something beyond the "no-code" tools Salesforce already provided, or that they'd do better at converting heroku customers to salesforce customers than they have been. It does seem to be true that salesforce stopped really investing in new heroku features or improvements some years ago, and seem to be looking to minimize costs while continuing to collect revenue, I agree. (Sometimes I'm not even sure how much they care about continuing to collect revenue...)
collectedparts 4y ago
This happened to me as well: projects with paid dynos but free databases, databases got nuked. How Heroku missed this is beyond me. They managed to screw over paying customers in their broad attempt to stop freeloaders. These are good accounts with credit cards on file. Why not just autocovert me to the lowest tier paid database? FWIW I was able to get them to restore my databases. But I also had free Heroku Redis on one my projects and that, they assured me, is gone forever.
- sieabahlpark 4y ago
  Redis is a memory store, you're the one who fucked up treating a cache as a persistence layer.
  - robertlagrant 4y ago
    You can use Redis as a cache, but it's not fundamentally a cache.
  - itake 4y ago
    Redis offers disk persistence [0]. Why can't it be used as a persistence layer? [0] - https://redis.io/docs/management/persistence/ https://redis.io/docs/management/persistence/
    - ceejayoz 4y ago
      Redis offers disk persistence; Heroku's free Redis offering did not.
  - deleted 4y ago
    [deleted]
- fireworks 4y ago
  Right? Like just charge me and tell me. Also, while I still think it is way too aggressive how they deprecated free in general, at the very least... maybe uh... I don't know, let me know if you are going to delete my database? Absolutely insane.
cameronfraser 4y ago
Same happened to me, I didn't receive any email notification or anything, found out when things started failing.
rovr138 4y ago
Well, I moved everything I had to a VPS for the time being because I left it to the end. So I take this opportunity to ask, what alternatives have people moved to? I really haven't gone back to look what's out there. Not looking for free, just alternatives to review.
- donnieashok 4y ago
  I went full blast into SQLite, agreed there's a migration job in between. But it's totally worth it. In fact my app is now 5x faster.
- dcow 4y ago
  You can host static sites on DigitalOcean's app platform for free. Their lowest tier offering for non-static sites is like $5/mo. Might be worth checking out.
- xp84 4y ago
  For what kind of an app? If I was tasked to put an app someplace today, the answer would depend if it was easily containerized, and what kind of scale you're talking about. For small apps, if containerized, I would host on Google Cloud Platform using App Engine Flexible Environment: https://cloud.google.com/appengine/docs/flexible https://cloud.google.com/appengine/docs/flexible Add on their managed database, of course. But also, random VPSs are so cheap that if it works fine as is, you could do a lot worse than just running it on Linode, Digital Ocean, Hetzner, etc.
marvinkennis 4y ago
Heroku sent me repeated messages that they would shut down my account due to inactivity. I was fine with that because finding and turning off the supposedly active Dyno was impossible. Guess what? Still charging me $27 a month for a server I can't even manage.
- dns_snek 4y ago
  If this is true I think it deserves spotlight in a separate post.
JEDI-HACKER 4y ago
Check your junk mail folder.
- javawizard 4y ago
  > They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent. Junk mail wasn't the issue in this case.
- deleted 4y ago
  [deleted]
codegeek 4y ago
"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even though RDS has backups and restore options. I want that database file.
- Max-q 4y ago
  After the Salesforce takeover? The one in 2010? So Heroku was good in like a year!?
  - hot_gril 4y ago
    Heh. I always loved Heroku and have complained that Salesforce is messing it up recently, but turns out the first time I used Heroku was like 5 years after the acquisition. I think Salesforce just started inserting their own branding more recently.
    - mst 4y ago
      At a guess, Salesforce bought it hoping it would take over the entire space rather than merely having the capacity to be one profitable provider among many, and lost interest once it became clear it was going to be the latter rather than the former. (guess based only on having watched things unfold from the outside without ever being a user/customer so please do add salt to taste)
      - hot_gril 4y ago
        As a customer with no knowledge of the inside, this is all I've noticed, which probably misses some things: 1. GitHub integration breach, which wasn't handled well but luckily didn't affect me. 2. No more free-tier DBs :( 3. Salesforce logo, which ofc doesn't matter. Hearing vague negative things here makes me nervous about the future because I really like Heroku and don't want to be stuck using AWS directly.
m00x 4y ago
I don't really understand. You were paying for the dyno, but they deleted your instance anyway?
- riebschlager 4y ago
  They were likely paying for a web dyno, but were using a free database instance.
- jack_riminton 4y ago
  Free db but paid dyno in this case I think
- danpalmer 4y ago
  They deleted the database. On Heroku, databases and dynos are completely distinct entities, each with their own payment plans. When you set up a dyno and “add” a database, as most people do, it’s really easy to think that they are part of the same plan, and conclude that the “pro” or paid dyno is actually the combo of the dyno and database. They are not. I understand why they did this, databases can be shared between applications, that’s handy, but this is a sharp edge.
  - favorited 4y ago
    The least they could have done is disconnect the database without deleting the data, so users would at least have a chance to make the decision to either delete or start paying.
    - tetha 4y ago
      This is something I've put into most workflows at $(WORKSPACE=/=heroku) that end up disabling a customers systems. It depends a bit on how hard we have to kill a system for various factors. But for the softest setting, we usually start pulling DNS or similar access configs for a minute, wait for a minute, disable for 2, wait for 2, ... and the 30 minutes never end. This gets people on the phone with support quickly if they don't react to their account manager or mails. Like, we've had a possible security incident coming from a customer system and they didn't react - because they tried to hide their processes internally. Scream-Testing got us on calls with their InfoSec in a hurry.
brightball 4y ago
Best option is https://www.crunchydata.com/ https://www.crunchydata.com/ IMO.
robbiep 4y ago
I also had this happen - unbelievable really
mtmail 4y ago
For reference here's the email another user received in November. "Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651 https://news.ycombinator.com/item?id=33755651
- lalopalota 4y ago
  For me, the messaging in the announcements, dashboard, emails, and discussions here was clear and obvious that free databases were going to be deleted. If someone managed to miss all that, that's on them.
  - fireworks 4y ago
    No in-app notifications, no emails, and I was travelling and not following the news closely. I don't think I should have to rely on forums to know if I'm going to have all my data deleted without warning as a paying customer anyway.
numpad0 4y ago
Heroku also had weird business with Skeb.jp on 12/23-24 JST that was literally solved under the table - after Skeb redeployed to AWS. It seemed like a policy decision being made at Salesforce which owns Heroku. Whether such behavior adds confidence to their business users, I don’t know.
tflinton 4y ago
Heroku could have done better. At a bare minimum offering an option to download the existing data for 1-year or convert it to a paid db would have been better options and not brought this type of despise from the community. I still use Heroku on a daily basis and wouldn't say this has caused me to re-evaluate my decision to stay with them, but then again i'm not using anything free from them.
celestialcheese 4y ago
Yup - this happened to me too. It was wildly frustrating, but they did send few emails looking back - it was just a really small project and it wasn't on my radar. What pissed me off most is that I WAS paying for the account. I was paying $17/mo for redis and dynos, so there was an active card on file. Why not just start charging for the postgres db, and only delete if there's no active billing? Heroku was already a no-go for me with new projects, I just keep old things running in there since it's too much work to migrate off. This just cements that for me.
- mst 4y ago
  > Why not just start charging for the postgres db, and only delete if there's no active billing? Probably because the original terms you agreed to were written not anticipating Salesforce perpetrating this and so while I would not at all be surprised if the vast majority of customers in your position would've been entirely happy with it they probably didn't have a legal path to do so.
  - jfoster 4y ago
    > they probably didn't have a legal path to do so. That makes no sense. Most large companies change both agreements and prices via notification emails all the time. Probably even SalesForce does so.
    - mst 4y ago
      Given a Heroku employee showed up in a different subthread of this comment section saying thay wanted to change things but couldn't, in spite of a long discussion between the technical and legal sides trying to find a way to do so, it seems to me to be at least plausible that in this case for whatever reason that wasn't an option. If you want to do a close read of the terms and conditions/contract language as of a reasonable guess as to when OP signed up and link the section you believe does give them the right to make such a change unilaterally, I'd be happy to read it and discuss further. My current position in the mean time will, however, remain that while there's always a chance (often a pretty good one) that my hypothesis is wrong, claiming it makes no sense at all is a stronger claim than is justified by the information forming the basis of this discussion.
tchock23 4y ago
I also didn’t receive a notification about it, but some good samaritan HN user posted days before to give people a heads up. That’s the only way I found out.
favorited 4y ago
Same exact thing happened to me. Out of nowhere, my (paid) app started failing. It was only after signing in did I see that the entire postgres database had been removed. No warning that this was going to happen, no ability to recover a backup. Just... deleted. Not the end of the world for me, because 95% of the data I had in there had already been processed, but I did lose some. Complete joke of a service.
deepsun 4y ago
I remember in Google one project sent some announcement to incorrect recipients. Upon investigation it turned on that the bug was in all SQL queries that selected relevant customers, but it was broken for years and no one complained before our investigation.
sneak 4y ago
Anything stored in a cloud provider you're not paying a minimum of $10k a month should be assumed to be subject to evaporation without recourse at any time. Make offsite backups. For this use case, a raspberry pi 4 with a 1TB SD card hidden somewhere in your home with a cronjob is probably more than enough.
bryanrasmussen 4y ago
reading this I keep thinking the real questions are: 1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they? 2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were not being sent. But maybe it wasn't that important for Heroku. Maybe it was not known that notices were not being sent for a while - or was it known immediately but things on other parts of business chugged along anyway. 3. If they knew notices were not being sent and they went ahead and deleted db anyway, seems messed up, but that would probably be ok with people if they had data retention for people who did not get notices sent. 4. The whole thing your stuff can be deleted at any time without telling you is basically probably true almost everywhere in that notice sending can have a bug and deletion of stuff is probably not adequately tied to notice sending so that if notice not sent automatic deletion is stopped. Which I'm thinking is probably everywhere - if you work somewhere with automatic deletion and a notice sending module - what happens? Is this scenario handled? 5. answer to this is probably not, but is there a legal issue if notices not sent and stuff deleted, issue might be if some notices were sent - if account A gets notice about deletion and is thus able to act on it and account B does not get notice and is thus not able to act on it there might be a ground for action. Probably not, but when something seems unfair there might be a law that can be stretched to fit it.
- mst 4y ago
  My guess would be that a tiny percentage of accounts didn't get their notices sent, nobody complained about it because they didn't realise they were supposed to be getting them (or at least for whatever reason no such complaints were escalated to the people who would go "oh shit" upon seeing one) and some unfortunate combination of technical factors meant whatever internal monitoring existed didn't pick up the omissions. Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.
MzHN 4y ago
We also had some of our free databases on our Team tier projects nuked, which Heroku said would instead be automatically upgraded to lowest paid tier. No warning emails either, to anyone on our team. My personal account did get a bunch of warning emails but nothing on our business accounts.
latortuga 4y ago
This happened to me but I filed a ticket and they were able to restore it. I guess maybe it was within the 30 day window?
jacobsenscott 4y ago
On heroku be sure you enable database backups with heroku pg:backups:schedule, and also script syncing those backups on a regular bases to some "off site" (non heroku) storage like your own s3 buckets. It is easy to get the URLs to you backups with `heroku pg:backups:url`. I have no faith that if a rouge employee clicks the "delete app" button (because did you know to give an employee the ability to update the ssl cert on your web app you also need to give them permission to delete the whole damn app?) you'll ever be able to get your database back (although you might have a 30 day window to do so, but I wouldn't trust it.)
dboreham 4y ago
They were charging money for thing A, that connected to (free) thing B, and rather than just begin charging the same customer, using same credit card, for thing B, they deleted it. I would love to see the recording of that zoom meeting...
- jfoster 4y ago
  Is there any way for it to make business sense, or was it pure incompetence? I wonder how many databases they deleted that they could have instead started charging for. Seems to be incredibly destructive for all involved. (SF shareholders, Heroku customers, etc.)
kragen 4y ago
does seem like a cautionary story that should rule heroku out for anything except quick experiments
yonrg 4y ago
I can just second most of those comments here. One thing should be clear: It doesn't matter if you have a payed or free plan with someone who stores your data. Always make backups! There are so many things what can go wrong
purpleblue 4y ago
You didn't have a full backup of your database? That's on you. Disaster Recovery is an important part of any project. Heroku could have blipped out of existence by accident and you would still have the same problem.
hot_gril 4y ago
A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CAP theorem). Today I have enough experience to check the fine print for that kind of detail, but anyway such a big change should come with big bold letters IMO. [Edit: To this day I'm still puzzled by what I'm about to describe, so idk if it's Heroku's fault or mine.] I got a call from my colleague one day saying our database had gone back in time. Evidently we lost an hour of records. The code wasn't even capable of deleting rows, and nobody had direct DB access but me, so after leafing through the docs I suspected a failover event caused it. Premium DBs also let you roll back the DB to a previous point in time, and we were able to recover most of our data this way, like Back to the Future. If this really was a failover event, it's super weird if that the backup was more up to date than the standby master, and that a whole hour (rather than minute) was lost.
- nightpool 4y ago
  Having a HA follower is the only different between Premium and Standard tiers, so I'm not really sure what else you expected them to do in this case. Like, premium-6 is 2x the cost of the standard-6 plan explicitly because of the HA follower.
  - hot_gril 4y ago
    Yes. I was inexperienced, saw "high availability," and didn't realize the standby could fall behind and lose data.
    - hot_gril 4y ago
      Also, it's worded like a strict improvement when really it's a tradeoff. You're sacrificing the guarantee of persistence for more availability. I feel like most people who know what this means are not going to want it.
benediktdeicke 4y ago
Oh crap! What a horror story. We recently migrated from Heroku Postgres to Crunchy Bridge and can totally recommend it. So maybe set up your new database there instead.
rurp 4y ago
I remember being quite surprised when I first learned that Heroku was owned by Salesforce, because I had such a different impression of the two companies. I haven't used it in a few years, but Heroku used to be a great platform for certain types of projects. Unfortunately, the number of concerning stories I have heard about them in recent years has discouraged me from ever using Heroku for a future project.
menzoic 4y ago
Times like this are great opportunities for growth. This can help you build consistent habits in the future to have more durable storage practices.
coffeeblack 4y ago
Never trust any company or hardware. Either can fail you at any moment. Always keep your own backup. Still sucks though.
- thinking001001 4y ago
  This is probably the best advice
nineteen999 4y ago
> I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days. Maybe people shouldn't, I don't know because I've never used it, but ... > I learned my lesson about not doing an offsite backup regularly. This is the bigger lesson, no matter whether you are using third party database hosting, or hosting yourself, whether it's a NoSQL database or a SQL one. Once the data is gone, its gone.
tevon 4y ago
The reputation hit to Heroku is real. Its so too bad that they've made these changes, and stopped pushing new features. I built my prior business on heroku and it was wonderful. Now with this current company I've switched to Render (through am looking for something else). For me the writing was on the wall for Heroku and it wasn't worth it to me to dig in there if new features and support wasn't a given. Curious for anyone with an inside perspective. What happened? Heroku was so far ahead of the pack for awhile, then suddenly stopped staying a step ahead. Was this intentional? Why?
- strix_varius 4y ago
  I explained a little here: https://news.ycombinator.com/item?id=34604723 https://news.ycombinator.com/item?id=34604723
ioseph 4y ago
Happened to myself as well, luckily it was just our staging environment but it was while I was on leave. I feel one should reasonably expect their configuration to be stable to 3 months and 30 day data retention is just insulting.
yellowapple 4y ago
Something similar happened to me on IBM Cloud, back when it was called Bluemix. This was early in my career for an early-stage startup (that eventually shuttered - not due to this issue, but rather due to a lack of investor interest and us running out of runway as a result). Easy enough setup: staging and production, each with two Docker containers, one for the (Elixir/Sugar) app and one with the (Postgres) database. Well, one day the production Postgres container just... vanishes. All the storage is gone. After weeks back and forth with IBM's support, they confirm that the loss is permanent. No explanation, no refunds, just "lol fuck you". Naturally, we didn't have backups yet, so the few users we did have now had to start from scratch. I, too, learned my lesson about not doing an offsite backup regularly - and the bigger lesson that Bluemix should've been a last resort option. Funny enough, we had migrated to that from Heroku; we ended up migrating again to AWS (specifically: Elastic Beanstalk).
marcelaguiar 4y ago
Same here. I have 2 big gripes: - I did not receive any email warnings that this would happen. I have a valid email registered with them because I do receive non-promotional emails. However nothing for this. After getting in contact with customer service their rebuttal was that I also would've known by checking the forums/blog. Who on earth is doing that? It's a terrible response. - They detached the DBs at the start of December. I wasn't going to fix my broken personal projects over the holidays. I told myself I would take a look come January. Which by that point it was past the 30 day grace period. In the end, I've had to completely stop paying for the dynos. So Heroku lost business overall from me.
sebastien_b 4y ago
All your databases are belong to /dev/null
frietzkriesler2 4y ago
Bought some HDDs with some left over end of quarter money, found an old dell server, slapped em in installed it in a switch closet, and have the cloud perform backups to this local closet server. I even do this at home. I don't trust these guys with anything. Sorry for your loss
pgrenn 4y ago
Same thing happened to me. My personal rails website, which I kept personal blog postings in a postgres database got completely lost. Most of those post date back to 2014. I made some mistakes like not backing up my db, not keeping up on maintenance. But didn't think it would result in a complete loss of database considering I was still paying heroku. This app had been running since 2014. After this incident I removed my website from heroku.
- simonw 4y ago
  In case this is useful, I recovered some lost content from my blog using the Internet Archive a few years ago and published notes about how I did it here: https://simonwillison.net/2017/Oct/8/missing-content/ https://simonwillison.net/2017/Oct/8/missing-content/
  - pgrenn 4y ago
    Interesting, good to know. I do have a internet archive of my website as well.
pattrn 4y ago
Same exact thing happened to a friend of mine who was just about to launch. He didn't realize that he had only paid for the dynos, but was using a free tier DB. No communication at all, and they wiped out his data. Needless to say, he is no longer a customer. He's moved on to Render since then (and he's now backing up his data offsite). Painful lesson to learn, but at least he hadn't launched his product yet.
userbinator 4y ago
"Cloud is just a fancy word for someone else's computer."
reiderrider 4y ago
Similar issue with billing post the Salesforce acquisition. We’re on a Enterprise Heroku plan but the billing could only be handled within a Salesforce account (which we don’t have). Autopay disconnected and our app went offline for 3 hours until we could confirm payment over the phone. Billing went from self serve to having to call in to provide a credit card over the phone.
kakarukeys 4y ago
Not surprised to hear that. This happened to me in 2019, and for the whole Heroku account not just 1 database. See https://news.ycombinator.com/item?id=21969358 https://news.ycombinator.com/item?id=21969358. They were going downhill since Salesforce takeover.
kuon 4y ago
Heroku was so nice "back in the day", it was expensive but served me well. I started having some issues before the acquisition so I moved away. I guess I was lucky for that. They must have had some billing problem because I got charged for months after stopping everything, I was able to block the charges at the credit card level but they kept trying and I never had any answer nor explanation from them.
ehPReth 4y ago
What a sad, money grubbing set of actions from Heroku. Tarnishing their brand forever to save a few dollars. Shame on them for doing this. There were so many better ways to deal with this and still save money in the longer run such as, https://news.ycombinator.com/item?id=33759178 https://news.ycombinator.com/item?id=33759178.
6451937099 4y ago
[dead]
6451937099 4y ago
[dead]
93po 4y ago
Heroku banned my account with several income generating projects on it with zero notice. When I called them they treated me like a criminal and like they were doing me a massive fucking favor by even talking to me. I suspect I was supposed to get email notifications leading up to the ban to take corrective action but 100% did not get anything. For context, someone put in malicious DMCA complaints against my website. Heroku did a shit fucking job (i.e. nothing) verifying the complaints. They provided no ability to dispute that it was bullshit. I took my business elsewhere, not that they care.
anonym29 4y ago
Not your hardware, not your database.
icey 4y ago
The same thing happened to me, but I was fortunate enough to catch it in time for them to restore it. Very disappointing because I was paying Heroku over $100/month for various things and didn’t realize an actively used project was on a free tier.
nzoschke 4y ago
Heroku also has a free or cheap tool to back up databases to s3 and download or copy the backup. Totally understand expectations that a managed database service will keep your data but if it’s truly mission critical take backups…
dtjohnnymonkey 4y ago
Mine was also deleted recently, I just had assumed I wasn’t reading their emails carefully, but good to know I wasn’t the only one.
t312227 4y ago
[flagged]
- spuz 4y ago
  He said he used a paid dyno
hyperman1 4y ago
I am wondering more and more if any trust in a single cloud provider isn't simply an unacceptable risk. The power balance means that a minor error on their side is generally business-ending for you. This is different from almost any other supplier, which can generally be replaced, even if the business is on hold for a few days. So any data recovery plan that should be able answer: what if the relation with our cloud provider disappears unexpectedly. This is not a warning against only heroku. Google is famous for terminating without any recourse random accounts because they felt like it that day. Amazon and especially Microsoft seem more dependable, but even they had their share of business-killing behaviour. So e.g. a backup with another cloud vendor is a requirement for almost any business. And of course, validate it. Easyer said than done at scale, of course, but even a partially failed backup is better than nothing.
- jfoster 4y ago
  Even the seemingly independent ones can get acquired by Google and eventually killed. Anyone remember Parse? It was quite successful until Facebook got hold of it and seemingly out of nowhere decided to destroy it.
carmenhchung 4y ago
This exact same thing happened to me with a client project I was working on - my client was paying $50/month for dynos, but we received no notifications about needing to pay for the database to avoid it being nuked! On the upside, when I reached out to Heroku, they did offer to recover the database for me - but by this point in time, I'd already moved on with setting up a new one from scratch (with some changes from the original), so this wasn't particularly helpful.
gloosx 4y ago
>your database might be nuked at any time What is not directly possessed by you is not fully YOURS, and probably it is (un)clearly stated in the "free tier" terms of use. Remember kids, when you encounter free cloud thing on top of the free host-able thing, just spend another day and host it yourself, cause you're just facing a sales funnel which will eventually collapse

460 points by fireworks 4y ago

gwbas1c 4y ago

What frustrated me about Heroku nuking free projects was that the paid version was, IMO, too expensive. (Granted, I understand that the free tier was probably a money sink for them.) $5-10 a month is pocket change in some contexts, but for a hobby or a one-off project, it's a huge chunk of change. I really wish they offered a lower-cost tier.

shortcake27 4y ago
Or let me share dynos across all my projects. $5-$10 per month for all projects, sounds good! $5 per month for each project adds up very quickly.
- throwawaykai 4y ago
  That was the new “eco” plan that was introduced alongside the ending of free plans — $5/mo for 1000 hours shared across all eco dynos.
  - gwillen 4y ago
    I definitely wish you guys had advertised that earlier / better. I feel like I got lucky because I put off migrating until just before the sunset, at which point I discovered eco dynos. I swear they either didn't exist or were harder to find when the sunset was first announced. Thanks for making them available at all, though. (Context: I maintain an app for an event that runs about one week a year. The ability to use free dynos the other 51 weeks means we retain the ability to do one-off analytics queries, minor development work, etc. during the off-season, without having to delete and recreate the app or something every year. Eco isn't quite as good as free, but it means we can still have separate staging and production instances during the off-season, without paying extra for staging to be idle, and without having to destroy and recreate staging every year.)
    - shortcake27 4y ago
      Wow, I also don’t recall seeing this either. It must have been marketed poorly. Same for Postgres, looks like there are now some more cost effective options which weren’t available when I migrated my stuff away.
m00x 4y ago
That's 2 cups of coffee, or 1 latte. It's objectively not a huge chunk of change unless you live in a very LCOL area. You can also go to any competitors, which have their smallest tier around the same price. Linode has a $5/m tier. You could also buy a Raspberry Pi for $50 and run it in your closet.
- gwbas1c 4y ago
  > or a hobby or a one-off project I don't want to pay $5 / month indefinitely to host a weekend project. I don't want to deal with the complexities of hosting a weekend project myself. I've done plenty of that. I just don't want to end up spending $100-200 / month to host a lifetime of weekend projects and experiments.
  - m00x 4y ago
    Then you can get a raspberry Pi and host it in your closet.
    - gwbas1c 4y ago
      > I don't want to deal with the complexities of hosting a weekend project myself. I've done plenty of that. That involves port forwarding, figuring out something like dyndns, and whatever I need to share domains on a single IP.
Towaway69 4y ago
My personal take is that I'd rather work with heroku than dealing with aws or gcs. Sure there are others out there that provide the same easy of use, I am just too lazy to switch ;) OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on.
- xp84 4y ago
  > I've never had them wipe a database on me, Give it time, apparently ;) If you're like me and are really into ease of use, I really endorse Digital Ocean. People don't seem to use it in Big Important Companies, but I never had a problem with anything I hosted there when I was my last, medium-sized company, and everything was so easy to manage in the GUI without having to learn proprietary APIs and whole stacks of infrastructure-as-code stuff like in AWS land.
  - Towaway69 4y ago
    DO is also good and less restrictive for a new user than aws (I.e. getting more resources). I remember having to beg to get more than 1(or 2) servers at aws as a new user. I haven't looked recently but does DO have the heroku click to add resources and git push to deploy offerings?
    - xp84 4y ago
      I think this product is the closest: https://www.digitalocean.com/products/app-platform https://www.digitalocean.com/products/app-platform
- gwbas1c 4y ago
  > OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on. About a year ago they shut down my personal blog based on a vague "violation of our terms." There was no violation, either. (I suspect they had to shut down phishing and other abusive sites, and cast a little too wide of a net.) When they claimed they turned it back on, I had to keep pestering them because they goofed something or other. Eventually I couldn't push updates via Github. (I wrote the blog engine as an exercise to learn NodeJS.) Again, if I could have bought an ultra-cheap tier, I would have been happy to pay.

swader999 4y ago

I'm curious how this affected your projects on paid dynos, can you provide more details around that?

fireworks 4y ago
So I did not realize this at first (would have helped if I were, you know, emailed some information about this), but my project was still using a free Postgres addon despite being on a paid dyno setup. So that I suppose was nuked along with the general free tier deprecation.
- m00x 4y ago
  Did they explain why the notifications didn't work?
  - fireworks 4y ago
    "I discussed this with our backend team and upon deeper investigation, we were able to confirm that there was some issue sending the notifications to your email." Nope.
    - dboreham 4y ago
      Guesses 1, 2 and 3 are : their MTA's IP was black listed by a reputable mail hosting provider's spam filtering system (because they send spam).
    - joemi 4y ago
      That phrasing doesn't rule out an issue on your email server's end, like the email could have bounced. (Though I'm guessing that's unlikely.)
      - fireworks 4y ago
        Would be surprised if that was this case - vanilla GSuite setup.

ericpauley 4y ago

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake. In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at night, but clearly that's gone now.

xp84 4y ago
I assumed the value prop of being on Heroku in 2023 was purely "not having to do the pain in the butt to migrate your legacy app that you made on Heroku in the pre-2010 era" - in other words, I would be really shocked if Heroku got significant new business or any growth at all, now that it's just expensive AWS with some basic CI integration points. I also assume Salesforce only bought them as a cash generator and has no interest in investing in it. So if they saved bottom line from this move, that's a win for them. (Feel free to correct my assumptions if I'm very wrong)
- jrochkind1 4y ago
  "Pre-2010"? Heroku only launched in April 2009. So I would assume the vast majority of current customers are not actually from a "pre-2010 era". Heroku didn't even have a postgres add-on until November 2010 -- the salesforce purchase went through in Dec 2010. In fact, the heroku "golden years" were mostly during the early salesforce period. (Heroku put matz, creator of ruby, on staff in July 2011; there was no Heroku API until 2014!). (source for timeline: https://www.heroku.com/about https://www.heroku.com/about) Heroku is still quite a bit of value-added over "AWS with some basic CI integration", people chose it and still choose it because it requires a lot lot lot less in-house expertise and management hours than AWS for most kinds of standard apps. (I'm not totally sure what AWS services/architecutres you are thinking of when you say that; but I'll say: for pretty much all of them.) (Heroku def has more competition in that space than it did 10 years ago, opinions differ on the relative merits) AWS of course already existed when Heroku was launched, so if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then? I think Salesforce thought it would somehow be more "synergistic" with their other offerings than it has turned out to be. That they'd get Salesforce customers on heroku when they needed something beyond the "no-code" tools Salesforce already provided, or that they'd do better at converting heroku customers to salesforce customers than they have been. It does seem to be true that salesforce stopped really investing in new heroku features or improvements some years ago, and seem to be looking to minimize costs while continuing to collect revenue, I agree. (Sometimes I'm not even sure how much they care about continuing to collect revenue...)

collectedparts 4y ago

This happened to me as well: projects with paid dynos but free databases, databases got nuked. How Heroku missed this is beyond me. They managed to screw over paying customers in their broad attempt to stop freeloaders. These are good accounts with credit cards on file. Why not just autocovert me to the lowest tier paid database? FWIW I was able to get them to restore my databases. But I also had free Heroku Redis on one my projects and that, they assured me, is gone forever.

sieabahlpark 4y ago
Redis is a memory store, you're the one who fucked up treating a cache as a persistence layer.
- robertlagrant 4y ago
  You can use Redis as a cache, but it's not fundamentally a cache.
- itake 4y ago
  Redis offers disk persistence [0]. Why can't it be used as a persistence layer? [0] - https://redis.io/docs/management/persistence/ https://redis.io/docs/management/persistence/
  - ceejayoz 4y ago
    Redis offers disk persistence; Heroku's free Redis offering did not.
- deleted 4y ago
  [deleted]
fireworks 4y ago
Right? Like just charge me and tell me. Also, while I still think it is way too aggressive how they deprecated free in general, at the very least... maybe uh... I don't know, let me know if you are going to delete my database? Absolutely insane.

cameronfraser 4y ago

Same happened to me, I didn't receive any email notification or anything, found out when things started failing.

rovr138 4y ago

Well, I moved everything I had to a VPS for the time being because I left it to the end. So I take this opportunity to ask, what alternatives have people moved to? I really haven't gone back to look what's out there. Not looking for free, just alternatives to review.

donnieashok 4y ago
I went full blast into SQLite, agreed there's a migration job in between. But it's totally worth it. In fact my app is now 5x faster.
dcow 4y ago
You can host static sites on DigitalOcean's app platform for free. Their lowest tier offering for non-static sites is like $5/mo. Might be worth checking out.
xp84 4y ago
For what kind of an app? If I was tasked to put an app someplace today, the answer would depend if it was easily containerized, and what kind of scale you're talking about. For small apps, if containerized, I would host on Google Cloud Platform using App Engine Flexible Environment: https://cloud.google.com/appengine/docs/flexible https://cloud.google.com/appengine/docs/flexible Add on their managed database, of course. But also, random VPSs are so cheap that if it works fine as is, you could do a lot worse than just running it on Linode, Digital Ocean, Hetzner, etc.

marvinkennis 4y ago

Heroku sent me repeated messages that they would shut down my account due to inactivity. I was fine with that because finding and turning off the supposedly active Dyno was impossible. Guess what? Still charging me $27 a month for a server I can't even manage.

dns_snek 4y ago
If this is true I think it deserves spotlight in a separate post.

JEDI-HACKER 4y ago

Check your junk mail folder.

javawizard 4y ago
> They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent. Junk mail wasn't the issue in this case.
deleted 4y ago
[deleted]

codegeek 4y ago

"I learned my lesson about not doing an offsite backup regularly" Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider. Heck, we wrote our own script to backup RDS databases offsite as well even though RDS has backups and restore options. I want that database file.

Max-q 4y ago
After the Salesforce takeover? The one in 2010? So Heroku was good in like a year!?
- hot_gril 4y ago
  Heh. I always loved Heroku and have complained that Salesforce is messing it up recently, but turns out the first time I used Heroku was like 5 years after the acquisition. I think Salesforce just started inserting their own branding more recently.
  - mst 4y ago
    At a guess, Salesforce bought it hoping it would take over the entire space rather than merely having the capacity to be one profitable provider among many, and lost interest once it became clear it was going to be the latter rather than the former. (guess based only on having watched things unfold from the outside without ever being a user/customer so please do add salt to taste)
    - hot_gril 4y ago
      As a customer with no knowledge of the inside, this is all I've noticed, which probably misses some things: 1. GitHub integration breach, which wasn't handled well but luckily didn't affect me. 2. No more free-tier DBs :( 3. Salesforce logo, which ofc doesn't matter. Hearing vague negative things here makes me nervous about the future because I really like Heroku and don't want to be stuck using AWS directly.

m00x 4y ago

I don't really understand. You were paying for the dyno, but they deleted your instance anyway?

riebschlager 4y ago
They were likely paying for a web dyno, but were using a free database instance.
jack_riminton 4y ago
Free db but paid dyno in this case I think
danpalmer 4y ago
They deleted the database. On Heroku, databases and dynos are completely distinct entities, each with their own payment plans. When you set up a dyno and “add” a database, as most people do, it’s really easy to think that they are part of the same plan, and conclude that the “pro” or paid dyno is actually the combo of the dyno and database. They are not. I understand why they did this, databases can be shared between applications, that’s handy, but this is a sharp edge.
- favorited 4y ago
  The least they could have done is disconnect the database without deleting the data, so users would at least have a chance to make the decision to either delete or start paying.
  - tetha 4y ago
    This is something I've put into most workflows at $(WORKSPACE=/=heroku) that end up disabling a customers systems. It depends a bit on how hard we have to kill a system for various factors. But for the softest setting, we usually start pulling DNS or similar access configs for a minute, wait for a minute, disable for 2, wait for 2, ... and the 30 minutes never end. This gets people on the phone with support quickly if they don't react to their account manager or mails. Like, we've had a possible security incident coming from a customer system and they didn't react - because they tried to hide their processes internally. Scream-Testing got us on calls with their InfoSec in a hurry.

brightball 4y ago

Best option is https://www.crunchydata.com/ https://www.crunchydata.com/ IMO.

robbiep 4y ago

I also had this happen - unbelievable really

mtmail 4y ago

For reference here's the email another user received in November. "Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651 https://news.ycombinator.com/item?id=33755651

lalopalota 4y ago
For me, the messaging in the announcements, dashboard, emails, and discussions here was clear and obvious that free databases were going to be deleted. If someone managed to miss all that, that's on them.
- fireworks 4y ago
  No in-app notifications, no emails, and I was travelling and not following the news closely. I don't think I should have to rely on forums to know if I'm going to have all my data deleted without warning as a paying customer anyway.

numpad0 4y ago

Heroku also had weird business with Skeb.jp on 12/23-24 JST that was literally solved under the table - after Skeb redeployed to AWS. It seemed like a policy decision being made at Salesforce which owns Heroku. Whether such behavior adds confidence to their business users, I don’t know.

tflinton 4y ago

Heroku could have done better. At a bare minimum offering an option to download the existing data for 1-year or convert it to a paid db would have been better options and not brought this type of despise from the community. I still use Heroku on a daily basis and wouldn't say this has caused me to re-evaluate my decision to stay with them, but then again i'm not using anything free from them.

celestialcheese 4y ago

Yup - this happened to me too. It was wildly frustrating, but they did send few emails looking back - it was just a really small project and it wasn't on my radar. What pissed me off most is that I WAS paying for the account. I was paying $17/mo for redis and dynos, so there was an active card on file. Why not just start charging for the postgres db, and only delete if there's no active billing? Heroku was already a no-go for me with new projects, I just keep old things running in there since it's too much work to migrate off. This just cements that for me.

mst 4y ago
> Why not just start charging for the postgres db, and only delete if there's no active billing? Probably because the original terms you agreed to were written not anticipating Salesforce perpetrating this and so while I would not at all be surprised if the vast majority of customers in your position would've been entirely happy with it they probably didn't have a legal path to do so.
- jfoster 4y ago
  > they probably didn't have a legal path to do so. That makes no sense. Most large companies change both agreements and prices via notification emails all the time. Probably even SalesForce does so.
  - mst 4y ago
    Given a Heroku employee showed up in a different subthread of this comment section saying thay wanted to change things but couldn't, in spite of a long discussion between the technical and legal sides trying to find a way to do so, it seems to me to be at least plausible that in this case for whatever reason that wasn't an option. If you want to do a close read of the terms and conditions/contract language as of a reasonable guess as to when OP signed up and link the section you believe does give them the right to make such a change unilaterally, I'd be happy to read it and discuss further. My current position in the mean time will, however, remain that while there's always a chance (often a pretty good one) that my hypothesis is wrong, claiming it makes no sense at all is a stronger claim than is justified by the information forming the basis of this discussion.

tchock23 4y ago

I also didn’t receive a notification about it, but some good samaritan HN user posted days before to give people a heads up. That’s the only way I found out.

favorited 4y ago

Same exact thing happened to me. Out of nowhere, my (paid) app started failing. It was only after signing in did I see that the entire postgres database had been removed. No warning that this was going to happen, no ability to recover a backup. Just... deleted. Not the end of the world for me, because 95% of the data I had in there had already been processed, but I did lose some. Complete joke of a service.

deepsun 4y ago

I remember in Google one project sent some announcement to incorrect recipients. Upon investigation it turned on that the bug was in all SQL queries that selected relevant customers, but it was broken for years and no one complained before our investigation.

sneak 4y ago

Anything stored in a cloud provider you're not paying a minimum of $10k a month should be assumed to be subject to evaporation without recourse at any time. Make offsite backups. For this use case, a raspberry pi 4 with a 1TB SD card hidden somewhere in your home with a cronjob is probably more than enough.

bryanrasmussen 4y ago

reading this I keep thinking the real questions are: 1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they? 2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were not being sent. But maybe it wasn't that important for Heroku. Maybe it was not known that notices were not being sent for a while - or was it known immediately but things on other parts of business chugged along anyway. 3. If they knew notices were not being sent and they went ahead and deleted db anyway, seems messed up, but that would probably be ok with people if they had data retention for people who did not get notices sent. 4. The whole thing your stuff can be deleted at any time without telling you is basically probably true almost everywhere in that notice sending can have a bug and deletion of stuff is probably not adequately tied to notice sending so that if notice not sent automatic deletion is stopped. Which I'm thinking is probably everywhere - if you work somewhere with automatic deletion and a notice sending module - what happens? Is this scenario handled? 5. answer to this is probably not, but is there a legal issue if notices not sent and stuff deleted, issue might be if some notices were sent - if account A gets notice about deletion and is thus able to act on it and account B does not get notice and is thus not able to act on it there might be a ground for action. Probably not, but when something seems unfair there might be a law that can be stretched to fit it.

mst 4y ago
My guess would be that a tiny percentage of accounts didn't get their notices sent, nobody complained about it because they didn't realise they were supposed to be getting them (or at least for whatever reason no such complaints were escalated to the people who would go "oh shit" upon seeing one) and some unfortunate combination of technical factors meant whatever internal monitoring existed didn't pick up the omissions. Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.

MzHN 4y ago

We also had some of our free databases on our Team tier projects nuked, which Heroku said would instead be automatically upgraded to lowest paid tier. No warning emails either, to anyone on our team. My personal account did get a bunch of warning emails but nothing on our business accounts.

latortuga 4y ago

This happened to me but I filed a ticket and they were able to restore it. I guess maybe it was within the 30 day window?

jacobsenscott 4y ago

On heroku be sure you enable database backups with heroku pg:backups:schedule, and also script syncing those backups on a regular bases to some "off site" (non heroku) storage like your own s3 buckets. It is easy to get the URLs to you backups with `heroku pg:backups:url`. I have no faith that if a rouge employee clicks the "delete app" button (because did you know to give an employee the ability to update the ssl cert on your web app you also need to give them permission to delete the whole damn app?) you'll ever be able to get your database back (although you might have a 30 day window to do so, but I wouldn't trust it.)

dboreham 4y ago

They were charging money for thing A, that connected to (free) thing B, and rather than just begin charging the same customer, using same credit card, for thing B, they deleted it. I would love to see the recording of that zoom meeting...

jfoster 4y ago
Is there any way for it to make business sense, or was it pure incompetence? I wonder how many databases they deleted that they could have instead started charging for. Seems to be incredibly destructive for all involved. (SF shareholders, Heroku customers, etc.)

kragen 4y ago

does seem like a cautionary story that should rule heroku out for anything except quick experiments

yonrg 4y ago

I can just second most of those comments here. One thing should be clear: It doesn't matter if you have a payed or free plan with someone who stores your data. Always make backups! There are so many things what can go wrong

purpleblue 4y ago

You didn't have a full backup of your database? That's on you. Disaster Recovery is an important part of any project. Heroku could have blipped out of existence by accident and you would still have the same problem.

hot_gril 4y ago

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CAP theorem). Today I have enough experience to check the fine print for that kind of detail, but anyway such a big change should come with big bold letters IMO. [Edit: To this day I'm still puzzled by what I'm about to describe, so idk if it's Heroku's fault or mine.] I got a call from my colleague one day saying our database had gone back in time. Evidently we lost an hour of records. The code wasn't even capable of deleting rows, and nobody had direct DB access but me, so after leafing through the docs I suspected a failover event caused it. Premium DBs also let you roll back the DB to a previous point in time, and we were able to recover most of our data this way, like Back to the Future. If this really was a failover event, it's super weird if that the backup was more up to date than the standby master, and that a whole hour (rather than minute) was lost.

nightpool 4y ago
Having a HA follower is the only different between Premium and Standard tiers, so I'm not really sure what else you expected them to do in this case. Like, premium-6 is 2x the cost of the standard-6 plan explicitly because of the HA follower.
- hot_gril 4y ago
  Yes. I was inexperienced, saw "high availability," and didn't realize the standby could fall behind and lose data.
  - hot_gril 4y ago
    Also, it's worded like a strict improvement when really it's a tradeoff. You're sacrificing the guarantee of persistence for more availability. I feel like most people who know what this means are not going to want it.

benediktdeicke 4y ago

Oh crap! What a horror story. We recently migrated from Heroku Postgres to Crunchy Bridge and can totally recommend it. So maybe set up your new database there instead.

rurp 4y ago

I remember being quite surprised when I first learned that Heroku was owned by Salesforce, because I had such a different impression of the two companies. I haven't used it in a few years, but Heroku used to be a great platform for certain types of projects. Unfortunately, the number of concerning stories I have heard about them in recent years has discouraged me from ever using Heroku for a future project.

menzoic 4y ago

Times like this are great opportunities for growth. This can help you build consistent habits in the future to have more durable storage practices.

coffeeblack 4y ago

Never trust any company or hardware. Either can fail you at any moment. Always keep your own backup. Still sucks though.

thinking001001 4y ago
This is probably the best advice

nineteen999 4y ago

> I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days. Maybe people shouldn't, I don't know because I've never used it, but ... > I learned my lesson about not doing an offsite backup regularly. This is the bigger lesson, no matter whether you are using third party database hosting, or hosting yourself, whether it's a NoSQL database or a SQL one. Once the data is gone, its gone.

tevon 4y ago

The reputation hit to Heroku is real. Its so too bad that they've made these changes, and stopped pushing new features. I built my prior business on heroku and it was wonderful. Now with this current company I've switched to Render (through am looking for something else). For me the writing was on the wall for Heroku and it wasn't worth it to me to dig in there if new features and support wasn't a given. Curious for anyone with an inside perspective. What happened? Heroku was so far ahead of the pack for awhile, then suddenly stopped staying a step ahead. Was this intentional? Why?

strix_varius 4y ago
I explained a little here: https://news.ycombinator.com/item?id=34604723 https://news.ycombinator.com/item?id=34604723

ioseph 4y ago

Happened to myself as well, luckily it was just our staging environment but it was while I was on leave. I feel one should reasonably expect their configuration to be stable to 3 months and 30 day data retention is just insulting.

yellowapple 4y ago

Something similar happened to me on IBM Cloud, back when it was called Bluemix. This was early in my career for an early-stage startup (that eventually shuttered - not due to this issue, but rather due to a lack of investor interest and us running out of runway as a result). Easy enough setup: staging and production, each with two Docker containers, one for the (Elixir/Sugar) app and one with the (Postgres) database. Well, one day the production Postgres container just... vanishes. All the storage is gone. After weeks back and forth with IBM's support, they confirm that the loss is permanent. No explanation, no refunds, just "lol fuck you". Naturally, we didn't have backups yet, so the few users we did have now had to start from scratch. I, too, learned my lesson about not doing an offsite backup regularly - and the bigger lesson that Bluemix should've been a last resort option. Funny enough, we had migrated to that from Heroku; we ended up migrating again to AWS (specifically: Elastic Beanstalk).

marcelaguiar 4y ago

Same here. I have 2 big gripes: - I did not receive any email warnings that this would happen. I have a valid email registered with them because I do receive non-promotional emails. However nothing for this. After getting in contact with customer service their rebuttal was that I also would've known by checking the forums/blog. Who on earth is doing that? It's a terrible response. - They detached the DBs at the start of December. I wasn't going to fix my broken personal projects over the holidays. I told myself I would take a look come January. Which by that point it was past the 30 day grace period. In the end, I've had to completely stop paying for the dynos. So Heroku lost business overall from me.

sebastien_b 4y ago

All your databases are belong to /dev/null

frietzkriesler2 4y ago

Bought some HDDs with some left over end of quarter money, found an old dell server, slapped em in installed it in a switch closet, and have the cloud perform backups to this local closet server. I even do this at home. I don't trust these guys with anything. Sorry for your loss

pgrenn 4y ago

Same thing happened to me. My personal rails website, which I kept personal blog postings in a postgres database got completely lost. Most of those post date back to 2014. I made some mistakes like not backing up my db, not keeping up on maintenance. But didn't think it would result in a complete loss of database considering I was still paying heroku. This app had been running since 2014. After this incident I removed my website from heroku.

simonw 4y ago
In case this is useful, I recovered some lost content from my blog using the Internet Archive a few years ago and published notes about how I did it here: https://simonwillison.net/2017/Oct/8/missing-content/ https://simonwillison.net/2017/Oct/8/missing-content/
- pgrenn 4y ago
  Interesting, good to know. I do have a internet archive of my website as well.

pattrn 4y ago

Same exact thing happened to a friend of mine who was just about to launch. He didn't realize that he had only paid for the dynos, but was using a free tier DB. No communication at all, and they wiped out his data. Needless to say, he is no longer a customer. He's moved on to Render since then (and he's now backing up his data offsite). Painful lesson to learn, but at least he hadn't launched his product yet.

userbinator 4y ago

"Cloud is just a fancy word for someone else's computer."

reiderrider 4y ago

Similar issue with billing post the Salesforce acquisition. We’re on a Enterprise Heroku plan but the billing could only be handled within a Salesforce account (which we don’t have). Autopay disconnected and our app went offline for 3 hours until we could confirm payment over the phone. Billing went from self serve to having to call in to provide a credit card over the phone.

kakarukeys 4y ago

Not surprised to hear that. This happened to me in 2019, and for the whole Heroku account not just 1 database. See https://news.ycombinator.com/item?id=21969358 https://news.ycombinator.com/item?id=21969358. They were going downhill since Salesforce takeover.

kuon 4y ago

Heroku was so nice "back in the day", it was expensive but served me well. I started having some issues before the acquisition so I moved away. I guess I was lucky for that. They must have had some billing problem because I got charged for months after stopping everything, I was able to block the charges at the credit card level but they kept trying and I never had any answer nor explanation from them.

ehPReth 4y ago

What a sad, money grubbing set of actions from Heroku. Tarnishing their brand forever to save a few dollars. Shame on them for doing this. There were so many better ways to deal with this and still save money in the longer run such as, https://news.ycombinator.com/item?id=33759178 https://news.ycombinator.com/item?id=33759178.

6451937099 4y ago

[dead]

6451937099 4y ago

[dead]

93po 4y ago

Heroku banned my account with several income generating projects on it with zero notice. When I called them they treated me like a criminal and like they were doing me a massive fucking favor by even talking to me. I suspect I was supposed to get email notifications leading up to the ban to take corrective action but 100% did not get anything. For context, someone put in malicious DMCA complaints against my website. Heroku did a shit fucking job (i.e. nothing) verifying the complaints. They provided no ability to dispute that it was bullshit. I took my business elsewhere, not that they care.

anonym29 4y ago

Not your hardware, not your database.

icey 4y ago

The same thing happened to me, but I was fortunate enough to catch it in time for them to restore it. Very disappointing because I was paying Heroku over $100/month for various things and didn’t realize an actively used project was on a free tier.

nzoschke 4y ago

Heroku also has a free or cheap tool to back up databases to s3 and download or copy the backup. Totally understand expectations that a managed database service will keep your data but if it’s truly mission critical take backups…

dtjohnnymonkey 4y ago

Mine was also deleted recently, I just had assumed I wasn’t reading their emails carefully, but good to know I wasn’t the only one.

t312227 4y ago

[flagged]

spuz 4y ago
He said he used a paid dyno

hyperman1 4y ago

I am wondering more and more if any trust in a single cloud provider isn't simply an unacceptable risk. The power balance means that a minor error on their side is generally business-ending for you. This is different from almost any other supplier, which can generally be replaced, even if the business is on hold for a few days. So any data recovery plan that should be able answer: what if the relation with our cloud provider disappears unexpectedly. This is not a warning against only heroku. Google is famous for terminating without any recourse random accounts because they felt like it that day. Amazon and especially Microsoft seem more dependable, but even they had their share of business-killing behaviour. So e.g. a backup with another cloud vendor is a requirement for almost any business. And of course, validate it. Easyer said than done at scale, of course, but even a partially failed backup is better than nothing.

jfoster 4y ago
Even the seemingly independent ones can get acquired by Google and eventually killed. Anyone remember Parse? It was quite successful until Facebook got hold of it and seemingly out of nowhere decided to destroy it.

carmenhchung 4y ago

This exact same thing happened to me with a client project I was working on - my client was paying $50/month for dynos, but we received no notifications about needing to pay for the database to avoid it being nuked! On the upside, when I reached out to Heroku, they did offer to recover the database for me - but by this point in time, I'd already moved on with setting up a new one from scratch (with some changes from the original), so this wasn't particularly helpful.

gloosx 4y ago

>your database might be nuked at any time What is not directly possessed by you is not fully YOURS, and probably it is (un)clearly stated in the "free tier" terms of use. Remember kids, when you encounter free cloud thing on top of the free host-able thing, just spend another day and host it yourself, cause you're just facing a sales funnel which will eventually collapse