13 ms·

Heroku has been running a second copy of my scheduler instance

297 points by haki 3y ago

maushu 3y ago
This is why I use the git commit short hash as part of the identification of the service (eg: container name). Allows for quick check of which is the running code version and identify problems like these.
- zacksiri 3y ago
  That’s smart. I like it.
- rcarmo 3y ago
  Wouldn’t help in this case if you redeployed the same version. I usually log the PID and some form of host ID.
- bomewish 3y ago
  So you have to constantly rename it with each commit or what?
vinhboy 3y ago
Similar thing happened to me. Took a long time to realize there was a zombie Heroku instance running jobs. That horror when you delete code but it's still running, you're just so confused... I've forgotten how I resolved it, but man it was a pain to diagnose.
akhayam 3y ago
Everyone can relate to the sad demise of Heroku at this time. Sad, because they could've dominated this space which still has a lot of issues. Yet-another-infrastructure-group nailed it in their graphic about "what your infrastructure says about you?"... "Heroku => "You are either negotiating the pricing or deplatforming" https://www.yaig.dev/ https://www.yaig.dev/
zatkin 3y ago
If Heroku is running some zombie instance of your app, what if you were to just kill the instance you control and let Heroku run the zombie instance? When your service goes fully down, you'll know they've killed the zombie instance, so you can then run a clean copy.
- badestrand 3y ago
  Not being able to update your code plus your app will completely crash randomly at some point between now and eternity - sounds great!
  - oneepic 3y ago
    It's salvageable. Maybe spin up some kind of Unix watch that pings the zombie and deploys the new version. (but then, what happens if you get another zombie... crap)
jchw 3y ago
In case it's not clear from the article, Nixpacks build OCI (essentially, Docker) images, so they are perfectly vendor neutral as far as I know. The ending ambiguously implies that Nixpacks may be undesirable vs using Docker, but actually it's just another way to build an image. Fly.io supports Nixpacks out of the box too, I believe, which makes it even easier to use them with Fly (although, you can also just push an OCI image directly to the fly.io registry and deploy with that, which I have been experimenting with in spare time as an option for using Nix itself to build an OCI image for Fly. I'm sure there will be an even nicer way some day, but it works for now...) I'm sure the author knew this, but I wanted to iterate it in case anyone misunderstood.
- shibel 3y ago
  OP here: I agree my phrasing is kind of problematic. Will rephrase shortly. Thank you for bringing that up.
sufehmi 3y ago
Clearly the author still hasn't learned his lesson: his "no more vendor lock" solution is another vendor(s) - which is always able to vendor-lock him, again. It is truly mind blowing on how many people keep falling for one of the oldest trick in IT : vendor lock-in. === No more vendor lock In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development. The landscape is very different now with providers like Render, Railway, and more
- emodendroket 3y ago
  So what should he be doing, building his own data center?
  - TylerE 3y ago
    Rent a $5/month VPS, install a few packages? Or just run a linux vm locally.
    - emodendroket 3y ago
      So, rely on yet another vendor, or do as I suggested except without any semblance of reliabiliy.
      - TylerE 3y ago
        Some one just starting out is not Google. They do not have Google's problems. They do not need Google's complexity, or a dozen layers of docker/terraform/k8s bullshit.
        akie 3y ago
        Dockerizing (what the author said he would do) is almost always worth it though.
        slyall 3y ago
        Agreed, instead of "a dozen layers of docker/terraform/k8s bullshit" they just need they application code and some config that will take them an hour to setup once.
dclowd9901 3y ago
Author and me have so much in common. From journalism to software engineering and our formative career shift years were spent on home made music apps. Now I’m wondering just how many of us are out there.
- shibel 3y ago
  Author here. Small world! Would love to see your app if you don’t mind sharing. I’m @SHxKM on Twitter if you’d like to connect.
  - dclowd9901 3y ago
    Unfortunately it’s abandonware now. It was called Showhopping and it would use the Scrobbler API and a list of your favorite bands and find out if any bands you might like (including the ones you listed) were playing in a given area any time soon. You could give it a location and a time range and it would give a list of shows playing on a map with a percentage back based on the likelihood that you’d enjoy the show. It was very lightweight with no sign up and I had it in mind people could use it like Yelp essentially, just ad-hoc. https://lifehacker.com/showhopping-finds-upcoming-shows-around-you-for-your-fa-5808835 https://lifehacker.com/showhopping-finds-upcoming-shows-arou... Anyway, nice to meet you! I enjoyed your write up.
teaearlgraycold 3y ago
I vote for moving to Render! My god I wish I could short Heroku stock.
- 8n4vidtmkvmk 3y ago
  Looks really expensive. And weird pricing. I click "pricing" and it gives me per user costs. Wat? I thought I was hosting stuff, not managing teams. Click on +Computing costs and it's like $0! $0! $0! Check the fineprint and its like 4GB RAM and 2 CPUs is $85/mo, or you can get the same at DigitalOcean for $24. Then add Postgres for another $95. Or if you're cheap like me, you self-host that in your $24 droplet and pay nothing extra. And then they want you to pay for Crons....instead of just running a cronjob on your dedicated compute? What is this world coming to?
  - toyg 3y ago
    Everyone charges for convenience. A lot of people don't want the headache to run and secure their own Postgres, for example. It's a trade-off, same as servicing your car vs just handing it off to a garage mechanic.
    - 8n4vidtmkvmk 3y ago
      Yeah I understand charging for convenience... kinda. Markup is too big IMO. Should be like 10-20% more not 3x. And then they should make up the difference in volume because why would you not choose the more convenient platform if they provide the same great hosting but managed or easier to use.
  - jmuguy 3y ago
    Its been a little while since I looked at Render and their pricing definitely used to be a lot simpler. Charging per user is just stupid. And then weird compute costs with plans like "Pro Plus" and "Pro Ultra". "Web Services" and "Private Services". WTF is this shit.
  - teaearlgraycold 3y ago
    It's a managed service. You pay a premium for it. Features like their blueprint file make it worth the price IMO - https://render.com/docs/blueprint-spec https://render.com/docs/blueprint-spec If you are working on a team and you want everyone to be able to spin up a copy of the production infrastruction for each pull request then PaaS is the way to go. If you just have some random one-off production environment and you are happy to either manage the state of a HDD or a k8s cluster yourself (which comes down to will more so than skill) then DO is the way to go.
lorenzk 3y ago
Not sure if you have done this already but: If you rotate your database, redis and email provider passwords/tokens, the zombie process might still be running but will not have any effects any more.
- domh 3y ago
  This is a really smart idea! If you can't kill the process, make it completely ineffective by making it's api calls fail.
- shibel 3y ago
  OP here: that’s a good idea that did come up to mind (specifically rotating the Redis queue credentials). I was just sure Heroku’s gonna take care of this shit by now. BTW, I’m not 100% sure it’ll work: if Heroku updates that zombie dyno’s environment variables, it may still have access to the new credentials :/ (My Redis instance is also a Heroku add-on)
  - gsa 3y ago
    Rotate a token, but also mangle it and change to the correct one in code. eg: change database url from `postgres://` to `invalid://` and replace it in code.
  - neurostimulant 3y ago
    > BTW, I’m not 100% sure it’ll work: if Heroku updates that zombie dyno’s environment variables, it may still have access to the new credentials :/ (My Redis instance is also a Heroku add-on) If the configuration only read once during startup, the celery process won't use the new configuration unless it's restarted, right?
    - DangitBobby 3y ago
      Heroku will automatically restart dynos if you change environment variables.
j45 3y ago
Maybe Dokku us an option to self host and maintain Heroku code
krokoo 3y ago
[dead]
marvinkennis 3y ago
Heroku has been an absolute dumpster fire. Every time their name comes up I can't help but mention that they deleted my account for inactivity, but have still been charging my credit card. No easy way to cancel because... they deleted my account.
- djbusby 3y ago
  Gotta chargeback and tell the CC company to block. Then file complaint with their states AG.
- ranting-moth 3y ago
  File a chargeback each time they charge your card. The merchant has to pay the transaction fees both ways and a fee for the chargeback.
  - joering2 3y ago
    But don't do it if its Google. I had been charged for g-suite for domain that expired and someone else picked it up (and parked), long story short after 3 months of back-and-forth with "support" I gave up and started with chargebacks. It was 7 of them, each month one for $6, until one sunny day I woke up and my own gmail account, my youtube TV and Google Play on 2 Android devices all got blocked with messages popping up I never seen before. Also my friend - who used my wi-fi plenty when we worked on a project together, got his gmail frozen with equally weird messages with no answer how to appeal or follow up. Just middle finger and good bye. edit: and no, they did not stop charging me for youtube TV. I actually had to get a new card number for my credit card for them to stop.
    - ranting-moth 3y ago
      Yes, chargeback is pretty much a nuclear option. But in his case they'd already closed his account and are not responding, so time for the nukes.
      - SOLAR_FIELDS 3y ago
frostyj 3y ago
I just hope you won't be billed twice too
pjd7 3y ago
I guess any "easy" way to solve this right now is to put some authentication on the celery communication bus? I wonder if flower is being used. Once auth is required for the celery processes, the old zombie wont have access. And if this occurs again OP could revoke access etc.
vultour 3y ago
> In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development Free PHP webhosts have been around for 15+ years
- winrid 3y ago
  you could even just throw up the job as an endpoint and have Zapier call it once a day or something! provided you can configure timeouts.
- govolckurself 3y ago
  But nobody was writing trendy Medium posts about PHP.
andrewstuart 3y ago
>> So is it Celery then? This awesome library has its fair share of complaints about duplicated task execution — most are due to misconfiguration. I scanned dozens of Stack Overflow threads and GitHub issues in Celery’s repository. Very few were relevant to my case, and none of the fixes were. I was so annoyed by the pain of configuring Celery I stopped my project and wrote my own job queuing system. Job and message queues are ridiculously over complicated - including the one I wrote. For trivial usage, job queuing should be zero config.
- andybak 3y ago
  Hell yeah. And people it them for low-load situations where they could just use a db table or even just spawining a new process. 30k lines of code and several binary dependencies to run a handful of tasks because "it's the best way to do it".
- fernandotakai 3y ago
  i wanna love celery but it's damn bloated that i gave up on using it. nowadays, if i need to connect to rabbitmq directly, i just use pika/aio-pika, which are lower level, but a LOT more reliable and lightweight.
- danjac 3y ago
  If you just need to send a set of emails once a week, the lowly cron job is fine. You don't need celerybeat. RQ or Huey are otherwise fine for anything more complicated. I've rarely run into a situation where Celery is better, but teams just use it because it's the "default".
- jwmoz 3y ago
  +1 Celery is awful. Gave up on it a few years ago Now just use simple asyncio scripts.
almost 3y ago
It’s such a pity that Heroku has fallen so much. But I definitely wouldn’t run anything I cared about on Heroku these days. I ended up with my apps on NorthFlank which I’ve been very happy with. Railway also looks like a good choice, especially for those looking for a free plan.
amir734jj 3y ago
Heroku really helped me get started. I then stated experimenting with running docker compose locally and finally was able to run my own kubernetes. You don't need Heroku as you progress. It takes time. Be patient.
rcarmo 3y ago
Friendly reminder that if you want Heroku-like deployments for simple projects you can go and roll your own with https://github.com/piku https://github.com/piku :)
- boxed 3y ago
  Or dokku, which is to a very high degree compatible with heroku too.
  - Traubenfuchs 3y ago
    Dokku is amazing if you just want to get your stuff deployed behind a https domain. It allows the average developer (me) to spin up docker containers behind HTTPS subdomains together with postgres and other popular services from nothing within less than half an hour.
    - boxed 3y ago
      It supports kubernetes and stuff nowadays too.
  - hot_gril 3y ago
    Looks nice, I'll try it for my next little project.
- kasia66 3y ago
  or cloud66.com - creates an environment like Heroku but on your servers on any cloud, plus you get 1 free server per application.
dentarg 3y ago
Probably related to https://status.heroku.com/incidents/2531 https://status.heroku.com/incidents/2531 ("On April 6, 2023, between 21:20 UTC and 22:30 UTC, our Common Runtime customers were unable to provision dynos in the US and EU regions. This issue impacted starting/scaling dynos in existing apps, one-off dynos, builds, and release phases.")
rmnclmnt 3y ago
Great read, thanks! I haven't heavily used Celery anymore for the past few years, but I remember trying to setup a HA Celery Beat schedule process with Kubernetes in 2019 and hitting this exact same issue! There was an alternative scheduler[1] storing its state in Redis instead of a local file, after a few fixes it was working pretty well: multiple beat processes running concurrently and not firing twice a scheduled task. Don't know if that still works though. [1] https://github.com/mixkorshun/celery-beatx https://github.com/mixkorshun/celery-beatx
h1fra 3y ago
The article doesn't mention billing, I hope you weren't billed twice? :o
- amerine 3y ago
  Please pay attention to this comment OP. The zombie dynos were likely not “closed”. Look at your bill closely.
zacksiri 3y ago
When heroku first came out they introduced so many wonderful new things that blew the development community away. Many people including myself were inspired by their work. They did serverless before serverless even existed as a marketing concept. But I guess having salesforce buying them out pretty much stalled their progress, and over time leading them to where they are today. Anyway this is why we're building something to continue to innovate in this space, there is a lot to do. https://instellar.app https://instellar.app
eikaramba 3y ago
https://caprover.com https://caprover.com FTW - just host everything yourself. you can host multiple services without any problem on a 5$ VPS nowadays, especially if you using just a simple SSR webapp, some lightweight backend and sqlite db. as soon as your hardware is not keeping up with your demand, you make enough money to scale vertically.
matus_congrady 3y ago
There's been a lot of complaints about Heroku recently, and I think there will be a lot more. I believe that Heroku under the Salesforce's management is a different product that a lot of developers are used to. It's now in a Cash cow stage: https://en.wikipedia.org/wiki/Cash_cow https://en.wikipedia.org/wiki/Cash_cow. Salesforce is trying to extract as much revenue as possible from the company they bought. And guess what - the real margins doesn't come from SMBs and individuals - they come from larger companies with large contracts. Heroku is no longer building a product for ordinary developer's side-project, or for SMBs or early-stage startups. They are building a product for larger accounts paying 25k+/month. Customers that are expecting the old Heroku experience just need to look somewhere else. There are a lot of alternatives (such as render.com and many others). But the truth is, none of them is as good as Heroku. Luckily, these PaaS alternatives are getting better as we speak. There are also different alternatives, such as https://stacktape.com https://stacktape.com (disclaimer: I'm a founder). For example, Stacktape offers PaaS-like experience directly within your own AWS account. It offers developer-friendly experience, and allows developers to deploy their apps entirely on their own, while at the same time offering the full power and flexibility of AWS.
abatilo 3y ago
In 2017, I worked at a company that built a websocket based app and ran it on Heroku. We had this exact same problem and all 5 of the engineers at the time spent all day until we realized that we had socket connections still connected to some "ghost dyno". I can't remember exactly what we did but I think we just kept deploying over and over again until the old one finally went away.
aeyes 3y ago
Even though it won't help you now, instead of celery-once I highly recommend using the celery-redbeat scheduler which uses a distributed lock to ensure that only one scheduler instance is active. It's plug&play and compatible with all the traditional task scheduling methods which beat supports.
- bastawhiz 3y ago
  That doesn't help in this particular case when the zombie node is running old code that can't be modified.
EntrePrescott 3y ago
Kinda refreshing - within the ever growing genre of stories about Heroku's downfall into junk status - to find such an article with a higher level of detail and analysis of the origins of the problem. There's one thing that somewhat surprised me though: this line: > "In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development" Was it? Really? I mean, I'm not saying the author did anything wrong in choosing Heroku at the time (after all, who could have known back then what a dumpster fire Heroku would turn out to be), and the commercial success Heroku achieved before going down the gutter shows that it must have had some good reasons going for it, at least on paper before the users are confronted with problems that make them judge the situation differently… … but I don't really get where that "pretty much the only option" vibe would come from. Maybe It's just me having completely different selection criteria, but from my perspective, there were lots of other options to "get started with web-development" and Heroku was just one among many. I don't see at all by what measure Heroku would have seemed like "pretty much the only option"?
- shibel 3y ago
  Hi. I’m the author. First off, happy to hear you find that refreshing - it’s a really nice compliment to hear! I, like you, hate blanket statements. Perhaps I should I’ve emphasized a newbie like me in that sentence? I really knew next to nothing about how web development, and specifically serving your site from a server, works.
- jazzyjackson 3y ago
  Assuming they wanted a webserver + database and wanted to learn web stuff, not server admin / unix stuff to configure a LAMP stack or whathaveyou + certbot etc. Deploying on your own domain of legitimately daunting without a lot of prerequisite knowledge. Heroku automated a lot of that and some people learned it as the only way to deploy without learning the manual process first, and that's fine IMO.
- danjac 3y ago
  Let's say you have never deployed anything to production before, and you've worked through a tutorial in Rails or Django or whatever and just built your first web app, and it's 2018, what are your options? There's the zoo of AWS. Maybe there's something there you can use - Lightsail came out I think in 2016 - but good luck trying to find it out of all the plethora of weirdly named services. Again, total beginner here, and AWS can be daunting even to experienced developers. You can go with Linode or Digital Ocean. You would need to set up a VM with all the dependencies and make it secure. Oh and you'll need a database - should that sit on another VM? What about backup and restore? What about deployment? Sure, you can use Ansible, but that's yet another learning curve. Render launched in 2019. Fly.io came out 2020. As far as I remember, there wasn't a competitor that offered the sheer convenience for the hobbyist beginner on the level of Heroku.
remram 3y ago
An option might be to fix your message IDs, e.g. something like 20230413-userid@yourapp. Let the recipient's email server de-duplicate if you are not sure you can do better than at-least-once delivery on your side.

297 points by haki 3y ago

This is why I use the git commit short hash as part of the identification of the service (eg: container name). Allows for quick check of which is the running code version and identify problems like these.

zacksiri 3y ago
That’s smart. I like it.
rcarmo 3y ago
Wouldn’t help in this case if you redeployed the same version. I usually log the PID and some form of host ID.
bomewish 3y ago
So you have to constantly rename it with each commit or what?

vinhboy 3y ago

Similar thing happened to me. Took a long time to realize there was a zombie Heroku instance running jobs. That horror when you delete code but it's still running, you're just so confused... I've forgotten how I resolved it, but man it was a pain to diagnose.

akhayam 3y ago

Everyone can relate to the sad demise of Heroku at this time. Sad, because they could've dominated this space which still has a lot of issues. Yet-another-infrastructure-group nailed it in their graphic about "what your infrastructure says about you?"... "Heroku => "You are either negotiating the pricing or deplatforming" https://www.yaig.dev/ https://www.yaig.dev/

zatkin 3y ago

If Heroku is running some zombie instance of your app, what if you were to just kill the instance you control and let Heroku run the zombie instance? When your service goes fully down, you'll know they've killed the zombie instance, so you can then run a clean copy.

badestrand 3y ago
Not being able to update your code plus your app will completely crash randomly at some point between now and eternity - sounds great!
- oneepic 3y ago
  It's salvageable. Maybe spin up some kind of Unix watch that pings the zombie and deploys the new version. (but then, what happens if you get another zombie... crap)

jchw 3y ago

In case it's not clear from the article, Nixpacks build OCI (essentially, Docker) images, so they are perfectly vendor neutral as far as I know. The ending ambiguously implies that Nixpacks may be undesirable vs using Docker, but actually it's just another way to build an image. Fly.io supports Nixpacks out of the box too, I believe, which makes it even easier to use them with Fly (although, you can also just push an OCI image directly to the fly.io registry and deploy with that, which I have been experimenting with in spare time as an option for using Nix itself to build an OCI image for Fly. I'm sure there will be an even nicer way some day, but it works for now...) I'm sure the author knew this, but I wanted to iterate it in case anyone misunderstood.

shibel 3y ago
OP here: I agree my phrasing is kind of problematic. Will rephrase shortly. Thank you for bringing that up.

sufehmi 3y ago

Clearly the author still hasn't learned his lesson: his "no more vendor lock" solution is another vendor(s) - which is always able to vendor-lock him, again. It is truly mind blowing on how many people keep falling for one of the oldest trick in IT : vendor lock-in. === No more vendor lock In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development. The landscape is very different now with providers like Render, Railway, and more

emodendroket 3y ago
So what should he be doing, building his own data center?
- TylerE 3y ago
  Rent a $5/month VPS, install a few packages? Or just run a linux vm locally.
  - emodendroket 3y ago
    So, rely on yet another vendor, or do as I suggested except without any semblance of reliabiliy.
    - TylerE 3y ago
      Some one just starting out is not Google. They do not have Google's problems. They do not need Google's complexity, or a dozen layers of docker/terraform/k8s bullshit.
      - akie 3y ago
        Dockerizing (what the author said he would do) is almost always worth it though.
      - slyall 3y ago
        Agreed, instead of "a dozen layers of docker/terraform/k8s bullshit" they just need they application code and some config that will take them an hour to setup once.

dclowd9901 3y ago

Author and me have so much in common. From journalism to software engineering and our formative career shift years were spent on home made music apps. Now I’m wondering just how many of us are out there.

shibel 3y ago
Author here. Small world! Would love to see your app if you don’t mind sharing. I’m @SHxKM on Twitter if you’d like to connect.
- dclowd9901 3y ago
  Unfortunately it’s abandonware now. It was called Showhopping and it would use the Scrobbler API and a list of your favorite bands and find out if any bands you might like (including the ones you listed) were playing in a given area any time soon. You could give it a location and a time range and it would give a list of shows playing on a map with a percentage back based on the likelihood that you’d enjoy the show. It was very lightweight with no sign up and I had it in mind people could use it like Yelp essentially, just ad-hoc. https://lifehacker.com/showhopping-finds-upcoming-shows-around-you-for-your-fa-5808835 https://lifehacker.com/showhopping-finds-upcoming-shows-arou... Anyway, nice to meet you! I enjoyed your write up.

teaearlgraycold 3y ago

I vote for moving to Render! My god I wish I could short Heroku stock.

8n4vidtmkvmk 3y ago
Looks really expensive. And weird pricing. I click "pricing" and it gives me per user costs. Wat? I thought I was hosting stuff, not managing teams. Click on +Computing costs and it's like $0! $0! $0! Check the fineprint and its like 4GB RAM and 2 CPUs is $85/mo, or you can get the same at DigitalOcean for $24. Then add Postgres for another $95. Or if you're cheap like me, you self-host that in your $24 droplet and pay nothing extra. And then they want you to pay for Crons....instead of just running a cronjob on your dedicated compute? What is this world coming to?
- toyg 3y ago
  Everyone charges for convenience. A lot of people don't want the headache to run and secure their own Postgres, for example. It's a trade-off, same as servicing your car vs just handing it off to a garage mechanic.
  - 8n4vidtmkvmk 3y ago
    Yeah I understand charging for convenience... kinda. Markup is too big IMO. Should be like 10-20% more not 3x. And then they should make up the difference in volume because why would you not choose the more convenient platform if they provide the same great hosting but managed or easier to use.
- jmuguy 3y ago
  Its been a little while since I looked at Render and their pricing definitely used to be a lot simpler. Charging per user is just stupid. And then weird compute costs with plans like "Pro Plus" and "Pro Ultra". "Web Services" and "Private Services". WTF is this shit.
- teaearlgraycold 3y ago
  It's a managed service. You pay a premium for it. Features like their blueprint file make it worth the price IMO - https://render.com/docs/blueprint-spec https://render.com/docs/blueprint-spec If you are working on a team and you want everyone to be able to spin up a copy of the production infrastruction for each pull request then PaaS is the way to go. If you just have some random one-off production environment and you are happy to either manage the state of a HDD or a k8s cluster yourself (which comes down to will more so than skill) then DO is the way to go.

lorenzk 3y ago

Not sure if you have done this already but: If you rotate your database, redis and email provider passwords/tokens, the zombie process might still be running but will not have any effects any more.

domh 3y ago
This is a really smart idea! If you can't kill the process, make it completely ineffective by making it's api calls fail.
shibel 3y ago
OP here: that’s a good idea that did come up to mind (specifically rotating the Redis queue credentials). I was just sure Heroku’s gonna take care of this shit by now. BTW, I’m not 100% sure it’ll work: if Heroku updates that zombie dyno’s environment variables, it may still have access to the new credentials :/ (My Redis instance is also a Heroku add-on)
- gsa 3y ago
  Rotate a token, but also mangle it and change to the correct one in code. eg: change database url from `postgres://` to `invalid://` and replace it in code.
- neurostimulant 3y ago
  > BTW, I’m not 100% sure it’ll work: if Heroku updates that zombie dyno’s environment variables, it may still have access to the new credentials :/ (My Redis instance is also a Heroku add-on) If the configuration only read once during startup, the celery process won't use the new configuration unless it's restarted, right?
  - DangitBobby 3y ago
    Heroku will automatically restart dynos if you change environment variables.

j45 3y ago

Maybe Dokku us an option to self host and maintain Heroku code

krokoo 3y ago

[dead]

marvinkennis 3y ago

Heroku has been an absolute dumpster fire. Every time their name comes up I can't help but mention that they deleted my account for inactivity, but have still been charging my credit card. No easy way to cancel because... they deleted my account.

djbusby 3y ago
Gotta chargeback and tell the CC company to block. Then file complaint with their states AG.
ranting-moth 3y ago
File a chargeback each time they charge your card. The merchant has to pay the transaction fees both ways and a fee for the chargeback.
- joering2 3y ago
  But don't do it if its Google. I had been charged for g-suite for domain that expired and someone else picked it up (and parked), long story short after 3 months of back-and-forth with "support" I gave up and started with chargebacks. It was 7 of them, each month one for $6, until one sunny day I woke up and my own gmail account, my youtube TV and Google Play on 2 Android devices all got blocked with messages popping up I never seen before. Also my friend - who used my wi-fi plenty when we worked on a project together, got his gmail frozen with equally weird messages with no answer how to appeal or follow up. Just middle finger and good bye. edit: and no, they did not stop charging me for youtube TV. I actually had to get a new card number for my credit card for them to stop.
  - ranting-moth 3y ago
    Yes, chargeback is pretty much a nuclear option. But in his case they'd already closed his account and are not responding, so time for the nukes.
    - SOLAR_FIELDS 3y ago

frostyj 3y ago

I just hope you won't be billed twice too

pjd7 3y ago

I guess any "easy" way to solve this right now is to put some authentication on the celery communication bus? I wonder if flower is being used. Once auth is required for the celery processes, the old zombie wont have access. And if this occurs again OP could revoke access etc.

vultour 3y ago

> In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development Free PHP webhosts have been around for 15+ years

winrid 3y ago
you could even just throw up the job as an endpoint and have Zapier call it once a day or something! provided you can configure timeouts.
govolckurself 3y ago
But nobody was writing trendy Medium posts about PHP.

andrewstuart 3y ago

>> So is it Celery then? This awesome library has its fair share of complaints about duplicated task execution — most are due to misconfiguration. I scanned dozens of Stack Overflow threads and GitHub issues in Celery’s repository. Very few were relevant to my case, and none of the fixes were. I was so annoyed by the pain of configuring Celery I stopped my project and wrote my own job queuing system. Job and message queues are ridiculously over complicated - including the one I wrote. For trivial usage, job queuing should be zero config.

andybak 3y ago
Hell yeah. And people it them for low-load situations where they could just use a db table or even just spawining a new process. 30k lines of code and several binary dependencies to run a handful of tasks because "it's the best way to do it".
fernandotakai 3y ago
i wanna love celery but it's damn bloated that i gave up on using it. nowadays, if i need to connect to rabbitmq directly, i just use pika/aio-pika, which are lower level, but a LOT more reliable and lightweight.
danjac 3y ago
If you just need to send a set of emails once a week, the lowly cron job is fine. You don't need celerybeat. RQ or Huey are otherwise fine for anything more complicated. I've rarely run into a situation where Celery is better, but teams just use it because it's the "default".
jwmoz 3y ago
+1 Celery is awful. Gave up on it a few years ago Now just use simple asyncio scripts.

almost 3y ago

It’s such a pity that Heroku has fallen so much. But I definitely wouldn’t run anything I cared about on Heroku these days. I ended up with my apps on NorthFlank which I’ve been very happy with. Railway also looks like a good choice, especially for those looking for a free plan.

amir734jj 3y ago

Heroku really helped me get started. I then stated experimenting with running docker compose locally and finally was able to run my own kubernetes. You don't need Heroku as you progress. It takes time. Be patient.

rcarmo 3y ago

Friendly reminder that if you want Heroku-like deployments for simple projects you can go and roll your own with https://github.com/piku https://github.com/piku :)

boxed 3y ago
Or dokku, which is to a very high degree compatible with heroku too.
- Traubenfuchs 3y ago
  Dokku is amazing if you just want to get your stuff deployed behind a https domain. It allows the average developer (me) to spin up docker containers behind HTTPS subdomains together with postgres and other popular services from nothing within less than half an hour.
  - boxed 3y ago
    It supports kubernetes and stuff nowadays too.
- hot_gril 3y ago
  Looks nice, I'll try it for my next little project.
kasia66 3y ago
or cloud66.com - creates an environment like Heroku but on your servers on any cloud, plus you get 1 free server per application.

dentarg 3y ago

Probably related to https://status.heroku.com/incidents/2531 https://status.heroku.com/incidents/2531 ("On April 6, 2023, between 21:20 UTC and 22:30 UTC, our Common Runtime customers were unable to provision dynos in the US and EU regions. This issue impacted starting/scaling dynos in existing apps, one-off dynos, builds, and release phases.")

rmnclmnt 3y ago

Great read, thanks! I haven't heavily used Celery anymore for the past few years, but I remember trying to setup a HA Celery Beat schedule process with Kubernetes in 2019 and hitting this exact same issue! There was an alternative scheduler[1] storing its state in Redis instead of a local file, after a few fixes it was working pretty well: multiple beat processes running concurrently and not firing twice a scheduled task. Don't know if that still works though. [1] https://github.com/mixkorshun/celery-beatx https://github.com/mixkorshun/celery-beatx

h1fra 3y ago

The article doesn't mention billing, I hope you weren't billed twice? :o

amerine 3y ago
Please pay attention to this comment OP. The zombie dynos were likely not “closed”. Look at your bill closely.

zacksiri 3y ago

When heroku first came out they introduced so many wonderful new things that blew the development community away. Many people including myself were inspired by their work. They did serverless before serverless even existed as a marketing concept. But I guess having salesforce buying them out pretty much stalled their progress, and over time leading them to where they are today. Anyway this is why we're building something to continue to innovate in this space, there is a lot to do. https://instellar.app https://instellar.app

eikaramba 3y ago

https://caprover.com https://caprover.com FTW - just host everything yourself. you can host multiple services without any problem on a 5$ VPS nowadays, especially if you using just a simple SSR webapp, some lightweight backend and sqlite db. as soon as your hardware is not keeping up with your demand, you make enough money to scale vertically.

matus_congrady 3y ago

There's been a lot of complaints about Heroku recently, and I think there will be a lot more. I believe that Heroku under the Salesforce's management is a different product that a lot of developers are used to. It's now in a Cash cow stage: https://en.wikipedia.org/wiki/Cash_cow https://en.wikipedia.org/wiki/Cash_cow. Salesforce is trying to extract as much revenue as possible from the company they bought. And guess what - the real margins doesn't come from SMBs and individuals - they come from larger companies with large contracts. Heroku is no longer building a product for ordinary developer's side-project, or for SMBs or early-stage startups. They are building a product for larger accounts paying 25k+/month. Customers that are expecting the old Heroku experience just need to look somewhere else. There are a lot of alternatives (such as render.com and many others). But the truth is, none of them is as good as Heroku. Luckily, these PaaS alternatives are getting better as we speak. There are also different alternatives, such as https://stacktape.com https://stacktape.com (disclaimer: I'm a founder). For example, Stacktape offers PaaS-like experience directly within your own AWS account. It offers developer-friendly experience, and allows developers to deploy their apps entirely on their own, while at the same time offering the full power and flexibility of AWS.

abatilo 3y ago

In 2017, I worked at a company that built a websocket based app and ran it on Heroku. We had this exact same problem and all 5 of the engineers at the time spent all day until we realized that we had socket connections still connected to some "ghost dyno". I can't remember exactly what we did but I think we just kept deploying over and over again until the old one finally went away.

aeyes 3y ago

Even though it won't help you now, instead of celery-once I highly recommend using the celery-redbeat scheduler which uses a distributed lock to ensure that only one scheduler instance is active. It's plug&play and compatible with all the traditional task scheduling methods which beat supports.

bastawhiz 3y ago
That doesn't help in this particular case when the zombie node is running old code that can't be modified.

EntrePrescott 3y ago

Kinda refreshing - within the ever growing genre of stories about Heroku's downfall into junk status - to find such an article with a higher level of detail and analysis of the origins of the problem. There's one thing that somewhat surprised me though: this line: > "In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development" Was it? Really? I mean, I'm not saying the author did anything wrong in choosing Heroku at the time (after all, who could have known back then what a dumpster fire Heroku would turn out to be), and the commercial success Heroku achieved before going down the gutter shows that it must have had some good reasons going for it, at least on paper before the users are confronted with problems that make them judge the situation differently… … but I don't really get where that "pretty much the only option" vibe would come from. Maybe It's just me having completely different selection criteria, but from my perspective, there were lots of other options to "get started with web-development" and Heroku was just one among many. I don't see at all by what measure Heroku would have seemed like "pretty much the only option"?

shibel 3y ago
Hi. I’m the author. First off, happy to hear you find that refreshing - it’s a really nice compliment to hear! I, like you, hate blanket statements. Perhaps I should I’ve emphasized a newbie like me in that sentence? I really knew next to nothing about how web development, and specifically serving your site from a server, works.
jazzyjackson 3y ago
Assuming they wanted a webserver + database and wanted to learn web stuff, not server admin / unix stuff to configure a LAMP stack or whathaveyou + certbot etc. Deploying on your own domain of legitimately daunting without a lot of prerequisite knowledge. Heroku automated a lot of that and some people learned it as the only way to deploy without learning the manual process first, and that's fine IMO.
danjac 3y ago
Let's say you have never deployed anything to production before, and you've worked through a tutorial in Rails or Django or whatever and just built your first web app, and it's 2018, what are your options? There's the zoo of AWS. Maybe there's something there you can use - Lightsail came out I think in 2016 - but good luck trying to find it out of all the plethora of weirdly named services. Again, total beginner here, and AWS can be daunting even to experienced developers. You can go with Linode or Digital Ocean. You would need to set up a VM with all the dependencies and make it secure. Oh and you'll need a database - should that sit on another VM? What about backup and restore? What about deployment? Sure, you can use Ansible, but that's yet another learning curve. Render launched in 2019. Fly.io came out 2020. As far as I remember, there wasn't a competitor that offered the sheer convenience for the hobbyist beginner on the level of Heroku.

remram 3y ago

An option might be to fix your message IDs, e.g. something like 20230413-userid@yourapp. Let the recipient's email server de-duplicate if you are not sure you can do better than at-least-once delivery on your side.