4 ms·

Why is celery awful?

tclancy 10mo ago
Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times. And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues. https://flower.readthedocs.io/en/latest/ https://flower.readthedocs.io/en/latest/
- sgt 10mo ago
  Although we could say the same thing about Kafka, couldn't we? It's made for much higher throughput and has usually other use cases, but it's also great until it's not great.
  - freedomben 10mo ago
    At least the last time I used Kafka (which was several years ago so things might have changed) it wasn't at all easy to get started. It was a downright asshole in fact. If you pursue a relationship with an asshole, you shouldn't be surprised when they become cold to you
    - sgt 10mo ago
      Yes, absolutely. It's still pretty much that way. Especially if you want to make changes to a running installation, add nodes etc.
akoumjian 10mo ago
Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples: - your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff. Honestly, it's a great education.
- saaspirant 10mo ago
  From your experience, what is a better alternative guys?
  - leobuskin 10mo ago
    There’s no alternative (while prototyping), and anything else is better (when you properly defined your case).
    - walthamstow 10mo ago
      DjangoQ2 is a fine alternative during early development
  - boxed 10mo ago
    Not the comment that you replied to but I use my own Urd. It's a fancier Cron that you can stop fast. Which is imo what you normally want. Task queues are like email. It's what everyone is used to so people ask for more of it, but it's not actually good/the right tool.
- ffsm8 10mo ago
  > your side effects (e.g. database writes) aren't idempotent What does idempotent mean in this context, or did you mean atomic/rollback on error? I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
  - teaearlgraycold 10mo ago
    In my experience async job idempotency is implemented as upserts. Insert all job outputs on the first run. Do (mostly) nothing on subsequent runs. Maybe increment a counter or timestamp.
  - jon-wood 10mo ago
leobuskin 10mo ago
It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).
JimDabell 10mo ago
> The Many Problems with Celery: — https://steve.dignam.xyz/2023/05/20/many-problems-with-celery/ https://steve.dignam.xyz/2023/05/20/many-problems-with-celer... > The problems with (Python’s) Celery: — https://docs.hatchet.run/blog/problems-with-celery https://docs.hatchet.run/blog/problems-with-celery > Dramatiq motivation: — https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html Here are some alternatives: Dramatiq: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq RQ: https://github.com/rq/rq https://github.com/rq/rq Huey: https://github.com/coleifer/huey https://github.com/coleifer/huey Hatchet: https://github.com/hatchet-dev/hatchet https://github.com/hatchet-dev/hatchet
- hda111 10mo ago
  django-q2: https://github.com/django-q2/django-q2 https://github.com/django-q2/django-q2
- hintoftime 10mo ago
  Would you consider tools like Temporal, DBOS, Absurd Workflows, PGQueuer as alternatives? https://temporal.io/ https://temporal.io/ https://docs.dbos.dev/ https://docs.dbos.dev/ https://news.ycombinator.com/item?id=45797228 https://news.ycombinator.com/item?id=45797228 https://python-absurd-client.readthedocs.io/en/latest/quickstart.html https://python-absurd-client.readthedocs.io/en/latest/quicks... https://pgqueuer.readthedocs.io/en/latest/ https://pgqueuer.readthedocs.io/en/latest/
  - BowBun 10mo ago
    Temporal is an AMAZING piece of software, however I don't believe it's a replacement for something more simple like Celery. Even if you write helpers, the overhead to setting up workflows, invoking them, etc. is just too much for simple jobs like sending an email (imo). I would love to work in a codebase that had access to both, depending on the complexity of what you're trying to background.

Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times. And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues. https://flower.readthedocs.io/en/latest/ https://flower.readthedocs.io/en/latest/

sgt 10mo ago
Although we could say the same thing about Kafka, couldn't we? It's made for much higher throughput and has usually other use cases, but it's also great until it's not great.
- freedomben 10mo ago
  At least the last time I used Kafka (which was several years ago so things might have changed) it wasn't at all easy to get started. It was a downright asshole in fact. If you pursue a relationship with an asshole, you shouldn't be surprised when they become cold to you
  - sgt 10mo ago
    Yes, absolutely. It's still pretty much that way. Especially if you want to make changes to a running installation, add nodes etc.

akoumjian 10mo ago

Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples: - your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff. Honestly, it's a great education.

saaspirant 10mo ago
From your experience, what is a better alternative guys?
- leobuskin 10mo ago
  There’s no alternative (while prototyping), and anything else is better (when you properly defined your case).
  - walthamstow 10mo ago
    DjangoQ2 is a fine alternative during early development
- boxed 10mo ago
  Not the comment that you replied to but I use my own Urd. It's a fancier Cron that you can stop fast. Which is imo what you normally want. Task queues are like email. It's what everyone is used to so people ask for more of it, but it's not actually good/the right tool.
ffsm8 10mo ago
> your side effects (e.g. database writes) aren't idempotent What does idempotent mean in this context, or did you mean atomic/rollback on error? I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
- teaearlgraycold 10mo ago
  In my experience async job idempotency is implemented as upserts. Insert all job outputs on the first run. Do (mostly) nothing on subsequent runs. Maybe increment a counter or timestamp.
- jon-wood 10mo ago

leobuskin 10mo ago

It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).

JimDabell 10mo ago

> The Many Problems with Celery: — https://steve.dignam.xyz/2023/05/20/many-problems-with-celery/ https://steve.dignam.xyz/2023/05/20/many-problems-with-celer... > The problems with (Python’s) Celery: — https://docs.hatchet.run/blog/problems-with-celery https://docs.hatchet.run/blog/problems-with-celery > Dramatiq motivation: — https://dramatiq.io/motivation.html https://dramatiq.io/motivation.html Here are some alternatives: Dramatiq: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq RQ: https://github.com/rq/rq https://github.com/rq/rq Huey: https://github.com/coleifer/huey https://github.com/coleifer/huey Hatchet: https://github.com/hatchet-dev/hatchet https://github.com/hatchet-dev/hatchet

hda111 10mo ago
django-q2: https://github.com/django-q2/django-q2 https://github.com/django-q2/django-q2
hintoftime 10mo ago
Would you consider tools like Temporal, DBOS, Absurd Workflows, PGQueuer as alternatives? https://temporal.io/ https://temporal.io/ https://docs.dbos.dev/ https://docs.dbos.dev/ https://news.ycombinator.com/item?id=45797228 https://news.ycombinator.com/item?id=45797228 https://python-absurd-client.readthedocs.io/en/latest/quickstart.html https://python-absurd-client.readthedocs.io/en/latest/quicks... https://pgqueuer.readthedocs.io/en/latest/ https://pgqueuer.readthedocs.io/en/latest/
- BowBun 10mo ago
  Temporal is an AMAZING piece of software, however I don't believe it's a replacement for something more simple like Celery. Even if you write helpers, the overhead to setting up workflows, invoking them, etc. is just too much for simple jobs like sending an email (imo). I would love to work in a codebase that had access to both, depending on the complexity of what you're trying to background.