5 ms·

The Many Problems with Celery

31 points by sbdchd 3y ago

fikka0 3y ago
Steve, this is such a great piece of knowledge! I see you mention lack of monitoring solutions, I'm creating a Celery monitoring tool at the moment, it's already a working/useful MVP, I'd love to get your opinion on it, is there any way to connect?
Something1234 3y ago
Celery has so many issues and this outlines just a few of them. I was working on celery for a while on one of my personal projects and it's just absolutely insane what you have to do to get it working with django. It's so incredibly over engineered nad powered by magic.
dikei 3y ago
I think Celery is doing itself a disservice by supporting so many brokers and result stores with different behaviors about delivery guarantee, leading to leaky abstraction everywhere.
iancmceachern 3y ago
I came here to air my grievances with the vegetable. Alas, not tonight.
- 1letterunixname 3y ago
  Have you tried it with peanut butter? It causes all jobs to work swimmingly.
  - shaftway 3y ago
    The problem is that it's difficult to add raisins without them being consumed sequentially.
    - iancmceachern 3y ago
      The problem in my view is those tooth floss like strings. They're worse with peanut butter. If one could figure out celery without those terrible strings I'd happily reconsider my opinion on celery.
      - 1letterunixname 3y ago
        Very thin slices.
z3ugma 3y ago
For folks who have tried celery, and were dissatisfied with that, what did you switch to? What else is there for python task queues?
1letterunixname 3y ago
There should be a distinction between a work queue and a job scheduler. The work queue dispatches jobs but doesn't know enough to retry them while the scheduler decides what to retry. In Rails land, I would opt for Backburner over the commercial choices because it uses beanstalkd. For anything requiring HA, I would look at some sort of queue using an ESB as a transport. Short of HA, shard for LB.
Beestie 3y ago
Celery? Not knowing this was a post about software, I figured it was about the vegatable. Can't wait until software naming reaches the same absurdity level as racehorse names. /rant
- Clamchop 3y ago
  I'm personally a fan of Pickle and Cucumber, completely unrelated naturally.

31 points by sbdchd 3y ago

Steve, this is such a great piece of knowledge! I see you mention lack of monitoring solutions, I'm creating a Celery monitoring tool at the moment, it's already a working/useful MVP, I'd love to get your opinion on it, is there any way to connect?

Something1234 3y ago

Celery has so many issues and this outlines just a few of them. I was working on celery for a while on one of my personal projects and it's just absolutely insane what you have to do to get it working with django. It's so incredibly over engineered nad powered by magic.

dikei 3y ago

I think Celery is doing itself a disservice by supporting so many brokers and result stores with different behaviors about delivery guarantee, leading to leaky abstraction everywhere.

iancmceachern 3y ago

I came here to air my grievances with the vegetable. Alas, not tonight.

1letterunixname 3y ago
Have you tried it with peanut butter? It causes all jobs to work swimmingly.
- shaftway 3y ago
  The problem is that it's difficult to add raisins without them being consumed sequentially.
  - iancmceachern 3y ago
    The problem in my view is those tooth floss like strings. They're worse with peanut butter. If one could figure out celery without those terrible strings I'd happily reconsider my opinion on celery.
    - 1letterunixname 3y ago
      Very thin slices.

z3ugma 3y ago

For folks who have tried celery, and were dissatisfied with that, what did you switch to? What else is there for python task queues?

1letterunixname 3y ago

There should be a distinction between a work queue and a job scheduler. The work queue dispatches jobs but doesn't know enough to retry them while the scheduler decides what to retry. In Rails land, I would opt for Backburner over the commercial choices because it uses beanstalkd. For anything requiring HA, I would look at some sort of queue using an ESB as a transport. Short of HA, shard for LB.

Beestie 3y ago

Celery? Not knowing this was a post about software, I figured it was about the vegatable. Can't wait until software naming reaches the same absurdity level as racehorse names. /rant

Clamchop 3y ago
I'm personally a fan of Pickle and Cucumber, completely unrelated naturally.