4 ms·

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've neve

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is. Does anyone have experience with both and is able to give a quick comparison? Thanks!

BowBun 9mo ago
Very, very different tools, though they cover similar areas. Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice. Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq) Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.
- kfajdsl 9mo ago
  I’ll say that, I think this varies by language/SDK - at least with the Temporal TypeScript SDK, a simple single idempotent step background task is however many lines of code to do the actual work in an activity, and then the only boilerplate is like 3 lines of code for a simple workflow function to call the activity.
owaislone 9mo ago
I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?
- TkTech 9mo ago
  Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.
  - hangonhn 9mo ago
    Yeah that list right there. That's exactly it. We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in. Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.
    - brianwawok 9mo ago
      But if you are too small for celery, it seems a hard sell to buy a premium message queue? My top problem with my celery setup has always been visibility. AI and I spent and afternoon setting up a Prometheus / grafana server, and wiring celery into it. Has been a game changer. When things go crazy in prod, I can usually single it down to a specific task for a specific user. Has made my troubleshooting go from days to minutes. The actual queue and execute part has always been easy / worked well.
  - xmcqdpt2 9mo ago
    For an even lighter system than Celery, I'm a big fan of https://python-rq.org/ https://python-rq.org/ It's super low on the dependencies and integrates nicely as a library into python applications. It's very bare bones.
    - TkTech 9mo ago
      I kinda have my own https://github.com/tktech/chancy https://github.com/tktech/chancy :P
tecoholic 9mo ago
We migrated from Celery to Prefect a couple of years back and have been very happy. But ours is a small op which handles tasks in 1000s and not millions. It’s been night and day in terms of visibility and tracking. I would definitely recommend it. It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.

BowBun 9mo ago

Very, very different tools, though they cover similar areas. Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice. Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq) Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.

kfajdsl 9mo ago
I’ll say that, I think this varies by language/SDK - at least with the Temporal TypeScript SDK, a simple single idempotent step background task is however many lines of code to do the actual work in an activity, and then the only boilerplate is like 3 lines of code for a simple workflow function to call the activity.

owaislone 9mo ago

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

TkTech 9mo ago
Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.
- hangonhn 9mo ago
  Yeah that list right there. That's exactly it. We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in. Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.
  - brianwawok 9mo ago
    But if you are too small for celery, it seems a hard sell to buy a premium message queue? My top problem with my celery setup has always been visibility. AI and I spent and afternoon setting up a Prometheus / grafana server, and wiring celery into it. Has been a game changer. When things go crazy in prod, I can usually single it down to a specific task for a specific user. Has made my troubleshooting go from days to minutes. The actual queue and execute part has always been easy / worked well.
- xmcqdpt2 9mo ago
  For an even lighter system than Celery, I'm a big fan of https://python-rq.org/ https://python-rq.org/ It's super low on the dependencies and integrates nicely as a library into python applications. It's very bare bones.
  - TkTech 9mo ago
    I kinda have my own https://github.com/tktech/chancy https://github.com/tktech/chancy :P

tecoholic 9mo ago

We migrated from Celery to Prefect a couple of years back and have been very happy. But ours is a small op which handles tasks in 1000s and not millions. It’s been night and day in terms of visibility and tracking. I would definitely recommend it. It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.