3 ms·

I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into var

I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)

roflyear 4y ago
Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting). I'm sure if I really tried, I can get celery to be very reliable... but I never really got there. Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/
- nickjj 4y ago
  What type of situations did you run into while using it? I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable. One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.
  - roflyear 4y ago
    Celery has a lot of gotchas, but primarily the issue I have had is with resource consumption. Maybe it is a problem with my config, but I haven't had this issue with other solutions.

roflyear 4y ago

Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting). I'm sure if I really tried, I can get celery to be very reliable... but I never really got there. Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/

nickjj 4y ago
What type of situations did you run into while using it? I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable. One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.
- roflyear 4y ago
  Celery has a lot of gotchas, but primarily the issue I have had is with resource consumption. Maybe it is a problem with my config, but I haven't had this issue with other solutions.