4 ms·

A long time ago I wrote a blog post about Celery use cases at https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-celery-in-a-flask-application https://n

by nickjj 4y ago

A long time ago I wrote a blog post about Celery use cases at https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-celery-in-a-flask-application https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-c.... It applies to Django, Flask or any Python system using it. All of it still applies today. It covers a few use cases on the before vs after of using Celery and touches base on why I'd consider using Celery over other solutions such as async / await. The TL;DR is Celery brings a lot to the table around tracking and retrying jobs. It's also nice to separate your web and worker workloads into different processes since they have much different requirements usually.

emptysea 4y ago
I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)
- roflyear 4y ago
  Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting). I'm sure if I really tried, I can get celery to be very reliable... but I never really got there. Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/
  - nickjj 4y ago
    What type of situations did you run into while using it? I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable. One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.
    - roflyear 4y ago
      Celery has a lot of gotchas, but primarily the issue I have had is with resource consumption. Maybe it is a problem with my config, but I haven't had this issue with other solutions.

by nickjj 4y ago

emptysea 4y ago

I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)

roflyear 4y ago
Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting). I'm sure if I really tried, I can get celery to be very reliable... but I never really got there. Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/ https://schedule.readthedocs.io/en/stable/
- nickjj 4y ago
  What type of situations did you run into while using it? I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable. One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.
  - roflyear 4y ago
    Celery has a lot of gotchas, but primarily the issue I have had is with resource consumption. Maybe it is a problem with my config, but I haven't had this issue with other solutions.