3 ms·

What do you recommend for the actual job execution for production environment? Is Celery sturdy enough?

bdcravens 8y ago
I can't speak to Celery; I've used Sidekiq, which is the Ruby equivalent, and I know they are similar. We run a tremendous workload on Sidekiq (millions of events a week) in production with no issues.
cimmanom 8y ago
Yes, celery is very commonly used in production for python applications.
shoo 8y ago
> Is Celery sturdy enough? It probably depends on what your exact requirements are, but celery is likely fine. My last project had celery doing batch job processing for a line of business enterprise web app. It was fine and flexible enough to do what we needed (thousands of jobs a week, job scheduling, rabbitmq broker & postgres result store, in use for years). One thing to be aware of: if you're not running on windows, celery worker processes are forked, and by default python's process abstractions (subprocess etc) will fight against you to prevent you launching processes from a celery worker. This can be worked around but is a bit irritating. This is probably obvious but you want to ensure your celery worker processes are properly run as e.g. services with good monitoring, and configured to automatically restart if they crash (due to defects in application specific task logic, say). Some tips / past discussions: https://khashtamov.com/en/celery-best-practices-practical-approach/ https://khashtamov.com/en/celery-best-practices-practical-ap... https://denibertovic.com/posts/celery-best-practices/ https://denibertovic.com/posts/celery-best-practices/ https://news.ycombinator.com/item?id=7909201 https://news.ycombinator.com/item?id=7909201 Sadly, the main issue I see with celery in the past is that it is a popular open source project with no income stream to fund development, so at times swathes of reported bugs have been closed with "won't fix ; we don't have the resources".
toomuchtodo 8y ago
Celery is sturdy enough for production. Suggest RabbitMQ as the queue, equally sturdy. https://zapier.com/engineering/automating-billions-of-tasks/ https://zapier.com/engineering/automating-billions-of-tasks/

I can't speak to Celery; I've used Sidekiq, which is the Ruby equivalent, and I know they are similar. We run a tremendous workload on Sidekiq (millions of events a week) in production with no issues.

cimmanom 8y ago

Yes, celery is very commonly used in production for python applications.

shoo 8y ago

> Is Celery sturdy enough? It probably depends on what your exact requirements are, but celery is likely fine. My last project had celery doing batch job processing for a line of business enterprise web app. It was fine and flexible enough to do what we needed (thousands of jobs a week, job scheduling, rabbitmq broker & postgres result store, in use for years). One thing to be aware of: if you're not running on windows, celery worker processes are forked, and by default python's process abstractions (subprocess etc) will fight against you to prevent you launching processes from a celery worker. This can be worked around but is a bit irritating. This is probably obvious but you want to ensure your celery worker processes are properly run as e.g. services with good monitoring, and configured to automatically restart if they crash (due to defects in application specific task logic, say). Some tips / past discussions: https://khashtamov.com/en/celery-best-practices-practical-approach/ https://khashtamov.com/en/celery-best-practices-practical-ap... https://denibertovic.com/posts/celery-best-practices/ https://denibertovic.com/posts/celery-best-practices/ https://news.ycombinator.com/item?id=7909201 https://news.ycombinator.com/item?id=7909201 Sadly, the main issue I see with celery in the past is that it is a popular open source project with no income stream to fund development, so at times swathes of reported bugs have been closed with "won't fix ; we don't have the resources".

toomuchtodo 8y ago

Celery is sturdy enough for production. Suggest RabbitMQ as the queue, equally sturdy. https://zapier.com/engineering/automating-billions-of-tasks/ https://zapier.com/engineering/automating-billions-of-tasks/