4 ms·
Celery was never reliable for me. I used Celery as a part of Django. It kept running into weird issues. Celery-Flower used to monitor the Celery threads is not
by sharmi 6y ago
Celery was never reliable for me. I used Celery as a part of Django. It kept running into weird issues. Celery-Flower used to monitor the Celery threads is not maintained and that became a huge issue for me as I had no way of tracking any issue.
I have also used Celery parallelize crawling of multiple websites in parallel. Beyond a small number of workers, Celery would go into a frozen state (My data flow was one way. No chance that I know of deadlocking.) The code felt too convoluted to dive in and understand.
I have since shifted to Dramatiq ( https://dramatiq.io/ https://dramatiq.io/ ). It has all the features I like of Celery (Chaining of async methods with error handling when any one link on the chain fails), with super reliability. It also has Django support.
Having said that, it must be recognized that Dramatiq was built from the success and learnings gained from Celery. Celery is a wonderful piece of work still in active use in almost all Python shops. I am just thankful that we now have so many alternatives.
PS: The Django-Q mentioned in another comment looks very well structured, comprehensive and actively developed. I shall give it a try. For now, so happy with Dramatiq.
- whalesalad 6y agoI love everything about Dramatiq (and am a contributor to the project, and use it in a few prod apps) except for the decision to use actor nomenclature. It makes sense at first glance, but in reality dramatiq is not an actor system. Actors do not live and die and get managed by supervision as they would in an environment like Erlang. Instead, messages just go onto a queue and get popped off by a worker. I guess as long as you go into it eyes wide open - dramatiq is not an actor framework - you will be a-ok.
- actionscripted 6y agoCelery Flower is absolutely maintained! Last commit and last release were a few months ago: * https://github.com/mher/flower https://github.com/mher/flower * https://pypi.org/project/flower/#history https://pypi.org/project/flower/#history
- sharmi 6y agoI faced some issues a year or two ago. It wasn't then. It was one of the pain points many users faced. One of the issues I faced was scheduled tasks wouldn't show up in Flower. There was no way for me to know if a task was run without monitoring the database. I didn't follow after that as I moved to Dramatiq. I am so glad to hear that the community has listened and picked it up again. Thank you for pointing it out.
- paulie_a 6y agoI am going to take a look at Django-q I've used celery very successfully in the past once setup. About ten years ago I had problems connecting to rabbitmq. I finally found a solution and posted it on the GitHub project page. The funny thing is two years later I ran into the same issue. I forgot about that post and ended up stumbling on my post, and at first didn't realize it was mine. I still get an occasional email about that solution.