4 ms·
In my experience Celery brings more cost in complexity than value. It's got several abstraction layers to make a variety of backends (Redis, Mongo, relational
by bjt 11y ago
In my experience Celery brings more cost in complexity than value. It's got several abstraction layers to make a variety of backends (Redis, Mongo, relational DBs) all look more or less like AMQP.
My life got better when I stopped using Celery and instead started using Redis or RabbitMQ directly. It's not that hard.
- 1_player 11y agoI have been working on a Django application with two periodic background tasks to synchronize some data with some hardware, I was using Celery+Redis to handle the schedulation. The solutions for running cron-like tasks on Python are not very mature, everybody seems to be using Celery. Once deployed in production I spent countless evenings debugging why the tasks weren't running, why upon timeouts tasks weren't terminated, and why the task queue was getting bigger and bigger. 50% were configuration problems, due to the _bad_ and confusing documentation, 50% were due to introducing a sheer amount of complexity with Celery that I often spent my time reading through the source code to understand what was going on. And Python, with all its magic methods and abstractions, made that very hard. One day I just rewrote everything with three Django management commands: two for the background tasks, one for the schedulation (with multiprocessing). I haven't heard from the client since.