4 ms·

Three quick tips from two years with Celery

47 points by taylorhughes 11y ago

rizwan 11y ago
Forgive me for asking, but how is "no default timeout" a sensible default for any technology?
OrangeTux 11y ago
And do not use Redis as a broker under a high load. It will random crash or refuse to set workers to work, while queue is filling up with tasks.
- mooted1 11y ago
  At Uber we use redis as a broker for 10k+ requests per second (pushing 50k one some days) with up to several million items backed up. This is a single redis instance with replication turned on and RDB snapshots for durability. Curious how our use patterns are different that causes what you're seeing.
  - bliti 11y ago
    Tell me about the hardware specs where your Redis runs.
  - andy_ppp 11y ago
    Is that with Celery or are you using your own queue management?
- taylorhughes 11y ago
  Good point. We've run into some issues with dropping tasks when the redis host is underwater; can imagine that becoming problematic.
  - sandGorgon 11y ago
    We recently built out a small background task runner - we were told to use RQ+Redis rather than Celery because (and I'm repeating hearsay here) - 1. Celery's code is very messy and 2. because of the way it keeps jobs in memory, it is very prone to memory leaks. Is there any truth to this ?
    - caio1982 11y ago
      I don't know how much of it is truth but I've definitely had problems with Celery and memory leaks in the past myself. We can add this up to the hearsay pile though.
      - gsharma 11y ago
        What do you use as a broker? We have seen these issues too and use Redis as the broker.
    - phunge 11y ago
      Celery works, that's been my experience. It's buggy, complicated, and we've had unresolved issues, but it more or less does the job. The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
      - sandGorgon 11y ago
        have you looked at RQ - would love to get a comparitive opinion.
    - mpdehaan2 11y ago
      My experience is that Redis slows down considerably with a large amount of data in the queue, and also consumes large amounts of RAM. I'm not sure I'd look to something that explicitly uses Redis as an alternative unless I could be sure the queue would be reasonably small. Which, if the jobs are not going to be overwhelmed with backed up jobs, maybe that's fine. Also, maybe we were doing it wrong.
humbertomn 11y ago
For the retry part, I prefer to use a database table + cron task to do it... storing failed attempts and making x new attempts in predefined date and times, not having it permanently on a celery queue.
simonpantzare 11y ago
-Ofair disables task prefetching which will affect throughput negatively if most of your tasks run for a short time (http://celery.readthedocs.org/en/latest/userguide/optimizing.html#prefetch-limits http://celery.readthedocs.org/en/latest/userguide/optimizing...).
- mlissner 11y ago
  yeah, I don't get this suggestion at all. They say that: > This option comes with a coordination penalty, but results in a much more predictable behavior In general, I'd much rather the greater throughput over predictable behavior. In general, I'm not watching my queue and having it be predictable isn't a very high priority.
  - taylorhughes 11y ago
    Ofair will increase throughput in a queue where you have mixed-length tasks, because in a default configuration long tasks will block the queue from starting new tasks. I believe this is because the main process distributes some number of tasks, waits for them to finish, then distributes another set of tasks, and so on. (Would love more clarity on this — I haven't dug into it too much.) Assuming most Celery task queue work involves network requests that can be unpredictable, I think Ofair is a more sensible/understandable default.
bjt 11y ago
In my experience Celery brings more cost in complexity than value. It's got several abstraction layers to make a variety of backends (Redis, Mongo, relational DBs) all look more or less like AMQP. My life got better when I stopped using Celery and instead started using Redis or RabbitMQ directly. It's not that hard.
- 1_player 11y ago
  I have been working on a Django application with two periodic background tasks to synchronize some data with some hardware, I was using Celery+Redis to handle the schedulation. The solutions for running cron-like tasks on Python are not very mature, everybody seems to be using Celery. Once deployed in production I spent countless evenings debugging why the tasks weren't running, why upon timeouts tasks weren't terminated, and why the task queue was getting bigger and bigger. 50% were configuration problems, due to the _bad_ and confusing documentation, 50% were due to introducing a sheer amount of complexity with Celery that I often spent my time reading through the source code to understand what was going on. And Python, with all its magic methods and abstractions, made that very hard. One day I just rewrote everything with three Django management commands: two for the background tasks, one for the schedulation (with multiprocessing). I haven't heard from the client since.
andy_ppp 11y ago
I've found a few problems with celery myself. It has an absolutely amazing feature set and seems like a clever and very clean abstraction. However, when it starts hiding exceptions within your tasks if you use JSON for messages or if you are having problems with it doing things you were not expecting... Like randomly not passing information between dependant tasks... Gevent and celery I have found don't play well together even if you disable Gevent for the worker. Maybe you eventually realise that writing your own task for rabbitmq and your own workers is simpler than relying on someone elses complex code. Celery will 100% do what you want but it's definitely been a steeper learning curve than it should be and has lots of gotchas and many hours of hunting around to deal with obscure issues. I do wonder if there is a reasonable heuristic, apart from experience, to learn when a piece of software may be too complex for at least me to use well.

47 points by taylorhughes 11y ago

Forgive me for asking, but how is "no default timeout" a sensible default for any technology?

And do not use Redis as a broker under a high load. It will random crash or refuse to set workers to work, while queue is filling up with tasks.

mooted1 11y ago
At Uber we use redis as a broker for 10k+ requests per second (pushing 50k one some days) with up to several million items backed up. This is a single redis instance with replication turned on and RDB snapshots for durability. Curious how our use patterns are different that causes what you're seeing.
- bliti 11y ago
  Tell me about the hardware specs where your Redis runs.
- andy_ppp 11y ago
  Is that with Celery or are you using your own queue management?
taylorhughes 11y ago
Good point. We've run into some issues with dropping tasks when the redis host is underwater; can imagine that becoming problematic.
- sandGorgon 11y ago
  We recently built out a small background task runner - we were told to use RQ+Redis rather than Celery because (and I'm repeating hearsay here) - 1. Celery's code is very messy and 2. because of the way it keeps jobs in memory, it is very prone to memory leaks. Is there any truth to this ?
  - caio1982 11y ago
    I don't know how much of it is truth but I've definitely had problems with Celery and memory leaks in the past myself. We can add this up to the hearsay pile though.
    - gsharma 11y ago
      What do you use as a broker? We have seen these issues too and use Redis as the broker.
  - phunge 11y ago
    Celery works, that's been my experience. It's buggy, complicated, and we've had unresolved issues, but it more or less does the job. The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
    - sandGorgon 11y ago
      have you looked at RQ - would love to get a comparitive opinion.
  - mpdehaan2 11y ago
    My experience is that Redis slows down considerably with a large amount of data in the queue, and also consumes large amounts of RAM. I'm not sure I'd look to something that explicitly uses Redis as an alternative unless I could be sure the queue would be reasonably small. Which, if the jobs are not going to be overwhelmed with backed up jobs, maybe that's fine. Also, maybe we were doing it wrong.

humbertomn 11y ago

For the retry part, I prefer to use a database table + cron task to do it... storing failed attempts and making x new attempts in predefined date and times, not having it permanently on a celery queue.

simonpantzare 11y ago

-Ofair disables task prefetching which will affect throughput negatively if most of your tasks run for a short time (http://celery.readthedocs.org/en/latest/userguide/optimizing.html#prefetch-limits http://celery.readthedocs.org/en/latest/userguide/optimizing...).

mlissner 11y ago
yeah, I don't get this suggestion at all. They say that: > This option comes with a coordination penalty, but results in a much more predictable behavior In general, I'd much rather the greater throughput over predictable behavior. In general, I'm not watching my queue and having it be predictable isn't a very high priority.
- taylorhughes 11y ago
  Ofair will increase throughput in a queue where you have mixed-length tasks, because in a default configuration long tasks will block the queue from starting new tasks. I believe this is because the main process distributes some number of tasks, waits for them to finish, then distributes another set of tasks, and so on. (Would love more clarity on this — I haven't dug into it too much.) Assuming most Celery task queue work involves network requests that can be unpredictable, I think Ofair is a more sensible/understandable default.

bjt 11y ago

In my experience Celery brings more cost in complexity than value. It's got several abstraction layers to make a variety of backends (Redis, Mongo, relational DBs) all look more or less like AMQP. My life got better when I stopped using Celery and instead started using Redis or RabbitMQ directly. It's not that hard.

1_player 11y ago
I have been working on a Django application with two periodic background tasks to synchronize some data with some hardware, I was using Celery+Redis to handle the schedulation. The solutions for running cron-like tasks on Python are not very mature, everybody seems to be using Celery. Once deployed in production I spent countless evenings debugging why the tasks weren't running, why upon timeouts tasks weren't terminated, and why the task queue was getting bigger and bigger. 50% were configuration problems, due to the _bad_ and confusing documentation, 50% were due to introducing a sheer amount of complexity with Celery that I often spent my time reading through the source code to understand what was going on. And Python, with all its magic methods and abstractions, made that very hard. One day I just rewrote everything with three Django management commands: two for the background tasks, one for the schedulation (with multiprocessing). I haven't heard from the client since.

andy_ppp 11y ago

I've found a few problems with celery myself. It has an absolutely amazing feature set and seems like a clever and very clean abstraction. However, when it starts hiding exceptions within your tasks if you use JSON for messages or if you are having problems with it doing things you were not expecting... Like randomly not passing information between dependant tasks... Gevent and celery I have found don't play well together even if you disable Gevent for the worker. Maybe you eventually realise that writing your own task for rabbitmq and your own workers is simpler than relying on someone elses complex code. Celery will 100% do what you want but it's definitely been a steeper learning curve than it should be and has lots of gotchas and many hours of hunting around to deal with obscure issues. I do wonder if there is a reasonable heuristic, apart from experience, to learn when a piece of software may be too complex for at least me to use well.