4 ms·

The overhead is messaging to rabbit, having that create ids for it and queue it; then celery pulls the next job from the exchange, unpacks the job description w

The overhead is messaging to rabbit, having that create ids for it and queue it; then celery pulls the next job from the exchange, unpacks the job description with its arguments, and then calls the function (wrapped in exception handlers etc. etc.) Celery is suitable for "send an email" or "rebuild a value and put in memcache" but not suitable for "when the OS finally returns the file handle I requested, could you read the file into memory please" Totally different time scales and use-cases.

kerkeslager 10y ago
I guess what I'm confused about is the "waiting for the result" in the parent comment. Celery is pretty much entirely fire-and-forget, and you're usually firing to an entirely different server. If this is being used to achieve parallelization instead of asynchronous processing, then it's no wonder that the parent is having trouble, because they're using Celery for a task for which it's wholly inappropriate.
- crucialfelix 10y ago
  Agreed. There are a lot of bells and whistles with celery and I've tried them and now avoid them all. I don't nest or chain tasks. I don't use periodic tasks because they can pile up badly and block everything. I have my own simple loop that calls the most urgent periodic-task. I built an angular button that would call a task and would show spinning in-progress and then show when the task completes. Works on development, didn't work on production. why why why ? well, no time to fiddle with celery so I moved on. over the years I've come to mistrust its complexity.
  - kerkeslager 10y ago
    > I don't nest or chain tasks. Yeah, I think nested/chained tasks are pointless. The point of celery is to offload the task to an asychronous worker (probably on another machine) so if you're already on the worker there's no point sending another task to the same worker--it doesn't achieve anything above doing the task inline. > I don't use periodic tasks because they can pile up badly and block everything. This sounds like you're either running the periodic task too frequently, or you need more workers. I've seen a few cases where people tried running a 20-minute task every second on a single worker--of course this is going to pile up. Periodic tasks are among the most predictable, so they're easiest to prevent from piling up: simply make sure that the following relation holds true: w > t / p w = number of workers t = how long it takes for the task to run p = the period (a task that runs ever 20 minutes has period 20 minutes) The > should be greater than by an order of magnitude to be safe in case t increases.
    - crucialfelix 10y ago
      w > t / p .... works fine. as long as nothing ever goes wrong ;) Then you have a rabbit queue that has hundreds of rescheds piled up and I have to manually unclog the queue. Also increasing workers isn't so simple. There is only so much memory on a machine, and I don't need or want to add machines. A more efficient loop was the better solution. The machine is now full of work all the time and stable in memory, CPU usage and db access. This is no longer a problem now that I don't use periodic tasks. "periodic" doesn't describe the nature of the work to be done. In any case the main problem I needed to solve is that I don't have single tasks that need to happen every p seconds. I have 40 tasks that have different min-max-acceptable-backlog-queue-length, different min-max time periods that they could run in. Different day and night cycle ranges. Some really need to go every 2 minutes, but only if there are emails to send. Some are things to do whenever there is spare time. Some image processing queues need to be done very quickly, for others it can chew on it all night when there are spare cycles.

kerkeslager 10y ago

I guess what I'm confused about is the "waiting for the result" in the parent comment. Celery is pretty much entirely fire-and-forget, and you're usually firing to an entirely different server. If this is being used to achieve parallelization instead of asynchronous processing, then it's no wonder that the parent is having trouble, because they're using Celery for a task for which it's wholly inappropriate.

crucialfelix 10y ago
Agreed. There are a lot of bells and whistles with celery and I've tried them and now avoid them all. I don't nest or chain tasks. I don't use periodic tasks because they can pile up badly and block everything. I have my own simple loop that calls the most urgent periodic-task. I built an angular button that would call a task and would show spinning in-progress and then show when the task completes. Works on development, didn't work on production. why why why ? well, no time to fiddle with celery so I moved on. over the years I've come to mistrust its complexity.
- kerkeslager 10y ago
  > I don't nest or chain tasks. Yeah, I think nested/chained tasks are pointless. The point of celery is to offload the task to an asychronous worker (probably on another machine) so if you're already on the worker there's no point sending another task to the same worker--it doesn't achieve anything above doing the task inline. > I don't use periodic tasks because they can pile up badly and block everything. This sounds like you're either running the periodic task too frequently, or you need more workers. I've seen a few cases where people tried running a 20-minute task every second on a single worker--of course this is going to pile up. Periodic tasks are among the most predictable, so they're easiest to prevent from piling up: simply make sure that the following relation holds true: w > t / p w = number of workers t = how long it takes for the task to run p = the period (a task that runs ever 20 minutes has period 20 minutes) The > should be greater than by an order of magnitude to be safe in case t increases.
  - crucialfelix 10y ago
    w > t / p .... works fine. as long as nothing ever goes wrong ;) Then you have a rabbit queue that has hundreds of rescheds piled up and I have to manually unclog the queue. Also increasing workers isn't so simple. There is only so much memory on a machine, and I don't need or want to add machines. A more efficient loop was the better solution. The machine is now full of work all the time and stable in memory, CPU usage and db access. This is no longer a problem now that I don't use periodic tasks. "periodic" doesn't describe the nature of the work to be done. In any case the main problem I needed to solve is that I don't have single tasks that need to happen every p seconds. I have 40 tasks that have different min-max-acceptable-backlog-queue-length, different min-max time periods that they could run in. Different day and night cycle ranges. Some really need to go every 2 minutes, but only if there are emails to send. Some are things to do whenever there is spare time. Some image processing queues need to be done very quickly, for others it can chew on it all night when there are spare cycles.