4 ms·

I used this in a project at a previous job -- I have to say, while the API is simple and useful enough for small projects, it raises some issues with how it's d

I used this in a project at a previous job -- I have to say, while the API is simple and useful enough for small projects, it raises some issues with how it's designed. Instead of relying on trusted exposed endpoints and just invoking them by URL, it does a bytecode dump of task functions and stores those in Redis before restoring them from bytecode at execution time. This has a few drawbacks: - Payloads in the queue are potentially a fair bit larger for complex jobs - Serialization for stuff that has decorators (and especially stateful one, like `lru_cache`) is not really possible, even with `dill` instead of `pickle` - It's not trivial, but this exposes a different set of security risks compared to the alternative I don't want to say this is a bad piece of software, it's super easy to set up and way more lightweight than Celery for example, but it's not my tool of choice having worked with the alternatives.

yamrzou 7y ago
Which alternatives worked for you?
- wakatime 7y ago
  Can't speak for op, but for us the ones based on RabbitMQ have performed well: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq Stay away from Celery if you can, or stick to v3.2 because 4.x has a ton of bugs.
  - scaryclam 7y ago
    Celery has always been a massive resource drain, and for pretty much zero gain. I'd suggest just saying "stay away from Celery" as it's easy to get going with, but really hard to scale or work with when you need what it promises. There are better options.
  - _verandaguy 7y ago
    Agree that you should stay away from Celery in most cases, but it does integrate well with Django, and pre-4.x is stable enough. It's just a fairly higher amount of boilerplate and setup compared to most of the alternatives either way.
- _verandaguy 7y ago
  So far I've mostly used Celery in Python, but at jobs past there was also Sidekiq (granted, the use cases for that were a bit different, but it's kind of in the same bucket). Haven't had the opportunity to work with them, but I've heard good things about SQS and other cloud queues, but I think those are fundamentally message queues, so you'd be responsible for writing a job queue layer over it, which can be a huge hassle.
IgorPartola 7y ago
Wait is that how RQ works?! How does it handle code updates? What happens if I update the task code while there are tasks still enqueued?
- _verandaguy 7y ago
  Can't speak to the code updates since I never went that in-depth into the code. The pickle logic lives here: https://github.com/rq/rq/blob/80c82f731f57186f8c97239b38100f4f793e19cf/rq/job.py https://github.com/rq/rq/blob/80c82f731f57186f8c97239b38100f... so if you've got a minute you can skim that and hopefully get an answer to your question.
deckar01 7y ago
I had to setup a task queue in an old, resource constrained environment once. I ended up forking the client process into a worker when the queue wasn't already running. Any time we need an expensive method call to be throttled asynchronously we just have to add a decorator to it. https://github.com/deckar01/low_queue https://github.com/deckar01/low_queue

yamrzou 7y ago

Which alternatives worked for you?

wakatime 7y ago
Can't speak for op, but for us the ones based on RabbitMQ have performed well: https://github.com/Bogdanp/dramatiq https://github.com/Bogdanp/dramatiq Stay away from Celery if you can, or stick to v3.2 because 4.x has a ton of bugs.
- scaryclam 7y ago
  Celery has always been a massive resource drain, and for pretty much zero gain. I'd suggest just saying "stay away from Celery" as it's easy to get going with, but really hard to scale or work with when you need what it promises. There are better options.
- _verandaguy 7y ago
  Agree that you should stay away from Celery in most cases, but it does integrate well with Django, and pre-4.x is stable enough. It's just a fairly higher amount of boilerplate and setup compared to most of the alternatives either way.
_verandaguy 7y ago
So far I've mostly used Celery in Python, but at jobs past there was also Sidekiq (granted, the use cases for that were a bit different, but it's kind of in the same bucket). Haven't had the opportunity to work with them, but I've heard good things about SQS and other cloud queues, but I think those are fundamentally message queues, so you'd be responsible for writing a job queue layer over it, which can be a huge hassle.

IgorPartola 7y ago

Wait is that how RQ works?! How does it handle code updates? What happens if I update the task code while there are tasks still enqueued?

_verandaguy 7y ago
Can't speak to the code updates since I never went that in-depth into the code. The pickle logic lives here: https://github.com/rq/rq/blob/80c82f731f57186f8c97239b38100f4f793e19cf/rq/job.py https://github.com/rq/rq/blob/80c82f731f57186f8c97239b38100f... so if you've got a minute you can skim that and hopefully get an answer to your question.

deckar01 7y ago

I had to setup a task queue in an old, resource constrained environment once. I ended up forking the client process into a worker when the queue wasn't already running. Any time we need an expensive method call to be throttled asynchronously we just have to add a decorator to it. https://github.com/deckar01/low_queue https://github.com/deckar01/low_queue