4 ms·

Shameless plug: Our team at Close has been inspired by RQ when we created TaskTiger – https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger.

Shameless plug: Our team at Close has been inspired by RQ when we created TaskTiger – https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger. We’ve been running it in production for a few years now, processing ~4M tasks a day and it’s been a wonderful tool to work with. Curious to hear what y’all think!

mixmastamyk 7y ago
What’s the difference?
hu3 7y ago
Curious. What's the ROI on developing a task queue system for 50 req/sec? Please don't take my question negatively. I acknowledge that not everything is about ROI. For example I often let my team scratch their own itches by developing tools just for the sake of having fun and learning new toys. Perhaps this is the case with TaskTiger?
- wojcikstefan 7y ago
  It was definitely fun to work on it :) That said, I think we've also seen quite a big positive ROI as well. It is true that we've spent quite some time working on TaskTiger where we could've used an off-the-shelf task queue system. However, since we've created it, it allowed us to develop new features a lot faster and spend a lot less time debugging issues. We were using Celery before and it was a big pain with memory leaks, periodic tasks not working as expected, some tasks being lost for no apparent reason, and poor visibility into both aggregate statistics and individual tasks. We spent too much time debugging Celery's issues instead of working on our core business. Here's some more context: https://github.com/closeio/tasktiger/issues/141#issuecomment-545597533 https://github.com/closeio/tasktiger/issues/141#issuecomment... Advanced features like scheduled tasks, unique queues, and task locks have helped us move faster, too.
wakatime 7y ago
We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramatiq + Celery<4.0 in use.
- sdan 7y ago
  Do you use Dramatiq or Celery? Do either fix zombie tasks/zombie workers?
  - wakatime 7y ago
    Both, but unfortunately still mostly using Celery 3.2. Never had any problems with zombie tasks, since that's related to how you're storing tasks and queues and RabbitMQ is excellent at that. Zombie tasks are never a problem when using RabbitMQ. Especially never had problems with zombie workers. That wouldn't be a problem with your choice of queue library, more an infra problem.
    - sdan 7y ago
      Interesting. At the moment I'm noticing some zombie workers popping up here and there with RQ and was thinking of migrating soon to Celery or Dramatiq. I think I'll experiment a bit and decide between both (but Celery seems like the more popular option, so I'm a bit biased for going for that).
- wojcikstefan 7y ago
  Interesting, we haven't had issues with zombie tasks at all (and we DID have issues with them when using Celery). Did you manage to find out what was causing them? > and some small but annoying bugs while testing Do you recall any details? Appreciate you trying TaskTiger, even if you've moved on since!
  - wakatime 7y ago
    Here's the list of my issues: https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Aalanhamlett https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q... It came down to I like the features of RabbitMQ: * RabbitMQ scales messages without needing tons of RAM * I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1] * Queue and task visibility is better in RabbitMQ * Support for purging all tasks in a queue * Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking? Things I did like about Tasktiger: * no feature bloat and it's possible to actually read it's source code[2]. [1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine. [2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.

mixmastamyk 7y ago

What’s the difference?

hu3 7y ago

Curious. What's the ROI on developing a task queue system for 50 req/sec? Please don't take my question negatively. I acknowledge that not everything is about ROI. For example I often let my team scratch their own itches by developing tools just for the sake of having fun and learning new toys. Perhaps this is the case with TaskTiger?

wojcikstefan 7y ago
It was definitely fun to work on it :) That said, I think we've also seen quite a big positive ROI as well. It is true that we've spent quite some time working on TaskTiger where we could've used an off-the-shelf task queue system. However, since we've created it, it allowed us to develop new features a lot faster and spend a lot less time debugging issues. We were using Celery before and it was a big pain with memory leaks, periodic tasks not working as expected, some tasks being lost for no apparent reason, and poor visibility into both aggregate statistics and individual tasks. We spent too much time debugging Celery's issues instead of working on our core business. Here's some more context: https://github.com/closeio/tasktiger/issues/141#issuecomment-545597533 https://github.com/closeio/tasktiger/issues/141#issuecomment... Advanced features like scheduled tasks, unique queues, and task locks have helped us move faster, too.

wakatime 7y ago

We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramatiq + Celery<4.0 in use.

sdan 7y ago
Do you use Dramatiq or Celery? Do either fix zombie tasks/zombie workers?
- wakatime 7y ago
  Both, but unfortunately still mostly using Celery 3.2. Never had any problems with zombie tasks, since that's related to how you're storing tasks and queues and RabbitMQ is excellent at that. Zombie tasks are never a problem when using RabbitMQ. Especially never had problems with zombie workers. That wouldn't be a problem with your choice of queue library, more an infra problem.
  - sdan 7y ago
    Interesting. At the moment I'm noticing some zombie workers popping up here and there with RQ and was thinking of migrating soon to Celery or Dramatiq. I think I'll experiment a bit and decide between both (but Celery seems like the more popular option, so I'm a bit biased for going for that).
wojcikstefan 7y ago
Interesting, we haven't had issues with zombie tasks at all (and we DID have issues with them when using Celery). Did you manage to find out what was causing them? > and some small but annoying bugs while testing Do you recall any details? Appreciate you trying TaskTiger, even if you've moved on since!
- wakatime 7y ago
  Here's the list of my issues: https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Aalanhamlett https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q... It came down to I like the features of RabbitMQ: * RabbitMQ scales messages without needing tons of RAM * I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1] * Queue and task visibility is better in RabbitMQ * Support for purging all tasks in a queue * Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking? Things I did like about Tasktiger: * no feature bloat and it's possible to actually read it's source code[2]. [1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine. [2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.