3 ms·

We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramati

We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramatiq + Celery<4.0 in use.

sdan 7y ago
Do you use Dramatiq or Celery? Do either fix zombie tasks/zombie workers?
- wakatime 7y ago
  Both, but unfortunately still mostly using Celery 3.2. Never had any problems with zombie tasks, since that's related to how you're storing tasks and queues and RabbitMQ is excellent at that. Zombie tasks are never a problem when using RabbitMQ. Especially never had problems with zombie workers. That wouldn't be a problem with your choice of queue library, more an infra problem.
  - sdan 7y ago
    Interesting. At the moment I'm noticing some zombie workers popping up here and there with RQ and was thinking of migrating soon to Celery or Dramatiq. I think I'll experiment a bit and decide between both (but Celery seems like the more popular option, so I'm a bit biased for going for that).
wojcikstefan 7y ago
Interesting, we haven't had issues with zombie tasks at all (and we DID have issues with them when using Celery). Did you manage to find out what was causing them? > and some small but annoying bugs while testing Do you recall any details? Appreciate you trying TaskTiger, even if you've moved on since!
- wakatime 7y ago
  Here's the list of my issues: https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Aalanhamlett https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q... It came down to I like the features of RabbitMQ: * RabbitMQ scales messages without needing tons of RAM * I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1] * Queue and task visibility is better in RabbitMQ * Support for purging all tasks in a queue * Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking? Things I did like about Tasktiger: * no feature bloat and it's possible to actually read it's source code[2]. [1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine. [2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.

We tried Tasktiger but ran into zombie tasks and some small but annoying bugs while testing. Around that same time, we found Dramatiq and currently have Dramatiq + Celery<4.0 in use.

sdan 7y ago

Do you use Dramatiq or Celery? Do either fix zombie tasks/zombie workers?

wakatime 7y ago
Both, but unfortunately still mostly using Celery 3.2. Never had any problems with zombie tasks, since that's related to how you're storing tasks and queues and RabbitMQ is excellent at that. Zombie tasks are never a problem when using RabbitMQ. Especially never had problems with zombie workers. That wouldn't be a problem with your choice of queue library, more an infra problem.
- sdan 7y ago
  Interesting. At the moment I'm noticing some zombie workers popping up here and there with RQ and was thinking of migrating soon to Celery or Dramatiq. I think I'll experiment a bit and decide between both (but Celery seems like the more popular option, so I'm a bit biased for going for that).

wojcikstefan 7y ago

Interesting, we haven't had issues with zombie tasks at all (and we DID have issues with them when using Celery). Did you manage to find out what was causing them? > and some small but annoying bugs while testing Do you recall any details? Appreciate you trying TaskTiger, even if you've moved on since!

wakatime 7y ago
Here's the list of my issues: https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Aalanhamlett https://github.com/closeio/tasktiger/issues?utf8=%E2%9C%93&q... It came down to I like the features of RabbitMQ: * RabbitMQ scales messages without needing tons of RAM * I don't have to decide between not persisting messages to disk with Redis vs only using half the machine's RAM [1] * Queue and task visibility is better in RabbitMQ * Support for purging all tasks in a queue * Tasktiger had lower throughput than Celery and Dramatiq, maybe needs lazy-forking? Things I did like about Tasktiger: * no feature bloat and it's possible to actually read it's source code[2]. [1]: To enable persisting to disk (Redis fork + save snapshotting) you must limit Redis to only using half the available RAM on a machine. [2]: Celery is split up into multiple convoluted, bloated, difficult to read repos.