3 ms·

I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the p

I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the problems you found with Bull? Writing a queue that atomically processes items is non-trivial, so when I wanted a feature in Bull I just improved it, the devs are very receptive. Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS. EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.

eropple 8y ago
If I had to choose between Kue and Bull, I'd choose Bull. But I found it frustrating to work with. - I want to support multiple backends. Bull is a Redis queue. Which is fine, as far as it goes, but I have some early designs on an AWS-backed SQS/Dynamo backend that obviates the need for Redis entirely, among other things. ElastiCache is not the worst thing in the world to have to run but not having to directly run anything is, in many environments, preferable, so that's a route I want to go down. - Separate queues for each type of job (per-key) isn't sufficient control. I think it's better to categorize jobs by priority. And while one could multiplex on top of a single queue it, tbh, makes it pretty rough to handle stuff like "these jobs are all of similar priority but have different retry counts and strategies". That multiplexer then isn't integrated with metrics, frontends, etc. and it becomes a "I guess we just reimplement everything except the basic transport on top of Bull" situation. - Also, the basic transport itself is arguable. TaskBotJS uses BRPOP in Redis and I think it's better in the general case to go "if the process hard crashes and doesn't have time to respond to a stop signal I may lose a job". When necessary, an administrator can explicitly opt-in to more complicated and more resource-intensive completely reliable queueing. IMO, RPOPLPUSH plus a heartbeat is a better way to do this than expiring locks, too; expecting a developer to renew the job lock or tune its timing is a bigger ask than "the worker's still working on it--whoops, the worker went away, requeue the job". - Bull uses functions over classes for jobs. In many ways that's defensible; I don't think it scales (people-scales, not computer-scales) across a large dev team without doing a lot of work on top of it. TaskBotJS expects you to provide dependencies to your jobs that it can inject into the worker state--every Kue/Bull-based processor I've seen tends to use globals wrapped in the job closure for stuff like database connections, and I do not subscribe to this point of view and I think you shouldn't either. Any one thing (save pluggable backends) I could have implemented on top of Bull. IMO it'd have been worse to do so, but I could have. But the sum total of things I think could be done better made it easier to write my own, and then to refine it over time into TaskBotJS. Don't get me wrong: I think Bull is fine. The compromises it forces upon you are ones I don't want to make and I don't think developers should be forced to, but of course you can get by with it. But I wouldn't have built TaskBotJS if I didn't think it was better. Throw shade if you'd like; I made the tool before I made the product and I really don't expect it to make more money than enough to keep me focused on its maintenance.
- doublerebel 8y ago
  My priorities as a business needing a job queue are: - Does it run my jobs reliably? - Does it run my jobs with good performance? - Are failed jobs obvious and simple to report/retry/log? Bull's tests and history shows that these needs are addressed. For a backend, the Redis strategy employed by Bull is battle-tested and used by many other related projects. I can't say that about any other backend. I haven't found this to be a common, real issue in practice. For multiple queues, I can easily run multiple namespaced Bull instances, so that is not an issue. Bull uses subscriptions and RPOPLPUSH, so I'm not sure where the polling problem is that you mention. Bull's performance testing is very competitive. Functions vs classes is an interesting dev choice... In JavaScript there is really not much of a difference. I would argue that functions are more lightweight, which is good for a job queue, but I don't actually think a job queue needs an opinion on client JS architecture. If you have performance numbers and a test suite for TaskBotJS, that would go a long way to support the claims.
  - smt88 8y ago
    My priorities are similar, but #1 for me is how easy it is to maintain. It sounds like most of the benefits over Bull are focused on that area.
    - eropple 8y ago
      That's a fair read, IMO. My contention is that by the time you've built something on top of Kue or Bull that feels good to a developer...you're probably going to have built most of TaskBotJS.

eropple 8y ago

If I had to choose between Kue and Bull, I'd choose Bull. But I found it frustrating to work with. - I want to support multiple backends. Bull is a Redis queue. Which is fine, as far as it goes, but I have some early designs on an AWS-backed SQS/Dynamo backend that obviates the need for Redis entirely, among other things. ElastiCache is not the worst thing in the world to have to run but not having to directly run anything is, in many environments, preferable, so that's a route I want to go down. - Separate queues for each type of job (per-key) isn't sufficient control. I think it's better to categorize jobs by priority. And while one could multiplex on top of a single queue it, tbh, makes it pretty rough to handle stuff like "these jobs are all of similar priority but have different retry counts and strategies". That multiplexer then isn't integrated with metrics, frontends, etc. and it becomes a "I guess we just reimplement everything except the basic transport on top of Bull" situation. - Also, the basic transport itself is arguable. TaskBotJS uses BRPOP in Redis and I think it's better in the general case to go "if the process hard crashes and doesn't have time to respond to a stop signal I may lose a job". When necessary, an administrator can explicitly opt-in to more complicated and more resource-intensive completely reliable queueing. IMO, RPOPLPUSH plus a heartbeat is a better way to do this than expiring locks, too; expecting a developer to renew the job lock or tune its timing is a bigger ask than "the worker's still working on it--whoops, the worker went away, requeue the job". - Bull uses functions over classes for jobs. In many ways that's defensible; I don't think it scales (people-scales, not computer-scales) across a large dev team without doing a lot of work on top of it. TaskBotJS expects you to provide dependencies to your jobs that it can inject into the worker state--every Kue/Bull-based processor I've seen tends to use globals wrapped in the job closure for stuff like database connections, and I do not subscribe to this point of view and I think you shouldn't either. Any one thing (save pluggable backends) I could have implemented on top of Bull. IMO it'd have been worse to do so, but I could have. But the sum total of things I think could be done better made it easier to write my own, and then to refine it over time into TaskBotJS. Don't get me wrong: I think Bull is fine. The compromises it forces upon you are ones I don't want to make and I don't think developers should be forced to, but of course you can get by with it. But I wouldn't have built TaskBotJS if I didn't think it was better. Throw shade if you'd like; I made the tool before I made the product and I really don't expect it to make more money than enough to keep me focused on its maintenance.

doublerebel 8y ago
My priorities as a business needing a job queue are: - Does it run my jobs reliably? - Does it run my jobs with good performance? - Are failed jobs obvious and simple to report/retry/log? Bull's tests and history shows that these needs are addressed. For a backend, the Redis strategy employed by Bull is battle-tested and used by many other related projects. I can't say that about any other backend. I haven't found this to be a common, real issue in practice. For multiple queues, I can easily run multiple namespaced Bull instances, so that is not an issue. Bull uses subscriptions and RPOPLPUSH, so I'm not sure where the polling problem is that you mention. Bull's performance testing is very competitive. Functions vs classes is an interesting dev choice... In JavaScript there is really not much of a difference. I would argue that functions are more lightweight, which is good for a job queue, but I don't actually think a job queue needs an opinion on client JS architecture. If you have performance numbers and a test suite for TaskBotJS, that would go a long way to support the claims.
- smt88 8y ago
  My priorities are similar, but #1 for me is how easy it is to maintain. It sounds like most of the benefits over Bull are focused on that area.
  - eropple 8y ago
    That's a fair read, IMO. My contention is that by the time you've built something on top of Kue or Bull that feels good to a developer...you're probably going to have built most of TaskBotJS.