4 ms·

Whee! My first Show HN. I've only been here, like, seven years and all. But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a while. I

Whee! My first Show HN. I've only been here, like, seven years and all. But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a while. I've found Kue and Bull really frustrating to use (no shade intended; they feel like they made a lot more sense in a pre-ES6 universe?) and `node-resque`, while scratching a little more of that itch, just made me miss the comfortable guardrails of Ruby offline job queues. So I made one. As my day job is devops and devops accessories, making it easy to run and administer TaskBotJS was a big priority for me and I've tried to get the best of both worlds: mostly turn-key to run in simpler cases but less prescriptive and more pluggable at scale. TaskBotJS is pretty extensible and supports job middleware and event listeners; I have a Sentry plugin coming in the near future as an example (and also 'cause I like Sentry). I also like to write documentation[0]. While it's not yet complete, what's there is pretty exhaustive and I try to explain my thinking as well as just "what it does"; capturing the whys is as important as the whats, to me, to better foster conversation around the project and to make for more constructive issues when systemic, design-focused concerns are raised. It's running in at least two production environments (only one of which is mine; I dogfood but I wouldn't have released it as a "you should check this out" version if somebody else wasn't checking me) and is being rolled out to at least one more in the near-ish future. Thanks for taking a peek, and I'd love to hear any actionable feedback that comes to mind. [0] - https://github.com/eropple/taskbotjs/wiki https://github.com/eropple/taskbotjs/wiki

Bogdanp 8y ago
Congrats on shipping! As someone who's written similar software[1] (for Python), I can appreciate the hard work that went into this. One piece of feedback regarding the docs: the documentation section in your README has two links, one to the wiki and the other to the website. I had initially scanned the section and followed the link to the website (I don't automatically associate "wiki" with "documentation") and was then unable to find your documentation for a good couple of minutes. You should make the wiki link more prominent. [1]: https://dramatiq.io https://dramatiq.io
ra00l 8y ago
I'm planning to use a similar model OSS / commercial as well. Mike and Sidekiq is my inspiration for this. Good luck going further with the project and congrats for the launch!
disiplus 8y ago
sorry i don't get it. why would somebody use this over a rabbitmq.
- DandyDev 8y ago
  Read and find out
- deleted 8y ago
  [deleted]
- ChrisCinelli 8y ago
  I think it is a legit question... I poked a little around and I did not find an answer.
doublerebel 8y ago
I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the problems you found with Bull? Writing a queue that atomically processes items is non-trivial, so when I wanted a feature in Bull I just improved it, the devs are very receptive. Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS. EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.
- eropple 8y ago
  If I had to choose between Kue and Bull, I'd choose Bull. But I found it frustrating to work with. - I want to support multiple backends. Bull is a Redis queue. Which is fine, as far as it goes, but I have some early designs on an AWS-backed SQS/Dynamo backend that obviates the need for Redis entirely, among other things. ElastiCache is not the worst thing in the world to have to run but not having to directly run anything is, in many environments, preferable, so that's a route I want to go down. - Separate queues for each type of job (per-key) isn't sufficient control. I think it's better to categorize jobs by priority. And while one could multiplex on top of a single queue it, tbh, makes it pretty rough to handle stuff like "these jobs are all of similar priority but have different retry counts and strategies". That multiplexer then isn't integrated with metrics, frontends, etc. and it becomes a "I guess we just reimplement everything except the basic transport on top of Bull" situation. - Also, the basic transport itself is arguable. TaskBotJS uses BRPOP in Redis and I think it's better in the general case to go "if the process hard crashes and doesn't have time to respond to a stop signal I may lose a job". When necessary, an administrator can explicitly opt-in to more complicated and more resource-intensive completely reliable queueing. IMO, RPOPLPUSH plus a heartbeat is a better way to do this than expiring locks, too; expecting a developer to renew the job lock or tune its timing is a bigger ask than "the worker's still working on it--whoops, the worker went away, requeue the job". - Bull uses functions over classes for jobs. In many ways that's defensible; I don't think it scales (people-scales, not computer-scales) across a large dev team without doing a lot of work on top of it. TaskBotJS expects you to provide dependencies to your jobs that it can inject into the worker state--every Kue/Bull-based processor I've seen tends to use globals wrapped in the job closure for stuff like database connections, and I do not subscribe to this point of view and I think you shouldn't either. Any one thing (save pluggable backends) I could have implemented on top of Bull. IMO it'd have been worse to do so, but I could have. But the sum total of things I think could be done better made it easier to write my own, and then to refine it over time into TaskBotJS. Don't get me wrong: I think Bull is fine. The compromises it forces upon you are ones I don't want to make and I don't think developers should be forced to, but of course you can get by with it. But I wouldn't have built TaskBotJS if I didn't think it was better. Throw shade if you'd like; I made the tool before I made the product and I really don't expect it to make more money than enough to keep me focused on its maintenance.
evantahler 8y ago
Nice! We need more work in this space. I'm one of the node-resque authors. I'd love to hear more about what you felt was missing from the ruby version... clearly we copied a lot :D
k_vi 8y ago
I'm using Kue heavily and so far had no problems using it. Can you elaborate on 'pre-ES6 universe' or any other issues you had with it?
nodesocket 8y ago
TaskBotJS looks great. Does it support recurring scheduled tasks? For example: run task foo at 13:00 every day for 14 days. run task bar at 06:30 every day forever (until cancelled).

Bogdanp 8y ago

Congrats on shipping! As someone who's written similar software[1] (for Python), I can appreciate the hard work that went into this. One piece of feedback regarding the docs: the documentation section in your README has two links, one to the wiki and the other to the website. I had initially scanned the section and followed the link to the website (I don't automatically associate "wiki" with "documentation") and was then unable to find your documentation for a good couple of minutes. You should make the wiki link more prominent. [1]: https://dramatiq.io https://dramatiq.io

ra00l 8y ago

I'm planning to use a similar model OSS / commercial as well. Mike and Sidekiq is my inspiration for this. Good luck going further with the project and congrats for the launch!

disiplus 8y ago

sorry i don't get it. why would somebody use this over a rabbitmq.

DandyDev 8y ago
Read and find out
deleted 8y ago
[deleted]
ChrisCinelli 8y ago
I think it is a legit question... I poked a little around and I did not find an answer.

doublerebel 8y ago

I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the problems you found with Bull? Writing a queue that atomically processes items is non-trivial, so when I wanted a feature in Bull I just improved it, the devs are very receptive. Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS. EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.

eropple 8y ago
If I had to choose between Kue and Bull, I'd choose Bull. But I found it frustrating to work with. - I want to support multiple backends. Bull is a Redis queue. Which is fine, as far as it goes, but I have some early designs on an AWS-backed SQS/Dynamo backend that obviates the need for Redis entirely, among other things. ElastiCache is not the worst thing in the world to have to run but not having to directly run anything is, in many environments, preferable, so that's a route I want to go down. - Separate queues for each type of job (per-key) isn't sufficient control. I think it's better to categorize jobs by priority. And while one could multiplex on top of a single queue it, tbh, makes it pretty rough to handle stuff like "these jobs are all of similar priority but have different retry counts and strategies". That multiplexer then isn't integrated with metrics, frontends, etc. and it becomes a "I guess we just reimplement everything except the basic transport on top of Bull" situation. - Also, the basic transport itself is arguable. TaskBotJS uses BRPOP in Redis and I think it's better in the general case to go "if the process hard crashes and doesn't have time to respond to a stop signal I may lose a job". When necessary, an administrator can explicitly opt-in to more complicated and more resource-intensive completely reliable queueing. IMO, RPOPLPUSH plus a heartbeat is a better way to do this than expiring locks, too; expecting a developer to renew the job lock or tune its timing is a bigger ask than "the worker's still working on it--whoops, the worker went away, requeue the job". - Bull uses functions over classes for jobs. In many ways that's defensible; I don't think it scales (people-scales, not computer-scales) across a large dev team without doing a lot of work on top of it. TaskBotJS expects you to provide dependencies to your jobs that it can inject into the worker state--every Kue/Bull-based processor I've seen tends to use globals wrapped in the job closure for stuff like database connections, and I do not subscribe to this point of view and I think you shouldn't either. Any one thing (save pluggable backends) I could have implemented on top of Bull. IMO it'd have been worse to do so, but I could have. But the sum total of things I think could be done better made it easier to write my own, and then to refine it over time into TaskBotJS. Don't get me wrong: I think Bull is fine. The compromises it forces upon you are ones I don't want to make and I don't think developers should be forced to, but of course you can get by with it. But I wouldn't have built TaskBotJS if I didn't think it was better. Throw shade if you'd like; I made the tool before I made the product and I really don't expect it to make more money than enough to keep me focused on its maintenance.

evantahler 8y ago

Nice! We need more work in this space. I'm one of the node-resque authors. I'd love to hear more about what you felt was missing from the ruby version... clearly we copied a lot :D

k_vi 8y ago

I'm using Kue heavily and so far had no problems using it. Can you elaborate on 'pre-ES6 universe' or any other issues you had with it?

nodesocket 8y ago

TaskBotJS looks great. Does it support recurring scheduled tasks? For example: run task foo at 13:00 every day for 14 days. run task bar at 06:30 every day forever (until cancelled).