3 ms·

Kue - redis backed priority job queue for node.js

39 points by tjholowaychuk 15y ago

markbao 15y ago
Incredibly cool, and thanks for this TJ (as well as your other great work for the Node.js community.) Have you seen coffee-resque[1], a CoffeeScript interface to the Resque queueing system originally written in Ruby? It's similar, and both Resque (through resque-web) and your app have really nice interfaces. Did you build this because Resque wasn't everything you wanted? [1]: https://github.com/technoweenie/coffee-resque https://github.com/technoweenie/coffee-resque
- tjholowaychuk 15y ago
  thanks! few reasons: a) I really wanted the logging / progress indication (haven't seen that personally) b) I don't read/write coffeescript c) I don't think coffee-resque has a UI (unless the ruby one works?)
  - tjholowaychuk 15y ago
    and also because it really only took less than a day to write, the UI was the primary focus, the rest is really simple
    - paulitex 15y ago
      minor typo: A key feature of Kue is it's (should be its) clean user-interface for viewing
  - markbao 15y ago
    Very cool, good to know. Yeah, it seems like it was a simple endeavor—Redis makes creating a queueing system quite easy. The Ruby `resque-web` interface works and is beautiful[1] with live polling. [1]: http://fhwang.net/image/image/124/4b0a538a5aa9a3e7dc1ff81bc4d94de4cdeb9e16.jpg http://fhwang.net/image/image/124/4b0a538a5aa9a3e7dc1ff81bc4...
    - tjholowaychuk 15y ago
      ya it's not to shabby, Kue is all live as well.
darrellsilver 15y ago
Awesome, and awesome looking interface. Reminds me of Our project Norc, (github.com/darrellsilver/norc). Kue has intermediate statusnof the job, which would be a great feature for Norc.
gourneau 15y ago
Looks nice, does it support clusters of machines?
- tjholowaychuk 15y ago
  yup, I don't see why not, I plan on integrating cluster (one of our projects). each worker uses its own connection with WATCH to implement ZPOP so it should scale fine. we'll certainly need it to :)
mcs 15y ago
waves the Node.JS flag
chewbranca 15y ago
I keep seeing these (admittedly cool) redis based job queues popping up for different languages, such as resque, pyres, now kue, etc, however I'm disappointed to not see any standardization among them. Obviously they're different projects created by different people, but it would be great to have a common protocol amongst them, especially since they're all just json based messages. I want to be able to use a job queue as a standard way to communicate between various language backends, so I can use different tools for different jobs. I've started looking into beanstalkd as it has an impressive list of client libraries: https://github.com/kr/beanstalkd/wiki/Client-Libraries https://github.com/kr/beanstalkd/wiki/Client-Libraries. I'm curious to hear if anyone has any thoughts/experiences/recommendations on using open source job queues to facilitate background processing between different languages and backends. I'm already using redis and resque, something like rabbitmq is overkill, and I just started looking into beanstalkd.
- tjholowaychuk 15y ago
  i agree interface wise it would be interesting, though i think many have slightly different goals and techniques, and frankly they are really basic, but UI-wise it could be neat
- technoweenie 15y ago
  coffee-resque and resque work together really well. I didn't mess with the python resque lib much, but I was able to get it to read jobs queued by ruby. With a little work, resque could work as a cross platform queue if you wanted. The main issue I ran into is the way the libs bind resque jobs to objects. For instance, the ruby resque library queues jobs by the ruby class name (Foo::Jobs::Bar), which doesn't translate well to python or node. A good first step would be defining a spec based on how resque or kue work so that alternate language ports can be written. In the case of resque, other interfaces like its failure or plugin system should also be documented, so that those libs could be ported too.
heresy 15y ago
One thing I'm not sure about with these job queues - how do you handle cases like having to drop down to C++ to do a chunk of work that cannot be done in JS, without blocking the interpreter? Or would you just submit that work to a separate external process over something like RabbitMQ, and handle the return when its done?
- tjholowaychuk 15y ago
  few ways, you could still utilize child processes of course, so web workers or something. Or just chunk that work up and nextTick() increasing concurrency, OR just increase the total number of workers for more parallelism.
dfield 15y ago
Can someone please make this work with Hadoop?
shapeshed 15y ago
another library from TJ that looks super useful
Cherian 15y ago
This look cool. One question though: How is this different from gearman? Is it the connector with nodejs?

39 points by tjholowaychuk 15y ago

Incredibly cool, and thanks for this TJ (as well as your other great work for the Node.js community.) Have you seen coffee-resque[1], a CoffeeScript interface to the Resque queueing system originally written in Ruby? It's similar, and both Resque (through resque-web) and your app have really nice interfaces. Did you build this because Resque wasn't everything you wanted? [1]: https://github.com/technoweenie/coffee-resque https://github.com/technoweenie/coffee-resque

tjholowaychuk 15y ago
thanks! few reasons: a) I really wanted the logging / progress indication (haven't seen that personally) b) I don't read/write coffeescript c) I don't think coffee-resque has a UI (unless the ruby one works?)
- tjholowaychuk 15y ago
  and also because it really only took less than a day to write, the UI was the primary focus, the rest is really simple
  - paulitex 15y ago
    minor typo: A key feature of Kue is it's (should be its) clean user-interface for viewing
- markbao 15y ago
  Very cool, good to know. Yeah, it seems like it was a simple endeavor—Redis makes creating a queueing system quite easy. The Ruby `resque-web` interface works and is beautiful[1] with live polling. [1]: http://fhwang.net/image/image/124/4b0a538a5aa9a3e7dc1ff81bc4d94de4cdeb9e16.jpg http://fhwang.net/image/image/124/4b0a538a5aa9a3e7dc1ff81bc4...
  - tjholowaychuk 15y ago
    ya it's not to shabby, Kue is all live as well.

darrellsilver 15y ago

Awesome, and awesome looking interface. Reminds me of Our project Norc, (github.com/darrellsilver/norc). Kue has intermediate statusnof the job, which would be a great feature for Norc.

gourneau 15y ago

Looks nice, does it support clusters of machines?

tjholowaychuk 15y ago
yup, I don't see why not, I plan on integrating cluster (one of our projects). each worker uses its own connection with WATCH to implement ZPOP so it should scale fine. we'll certainly need it to :)

mcs 15y ago

waves the Node.JS flag

chewbranca 15y ago

I keep seeing these (admittedly cool) redis based job queues popping up for different languages, such as resque, pyres, now kue, etc, however I'm disappointed to not see any standardization among them. Obviously they're different projects created by different people, but it would be great to have a common protocol amongst them, especially since they're all just json based messages. I want to be able to use a job queue as a standard way to communicate between various language backends, so I can use different tools for different jobs. I've started looking into beanstalkd as it has an impressive list of client libraries: https://github.com/kr/beanstalkd/wiki/Client-Libraries https://github.com/kr/beanstalkd/wiki/Client-Libraries. I'm curious to hear if anyone has any thoughts/experiences/recommendations on using open source job queues to facilitate background processing between different languages and backends. I'm already using redis and resque, something like rabbitmq is overkill, and I just started looking into beanstalkd.

tjholowaychuk 15y ago
i agree interface wise it would be interesting, though i think many have slightly different goals and techniques, and frankly they are really basic, but UI-wise it could be neat
technoweenie 15y ago
coffee-resque and resque work together really well. I didn't mess with the python resque lib much, but I was able to get it to read jobs queued by ruby. With a little work, resque could work as a cross platform queue if you wanted. The main issue I ran into is the way the libs bind resque jobs to objects. For instance, the ruby resque library queues jobs by the ruby class name (Foo::Jobs::Bar), which doesn't translate well to python or node. A good first step would be defining a spec based on how resque or kue work so that alternate language ports can be written. In the case of resque, other interfaces like its failure or plugin system should also be documented, so that those libs could be ported too.

heresy 15y ago

One thing I'm not sure about with these job queues - how do you handle cases like having to drop down to C++ to do a chunk of work that cannot be done in JS, without blocking the interpreter? Or would you just submit that work to a separate external process over something like RabbitMQ, and handle the return when its done?

tjholowaychuk 15y ago
few ways, you could still utilize child processes of course, so web workers or something. Or just chunk that work up and nextTick() increasing concurrency, OR just increase the total number of workers for more parallelism.

dfield 15y ago

Can someone please make this work with Hadoop?

shapeshed 15y ago

another library from TJ that looks super useful

Cherian 15y ago

This look cool. One question though: How is this different from gearman? Is it the connector with nodejs?