2 ms·

I guess this should be clarified more in the README - this uses zmq (and potentially other brokers) as the queue, and uses the database for tracking results, sa

• meritt 12y ago
  Good to know, so more like a crontab entry for the database. Where's the actual queue though? zmq isn't a message queue, it's more akin to a protocol. e.g. if I have N workers and M jobs, and M>N, where are all the >N indexed jobs kept while the others are being processed? Like it appears you're using PUSH/PULL pattern, which means the jobs queue up in the publisher's memory (on a per PULL-peer basis..). So if the publisher crashes, those jobs are lost. If the number of jobs queued exceeds the high-water mark, those additional pushes are either going to discard messages or block. If it blocks, it looks like your code is going to have lots of other issues... If the worker crashes, I believe that queue is also going to be lost.
  • mpconlen 12y ago
    You are right about the push/pull pattern, in the case the publisher crashes those jobs will be lost. That could be an issue depending on the use case and would be a great thing to address. If I am understanding the other notes correctly: Neither the worker nor publisher should ever really block (even in cases where M>N, N>M, publisher crashed, etc), mostly due to the async zmq bindings for node. If there are a ton of messages being sent to a single worker this worker will pull down messages as fast as it can and the rest will still be in publisher queue. If a worker crashes this is happening separate from the publisher and won't affect the queue stored in the publisher's memory