4 ms·
I wouldn't really classify this example as a "task queue", it resembles more the RPC pattern really (and I would guess that this example does not persist the jo
by asksol 13y ago
I wouldn't really classify this example as a "task queue", it resembles more the RPC pattern really (and I would guess that this example does not persist the jobs in any way). Celery does have a very experimental filesystem based transport that could be used for your use case. I don't know if it fits on a Raspberry PI but celery does not have very high memory/space requirements (c.f. other Python libraries).
- jknupp 13y agoZeroMQ takes care of the queueing. Though I didn't delve into it in depth in this example, you can create quite sophisticated broker-less distributed systems pretty easily with ZeroMQ.
- asksol 13y agoRight, I understand that ZeroMQ is used to send and receive messages, what I mean is that there's no persistency involved so the tasks will not survive a system restart.
- jknupp 13y agoNo, I mean ZeroMQ takes care of the underlying queue for free (there is an underlying set of queues, which can be persisted if necessary)
- asksol 13y agoThat must be a very recent development for 0mq then, do you have any source for this new feature?
- batbomb 13y agoNo, it doesn't, although it might be possible to sort of emulate it by setting ZMQ_HWM to 1 and enabling ZMQ_SWAP, but I wouldn't bet on it. The best you can hope is to use the Titanic Service Protocol and just throw data into some sort of disk store. I've looked into doing this, but I settled on using RabbitMQ instead for persistence. Unless you're dealing with more than 10k messages a second, it's just as easy as ZeroMQ. After a few months of experimenting, I've come to the conclusion that some combination of ZeroMQ and RabbitMQ is likely the easiest solution currently for a combination of low-overhead distributed messaging and broker-assisted persistent messaging.
- cschmidt 13y agoAre you thinking of RabbigMQ perhaps?