4 ms·
Choosing a message queue can certainly be tricky. Not a complete list, but you should consider: a.) do you need persistence? some queues offer this, some do
by spotman 12y ago
Choosing a message queue can certainly be tricky.
Not a complete list, but you should consider:
a.) do you need persistence? some queues offer this, some do not. many systems can be configured in either mode. you might need persistence if the messages your processing are vital and shouldn't be missed in the case of a broker restart, for example. however if the messages are very volatile, then maybe persistence is overkill. Or possibly, the architecture your working with handles message loss somewhere else, so its not the message queues responsibility to worry about loss.
b.) do you need fault tolerance? some queues off this, some do not. just like persistence, its often something you can enable or disable. and like persistence, often queues can be setup so that while they might not be fault tolerant, the application that employs them is designed to work around queues going down.
c.) do you need high availability? fault tolerance and high availability are separate things. fault tolerance might just be a master falling back to a slave. high availability means this happening and there being no hiccup in service.
d.) performance. its something to consider, but your correct in thinking that this is not everything. but sometimes its worth considering. in one experiment I ran, Amazon SQS was about 40x slower than rabbitmq. This may or may not be worth the trade off for you, as products like SQS relieve the developer and operations a lot of responsibility, but if 10ms average queue time (or less) is important, forget SQS.
e.) client libraries. some queues have client libraries for every language in the sun. some only offer support for one. you would want to consider who and what will need to talk to the message queue, and if this is going to be made a nightmare if your using something very exotic or custom.
f.) creature comforts. some things come with nice rest api's, some come with web ui's, some come with only command line interfaces, and some come with none of that. Often things like ZeroMQ, Nanomsg, are put into these blogs and things comparing message queues. They really are not message queues in the traditional sense. They can only replace one after you've probably spent a considerable amount of time building tooling. For example, in zeromq there is no way to ask 'how many messages are pending in this queue' without writing that functionality yourself.
g.) queues-as-a-service. I mentioned Amazon SQS in point d, but they are something to consider if you don't want to run it yourself. Running a message queue can be easy, but running a message queue that is fault tolerant, highly available, deals with message loss/deliverability, and scales as your workload scales, can be an art form.
In summary, you should do your own research based on these needs and others I likely forgot. Sometimes its easiest to just pick a simple one, and then once you experience its shortcomings and learn more about your queue needs, the upgrade path will be much less mysterious.
- porker 12y agoThank you, that's a really great and helpful answer! And finally clarified the difference between ZeroMQ and the rest, which I've seen hinted at but never explained.
- walterbell 12y agoThanks for the detailed answer. On a tangential note, do you have an opinion on http://triceps.sourceforge.net/ http://triceps.sourceforge.net/ as a CEP engine, or recommendation for alternatives?
- hkarthik 12y agoExcellent response. Only thing I would add is this: h) compatibility. Make sure (and test!) the compatibility of the queue with your application stack. Even though a queue may advertise support for a certain stack, the libraries will range in quality and every queue has a "first class citizen" in terms of supported stack. If you're going to take a chance on an off-the-beaten path combination of application stack and queue, accept that you may need to extend or even fork the client library to make sure it fits your needs. Queues vary in complexity, and often a lot of complexity gets pushed to the client.