4 ms·
I see lots of questions here about which queuing implementations one ought to consider. Some things to think about: 1. Do you need transactionality? What if
by ShabbyDoo 15y ago
I see lots of questions here about which queuing implementations one ought to consider. Some things to think about:
1. Do you need transactionality? What if your consumer experiences an error after it takes a message off the queue but before it finishes up processing it? Is it ok for that message to disappear forever? The common use case to consider is a consumer which takes a message from a queue, does some processing, and updates a database. To avoid distributed transactions, people often write these consumers so that they check first to see if a message has already been processed. This way, one simply ensures that his DB transaction commits before committing to the consumption of the message.
2. What level of message durability is required? Is it ok for all enqueued messages to disappear when a power cord is inadvertently pulled? I used ActiveMQ to populate a user behavior data warehouse. As users did stuff on a website, the application servers would enqueue observations (went to the product detail page, etc.) and these observations would eventually wind-up in a big Oracle DB. A delay of a day or so didn't matter, but we didn't want to lose more than a fraction of a percent of our observations. So, the queue was in effect a temporary system of record, and we had to allow for reboots, power outages, etc.
3. High Availability. Many modern queueing implementations can be deployed in redundant, scalable "meshes", ActiveMQ included. I haven't kept up on the feature sets of RabbitMQ, JBoss's offering, etc. or I would comment more here.
There are obviously a bunch of other considerations, but these are often the ones which are ignored when comparing ZeroMQ to ActiveMQ, etc. There's no right or wrong answer. "Lighter" implementations are often appropriate for messages where, if the shit hits the fan with your environment, the messages are useless anyway. IIRC, eBay builds pages by firing off a bunch of async requests for page parts, waits a maximum of N milliseconds, and then renders the page based on which pieces made it back on time. There's no value in persistence or transactionality, but the mesh sure better scale up.