18 ms·
Adventures in message queues
- rdoherty 12y agoThis has me excited for many reasons. Redis is amazingly powerful, robust and reliable piece of technology. Also I love reading antirez's blog posts about the decisions behind Redis so I can't wait to learn more about queueing systems from him when discussing Disque.
- deleted 12y ago[deleted]
- jpfr 12y agoSome big project are currently making the switch to DDS-based pub/sub. [1,2] Now that everybody is making QoS guarantees in pub/sub and message queues, is there a real difference to the 10 year old tech deployed in boats, trains and tanks? [1] http://www.omg.org/spec/DDS/1.2/ http://www.omg.org/spec/DDS/1.2/ [2] http://design.ros2.org/articles/ros_on_dds.html http://design.ros2.org/articles/ros_on_dds.html
- kweinber 12y agoAnyone personally work on a DDS project is in actual production? I have never even seen one although I've worked in some of the industries where it is supposedly a success.
- jacques_chester 12y agoI wish to assure all and sundry that Adrian Colyer is not my secret crime-fighting identity, and vice versa :)
- turingbook 12y agoThanks for your insightful comment at: https://news.ycombinator.com/item?id=8709146 https://news.ycombinator.com/item?id=8709146
- antirez 12y agoSorry Jacques :-( Fixed... and thank you again for your comment.
- jacques_chester 12y agoHey, it gave me chance to crack a lame joke. I love those.
- jtchang 12y agoWhen I first installed Redis years ago I was astounded at how easy it was to get up and running. Compare this to the plethora of message brokers out there: the vast majority you will spend the better half of the day trying to figure out how to configure the damn thing. My overall impressions with message brokers is that RabbitMQ is a pain in the ass to setup, celery is my go to these days with beanstalkd being a close second if I don't want too many of celery's features.
- Gigablah 12y agoIs that so? RabbitMQ on a single node is easy, you can get up and running right after installation with just the defaults. I do agree there's a few gotchas with clustering though, such as firewall rules and inet_dist_listen_min/max, setcookie, etc.
- yen223 12y agoCelery isn't a message broker - they always recommend using Celery as a consumer for a RabbitMQ broker.
- lobster_johnson 12y agoRabbitMQ is ridiculously trivial to set up in a clustered configuration. Even more advanced things like HA policies and user permissions are super easy to do. RabbitMQ's problem is that has no tolerance for network partitions, and also tends to be very buggy in the presence of such.
- cdelsolar 12y agohow to set up rabbitmq: 1) sudo apt-get install rabbitmq 2) there is no step 2
- nemothekid 12y agoWhile I believe the top poster is being hyperbolic, I think you aren't being entirely honest. Looking over my chef script I have - multiple add_user, set_user_tags, add_vhost & set_permission commands. Then theres the application code of creating the queues and adding them to the appropriate hosts. Whats even more annoying is there isn't a simple way to backup/move nodes that doesn't involve either imaging the entire OS, or exporting every message (why can't I just flush and copy a directory?). All of these permissions are needed when all I really need is a default admin given that the node is already behind a firewall and anyone with permission to connect to the node either likely knows what they are doing or is an intruder and I should consider the box hosed either way. Somehow, I found Kafka easier to use than RabbitMQ.
- ChuckMcM 12y agoIn my experience there are three things that will break here; 1) At-most-once is a bridge to an elementary school which has an inter-dimensional connection to a universe filled with pit vipers. Kids will die, and there is nothing you can do to stop it. 2) Messages are removed when acknowledged or memory pressure forces them to be kicked out. Black Perl messages, those that sail in from out of nowhere, and lonely widows (processes that never find out their loved ones are dead) will result. 3) Messages are ordered using wall clock millisecond time. This will leave your messages struggling to find their place in line and messages that should be dead, not be dead (missing fragment problem). Obviously all these are simply probabilistic trade-offs based on most likely scenarios which result in arbitrarily small windows of vulnerability. No window is small enough at scale over time. Often when these things have bitten me it has been non-programming stuff. For example a clock that wouldn't follow NTP because it was too far ahead of what NTP thought the time was, an operator fixed that by turning time back 8 seconds. A client library that was told messages arrive at most one time, and so made a file deletion call on the arrival of a message, a restored node holding that message managed to shoot it out before the operator could tell it that it was coming back from a crash, poof damaged file. And one of my favorites in ordering, a system that rebooted after an initial crash (resetting its sequence count) and getting messages back into flight with the wrong sequence number but with legitimate sequence values. FWIW, these sorts of things are especially challenging for distributed storage systems because files are, at their most abstract, little finite state machines that walk through a very specific sequence of mutations the order of which is critical for correct operation. My advice for folks building such systems are never depend on the 'time', always assume at-least-once, and build in-band error detection and correction to allow for computing the correct result from message stream 'n' where two or more invariants in your message protocol have been violated. Good luck!
- ma2rten 12y agoHow does something like rabbitmq fare here?
- suchire 12y agoApparently with clustered masters, with network partition failover and healing, it by default drops messages, which means it's not even at-least-once. https://aphyr.com/posts/315-call-me-maybe-rabbitmq https://aphyr.com/posts/315-call-me-maybe-rabbitmq Not so great guarantees there.
- caf 12y agoI wonder what the point is in having "best effort FIFO"? If the application has to be able to deal with unordered messages anyway, you might as well not bother to try to maintain any kind of order. It's as well to be hung for a sheep as for a lamb.
- latch 12y agoIt lets the consumer use a fixed buffer to re-introduce ordering (possibly dropping any out of order messages that come outside of a window)
- ekimekim 12y agoIn general, you want to consume messages "fairly" ie. in a way that minimizes latency introduced by the queuing. Best-effort ordering gives you this most of the time, which is better than none of the time.
- antirez 12y agoExactly, imagine you use your message broker in order to coordinate a web app that needs to apply effects to photos in a web interface. It is definitely a good idea that users arriving before are served before, but violating this by a couple of milliseconds is not going to change the experience.
- pixelmonkey 12y agoSeems like a similar design to Apache Kafka, http://kafka.apache.org http://kafka.apache.org. AP, partial ordering (Kafka does ordering within "partitions", but not topics). One difference is that Disque "garbage collects" data once delivery semantics are achieved (client acks) whereas Kafka holds onto all messages within an SLA/TTL, allowing reprocessing. Disque tries to handle at-most-once in the server whereas Kafka leaves it to the client. Will be good to have some fresh ideas in this space, I think. A Redis approach to message queues will be interesting because the speed and client library support is bound to be pretty good.
- latch 12y agoUnodered, in-memory queues shouldn't be anyone's goto solution. I think there's a time and place for these, and having at-least-once delivery is a huge win over just using Redis, so I'm excited. Still, unless you know exactly what you're doing, you should pick a something with strong ordering guarantees and that won't reject messages under memory pressure (although, rejecting new messages under memory pressure is A LOT easier/better to handle than dropping old messages).
- bcg1 12y agoThis looks like a good effort, congratulations. Personally I'm torn on the usefulness of generic brokers for all circumstances... there are obvious advantages, but at the same time every messaging problem scales and evolves differently so a broker can quickly become just one more tail trying to wag the dog. I am also interested in the architecture of tools like ZeroMQ and nanomsg, where they provide messaging "primitives" and patterns that can easily be used to compose larger systems, including having central brokers if that floats your boat.
- deleted 12y ago[deleted]
- acolyer 12y agoCredit for the questions is due to jacques_chester, not me! See https://news.ycombinator.com/item?id=8709146 https://news.ycombinator.com/item?id=8709146
- isb 12y agoThis looks very cool. At-least once semantics are the way to go because most tasks require idempotence anyway and that helps in dealing with multiple delivery. Strict FIFO ordering is not always needed either as long as you avoid starvation - most of the time you need "reliable" deferred execution ("durable threads"). I started prototyping something along these lines on top of riak (it is incomplete - missing leases etc but that should be straightforward to add): https://github.com/isbo/carousel https://github.com/isbo/carousel It is a durable and loosely FIFO queue. It is AP because of Riak+CRDTs. It is a proof of concept - would be nice to build it on top of riak_core instead of as a client library.
- sylvinus 12y agoFYI, Salvatore will speak at dotScale in Paris about Disque on June 8: http://dotscale.io http://dotscale.io
- andrewstuart 12y agoI didn't see any mention of dead letter queues. Does it support dead letters? This is an extremely useful feature of Amazon SQS.
- andrewstuart 12y agoIt would be nice to have a message queue system not built in Erlang or Java.
- bcg1 12y agohttp://zeromq.org/ http://zeromq.org/
- jsmthrowaway 12y agoDespite its name, ZeroMQ is an alternative sockets API and nothing, I repeat nothing, more.
- lobster_johnson 12y agoZeroMQ is not a message queue. It's in the name: It has zero message queue capabilities. It does implement socket primitives you can use to build a queue broker, but afaik no one has.
- tk120404 12y agohttps://github.com/prdn/pigato https://github.com/prdn/pigato
- bcg1 12y agohttps://github.com/zeromq/malamute https://github.com/zeromq/malamute
- lobster_johnson 12y agoThanks. Looks unfinished. No persistence yet, according to the author.
- X-Istence 12y agohttp://nats.io http://nats.io
- rb12345 12y ago
- arunoda 12y agoI think this has a lot of roots from NSQ. But, NSQ has no replication support. I think built in replication is very nice to have. Would like to try once this arrives.
- politician 12y agoNSQ writes messages sent to a topic onto all channels subscribing to the topic which can be used as a form of replication in a way that meshes well with its at-least-once semantics.
- arunoda 12y agoBut, that all happens inside one NSQ. I mean what if that NSQ server goes down. We don't have that kind of guarantee in NSQ. To get rid of that, we need to maintain the replication our own by publishing the message to two different NSQ servers.
- politician 12y agoGood points.
- mnutt 12y agoTrue, but with NSQ being brokerless, the lines are somewhat blurred between client and server. Each of our app servers has its own local nsqd. If that nsqd stops responding, we take the app server out of the load balancer. The local nsqd publishes to multiple channels, which get consumed by hosts in different datacenters. There are still potential failure cases: nsqd loses connection to consumers, messages start building up, then the machine somehow goes away. The only way to prevent it is to take the machine out of the load balancer any time nsqd messages start backing up, but we prioritize serving requests over sending messages.
- arunoda 12y agoYes. I didn't want to rant our NSQ. This is the reason we choose NSQ because of it's simplicity. We maintain NSQ close to the consumer. We don't our publisher to take responsibility to the message processing. Once it's push to the queue, we need to make sure it's getting process anyway. That's why we publish same message to multiple queues. All our DB operations are idempotent. So, we are okay with processing the same message multiple times.
- turingbook 12y ago>a few months ago I saw a comment in Hacker News, written by Adrian Colyer...was commenting how different messaging systems have very different set of features, properties, and without the details it is almost impossible to evaluate the different choices, and to evaluate if one is faster than the other because it has a better implementation, or simple offers a lot less guarantees. So he wrote a set of questions one should ask when evaluating a messaging system. I can not find the comment by @acolyer on HN. Who can help me?
- discardorama 12y agoI think Salvatore mis-remembered. The comment was by @jacques_chester (he made a joke about it below, but people didn't get it). The comment is in this thread (HN doesn't let me link directly): https://news.ycombinator.com/item?id=8708921 https://news.ycombinator.com/item?id=8708921 . Look for Jacque's comment in there, and Salvatore replied to it. I also wanted to find the original, and used Jacque's remark below as a starting point to start hunting for it.
- turingbook 12y agoThanks!
- antirez 12y agoSorry Jacques, Adrian. This great (IMHO) set of questions was written by Jacques indeed.
- JohnLen 12y agoHow does zero MQ stand up
- lobster_johnson 12y agoZeroMQ is not a message queue, nor is it a broker. You could build a message queue with it, but it's not really a valid comparison.
- JohnLen 12y agoI wasn't aware it's not a broker. Thanks for clearing that up.
- stingraycharles 12y agoIt's fairly trivial to build an in-memory broker with zmq, though, but you should rather consider zmq as building blocks for a message queue, or sockets on steroids. It fills a need for certain applications, and it is very good at what it does.
- X-Istence 12y agoThis reminds me of a talk at SCALE13x about NATS: http://nats.io http://nats.io It's fast and scaleable.
- stevewilhelm 12y agoThis might be why they sound similar: http://blogs.vmware.com/tribalknowledge/2010/03/vmware-hires-key-developer-for-redis.html http://blogs.vmware.com/tribalknowledge/2010/03/vmware-hires... and https://github.com/derekcollison?tab=repositories https://github.com/derekcollison?tab=repositories
- Lx1oG-AWb6h_ZG0 12y agoWill there be any way to set up machine affinity? I think Azure Service Bus uses this mechanism (by specifying a partition key for a message) to enable strict FIFO for a given partition.
- mappu 12y agoAsk HN: I'm in the market for a distributed message queue, for scheduling background tasks - Does anything support "regional" priorities, where jobs are popped based on a combination of age + geographic/latency factors? Also, what are recommended solutions for distributing job injection? My job injection is basically solely dependent on time, and so i envisage one node (raft consensus?) determining all jobs to inject into the queue. My queue volume is about 50 items/sec and nodes will be up to 400ms apart.
- Rapzid 12y agoThat sounds like a job for a scheduler, in the cluster sense. I'm actually looking around for a job framework in .NET that supports custom schedulers, but have yet to find something that supports resource constrained scheduling. It's all either about en-queueing something to be done NOW, or at a future date. I haven't seen anything that supports custom scheduler implementations on a per-job type basis. They don't really distinguish between logging work to be done and deciding whether or not it can be executed NOW.
- andrea_s 12y agoMaybe I'm missing something, but if it is important to guarantee that a certain message will be dispatched and processed by a worker, why wouldn't a RDBMS with appropriate transactional logic be the best solution?
- Spearchucker 12y agoIt would. There's an argument that something like only once, guaranteed and in-order delivery are business requirements which therefore have no place in the message layer - http://www.infoq.com/articles/no-reliable-messaging http://www.infoq.com/articles/no-reliable-messaging.
- aaa667 12y agoIs guaranteed at-most-once delivery impossible?
- antirez 12y agoExctly-once is impossible. Example, you build all the system to be totally consistent and transactional (a CP system basically). Then you deliver the message to the client, and it dies without to report you if the message was acknowledged or not. You have two options: 1. Re-issue the message. If you do that, it is possible that the now crashed client already processed it, and you end with multiple delivery. 2. Drop the message. If you do that, the client maybe did not processed the message, and you end with no delivery.
- deleted 12y ago[deleted]
- avar 12y agoExactly-once is possible in practice if you effectively bring the client under the umbrella of your transactions. E.g. you can (ab)use MySQL and other RDBMs to do something like: 1. Have a table where each row is a "job" or "message" and has a status which when enqueued is status = "unclaimed". You can also have a `last_changed` column. 2. Have workers consuming that table that GET_LOCK() a row and set status = "processing" and hold the lock for the duration of the processing. 3. When they're finished with the task they update status = "finished" and unlock the row (or equivalently, disconnect). This requires much tighter coupling between the queue and the queue consumer (each client must maintain a lock / connection for the duration of processing items). But it means that: * Nothing will ever pick up the item more than once due to the GET_LOCK(). * If the consumer dies the item is either just unlocked, or the status is "processing". You can as a matter of policy re-pick up "processing" with a `last_changed` in the distant past, alert on those items and manually inspect them. * If the consumer processes the item successfully it'll set the status to "finished" and nothing will process the item again. Now obviously this requires a lot more overhead & maintenance than what you have in mind, in particular it makes some of the failure cases be "the item is delayed due to a consumer dying and won't be processed until a human looks at whether it was actually finished". But this is the sort of pattern that you might use e.g. if you have a queue for sending out invoices. At work we use MySQL + https://metacpan.org/pod/Data::Consumer::MySQL https://metacpan.org/pod/Data::Consumer::MySQL to do this.
- antirez 12y agoI'm very sorry, credits for the questions goes to Jacques Chester, see https://news.ycombinator.com/item?id=8709146 https://news.ycombinator.com/item?id=8709146 I made an error cut&pasting the wrong name of Adrian (Hi Adrian, sorry for misquoting you!). Never blog and go to bed I guess, your post may magically be top news on HN...
- jraedisch 12y agoWe recently switched from RabbitMQ to Redis queuing because we were not able to implement a well enough priority queue with highly irregular workloads. Prefetch would not work since 2 minute workloads would block all following messages. Timeout queues would somewhat rebalance msgs, but large blocks of messages would be queued at the same time and therefor be processed as large blocks. Now our workers are listening to 10 queues/lists with different priorities with BRPOP and so far everything seems to work.