10 ms·
The Big Little Guide to Message Queues
- rowland66 6y agoI think that you description of exactly once messaging is inaccurate > This is the holy grail of messaging, and also the fountain of a lot of snake-oil. Exactly once message delivery is quite possible with messaging systems that support transactions. When combined with other transactional resources (e.g. database) and a distributed transaction monitor, exactly once messaging works well and is rock solid reliable. The grand-daddy of message brokers IBM MQ is absolutely capable of exactly once messaging.
- sudhirj 6y agoAWS makes the same claim with FIFO SQS, and maybe I’m getting it wrong, but these claims 1) have a lot of caveats and 2) only work inside the boundaries of the messaging system. There’s a note in the next paragraph about how systems manage to say that if you pass in the same message ID / token for X minutes they won’t be duplicated, and my ensuring FIFO there’s a side effect of not giving out the next message until the current one is acknowledged. This leads to a situation where there’s a guarantee of exactly once acknowledgement, but not necessarily exactly-once processing or delivery. Given that the semantics of at-most-once and at-least-once apply to processing and delivery, I personally don’t think the goalposts should move on exactly once. Systems claiming exactly-once lull developers into not planning for multiple deliveries on the subscriber, or the need to do multiple publishes, both of which can still happen.
- vp8989 6y agoIt's better to use a SQS standard queue and have the consuming system provide the exactly-once processing guarantee for various reasons. You will need to introduce something like Redis, if you are not already using it, but I still think it's net superior to using an SQS FIFO queue if you want exactly-once processing.
- sudhirj 6y agoMight not even need to use Redis. If the message has a proper idempotent ID a transactional database is more than enough. If the consumer is running MySQL/Postgres/DynamoDB etc nothing else is needed.
- sagichmal 6y agoEven in the face of network partitions?
- rowland66 6y agoSee my answer to WJW above. Yes, even in the face of network partition, but with system unavailability in the event of a network partition.
- kasey_junk 6y agoIt’s been an age since I’ve worked with IBM MQ and there are dials upon dials when setting up MQ based systems but it doesn’t off exactly once in the face of broker failure in most of its HA configurations and it uses deduplication at the protocol level to prevent duplicates. When people say “exactly once” is impossible they really mean in the face of failure at the queue level.
- tyu2 6y ago> When people say “exactly once” is impossible they really mean in the face of failure at the queue level. And what exactly is impossible with that? Just wait it out, i.e. like all the CP systems do (as per CAP).
- kasey_junk 6y agoThe premise is that unavailability is the same as zero delivered messages, not one. Note none of this is rigorously defined either in the article or with most message queues and the configuration of queues/brokers/clients means that there are all manner of edge cases around delivery guarantees in practice.
- WJW 6y ago"Wait it out" is only a valid strategy when message rates are low enough that you can buffer them all until the network partition goes away again. As an example, imagine a system sending a million 1 KB messages per second. To survive a 1 minute network outage it would need 60 GB extra storage to park the messages. If the outage lasts longer than it has space available, dropping messages becomes inevitable.
- kondro 6y agoNot quite, there are always a bunch of edge cases that inevitably make "exactly once" actually "almost always exactly once, but always at least once."
- WJW 6y agoIndeed. "exactly once" violates the CAP theorem, so if you actually make a system that can guarantee "exactly once" then you should apply for a Turing medal immediately.
- rowland66 6y agoI think that you are misunderstanding the CAP theorem. The CAP theorem states that in the event of a network partition, a system can either be consistent or available, but not both. So a messaging system that provided exactly once message delivery would not provide availability during a network partition. However, there are many applications for which consistency is more important than availability, especially if the period of unavailability can be limited.
- WJW 6y agoAh, but "especially if the period of unavailability can be limited" is exactly the type of edge case kasey_junk was talking about. Network partitions may persist for unbounded amounts of time as far as the CAP theorem is concerned, and an unspecified amount of packets may be dropped and/or delayed. It could be the case that every message you send gets dropped due to a persisting partition and in such a case none would arrive, thereby violating the "guarantee" of exactly-once delivery. In practice I agree that these problems are quite rare since most network are reasonably stable. However, especially at scale it's not rare to see messages dropped or delivered more than once. I have no doubt IBM MQ can achieve exactly-once most of the time, but no distributed system can achieve exactly-once delivery all of the time.
- rowland66 6y ago> It could be the case that every message you send gets dropped due to a persisting partition and in such a case none would arrive, thereby violating the "guarantee" of exactly-once delivery. That is not correct. All interactions between the client and the broker are performed in transactional units. If the transaction in which messages are sent fails to commit, then the messages are not sent, and all work is rolled back. Once a message is successfully send (that is, sent and transaction committed), it will be delivered once and only once to the receiver. Likewise on the receiving side, a message is delivered and the encompassing transaction is committed once and only once. A message may be delivered more than once if the encompassing transaction is later rolled back due to say network failure. But a message delivery in a transaction that does not commit is not a delivery. The benefit here is that application programmers don't need to concern themselves with message duplicate checking and the risk that duplicate checking is done incorrectly leading to bugs that are very difficult to identify.
- throw-away_42 6y agohttps://uxmovement.com/content/how-margins-and-line-lengths-affect-readability/ https://uxmovement.com/content/how-margins-and-line-lengths-...
- Normal_gaussian 6y agoHi, I'd just like to draw your attention to the community guidelines ( https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html ). I hope you'll find them as useful as I have in shaping my interaction with this community. Specifically I'd like to highlight the theme of substantiative comments and the discouragement of commenting on small issues like website formatting.
- sudhirj 6y agoI assume you’re saying the theme doesn’t work for you? Thanks for the feedback, will customise it to something more readable, better colours, nicer fonts etc soon.
- rualca 6y agoOn a side-note, it's customary to disclose the author's connection with his own submission. I understand that it's a bit on the nose to see user account sudhirj submitting a link to sudhirj.io. However, disclosures are also a clear indication on where to post questions and/or remarks directed at the author. Anyway, my 0.02€.
- sudhirj 6y agoWill post a note.
- Normal_gaussian 6y agoEverything worked great for me, its why I was surprised to find a somewhat rude link as the only comment once I'd read the article.
- lordkrandel 6y agoThis article is great! It was long I was looking for a simple article explaining what options are there. I've been developing custom semifunctional queue systems for years, and I always wanted to know why we could not use an existing one. It looks like DIY was the rule back then.
- sudhirj 6y agoThanks. Yeah, DIY was pretty much the norm. Jobs as table rows, in memory arrays, the whole deal. Spent quite a bit of my life writing glue and management code that should have just been queues.
- sudhirj 6y agoAuthor here. This was self-posted, so hope you find it useful. Please ask me anything.
- yogevyuval 6y agoWhat web framework is used for this blog? Really liked it
- sudhirj 6y agoIt’s Ghost (Ghost.org). The theme is Dawn (it’s on the official theme marketplace). My pandemic project is to offer Ghost hosting (https://ghosting.dev https://ghosting.dev) - let me know if you want a free personal setup.
- bambataa 6y agoThank you for putting this together. I am reading it with interest. I spotted one small typo and thought you would like to know: “like transferring information form your software into an email or an SMS on the cellphone network.”
- sudhirj 6y agoThanks, fixing.
- heipei 6y agoThanks for the guide, as others have said this is good introductory reading for new engineers. I know I really only learned about dedicated message queues 1-2 years into my career and felt really dumb for not having known/thought about message queuing before. One thing you might consider adding are more examples for different popular queuing systems and how they differ from one another. The software I always reach for is nsq (https://nsq.io/ https://nsq.io/) because it's meant to run co-located with the message producers and readers are supposed to connect to multiple instances where the messages are produced (using a lookup daemon). This is quite different from the queues on your list, so much so that I'd consider adding it just because it works so differently.
- indymike 6y agoI'm putting this one on my new hire required reading list. So often people end up re-inventing message queues and not even noticing they are doing it.
- sudhirj 6y agoInteresting. This is actually sort of a distillation of all the new hire training / discussions I’ve done. It’s as much a “things I wish someone had told me 15 years ago” thing too.
- brightball 6y agoThis is along the same lines, but something people don’t often think about when using and naming queues. https://www.brightball.com/articles/organizing-background-worker-queues https://www.brightball.com/articles/organizing-background-wo...
- athoscouto 6y agoWould you be willing to share the list with us?
- indymike 6y agoHere it is: https://mikeseidle.com/2021/01/03/my-developer-new-hire-reading-list/ https://mikeseidle.com/2021/01/03/my-developer-new-hire-read...
- indymike 6y agoHmm. I really should find time to get my personal site back online. I took it down a month ago to re-face it and life has intervened.
- loevborg 6y agoYou could just dump it in a gist
- yuppiepuppie 6y ago> people end up re-inventing message queues and not even noticing they are doing it Amazing how people sucked down this rabbit hole. My work is doing this right now, and its driving me mad as there are well supported alternatives out there. But I suppose there is something intrinsically fun about building a complex communications system.
- tyu2 6y ago> Given all the things that can go wrong, it's impossible for any messaging system to guarantee exactly-once delivery. It's possible to guarantee exactly-once delivery, just not in bounded time, but eventually. That's what FLP impossibility is about, impossibility to achieve consensus (e.g. exactly-once) in bounded time. But if you are talking in the context of semantics, it's a totally different thing, it's a computation model and it's absolutely possible to have exactly-once semantics, regardless whether you achieve it by waiting, like consensus-based systems, or by using special data types and operations that can be performed without waiting and eventually converge, like CRDT systems.
- sudhirj 6y agoDon’t quite get the gist of this... a consensus based system like Raft or a CRDT operation will converge on to exactly one state, yes. I haven’t seen these concepts applied to a message queue. One could build a consensus or CRDT system that used a normal at-least-once message broker as a component, for sure. Is that what we’re talking about?
- backing 6y agoAPI vs MQ : if there is an API you don't need MQ, or will you make a front MQ for every API you're calling ? This article is inconsiderate of the added complexity of MQs. How is the payment processor given as example decoupled from the main process from let's say an e-commerce website using a MQ ? If the payment processor isn't responding, the payment process on the main site is broken and a client can't purchase. If many clients try again to pay, the MQ will be flooded and attain its max limit. How do you deal with added complexity and extra error handling of the MQ ? Same question for integrating a new payment processor: with a different API or SDK to integrate, how would that not be the case with a MQ as stated in the article? A MQ is simply a buffer when you need to scale up service consumption. But services with APIs don't need it, they do rate limiting and if they're down you should better set the status on your program and stop communicating until it's up again. MQs are not automatic and this article is selling false promises and examples.
- yuppiepuppie 6y ago> if there is an API you don't need MQ Im not sure what you are talking about here. The two concepts are quite different and are meant to be used for different processes. > selling false promises AFAIK, the article wasnt selling anything. I think it was a nice intro and overview to message queues.
- vorpalhex 6y agoUsing an MQ with a payment processor is very common - payment processors do have transient outages and we still want to allow customers to transact. We may also use multiple payment processors in this case and fallback to a more expensive one if we see our preferred one go offline for a while. MQ's are distinct from APIs. APIs require both sides to be free to talk, while an MQ allows one side to send when it's able and the other to read when it's able which can happen out of sync. MQs do have limits but this is typically several orders of magnitude higher than an API - a few million queued messages is pretty trivial on any decent MQ cluster. An MQ isn't a buffer alone, it's also routing, retries, error handling, deadlettering, etc. This is the difference between your crappy in-memory blind buffer and something like RabbitMQ. Yes MQs still require maintenance/attention/etc, they're not magic.
- 6y ago
- billfruit 6y agoI'm speaking specifically of linux/unix message queues: Richard Stevens book APUE, seems to not like Message Queues, as much as FIFOs, especially IIRC, FIFOs work on file descriptors and therefore works nicely with poll/select and also since destruction of FIFOs is handled more gracefully as the connected programs terminates. Is that view still holds good? I wonder if that view still has currency?
- vorpalhex 6y agoWe normally use message queues when we have a distributed problem involving multiple nodes where you can't typically push a file descriptor. A FIFO is still basically a message queue though it has less semantics than we'd typically see.
- wbl 6y agoAll the System V IPC things are ugly ducklings imho.
- encoderer 6y agoGreat write-up. Back-pressure would be a useful concept to add.
- sudhirj 6y agoThought it was already implicit in the capacity decoupling, could you elaborate a bit more / add an example. I’ve always thought back pressure isn’t applicable to queues because there’s no pressure to begin with, front or otherwise.
- encoderer 6y agoIn most cases, queues are going to be a lot faster than whatever they are feeding into. If you are doing something like sending every SQS message to a Lambda with infinite concurrency then, sure, but many queues are feeding into a compute or persistence layer of some kind. Back-pressure here would mean that emitters of messages into your queue behave differently when there is an excessive number of messages waiting to be processed. For example: - If it's a job queue, load-shed less important jobs until the queue is healthy. - If you are ingesting a high volume of messages, be more aggressive with batching them together before inserting as a single queue message.
- sudhirj 6y agoAh, got it. I’m not entirely sure I want to put that in, though. The message I’m “preaching” is that you have full decoupling with queues - not sure that should be diluted. Wouldn’t this kind of checking defeat the purpose of using the queue in the first place? It seems unlikely that Amazon would stop selling stuff just because the order-confirmation-email-sending-service was backed up, for instance. The point of using SQS would be that the sales system wouldn’t have to care.
- encoderer 6y agoIn the universe of message queues, an email-per-sale queue is not going to be your problem. Think of the queue-as-message-bus pattern -- very common. You don't want to have a service with millions of pending messages to process, it forces you into painful, slow recovery, or making the tough choice of dropping everything. Anyway, great write up, and happy new year!
- drzaiusx11 6y agoGreat article, dead letter queues might be another thing add to later updates. Also Apache Pulsar might be worth mentioning.
- rad_gruchalski 6y agoWorth noting - Apache Kafka is not a queue. It's a distributed commit log.
- waynesonfire 6y agoIs that worth noting??
- rad_gruchalski 6y agoYes. Kafka does not have a concept of a queue. Messages are written to an append only log and the broker does not track the state of the message. The consumer is responsible for tracking the offset. Kafka is not a queue.
- sudhirj 6y agoMight add a note, but I don't want to muddy the waters. Technically, neither is the Redis Streams implementation. But they both quack like a queue, and are meant to be used like a queue, so will probably talk about implementation details in a deep dive of some sort. EventBridge isn't a queue either, actually.
- rad_gruchalski 6y ago> But they both quack like a queue, and are meant to be used like a queue Not necessarily. Nothing stops one from using Kafka topic in a fashion similar to a queue but even then, the way this would have been done differs significantly. Very simplified: in a queue, an available message is picked up by a consumer and must be acked / nacked within a certain period of time. If message is acked, it is considered processed and may be chosen to be removed, depending on the queue semantics. If nacked, it becomes available for reprocessing. Multiple consumers may be processing messages at completely different offsets and a failure of processing at an early offset does not prevent other consumers advancing the queue. In a commit log, if the consumer is not able to process the message, it has to keep retrying that one message until it can be processed OR ignore failure and publish the message back to the log at a new offset for future reprocessing. Regardless of that, any other consumer (or group) has its own view of the world so two consumers (or consumer groups) will process all messages from the topic. The semantics are completely different.
- dastbe 6y agowhile this is a good overview, i think this article paints too rosy a picture of message queues, specifically > We can put as many messages as we want into the tube (let's assume we have a infinitely long tube) at whatever speed is comfortable to us. and > The receiver will never be impacted by our actions—they will pull out as many messages as they want at whatever rate is comfortable to them. and > Neither the sender nor the receiver are concerned with how the other works. First, I would argue that the sender and receiver still absolutely have to know how the other works, because your API contract is now the structure of all messages that enter the queue. You also need to be aware of the semantics of both client and receiver: if I receive a message I can't process, can I get rid of it? Do I need to retry it forever? And as a sender, will my receiver retry this message or do I need to track success? In synchronous systems you have the opportunity provide clearer feedback in these scenarios, which help mitigate or at least share responsibility in mitigating poison pill scenarios. It's also the case that when you introduce a queue you're adding an arbitrary buffer. During short transient outages this is usually ok, as your system has some slack somewhere at some time to drive down the pileup, but during persistent outages you can end up driving up a difficult to overcome buffer. And worse, in the event of time sensitive message a substantial amount of that buffer may be otherwise useless. Unless your receiver knows whether a message can be safely ignored, you can encounter outage scenarios that take hours or even days to recover from and exacerbate impact after recovery compared to if you just had no queue and shed load in the first place.
- sudhirj 6y agoThanks for the points, yeah, definitely need to cover poison pills and dead letter queues. Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload. Will think about how to talk about recovery measures. The handling is very application specific, it seems that guidelines will boil down to 1) if your messages are important, suck it up and add more subscriber processing capacity or 2) if they're not important, purge the queue and restart your subscribers. Not sure I want to get into this kind level of advice without looking at the nature of the messages in the system.
- 6y ago
- cletus 6y agoMessage queues are one of these things that sound great in theory but IME has simply been a way of adding more failure modes and points of failure to chase the (largely unjustified) goal of systems decoupling. There are two types of people in the world: those who divide the world into two kinds of people and those that don't. As one of the former, I like to view MQs as being essentially one of two types: 1. Stateless 2. Stateful Stateless means if there's no one there to handle the message, it is simply lost. The benefit of course is you don't need to consider all the complexity of persisting messages. But a stateless MQ is really just a routing / service discovery architecture. Now that's completely fine to abstract that away but it's not really that exciting. It's still synchronous delivery, essentially. Stateful OTOH means you really can queue things up and deal with them later. Unfortunately you're now dealing with: - Reliability: what if the persist call fails? Do you have enough storage? Can your persistence medium keep up with the volume of reads and writes? - Is delivery guaranteed or is it simply best-effort? This is a big one. Best effort is MUCH less problematic but doesn't tend to be what people want. You see guaranteed delivery in a lot of enterprise MQ software (eg TIBCO certified messaging) where you have things like MQ integration with two-phase commits and the like. 3. What ordering guarantees are there? Guaranteed ordering or best-effort or no promise? Basically you've probably just created another database. Worse, that database may be different to your other databases. Also, is your MQ a FIFO or some form of priority queue? If it's a FIFO, how do you handle throttling and buffering? The post describes the apocryphal MQ as having infinite capacity. No MQ is. IME the supposed benefits have been outweighed by all the negatives.
- kag0 6y ago> Basically you've probably just created another database I think that's a fair assessment of persistent queues, using "database" in the most general sense. > Worse, that database may be different to your other databases. Different isn't inherently worse. It's just a different tool for a different job than a traditional RDBMS. The problem only occurs when we pretend they're the same.
- barrkel 6y agoThe decoupling narrative is oversold for queues. There's essential decoupling and accidental decoupling; decoupling you want, and decoupling which mostly just obscures your business logic. Resilience in the face of failure, where multiple systems are communicating, or there's a lot of long-running work which you want to continue as seamlessly as possible, is the biggest essential decoupling. You externalize transitions in the state machine (edges in the state graph) of the logic as serialized messages, so you can blow away services and bring them back and the global state machine can continue. Scaling from a single consumer to multiple consumers, from multiple CPUs to multiple machines, is mostly essential decoupling. Making decisions about how to parallelize subgraphs of your state machine, removing scaling bottlenecks, is an orthogonal problem to the correctness of the state machine on its own, and representing it as a state machine with messages in queues for edges helps with that orthogonality. You can scale things up without external persistent queues but you'll end up with queues somewhere, even if it's just worker queues for threads. Accidental decoupling is where you have a complex state machine encapsulating a business procedure with multiple steps, and it's coordinated as messages between and actions in multiple services. The business logic might say something like: take order from user; send email notification; complete billing steps; remove stock from inventory system; schedule delivery; dispatch stock; etc. All this logic needs to complete, in sequence, but without higher order workflow systems which encode the state machine, a series of messages and producers and consumers is like so much assembly code hiding the logic. It's easy to end up with the equivalent of COMEFROM code in a message system. https://en.wikipedia.org/wiki/COMEFROM https://en.wikipedia.org/wiki/COMEFROM
- sudhirj 6y agoThis is touched on in the article, but I should probably expand on it a bit more. A queue is a transport tool that allows moving information between the stages of the workflow, but using a queue doesn't mean you have an excellent workflow. The decoupling business advantages come mainly from having the workflow, not necessarily using a queue as a building block for it. When you do use a queue as a building block in a workflow you get some technical and operational advantages, maybe those are oversold. Will think about how to advocate business decoupling using a workflow as a the zeroth step before getting into queues.
- bob1029 6y agoMy guide to message queues: Never use them unless you have no other option. Message queues are an organizational band-aid for lack of architecture and agreeable contracts between systems. RPC is always going to be a more reliable approach. If something goes wrong with a complex system, I would much rather an exception be raised immediately and crater the caller than have a message wind up not being answered silently and having to remember to check other areas of the system after the fact. I have worked in environments that used messaging on a massive scale, and can vividly recall having to write adhoc software to reprocess 3+ gigabytes worth of messages that failed to be picked up by another system. A simple example would be a "Constraint Warning" message that might be produced by some factory tool. Perhaps you want to make sure all lots that tripped that warning are handled in an extra QC phase to double check things. Perhaps this is another system listening for these messages. What happens if that system fails to see these for whatever reason and you dont have a full-time employee monitoring dead letter queues? You might end up with weeks worth of broken products in your logistics chain and millions of dollars in damages before you figure out what went wrong.
- kag0 6y agocounterpoint: you can implement RPC using queues. queues are just one more tool in the toolbox. it's not the screwdriver's fault if it's used to hammer nails.
- sudhirj 6y agoThe example of factory QA is really an OLTP / near real time system, for which queues are definitely a bad idea. Any system where you "would much rather an exception be raised immediately and crater the caller" is by definition an OLTP type system, which is a really bad fit for queues. Lots of bad experiences with queues don't necessarily mean that queues aren't useful tools. They're just tools—they have a time and a place. If you've tried integrating with an email or SMS gateway, or a legacy rate limited bank you might have a different experience. Or dealing with slow job execution combined with spiky high speed job creation. Depends entirely on the problem, and probably the solution team and constraints as well. Don't think it makes sense to dismiss them out of hand for all problems, though—just as it doesn't make sense to claim that they should be used for each and every problem.
- deleted 6y ago[deleted]
- Spearchucker 6y agoSo today, if I want reliable messaging, I use asynchronous, non-blocking functions. Like async/await in .NET. If I want guaranteed delivery I write logic to provide that. Again, using .NET. If in-order delivery is required, again, .NET. Of course .NET can be replaced by anything that supports async, non-blocking functions. If instead I used message queuing instead of writing the code myself I'm adding yet another configuration item to my system, replete with security holes, patches, operational management, and sourcing the skills to procure, install and run it. Of course each has idiosyncracies you need to learn and understand - like Rabbit's memory-based queues. When the instance goes, so potentially does reliable messaging. I'm very much not a fan of message queues. Haven't been for a good many years.
- deleted 6y ago[deleted]
- barnacled 6y agoBefore I got my current job I reached the final round (of 4 or 5!!) at a startup where I was failed because I failed to say 'message queue' or mention RabbitMQ (the interview used to work for them I think) for a question of 'how would you implement whatsapp?' I was told I was a good enough lowly developer but not the right stuff for senior. So perhaps I'm bitter but, while they seem useful for specific purposes, it does feel a little religious and one of these presumed 'must have if we're serious' class of program when other solutions which architecturally resemble them might suit the problem far better. In my current job we interact with kafka and have definitely encountered many less than ideal aspects of its behaviour. But I guess that must be the junior developer talking again ;)
- teyc 6y agoNice write up. There should be a section on MQ gotchas. Here's my war story: Back in the days when we didn't know what we were doing, we had a system where MQ was used in Request/Response style for querying databases. On the beginning of this was a web endpoint. Frequently we end up in a cascading failure whenever we hit an overload situation, because the web client would time out, and retry another call. Meanwhile, there's a bunch of query messages enqueued to be executed against the DB despite the fact the client had already disconnected. If you are trying to web scale, you'd end up paying for IO twice, first to query the DB and second to persist the results in a queue. The big take away for me was to be wary of coupling an unreliable synchronous RPC with a reliable asynchronous MQ.
- baby 6y agoWhen reading the article I was wondering: what if clients just spam the queue (by retrying) because they don’t see any responses coming their way (because subscribers have a long backlog to get through first).