3 ms·
My guide to message queues: Never use them unless you have no other option. Message queues are an organizational band-aid for lack of architecture and agreeabl
by bob1029 6y ago
My guide to message queues: Never use them unless you have no other option.
Message queues are an organizational band-aid for lack of architecture and agreeable contracts between systems. RPC is always going to be a more reliable approach.
If something goes wrong with a complex system, I would much rather an exception be raised immediately and crater the caller than have a message wind up not being answered silently and having to remember to check other areas of the system after the fact.
I have worked in environments that used messaging on a massive scale, and can vividly recall having to write adhoc software to reprocess 3+ gigabytes worth of messages that failed to be picked up by another system.
A simple example would be a "Constraint Warning" message that might be produced by some factory tool. Perhaps you want to make sure all lots that tripped that warning are handled in an extra QC phase to double check things. Perhaps this is another system listening for these messages. What happens if that system fails to see these for whatever reason and you dont have a full-time employee monitoring dead letter queues? You might end up with weeks worth of broken products in your logistics chain and millions of dollars in damages before you figure out what went wrong.
- kag0 6y agocounterpoint: you can implement RPC using queues. queues are just one more tool in the toolbox. it's not the screwdriver's fault if it's used to hammer nails.
- sudhirj 6y agoThe example of factory QA is really an OLTP / near real time system, for which queues are definitely a bad idea. Any system where you "would much rather an exception be raised immediately and crater the caller" is by definition an OLTP type system, which is a really bad fit for queues. Lots of bad experiences with queues don't necessarily mean that queues aren't useful tools. They're just tools—they have a time and a place. If you've tried integrating with an email or SMS gateway, or a legacy rate limited bank you might have a different experience. Or dealing with slow job execution combined with spiky high speed job creation. Depends entirely on the problem, and probably the solution team and constraints as well. Don't think it makes sense to dismiss them out of hand for all problems, though—just as it doesn't make sense to claim that they should be used for each and every problem.
- bob1029 6y agoThere is some nuance here. My contention is with queues as a means to communicate between systems. Queues within systems are totally reasonable. For instance - You have 2 services: Bank Core Integration and Teller. The Bank Core Integration service contains the queue of items that need to ultimately be pushed to the underlying legacy, rate-limited system. The teller service operates in RPC terms with the bank core integration service, so callers into the problem area are isolated from the semantics. This might sound like a subtle difference, but it is huge in practice. If there is some problem with that queue, it is entirely contained to that one service and can be dealt with in isolation. Logs from that one service should comprehensively document the concern. If there was a message bus between these systems, you would have to go back and review message logs to see if things got missed between systems.
- sudhirj 6y agoMakes sense, yeah. If a system architecture demands that failures be cascaded downstream to all dependencies, then queues should not be used. There are definitely architectures for which this is a reasonable demand.
- Droobfest 6y agoThis eventually becomes a microservices vs monolith discussion though. Implementing both the message queue and the bank core integration service within 1 service makes for less communication/network overhead and is easier to debug, which is definitely valuable. Seperating them out means you can use off the shelf products for the message queue and is easier to scale, which can also be valuable.
- freewilly1040 6y agoThis is severely overgeneralized advice. Some traffic needs to be processed right now, and if it can't be processed within a narrow timeframe a failure needs to be acknowledged and handled. Other traffic can wait an hour, but needs to eventually succeed. Why conflate the two? > What happens if that system fails to see these for whatever reason and you dont have a full-time employee monitoring dead letter queues? "What if you fail to monitor correctly" is a bad scenario no matter the architecture, but doesn't critique message queues. Any sensible message queue will have lag and backlog metrics on subscriptions, which you can monitor just as easily as an RPC end point.
- staticassertion 6y ago> RPC is always going to be a more reliable approach. Seems like an odd statement. RPC seems considerably less reliable to me since it ties your business logic services states' together. > I would much rather an exception be raised immediately and crater the caller Why? The caller might not care. And what if you have a large graph of callers? You're going to end up with a cascade of failures, when really none of them are going to do anything except retry or drop - something that the failing service is capable of making the call to do in most cases. > I have worked in environments that used messaging on a massive scale, Same, and in my experience it was the RPCs that caused the most trouble. I don't think anecdotes are going to be helpful here. > Perhaps you want to make sure all lots that tripped that warning are handled If you want to make sure of something, and then you don't make sure of it, I don't see how that's the fault of a queue. Just... check?