11 ms·
How far can you go without a message queue?
- PaulHoule 4y agoI've never found message queues that hard to work with but it seems everyone else I've worked with inevitably winds up with a queue that messages go into and never get out of. I've often thought there is an analogy with broken "real-world" processes where the same thing happens.
- XorNot 4y agoThe problem is people leap to message queuing to make things fast, but don't build their recovery system first...but the recovery system is exactly what they need to build first. Whatever you're doing with queuing, you need to have a fallback so your service can "catch up" by pulling down what it thinks it missed. And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up. All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).
- trombone5000 4y ago> ...but the recovery system is exactly what they need to build first. I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.
- projektfu 4y agoMessage cues with ADHD like me. Add to inbox. Inbox full. Declare bankruptcy. Repeat.
- slowmotiony 4y agoWe use IBM MQ at work and I wouldn't exactly call it simple. There's sender channels, receiver channels, xmit queues, local queues, remote queues, dead letter queues, topics, subscriptions and all kinds of other things to learn about. Add that to the crappy IBM tools and their documentation that is full of dead links and you're in for a lot of learning through trial-and-error.
- jsymolon 4y ago> We use IBM MQ at work and I wouldn't exactly call it simple. 2nd'd & 3rd'd ... You hired a 50 man lumberjack crew to cut down a sapling. I've been spending a some time to remove IBM MQ out and going to "simpler" queuing solutions. First of all, cost. Guess what, we're not getting anything out of licensing per year to justify the cost. With improving the end points, we don't need all that complexity. With 5 9's uptime on the network and smarter end points, retrying messages isn't expensive. As always, YMMV.
- commandlinefan 4y agoI suffered under MQ for many years. I never understood what any of it was for, other than the actual queue part (which is what you can get for free from competing products like RabbitMQ, ActiveMQ, Kafka, etc.). And it wasn't because I didn't try to find some documentation...
- pearjuice 4y agoThat's because IBM is in the business of selling complex stuff and promising to make it simple with support contracts.
- kitd 4y agoIBM MQ is the granddaddy of queuing systems and predates most other queuing systems by decades (IIRC it is (ultimately) based on the iSeries/OS400 queuing system). Consequently, it carries a lot of baggage from that time which large ops teams like for tuning to their needs, but which modern systems have largely ignored or modern environments have made redundant.
- davewritescode 4y agoMost of the problems with message queues are from the fact that you will eventually get messages that you can't process now but might be able to later. Depending on your requirements you either stop the state of the whole system or use something like dead-lettering and/or delay-queues + retry count headers to make sure that you can reprocess failed messages without letting them accumulate until you're wasting CPU processing messages that will never work. The other problems with message queues is that you need to make sure you have alerting on all your queues otherwise you're just waiting for trouble.
- Thaxll 4y agoOne of the great thing about message queue is that it allows you to upgrade/update things downstream without loosing information. Work will be delayed but it will be done at some point.
- Spivak 4y agoAnd issues manifest as queue backups which a can turn what would otherwise be downtime into “the app is a little slow today.” Would never ever architect an app in production without leaning heavily on queues because it makes so many of the hard parts trivial.
- zihotki 4y agoThat's highly underestimated benefit of message queues. In addition to that it's also easy to mirror the queue and test new version in parallel with current.
- egberts1 4y agoZeroMQ; don’t leave home without it. Disclaimer: not affiliated with ZeroMQ, just a heavy user of BSON and ProtoBuf that goes with ZeroMQ. https://zeromq.org/get-started/ https://zeromq.org/get-started/
- tomnipotent 4y agoZeroMQ is not a messaging queue, but a messaging network library for broker-less communication.
- egberts1 4y agoZeroMQ can perform as a messaging queue just as well in Pub/Sub. In fact, ZeroMQ library (libzmq) supports over 13 different methods of queuing. https://stackoverflow.com/questions/50656232/guaranteed-delivery-messaging-should-i-use-mqtt-or-zeromq https://stackoverflow.com/questions/50656232/guaranteed-deli... https://zguide.zeromq.org/ https://zguide.zeromq.org/
- tomnipotent 4y agoMost people when talking about a message queue mean something with a centralized broker. ZeroMQ has no broker (hence the Zero in the name), but maintains local non-durable queues between participating devices. When all is said-and-done it's similar behavior, but implemented very differently.
- treis 4y agoThis doesn't actually tell you how far you can go without a message queue. Unsurprising because they're trying to sell one. But you can go very far without them by using the DB or redis as your message queue. Probably fine for 99%+ of applications. Don't add a message queue until you really need one. And if you do make sure you account for it being down, running locally, back pressure if the queue gets full, monitoring, and logging at least.
- worldsayshi 4y ago"But you can go very far without them by using the DB or redis as your message queue." So, you can go very far without a message queue... by using a simpler message queue?
- mhuffman 4y agoI mean, at some point you have to consider something a "proper" message queue. Otherwise, at the lower extreme your web server would act as a message queue.
- shapefrog 4y ago"Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker" Lets draw the line somewhere the other side of applications that advertise themselves as having that function?
- Spivak 4y agoThe number of people that think that Redis is a KV store and nothing else is too damn high.
- skyde 4y agoRedis is not a database. it doesn’t have proper transaction support. it’s basically a cache with a fancy API so you can incrementally update data structure instead of having to constantly write the whole thing you are trying to cache.
- debacle 4y agoAs far as message queues go, I'd be hard pressed to choose anything but rabbitmq. Easy to install, sane config out of the box, developer friendly definition, relatively flat learning curve with highly googleable answers, and widespread support for AMQP. There may be something faster/leaner or with some bells and whistles, but if you just need message queuing it's hard to beat.
- worldsayshi 4y agoI find rabbitmq easy-ish to use but I've been a bit surprised by the seeming lack of tools to support its use. For example, I have not been able to find an off the shelf solution for getting an overview visualization of the message flow. There are a few projects on github but they look like weekend experiments. Nothing that looks maintained.
- lsowen 4y agoThe management plugin [1] might be useful? It allows browsing of queues and clients, and includes statics and graphs for message throughput, etc 1. https://www.rabbitmq.com/management.html https://www.rabbitmq.com/management.html
- worldsayshi 4y agoYes it is very useful. Although it's a bit limited for getting an zoomed out view. It would be nice to have a flow chart. Perhaps I'm colored by playing Factorio like games. It feels like managing a message queue is quite similar to managing an abstract factory line. Would be nice if I could have a similar visualization.
- nvarsj 4y agoI think Redis Pub/Sub is up there too. Plus you get an amazing KV store.
- perlgeek 4y agoRabbitMQ as a stand-alone server is awesome. Getting RabbitMQ to work in a cluster is a nightmare. Be aware of that when selecting it as your message queue :-)
- xcskier56 4y agoI’m know there are other more complex applications of a message queue, but I really appreciate the fact that Rails has simplified this very common functionality so much that it’s not a “message queue” but simply background job processing. From a conceptual standpoint it makes it much more approachable for new devs. They have thought through and provide sane retry and dead letter defaults and can use many different backend solutions. It makes having this functionality out of the box dead simple and a no brainer
- teeray 4y agoHow do you deal with N-stage tasks in one of these message queues where you have to wind back the side-effects if something fails before the whole thing can be “done”?
- roomey 4y agoI'm not the op, but in my little app that uses rabbitmq, you have to never assume timely delivery and processing of messages. I use redis to keep a store of last updated timestamp property (that is, the last time the data was updated), and I essentially throw out a message if the timestamp is older. For any writing changes I make I have a validation loop in the code path as a secondary validation. I think once you get used to been defense about incoming messages you can handle most delays
- andrewingram 4y agoThe saga pattern, or use something like Temporal.
- danielovichdk 4y agoHow did the author create those really nice textbook drawings?
- glommer 4y agowith money! (We work with a design firm that provides illustrations for the blog)
- nesarkvechnep 4y agoWith Elixir or Erlang you can go pretty far without a message queue.
- glommer 4y agoErlang is dope!
- eatonphil 4y agoThis is a fine article but I find the jump from > This works reasonably well. The polling period of reading the events may become a problem, but you can make it better by using database triggers. to > But when we look at tailored backends in the open, especially in architectures for FAANG, MAMAA, or whatever the acronym is until the next time Zuck decides to pivot Facebook, things look different. unsatisfying. Somewhere I'd love the walkthrough of in what scenarios exactly does the database-as-event-store+ad-hoc-worker architecture fall over rather than immediately jumping to what "the pros" do. That's more of an appeal to authority rather than something I can learn from. Not saying you need to do this in your article but if not at least a link to read more would be welcome. :)
- glommer 4y agoThat's a really good point. I thought of that myself to be honest but the article was becoming too long and decided to cap. Great point about the link, that's great feedback!
- vpfaulkner 4y agoI've had a good experience using AWS SQS. It's fairly straightforward and can be paired with SNS if more complex workflows are necessary. I recently migrated to SQS from Rabbit MQ since we didn't need all of the features of Rabbit MQ and wanted something simpler.
- ingonealan3 4y agoAm I missing something about modern backend development, or does this article assume that all backends are built using a microservice architecture (and this implies some form of inter-service communication, for which message queues are a good fit)? To put it in another way: if I were to build an (fairly simple, monolithic) API, would I automatically benefit from using a queue? If yes, how exactly would a queue be used, and in what way would it bring a benefit?
- perlgeek 4y ago> if I were to build an (fairly simple, monolithic) API, would I automatically benefit from using a queue? Queues basically serve as a decoupling layer. If you need one, you'd benefit from one. Example one: you put incoming requests into a queue. If latency isn't a big concern, that means that your service can go down / be restarted for a moment while the queue buffers all the incoming requests. Example two: you write a HTTP API for an interactive web application. Fast response times are important, so you do only what you absolutely must inside the request handler, and everything else that can be done later (like sending mails) is put into a queue, and can be dealt with later either by another component, or by your service running "queue worker" mode instead of in "http responder" mode. If your services is just reading data from a DB and handing it back to the user, you usually don't benefit from a queue. But most apps tend to become more complicated with time...
- thinkingkong 4y agoA monolithic application can still have queues; there's no need for them to be coupled with a microservices architecture. They at least afford you a 'fire and forget' decoupling in parts of your code that might be nice. For example, take this action and fire this event. Then in another part of your application you can subscribe to events. They aren't _queues_ per say but they have a lot of the same DX. Depending on your runtime environment as well, you might benefit from splitting side-effect jobs from your API request/response cycle. That way you can still respond quickly while enqueueing work (locally) all still within a monolith.
- pigcat 4y agoI found the title a bit click-baity, I would have loved to learn how far I can go without a message queue. The jump from "saving events to be processed works pretty well" to "you should use message queue" wasn't explained at all.
- hcarvalhoalves 4y agoA queue is a pretty fundamental abstraction alongside heaps, files, etc. if you think about it. It sucks that it falls in that grey area where it’s badly supported both by the OS and most languages/runtimes, and you have to setup and manage a separate system to have one. I feel it should be as easy or well supported as sockets.
- effnorwood 4y ago[dead]
- LittlePeter 4y agoOff-topic: in the order book updates example, the top of the book is updated using one field per message: echo '{ "symbol": "AAPL", "bidPrice": 174.1 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidVolume": 1000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askPrice": 174.2 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askVolume": 2000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidPrice": 174.0 }' | rpk topic produce book-updates However some top of the book update can result in an entire level disappearing. In that case, both price and volume changed actually at the same time, yet according to this example it will be broken down into two messages? This will lead to inconsistent state of the book when I read it between two messages. Or did the example just happen to show one field per message, and multiple fields per message are possible too?
- dangerface 4y ago> So, if they make so much sense, why are queues left out of the first implementation Sorry what? How do they make so much sense? The article gives a full backstory and diagrams for event sourcing pattern it says every one knows but has no description as to why a message queue makes more sense than event sourcing? I am so fed up of reading half way through an article just for the author to throw away their attempt at being informative and go straight into a sales pitch. I think I am just going to say event sourcing makes so much sense and call it a day, I clearly don't need their product.