9 ms·
Introduction to Event-Driven Architectures with RabbitMQ
- Kiro 7y agoOT but anyone else hate debugging architectures like this compared to a monolith where you can just follow one big stack trace from start to end?
- afrodc_ 7y agoFrom my experience, debugging monoliths is much easier. Microservices a bit hard (mostly networking). Event Driven Architecture with microservices is even harder. My company's brief stint into building one out turned into a disaster of trying to isolate wtf was going on when there was some issue. Now that could have been the engineers/architects fault, but unless my team grows, I would hate having to deal with it again.
- BubRoss 7y agoI think it depends on the isolation of each part. If the incoming data is fine, then you have the problem narrowed down already. If it isn't, then you have to figure out why it was sent in the first place, probably by detecting something wrong in the sender at run time so you can see what led to the bad message.
- bigredhdl 7y agoThat is definitely the balancing act with events. They are great in certain spots, but can become a big pain if they are used for everything.
- awinder 7y agoI don't hate debugging in evented/microservice codebases, but it is more complicated to get a good debugging setup in place w/ distributed tracing. But after you do that you're often in a better place with logging, so, it's tradeoffs. It's all tradeoffs in fact, I'll take a monolith where it's appropriate for the space and I'll take evented/microservice architectures when the monolith pattern is no longer a good investment for the problem space. All these things exist for good reasons, don't be an emotional investor [of your own time/opinions] ;-).
- diminoten 7y agoNever been a problem for me, and I absolutely hate the maintenance nightmare that comes with one big repo. I feel like the more shortsighted/incentivized by sheer work volume a person is, the more they're into monorepos...
- mypalmike 7y agomonolith != monorepo
- british_india 7y agoNothing is worse than trying to debug Camel.
- callmeal 7y agoNot really. It's sometimes it's easier than debugging monoliths - just log the incoming message, log outgoing message. And you can narrow down the problem just as quickly as in a monolith.
- vemv 7y agoFair point, but also incomplete. Can you debug a forest by inspecting a single tree?
- vardump 7y agoSometimes when you inspect logs from a single component it feels like trying to solve a problem with the forest by inspecting all the ants in it. Lots of mostly irrelevant small detail without any way to place it in the big picture.
- nonconvergent 7y agowhich is why you want more tracing and less logging. log messages are always an art and I myself am a poor artist when it come to predicting what is both necessary and useful enough to log without overall increasing the log noise. Tracing will show you the distributed behavior, but the best log, at most, can only show you the current state of information on a single service on a single node even when you aggregate. Tracing also works hand in hand because the correlation id should associate with the relevant log message when you do decide to drill down.
- vemv 7y agoI would indicate that engineers' feelings ("hating" something) are quite orthogonal to business needs or technical challenges. Probably every technical decision maker pushing for microservices knows the perils of distributed debugging. They have some weight - just some.
- d_watt 7y ago> Probably every technical decision maker pushing for microservices knows the perils of distributed debugging That doesn't line up with my experience. I think humans have a pretty strong tendency to take for granted the advantages of a current situation when looking at something that solves a very specific new problem. "The grass is always greener" "Hating" something of course doesn't matter, what does matter is response time in being able to debug production systems, and developer efficiency, and satisfaction / employee retention. So it's not orthogonal, imo, as it's very likely correlated with an efficiency loss.
- sk5t 7y agoRight on. It takes quite a bit of experience to get to the point where you can sort of look into the future to think about what living with the new system will be like, once its warts and shortcomings are known. The time-sucking problems usually won't show up during a PoC where everything looks beautiful and easy.
- nicolasjudalet 7y agoMy experience with debugging with RabbitMQ was not bad, but we had to do a bit of work to ease the process. For example, we configured error queues where processors publish error log and input message information, which is enough to reproduce the bug and understand what happened.
- a-priori 7y agoThis isn't a sign that one architecture is better than the other. It only indicates that the tools are more mature for debugging monoliths than distributed architectures (including event-driven) for tracing causality and data flow through the whole system. The fact that there's great advantages to distributed architectures, yet they're harder to work with, is a signal that there's a market opportunity for better tooling around working with them.
- haggy 7y agoThis ^^ And with regards to tooling, there is already a lot of work being done around distributed tracing tools
- cloverich 7y agoI think it comes down to taking debugging, monitoring, and alerting seriously. I doubt any team would be better off with a distributed system if they don't spend a serious amount of time setting up logging, monitoring, tracing, making it easy to debug, having automated alerts, etc. Its what separates a great coder from a good engineer and if you don't have the latter on hand, you should steer far away from building systems like this.
- spoonfoe 7y agoThis. We've got a couple of these kinds of applications where I work. All built on RabbitMq / MassTransit. Without smart logging it would be real frustrating to track down issues. You could put breakpoints everywhere locally, but that's a pain. Tools like Seq are really helpful during development. ELK stack / splunk / log aggregator X are almost required on the non-local environments.
- iliketoworkhard 7y agoSplunk aggregation is a life saver. If you assign a unique id to each request, you can trace its path thru it's entire lifecycle, including asynchronous post-tasks. This can also help see average delay latencies between a sync task and its associated asynchronous post task/s.
- alexandercrohde 7y ago100% Every time I work somewhere I have to play shepherd and ask the very basics: - Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments - Have you thought about all the new problems that might happen: queue sending to dead endpoints, circular queue problem, queue being restarted somehow (e.g. deploys) and losing messages? - If the app fails post-queue, not surfacing the message to the user, do you have a plan to ensure somebody in engineering sees and fixes that error? And then goes back and remediates the broken request(s)? - Have you prepared code/logs to do distributed tracing? - If there's a dispute a week from now whether Joe didn't get an email because of a problem BEFORE or AFTER the queue, will you be able to tell from the logs? Many powerful engineering abstractions (threads, async, services) require one notch higher of engineering talent and allows for all sorts of new failure paths. The tradeoff must be taken very seriously. Most places I have worked at adopted complexity too soon.
- yowlingcat 7y agoFully agree here. Catastrophic failures can slip in silently on both ends of the curve -- if Joe doesn't get an email OR Joe gets too many emails, it's still bad. In my experience, beefing up logging to be able to do distributed tracing is an absolute must to prevent debugging pain.
- lcfcjs2 7y ago- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments Azure and AWS and every other service that provides a Service Bus service guarantees up-time. Their SLA should be your customers SLA.
- iliketoworkhard 7y agoGreat points! Would you mind elaborating to a relatively inexperienced engineer how the following scenarios can end up happening? > queue sending to dead endpoints > circular queue problem
- 7y ago
- LargeWu 7y agoIt can be very hard to know the state of a piece of information within your system unless you fanatically log its state at all times. And even then, you might have to look through lots of distributed logs to find that. Of course, logging the state of the world at every step also creates tons of noise. At a previous gig, we had a processing workflow where an entity might move through any of 20 processes tied together through queues. It was very, very difficult to track down problems with things dropping out, things getting stuck, etc.
- rmetzler 7y agoI just wonder, when you have a queueing system, can’t you use this for distributed logging?
- iddqd 7y agoImagine if you flipped on debug logging and suddenly your queueing system gets DDoS by log messages, causing more errors and cascading effects, taking down your product. It’s always a good idea to separate the two.
- rswail 7y ago> where an entity might move through any of 20 processes To me that's a design smell. The processes should be making requests of the entity via messaging. Each process should result in the entity emitting a change of state. That change of state should be the trigger for subsequent processes.
- LeonM 7y agoThe timing of this article couldn't have been more painful. I'm currently working on solving a problem with an insanely over complicated setup (for the task at hand) that is build by another engineer who has since left the company. It's a cluster of 3 virtualised machines running docker swarm where a RabbitMQ instance ties 40+ worker pods together. Once every 5 days or so the connection between RabbitMQ and (some of) the worker container stops working, causing the worker to crash and the queue message is lost. We are talking like 5 layers of virtualization and/or abstraction. It's impossible to debug. I honestly don't know how to explain this to my customer.
- deleted 7y ago[deleted]
- rmetzler 7y agoThe bug description sounds a little bit like a memory leak or connections aren’t closed properly.
- noncoml 7y agoThis. You are leaking connections probably.
- iddqd 7y agoYou might be better off focusing on improving fault tolerance instead of trying to hunt down the root cause.
- eternalban 7y agoRabbit supports the choice of durable exchanges, queues, and messages. There are (tunable, negotiable) performance penalties in terms of latency and throughput. We are partly in agreement regarding f/t. Logging comes into play when digging into recurring failure (bugs). Fault tolerance will not save you from bugs and bug hunting. [p.s. to add, performance tuning/trouble-shooting also require instrumentation. Instrumentation would appear to be a fundamental (and useful) capability.]
- 7y ago
- johnbrodie 7y agoI've found it very helpful in distributed environments to ensure that you are giving the incoming request an extra "Request ID"-style header. Make sure this header is propagated everywhere, and logged everywhere. Makes debugging much, much easier. Still gets hard at scale, which is why tools like New Relic have various "distributed tracing" features now.
- Redsquare 7y agoOr simply correlation + causation
- jjtheblunt 7y agoComparably OT, but i bet this is the essence of biology and cancer being tough : signals sent via cytokines and hormones _are_ messages, IIRC.
- davidw 7y agoI forget the exact quote, but it goes something like "the best distributed system is one you didn't build". Obviously there are reasons to build them that become inevitable once a system needs to grow beyond a certain size, but if you can put that off for a while, it's a win.
- nonconvergent 7y agoThat's why I get paid the medium bucks.
- orthecreedence 7y agoTo your point...even if you do want eventing, you can do it inside of a monolith and not involve the network at all, which will save a good amount of pain.
- segmondy 7y agoNot really, it's tougher, but if you're bringing queues into your workflow then it should be because you need it. You are having to deal with scaling, resiliency, parallelism, faster response time, etc. The benefit then out weighs the debugging overhead. But at that point you have a distributed system. If you have a Distributed system, you should have centralized logging and distributed tracing too so you can debug easier. Without those, you are going to go through the pain.
- shriek 7y agoOf course but you're often trading debug-ability over scalability whenever you're picking evented architecture. I do think tools can help here but it's just the nature of this architecture. You're reacting to 'events' instead of following procedural orders.
- rswail 7y agowe just finished implementation of an event sourced/CQRS solution using kafka. Yes, you need to have monitoring etc, but the testing and debugging were substantially easier because of how we broke down the "services". For each major entity (or aggregate) there was a service that subscribed to a number of command and event topics, it produced output to an event topic for the aggregate. We had FSMs for each of the aggregates, documenting the effect of each potential command or external event and the change of state and the resulting state changes (events) and/or commands. The architectural constraint meant that the infrastructure was the same for each aggregate, the testing of each was independent and could be mocked easily using topic producers. So as opposed to the "Big Ball Of Mud" we have a monitorable infrastructure (kafka + alerts/stats sent to a statsd integrated with AWS Cloudwatch), we have individual aggregate processing that only respond to incoming commands or events and have defined outputs for each potential incoming command/event. Much much easier to design, develop and debug. But the trick is at the start (like anything else). Analyzing the domain to determine the entities/aggregates, modelling the externally generated commands, modelling the FSMs for each aggregate etc.
- chadcmulligan 7y agoYah, its one big race condition
- danatcofo 7y agoaccount has been suspended. link now broken. /shrug
- adamnemecek 7y agohttp://web.archive.org/web/20190829150014/https://blog.theodo.com/2019/08/event-driven-architectures-rabbitmq/ http://web.archive.org/web/20190829150014/https://blog.theod...
- longcommonname 7y agoQueue driven systems really fascinate me, coming from a chemical engineering background I can't help but to see parallels to fluid dynamics and all that difficult math that comes from their analysis. I've always wanted to create some type of monitoring system that displays the entire system in that vein and then model or using control theory. Has anybody seen a project that does this?