15 ms·

To Message Bus or Not: Distributed Systems Design (2017)

290 points by gk1 8y ago

redact207 8y ago
This article talks about the benefit of a message bus being pub/sub. This definitely helps to decouple the internals of apps, which in turn makes things easier to maintain. There are many other benefits to using a message bus, and it's a better fit in general to distributed systems. The hard part is understanding how to structure apps to make them compatible with messaging. Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. You can't expect a reply at that point because the service that sent it may no longer be running. If you're on Node, try https://node-ts.github.io/bus/ https://node-ts.github.io/bus/ It's a service bus that manages the complexity of rabbit/sqs/whatever, so you can build simple message handlers and complex orchestration workflows
- jandrese 8y ago
  The downside of message busses is that they can be really hard to debug for third parties. I run into this in SystemD, where a process will hang at startup waiting for a particular signal, and I have to hunt down all of the locations that signal could have been generated from to see which one failed. Much much harder than the old system where you would find the line you stalled on and then read up a few lines to see what it was trying to do.
  - EpicEng 8y ago
    It also becomes difficult to reason about what code will actually run when a message goes out. Top level handlers are ok, but they can spawn messages which spawn messages which... lead to me running out of mental capacity.
- ledneb 8y ago
  > Imagine sending out a command to the bus and not knowing when it'll get processed I would love to hear how others are correlating output with commands in such architectures - especially if they can be displayed to users as a direct result of a command. Always felt like I'm missing a thing or two. It seems the choices are: * Manage work across domains (sagas, two phase commit, rpc) * Losen requirements (At some point in the future, stuff might happen. It may not be related to your command. Deal with it.) * Correlation and SLAs (correlate outcomes with commands, have clients wait a fixed period while collecting correlating outcomes) Is that a fair summary of where we can go? Any recommended reading?
  - jerf 8y ago
    My personal answer would be that commands (as defined as "things the issuer cares about the response to") don't belong on message busses, and there's probably an architectural mismatch somewhere. Message busses are at their best when everything is unidirectional and has no loops. If you need loops, and you often do, you're better off with something that is designed around that paradigm. To the extent that it scales poorly, well, yeah. You ask for more, it costs more. But probably less than trying to bang it on to an architecture where it doesn't belong. You want something more like a service registry like zookeeper for that, where you can obtain a destination for service and speak to it directly. You'll need to wrap other error handling around it, of course, but that almost goes without saying.
  - bunderbunder 8y ago
    I don't know about correlating output with commands, but if you're looking to correlate output with input, one option is to stick an ID on every message, and, for messages that are created in response to other messages, also list which one(s) it's responding to. I would say that loosening requirements is also a reasonable option. You can't assume that anything downstream will be up, or healthy, or whatever. On a system that's large enough to benefit from a message bus, you have to assume that failures are the exception and not the norm. And trying to get a system that acts like that is the case is likely to be more expensive than it's worth. For a decent blog post that touches on the subject, see "Starbucks Does Not Use Two-Phase Commit"[1]. [1]: https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html https://www.enterpriseintegrationpatterns.com/ramblings/18_s...
    - hadsed 8y ago
      Nice blog post! Certainly puts things into perspective in terms of how one should deal with errors, including sometimes just not caring about them much.
  - CorvusCrypto 8y ago
    My whole take on this: Commands can go through message busses and be managed easily or it could just be a sequence of async requests, but regardless of what drives commands and events at that point you should have a very solid CQRS architecture in mind. What should be acknowledged to the client is that the command was published and that's it. The problem is of course eventual consistency but it's a trade-off for being able to handle a huge amount of load by scaling separately both COMMAND handlers which perform data modification, and EVENT handlers that allow the side effects that must occur. In a typical web app setup I would define a request ID at time of client request. The request creates a COMMAND which carries with it a request ID as well as a command ID. This results in an action and then the EVENT is published with the request ID, command ID, and event ID. To monitor you collect the data and then look at the timestamp differences to monitor lag and dropped messages. With the events, you get all the data necessary to audit what request, and subsequent command, created a system change. To audit the full data change however, and not just which request caused what change, you need to have a very well-structured event model designed for what you want to audit. You can't guarantee when a command or subsequent event will be processed, but that's fine. That's the whole point around eventual consistency. It's a bit uncomfortable at first, but use the lag monitoring and traceability as a debug tool when needed and really it's no problem. Also just shift the client over to reading from your read-specific projections on refresh or periodically and data will eventually appear for the user. It's the reason sometimes a new order might not appear right away on your order history for instance on Amazon, and in reality it's fine 99% of the time. Never have your client wait on a result. Instead think: how can I design my application to not need to block on anything? It's doable though it is quite hard and if you've only designed synchronous systems it will feel so uncomfortable. And remember some things should not have CQRS design, backed by a message bus or not. These will be bottlenecks but they might be necessary. The whole system you design doesn't have to stick to a single paradigm. Need transaction IDs to be strictly consistent for a checkout flow to be secure and safe? Use good old synchronous methods to do it. Core in all of this is data design. If you design your entities, commands, or events poorly, you will suffer the consequences. You will often hear the word "idempotency" a lot in CQRS design. It's because idempotent commands are super important in preventing unintended side effects. Same with idempotent events, if possible. If you get duplicate events from a message bus, idempotency will save your arse if it's a critical part of the system. If it's something mild like an extra "addToCart" command or something, no big deal really, but imagine a duplicated "payOrder" command ;). To summarize, I correlate output to commands and requests by ensuring there are no unknown side-effects, critical synchronous components remain synchronous, designing architecture that compliments the data (not the other way around), and ensuring that the client is designed in such a way that eventual consistency doesn't matter from the user perspective when it comes into play.
- protomyth 8y ago
  > Imagine sending out a command to the bus and not knowing when it'll get processed That's one of the things about tuple spaces[0,1,2] that was described in the book Mirror Worlds[3]. The author gives a lengthy description of a tuple's life. In practice, it really depended on what pattern you used to process the tuples. 0) https://software-carpentry.org/blog/2011/03/tuple-spaces-or-good-ideas-dont-always-win.html https://software-carpentry.org/blog/2011/03/tuple-spaces-or-... 1) https://en.wikipedia.org/wiki/Tuple_space https://en.wikipedia.org/wiki/Tuple_space 2) http://wiki.c2.com/?TupleSpace http://wiki.c2.com/?TupleSpace 3) https://www.amazon.com/Mirror-Worlds-Software-Universe-Shoebox/dp/019507906X https://www.amazon.com/Mirror-Worlds-Software-Universe-Shoeb...
- chrisseaton 8y ago
  > Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. Isn’t that the situation on any network?
- chrischen 8y ago
  How does Bus compare to Celery?
  - zbentley 8y ago
    Celery is a job execution system, nothing more. It can be used synchronously or asynchronously, in a central-bus type broker layout or a different one. It is orthogonal to the architectural patterns being discussed here, but can be used to facilitate many of them when deployed purposefully.
    - chrischen 8y ago
      I meant specifically compared to https://node-ts.github.io/bus/ https://node-ts.github.io/bus/, not Messages Bus architecture.
- opportune 8y ago
  >Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. You can't expect a reply at that point because the service that sent it may no longer be running. That's because this is missing the point of a message bus. Namely, communication with a message bus is meant to be one way, there is not supposed to be a response. If you want a timely response, you should set up a loadbalanced autoscaling microservice that maybe hooks into some large backend store.
  - icxa 8y ago
    There is no explicit forbiddance of Request/Reply semantics in a message bus. Data on a message bus absolutely does not have to be one way.
    - opportune 8y ago
      As far as I know, typically message buses reply semantics are only acknowledgement that they've received the data. That's the way that Kafka and AWS Kinesis and Azure EventHubs work, though I'm not familiar with other message buses. You'll note that almost all architectural diagrams for these message buses, along with the article linked in this thread, show a one-way data flow in/out of a message bus. Do you know of any message buses that have more complicated reply semantics? I'm thinking maybe the javascript world is calling things message buses that the big data / distributed systems world wouldn't think are message buses
      - jeffasinger 8y ago
        Often times these "Service Bus" type frameworks implement request/reply by creating a queue per host, and in every message adding envelope information that includes the queue that the reply should be sent to. This does allow a request/reply pattern. I've never actually done this, but I imagine in environments where reliability is more important than latency, it probably works pretty well. One minor nitpick. The patterns you're talking about aren't typically implemented on Kinesis, since there's no concept of topics there. On AWS, you could do something like that with SNS + SQS.
bvrmn 8y ago
Message bus proponents never build large systems. The only sane way is to be pretty specific about data flow with clear mental model shared between developers and ops. Message bus hides producer-consumer relations and with multiple endpoints it's very hard to reason about the system as a whole.
- redact207 8y ago
  Why do you say that? I'm someone who's worked on a few very large message based systems in finance for years (100s of devs working together in multiple countries). I found that messaging and workflow orchestration were the things that helped keep things same.
  - PaulHoule 8y ago
    I've seen people succeed in a big way with message bus architectures and I have seen them fail in a big way. The master symptom I've seen is that people queue work and that work never gets unqueued. It's amazing how many real-life systems fail with the same symptom.
    - K2L8M11N2 8y ago
      Perhaps that could be solved with monitoring the number of pending requests and alerting over a certain threshold?
      - PaulHoule 8y ago
        That diagnoses the problem, sets the stage for solving it, but doesn't actually solve it.
        dajohnson89 8y ago
        Isn't that an "easy" problem to solve, in the sense that you just increase the processing capacity of the consumer?
    - 7y ago
ngrilly 8y ago
Must be noted that service meshes bring some of the message bus advantages to the point-to-point architecture.
- opportune 8y ago
  Somewhat, but it's a different use case. I'd say the main difference is that message buses are better for non-urgent, hopefully-soon-but-definitely-eventually workloads, while creating something like a message bus between services within a mesh will impose more urgency. Anecdotally I've heard that extremely chatty services (like something that approximates a message bus) are considered poor mesh design but I don't really understand why that is the case so long as the service architecture is kept clean
  - ngrilly 8y ago
    I agree. The advantages I was referring to are things like adding monitoring, which is mentioned in the article. Interesting point about chatty services :-)
lugg 8y ago
Whether to use pub/sub or rest is entirely domain dependent. So, who cares? Its an implementation detail. Just do whatever makes sense. The most sane systems I've seen make use of both. > In an architecture driven by a message bus it allows more ubiquitous access to data. Please stop mistaking design details for architecture. Lots of things allow more ubiquitous access to data. Talking about this stuff in this way is just going to wind up with you replacing MySQL with Kafka and never actually solving any real problems with your contexts/domains.
- rubyn00bie 8y ago
  Totally agreeing with you. <rant> A message bus is to do things like fan-out, fucking messages, not act-as-a-fucking-data-store. Situations that need a message bus are so fucking few and far between I can't believe they posted this article. Message bus is the fastest way to distribute your monolith... I don't know why people are always trying to get rid of REST like it's the problem (it's not; your code/infrastructure is)... </rant>
- closeparen 8y ago
  What does architecture mean to you? The set of components and how they’ll interact is what it seems to mean in my world.
  - lugg 8y ago
    I'm not sure I have a clear definition. What I do know is that I avoid confusing tools, frameworks, languages, libraries, or really anything you could call "software" with architecture. They're all just tools that enable me to draw boundaries around contexts. To attempt to define it, I would say architecture has far more to do with _where_ I draw bounded context lines than how or what tools I use to do so. I saved this talk[1] in my favorites because he manages to distill ways to induce the right boundaries out of business needs. (surprisingly not an uncle bob talk!) For a more concrete example I separate a CQRS/ES architecture philosophy from the implementation detail of a message bus in this comment I wrote further down in the thread [2]. [1] https://www.youtube.com/watch?v=ez9GWESKG4I https://www.youtube.com/watch?v=ez9GWESKG4I [2] https://news.ycombinator.com/item?id=19905016 https://news.ycombinator.com/item?id=19905016
carc 8y ago
A couple of big reasons don't like message buses (but open to hearing about why I'm wrong): -All "requests" will succeed even if malformed -Couples producers/consumers to the bus (unless you put in the extra work to wrap it in a very simple service)
- GordonS 8y ago
  > -All "requests" will succeed even if malformed Took me a minute to grok this, but I think you mean, message bus clients can send 'any old shit' and the broker will happily queue it? A lot of the more 'managed' (abstracted) clients in the statically typed world deal with this by structuring queues by the object type/interface. A bad actor could probably circumvent this if they really wanted, but for normal usage it will at least ensure that objects sent are of the expected type. This means that in real-world usage, this isn't an issue.
- GordonS 8y ago
  > Couples producers/consumers to the bus (unless you put in the extra work to wrap it in a very simple service) This is an area where adding a simple layer of abstraction is a good thing - most times, all you want is something like a `Send<T>`/`Send(obj)` method, which really will translate fairly universally across message busses. I used such a thing recently, switching out RabbitMQ for MQTT - all I really had to do was point to a new implementation of the interface!
- teyc 8y ago
  indeed. Here's an example of a case against messaging in distributed systems design. https://www.brisbanetimes.com.au/national/queensland/education-qld-school-child-abuse-hundreds-of-cases-not-passed-to-police-20150731-giotkw.html https://www.brisbanetimes.com.au/national/queensland/educati... A synchronous call would have failed when the operator issued the request, and the onus is on the operator to find alternate ways of contacting the police.
- EdwardDiego 8y ago
  > All "requests" will succeed even if malformed A good pattern is to use a schema to verify you're sending valid data in the producer (at the very least, in a unit test, if not at runtime) - for example, the Confluent wrapped Kafka ships with a schema registry and painless Avro based Kafka producers If overhead in the producer is stopping you from validating each record at creation time, then you can validate downstream, so long as your consumers agree to only consume the validated data.
cpitman 8y ago
In enterprise environments, I more often see overuse of message brokers. I am all for targeted use of messaging systems, but what I usually see is an all or nothing approach where as soon as a team starts using messaging they use it for _all_ interprocess communication. This has some very painful downsides. First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). I've had customers who start breaking apart their monoliths and replacing with microservices, and since the monolith used messaging they use messaging between microservices as well. Well, the brokers were already scaled to capacity for the old system, so going from "1" monolith service to "10" microservices means deploying 10x more messaging infrastructure. Traditional messaging brokers also don't scale all that well horizontally, so that 10x is probably more like 20x. Second, performance tuning becomes harder for people to understand. A core tenant of "lean" is to reduce the number/length of queues in a system to get more predictable latency and to make it easier to diagnose bottlenecks. But everyone always does the opposite with messaging: massive queues everywhere. By the time the alarms go off, multiple queues are backing up and no one is clear on where the actual bottleneck is. What I would like to see more of is _strategic_ use of messaging. For example, a work queue to scale out processing, but the workers use HTTP or some other synchronous method to call downstream services. Limiting messaging to very few points in the process still give a lot of the async benefits to the original client, while limiting how often requests hit the broker and making it easier to diagnose performance problems.
- _pmf_ 8y ago
  > First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). How is multiplexing supposed to work with plain HTTP without middleman? One of the main uses for message queues is efficient fan-out (m + n connections instead of m x n).
- lulf 8y ago
  I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Either “direct” client server as with HTTP , or via an intermediary such as the Apache Qpid Dispatch Router. With the dispatch router, you can use pattern matching to specify if addresses should be routed to a broker or be routed directly to a particular service. This is way you can get the semantics that best fits your use case.
ww520 8y ago
Besides the distributed case, message bus is invaluable in building crash-proof applications. I've used lightweight message bus within the app itself, for better crash recovery on long running tasks. E.g. Need to generate lots of emails and send them out. I would create a small command object for each email and queue it to the message bus, and let the message handler to handle email generation one by one. The lightweight command objects can be queued up quickly and the UI can return to the user right the way. The slow running email generation can be run in the background. In the event of a system crash, the message bus will be restored automatically and continue on the remaining tasks in the queue.
- sk5t 8y ago
  What if your process to generate the initial messages crashes halfway through?
  - ww520 8y ago
    You return the error to the user. Crash handling is a matter of managing user expectation. If the user hits a button and the UI shows the command as success, the system better ensures the command will complete eventually. If the user hits a button and the UI shows an error, well the user can try again when the system is up.
    - sk5t 8y ago
      Ah, let me restate the question: suppose the user has clicked a button to approve sending 5,000 emails; the server process puts 2,000 messages onto a queue for later processing--but then, something occurs to interrupt adding the remaining 3,000 messages to the queue. Presuming the queue doesn't support any transactionality beyond per-message, and optionally supposing a queue consumer has already started sending some of these emails, how do you recover from the fault and help the user send the rest of the email (without duplicating outbound email)?
      - hderms 8y ago
hestefisk 8y ago
My main gripe with service buses is that they can be very hard to deploy and test automatically. At least for traditional ‘middleware’ like WebSphere/MQ, Weblogic etc. It potentially adds another monolith to your architecture, which, whilst fancy, may not be required. Using ZeroMQ or similar ‘lightweight’ tech could be a better choice for small teams as it is easy to integrate into containers and testable.
Pamar 8y ago
I strongly suggest anyone interested in this type of architecture to read https://www.goodreads.com/search?q=Enterprise+Integration+Patterns%3A+Designing%2C+Building%2C+and+Deploying+Messaging+Solutions https://www.goodreads.com/search?q=Enterprise+Integration+Pa... - it is a really well though out catalog of patterns and solution for Message-based systems.
kbouck 8y ago
I'm curious to hear others' opinions on using a database as a message queue. One issue I have with most message brokers is that you can not perform adhoc queries or manipulations of the queue. When you've got a backlog situation, it's nice to be able to answer questions like: - how many of these queued msgs are for/from customer/partner X.
phoe-krk 8y ago
It's weird for me to not see even a single mention of any BEAM languages, such as Erlang or Elixir. They are naturally distributed and have discovery, networking and messaging built into the virtual machine itself.
scraegg 8y ago
The first sentence already is triggering me. No, it's not hard. Like most topics in software engineering there are 50+ years of pretty successful development, backed by science, backed by software from the 70s, backed by all seniority levels of engineers. The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right? The problem is that something can only be hip for 1-5 years. But really stable complex systems like a distributed network can only be developed in a time frame of 5-10 years. Therefore most of the hip stuff is unfinished or already flawed in its basic architecture. And usually if you switch from one hip stuff to another, you get the same McDonalds software menues, just with differently flavored ketchup and marketing. So if you feel something is hard it might be because you are not dealing with the actual problem. For instance you might think about doing Kafka, and that's fine. But be aware that email is shipping more messages than Kafka and its doing it longer than Kafka. For instance topologies: There is no point-to-point. There's star, meshed, bus etc. See here: https://en.wikipedia.org/wiki/Network_topology https://en.wikipedia.org/wiki/Network_topology If you don't know your topology it might be star or mesh. But it's still a topology or a mix of multiple topologies. And if you develop a distributed system you really need to think about how your network should be structured. Then if you know which topology fits your use case you can go and figure out the way this topology works and what the drawbacks are. Star networks (like k8s) for instance are easy to setup but have a clear bottle neck in the center. A Bus (like Kafka) is like a street. It works fine until it is full, and there are sadly some activity patterns were an overloaded Bus will cascade and the overload is visible even weeks later (although you have reduced the traffic already), if you don't reset it completely from time to time. It's not magic. You can look all of it up on wikipedia if you know the keywords. Also there is not a single "good" solution. It depends always on how well the rest of your system integrates with the pros and cons of your topology choice. And if you use multiple topologies in parallel you have a complexity overhead at some point, which is why working in big corps is usually so slow.
- geezerjay 8y ago
  > The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right? That's an awfully short-sighted comparison. There are far more job offers for deploying and managing systems with kubernetes and to develop front-ends with React than there are for developing TCP/IP infrastructure. It's fun to earn a living and enjoy the priviledges of having a high salary, and the odds you get that by studying solved problems that nowadays just work are not that high.

290 points by gk1 8y ago

This article talks about the benefit of a message bus being pub/sub. This definitely helps to decouple the internals of apps, which in turn makes things easier to maintain. There are many other benefits to using a message bus, and it's a better fit in general to distributed systems. The hard part is understanding how to structure apps to make them compatible with messaging. Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. You can't expect a reply at that point because the service that sent it may no longer be running. If you're on Node, try https://node-ts.github.io/bus/ https://node-ts.github.io/bus/ It's a service bus that manages the complexity of rabbit/sqs/whatever, so you can build simple message handlers and complex orchestration workflows

jandrese 8y ago
The downside of message busses is that they can be really hard to debug for third parties. I run into this in SystemD, where a process will hang at startup waiting for a particular signal, and I have to hunt down all of the locations that signal could have been generated from to see which one failed. Much much harder than the old system where you would find the line you stalled on and then read up a few lines to see what it was trying to do.
- EpicEng 8y ago
  It also becomes difficult to reason about what code will actually run when a message goes out. Top level handlers are ok, but they can spawn messages which spawn messages which... lead to me running out of mental capacity.
ledneb 8y ago
> Imagine sending out a command to the bus and not knowing when it'll get processed I would love to hear how others are correlating output with commands in such architectures - especially if they can be displayed to users as a direct result of a command. Always felt like I'm missing a thing or two. It seems the choices are: * Manage work across domains (sagas, two phase commit, rpc) * Losen requirements (At some point in the future, stuff might happen. It may not be related to your command. Deal with it.) * Correlation and SLAs (correlate outcomes with commands, have clients wait a fixed period while collecting correlating outcomes) Is that a fair summary of where we can go? Any recommended reading?
- jerf 8y ago
  My personal answer would be that commands (as defined as "things the issuer cares about the response to") don't belong on message busses, and there's probably an architectural mismatch somewhere. Message busses are at their best when everything is unidirectional and has no loops. If you need loops, and you often do, you're better off with something that is designed around that paradigm. To the extent that it scales poorly, well, yeah. You ask for more, it costs more. But probably less than trying to bang it on to an architecture where it doesn't belong. You want something more like a service registry like zookeeper for that, where you can obtain a destination for service and speak to it directly. You'll need to wrap other error handling around it, of course, but that almost goes without saying.
- bunderbunder 8y ago
  I don't know about correlating output with commands, but if you're looking to correlate output with input, one option is to stick an ID on every message, and, for messages that are created in response to other messages, also list which one(s) it's responding to. I would say that loosening requirements is also a reasonable option. You can't assume that anything downstream will be up, or healthy, or whatever. On a system that's large enough to benefit from a message bus, you have to assume that failures are the exception and not the norm. And trying to get a system that acts like that is the case is likely to be more expensive than it's worth. For a decent blog post that touches on the subject, see "Starbucks Does Not Use Two-Phase Commit"[1]. [1]: https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html https://www.enterpriseintegrationpatterns.com/ramblings/18_s...
  - hadsed 8y ago
    Nice blog post! Certainly puts things into perspective in terms of how one should deal with errors, including sometimes just not caring about them much.
- CorvusCrypto 8y ago
  My whole take on this: Commands can go through message busses and be managed easily or it could just be a sequence of async requests, but regardless of what drives commands and events at that point you should have a very solid CQRS architecture in mind. What should be acknowledged to the client is that the command was published and that's it. The problem is of course eventual consistency but it's a trade-off for being able to handle a huge amount of load by scaling separately both COMMAND handlers which perform data modification, and EVENT handlers that allow the side effects that must occur. In a typical web app setup I would define a request ID at time of client request. The request creates a COMMAND which carries with it a request ID as well as a command ID. This results in an action and then the EVENT is published with the request ID, command ID, and event ID. To monitor you collect the data and then look at the timestamp differences to monitor lag and dropped messages. With the events, you get all the data necessary to audit what request, and subsequent command, created a system change. To audit the full data change however, and not just which request caused what change, you need to have a very well-structured event model designed for what you want to audit. You can't guarantee when a command or subsequent event will be processed, but that's fine. That's the whole point around eventual consistency. It's a bit uncomfortable at first, but use the lag monitoring and traceability as a debug tool when needed and really it's no problem. Also just shift the client over to reading from your read-specific projections on refresh or periodically and data will eventually appear for the user. It's the reason sometimes a new order might not appear right away on your order history for instance on Amazon, and in reality it's fine 99% of the time. Never have your client wait on a result. Instead think: how can I design my application to not need to block on anything? It's doable though it is quite hard and if you've only designed synchronous systems it will feel so uncomfortable. And remember some things should not have CQRS design, backed by a message bus or not. These will be bottlenecks but they might be necessary. The whole system you design doesn't have to stick to a single paradigm. Need transaction IDs to be strictly consistent for a checkout flow to be secure and safe? Use good old synchronous methods to do it. Core in all of this is data design. If you design your entities, commands, or events poorly, you will suffer the consequences. You will often hear the word "idempotency" a lot in CQRS design. It's because idempotent commands are super important in preventing unintended side effects. Same with idempotent events, if possible. If you get duplicate events from a message bus, idempotency will save your arse if it's a critical part of the system. If it's something mild like an extra "addToCart" command or something, no big deal really, but imagine a duplicated "payOrder" command ;). To summarize, I correlate output to commands and requests by ensuring there are no unknown side-effects, critical synchronous components remain synchronous, designing architecture that compliments the data (not the other way around), and ensuring that the client is designed in such a way that eventual consistency doesn't matter from the user perspective when it comes into play.
protomyth 8y ago
> Imagine sending out a command to the bus and not knowing when it'll get processed That's one of the things about tuple spaces[0,1,2] that was described in the book Mirror Worlds[3]. The author gives a lengthy description of a tuple's life. In practice, it really depended on what pattern you used to process the tuples. 0) https://software-carpentry.org/blog/2011/03/tuple-spaces-or-good-ideas-dont-always-win.html https://software-carpentry.org/blog/2011/03/tuple-spaces-or-... 1) https://en.wikipedia.org/wiki/Tuple_space https://en.wikipedia.org/wiki/Tuple_space 2) http://wiki.c2.com/?TupleSpace http://wiki.c2.com/?TupleSpace 3) https://www.amazon.com/Mirror-Worlds-Software-Universe-Shoebox/dp/019507906X https://www.amazon.com/Mirror-Worlds-Software-Universe-Shoeb...
chrisseaton 8y ago
> Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. Isn’t that the situation on any network?
chrischen 8y ago
How does Bus compare to Celery?
- zbentley 8y ago
  Celery is a job execution system, nothing more. It can be used synchronously or asynchronously, in a central-bus type broker layout or a different one. It is orthogonal to the architectural patterns being discussed here, but can be used to facilitate many of them when deployed purposefully.
  - chrischen 8y ago
    I meant specifically compared to https://node-ts.github.io/bus/ https://node-ts.github.io/bus/, not Messages Bus architecture.
opportune 8y ago
>Imagine sending out a command to the bus and not knowing when it'll get processed. Perhaps under a second, perhaps next week. You can't expect a reply at that point because the service that sent it may no longer be running. That's because this is missing the point of a message bus. Namely, communication with a message bus is meant to be one way, there is not supposed to be a response. If you want a timely response, you should set up a loadbalanced autoscaling microservice that maybe hooks into some large backend store.
- icxa 8y ago
  There is no explicit forbiddance of Request/Reply semantics in a message bus. Data on a message bus absolutely does not have to be one way.
  - opportune 8y ago
    As far as I know, typically message buses reply semantics are only acknowledgement that they've received the data. That's the way that Kafka and AWS Kinesis and Azure EventHubs work, though I'm not familiar with other message buses. You'll note that almost all architectural diagrams for these message buses, along with the article linked in this thread, show a one-way data flow in/out of a message bus. Do you know of any message buses that have more complicated reply semantics? I'm thinking maybe the javascript world is calling things message buses that the big data / distributed systems world wouldn't think are message buses
    - jeffasinger 8y ago
      Often times these "Service Bus" type frameworks implement request/reply by creating a queue per host, and in every message adding envelope information that includes the queue that the reply should be sent to. This does allow a request/reply pattern. I've never actually done this, but I imagine in environments where reliability is more important than latency, it probably works pretty well. One minor nitpick. The patterns you're talking about aren't typically implemented on Kinesis, since there's no concept of topics there. On AWS, you could do something like that with SNS + SQS.

bvrmn 8y ago

Message bus proponents never build large systems. The only sane way is to be pretty specific about data flow with clear mental model shared between developers and ops. Message bus hides producer-consumer relations and with multiple endpoints it's very hard to reason about the system as a whole.

redact207 8y ago
Why do you say that? I'm someone who's worked on a few very large message based systems in finance for years (100s of devs working together in multiple countries). I found that messaging and workflow orchestration were the things that helped keep things same.
- PaulHoule 8y ago
  I've seen people succeed in a big way with message bus architectures and I have seen them fail in a big way. The master symptom I've seen is that people queue work and that work never gets unqueued. It's amazing how many real-life systems fail with the same symptom.
  - K2L8M11N2 8y ago
    Perhaps that could be solved with monitoring the number of pending requests and alerting over a certain threshold?
    - PaulHoule 8y ago
      That diagnoses the problem, sets the stage for solving it, but doesn't actually solve it.
      - dajohnson89 8y ago
        Isn't that an "easy" problem to solve, in the sense that you just increase the processing capacity of the consumer?
  - 7y ago

ngrilly 8y ago

Must be noted that service meshes bring some of the message bus advantages to the point-to-point architecture.

opportune 8y ago
Somewhat, but it's a different use case. I'd say the main difference is that message buses are better for non-urgent, hopefully-soon-but-definitely-eventually workloads, while creating something like a message bus between services within a mesh will impose more urgency. Anecdotally I've heard that extremely chatty services (like something that approximates a message bus) are considered poor mesh design but I don't really understand why that is the case so long as the service architecture is kept clean
- ngrilly 8y ago
  I agree. The advantages I was referring to are things like adding monitoring, which is mentioned in the article. Interesting point about chatty services :-)

lugg 8y ago

Whether to use pub/sub or rest is entirely domain dependent. So, who cares? Its an implementation detail. Just do whatever makes sense. The most sane systems I've seen make use of both. > In an architecture driven by a message bus it allows more ubiquitous access to data. Please stop mistaking design details for architecture. Lots of things allow more ubiquitous access to data. Talking about this stuff in this way is just going to wind up with you replacing MySQL with Kafka and never actually solving any real problems with your contexts/domains.

rubyn00bie 8y ago
Totally agreeing with you. <rant> A message bus is to do things like fan-out, fucking messages, not act-as-a-fucking-data-store. Situations that need a message bus are so fucking few and far between I can't believe they posted this article. Message bus is the fastest way to distribute your monolith... I don't know why people are always trying to get rid of REST like it's the problem (it's not; your code/infrastructure is)... </rant>
closeparen 8y ago
What does architecture mean to you? The set of components and how they’ll interact is what it seems to mean in my world.
- lugg 8y ago
  I'm not sure I have a clear definition. What I do know is that I avoid confusing tools, frameworks, languages, libraries, or really anything you could call "software" with architecture. They're all just tools that enable me to draw boundaries around contexts. To attempt to define it, I would say architecture has far more to do with _where_ I draw bounded context lines than how or what tools I use to do so. I saved this talk[1] in my favorites because he manages to distill ways to induce the right boundaries out of business needs. (surprisingly not an uncle bob talk!) For a more concrete example I separate a CQRS/ES architecture philosophy from the implementation detail of a message bus in this comment I wrote further down in the thread [2]. [1] https://www.youtube.com/watch?v=ez9GWESKG4I https://www.youtube.com/watch?v=ez9GWESKG4I [2] https://news.ycombinator.com/item?id=19905016 https://news.ycombinator.com/item?id=19905016

carc 8y ago

A couple of big reasons don't like message buses (but open to hearing about why I'm wrong): -All "requests" will succeed even if malformed -Couples producers/consumers to the bus (unless you put in the extra work to wrap it in a very simple service)

GordonS 8y ago
> -All "requests" will succeed even if malformed Took me a minute to grok this, but I think you mean, message bus clients can send 'any old shit' and the broker will happily queue it? A lot of the more 'managed' (abstracted) clients in the statically typed world deal with this by structuring queues by the object type/interface. A bad actor could probably circumvent this if they really wanted, but for normal usage it will at least ensure that objects sent are of the expected type. This means that in real-world usage, this isn't an issue.
GordonS 8y ago
> Couples producers/consumers to the bus (unless you put in the extra work to wrap it in a very simple service) This is an area where adding a simple layer of abstraction is a good thing - most times, all you want is something like a `Send<T>`/`Send(obj)` method, which really will translate fairly universally across message busses. I used such a thing recently, switching out RabbitMQ for MQTT - all I really had to do was point to a new implementation of the interface!
teyc 8y ago
indeed. Here's an example of a case against messaging in distributed systems design. https://www.brisbanetimes.com.au/national/queensland/education-qld-school-child-abuse-hundreds-of-cases-not-passed-to-police-20150731-giotkw.html https://www.brisbanetimes.com.au/national/queensland/educati... A synchronous call would have failed when the operator issued the request, and the onus is on the operator to find alternate ways of contacting the police.
EdwardDiego 8y ago
> All "requests" will succeed even if malformed A good pattern is to use a schema to verify you're sending valid data in the producer (at the very least, in a unit test, if not at runtime) - for example, the Confluent wrapped Kafka ships with a schema registry and painless Avro based Kafka producers If overhead in the producer is stopping you from validating each record at creation time, then you can validate downstream, so long as your consumers agree to only consume the validated data.

cpitman 8y ago

In enterprise environments, I more often see overuse of message brokers. I am all for targeted use of messaging systems, but what I usually see is an all or nothing approach where as soon as a team starts using messaging they use it for _all_ interprocess communication. This has some very painful downsides. First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). I've had customers who start breaking apart their monoliths and replacing with microservices, and since the monolith used messaging they use messaging between microservices as well. Well, the brokers were already scaled to capacity for the old system, so going from "1" monolith service to "10" microservices means deploying 10x more messaging infrastructure. Traditional messaging brokers also don't scale all that well horizontally, so that 10x is probably more like 20x. Second, performance tuning becomes harder for people to understand. A core tenant of "lean" is to reduce the number/length of queues in a system to get more predictable latency and to make it easier to diagnose bottlenecks. But everyone always does the opposite with messaging: massive queues everywhere. By the time the alarms go off, multiple queues are backing up and no one is clear on where the actual bottleneck is. What I would like to see more of is _strategic_ use of messaging. For example, a work queue to scale out processing, but the workers use HTTP or some other synchronous method to call downstream services. Limiting messaging to very few points in the process still give a lot of the async benefits to the original client, while limiting how often requests hit the broker and making it easier to diagnose performance problems.

_pmf_ 8y ago
> First, traditional message brokers (ie queues, topics, persistence, etc) introduce a potential bottleneck for all communication, and on top of that they are _slow_ compared to just using the network (ie HTTP, with no middleman). How is multiplexing supposed to work with plain HTTP without middleman? One of the main uses for message queues is efficient fan-out (m + n connections instead of m x n).
lulf 8y ago
I recommend looking at a standard messaging protocol like AMQP 1.0, which will allow you to implement the request-response pattern in an efficient manner without message brokers. Either “direct” client server as with HTTP , or via an intermediary such as the Apache Qpid Dispatch Router. With the dispatch router, you can use pattern matching to specify if addresses should be routed to a broker or be routed directly to a particular service. This is way you can get the semantics that best fits your use case.

ww520 8y ago

Besides the distributed case, message bus is invaluable in building crash-proof applications. I've used lightweight message bus within the app itself, for better crash recovery on long running tasks. E.g. Need to generate lots of emails and send them out. I would create a small command object for each email and queue it to the message bus, and let the message handler to handle email generation one by one. The lightweight command objects can be queued up quickly and the UI can return to the user right the way. The slow running email generation can be run in the background. In the event of a system crash, the message bus will be restored automatically and continue on the remaining tasks in the queue.

sk5t 8y ago
What if your process to generate the initial messages crashes halfway through?
- ww520 8y ago
  You return the error to the user. Crash handling is a matter of managing user expectation. If the user hits a button and the UI shows the command as success, the system better ensures the command will complete eventually. If the user hits a button and the UI shows an error, well the user can try again when the system is up.
  - sk5t 8y ago
    Ah, let me restate the question: suppose the user has clicked a button to approve sending 5,000 emails; the server process puts 2,000 messages onto a queue for later processing--but then, something occurs to interrupt adding the remaining 3,000 messages to the queue. Presuming the queue doesn't support any transactionality beyond per-message, and optionally supposing a queue consumer has already started sending some of these emails, how do you recover from the fault and help the user send the rest of the email (without duplicating outbound email)?
    - hderms 8y ago

hestefisk 8y ago

My main gripe with service buses is that they can be very hard to deploy and test automatically. At least for traditional ‘middleware’ like WebSphere/MQ, Weblogic etc. It potentially adds another monolith to your architecture, which, whilst fancy, may not be required. Using ZeroMQ or similar ‘lightweight’ tech could be a better choice for small teams as it is easy to integrate into containers and testable.

Pamar 8y ago

I strongly suggest anyone interested in this type of architecture to read https://www.goodreads.com/search?q=Enterprise+Integration+Patterns%3A+Designing%2C+Building%2C+and+Deploying+Messaging+Solutions https://www.goodreads.com/search?q=Enterprise+Integration+Pa... - it is a really well though out catalog of patterns and solution for Message-based systems.

kbouck 8y ago

I'm curious to hear others' opinions on using a database as a message queue. One issue I have with most message brokers is that you can not perform adhoc queries or manipulations of the queue. When you've got a backlog situation, it's nice to be able to answer questions like: - how many of these queued msgs are for/from customer/partner X.

phoe-krk 8y ago

It's weird for me to not see even a single mention of any BEAM languages, such as Erlang or Elixir. They are naturally distributed and have discovery, networking and messaging built into the virtual machine itself.

scraegg 8y ago

The first sentence already is triggering me. No, it's not hard. Like most topics in software engineering there are 50+ years of pretty successful development, backed by science, backed by software from the 70s, backed by all seniority levels of engineers. The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right? The problem is that something can only be hip for 1-5 years. But really stable complex systems like a distributed network can only be developed in a time frame of 5-10 years. Therefore most of the hip stuff is unfinished or already flawed in its basic architecture. And usually if you switch from one hip stuff to another, you get the same McDonalds software menues, just with differently flavored ketchup and marketing. So if you feel something is hard it might be because you are not dealing with the actual problem. For instance you might think about doing Kafka, and that's fine. But be aware that email is shipping more messages than Kafka and its doing it longer than Kafka. For instance topologies: There is no point-to-point. There's star, meshed, bus etc. See here: https://en.wikipedia.org/wiki/Network_topology https://en.wikipedia.org/wiki/Network_topology If you don't know your topology it might be star or mesh. But it's still a topology or a mix of multiple topologies. And if you develop a distributed system you really need to think about how your network should be structured. Then if you know which topology fits your use case you can go and figure out the way this topology works and what the drawbacks are. Star networks (like k8s) for instance are easy to setup but have a clear bottle neck in the center. A Bus (like Kafka) is like a street. It works fine until it is full, and there are sadly some activity patterns were an overloaded Bus will cascade and the overload is visible even weeks later (although you have reduced the traffic already), if you don't reset it completely from time to time. It's not magic. You can look all of it up on wikipedia if you know the keywords. Also there is not a single "good" solution. It depends always on how well the rest of your system integrates with the pros and cons of your topology choice. And if you use multiple topologies in parallel you have a complexity overhead at some point, which is why working in big corps is usually so slow.

geezerjay 8y ago
> The problem nowadays is that people don't want to learn the proven stuff. People want to learn the new hip framework. Learning Kubernetes and React.js is much more fun than learning how actual routing in TCP/IP works, right? That's an awfully short-sighted comparison. There are far more job offers for deploying and managing systems with kubernetes and to develop front-ends with React than there are for developing TCP/IP infrastructure. It's fun to earn a living and enjoy the priviledges of having a high salary, and the odds you get that by studying solved problems that nowadays just work are not that high.