9 ms·

I reckon your message broker might be a bad idea

63 points by tonyg 13y ago

cmbaus 13y ago
What the author is alluding to is that centralized messages queues can cause catastrophic failure over[1]. Industry knowledge of how to distribute loads has improved significantly in the past 10 years. I wish the author would touch on some other options like using a load balancer with consistent hashing[2]. [1] http://martinfowler.com/bliki/CatastrophicFailover.html http://martinfowler.com/bliki/CatastrophicFailover.html [2] https://en.wikipedia.org/wiki/Consistent_hashing https://en.wikipedia.org/wiki/Consistent_hashing
grumblestumble 13y ago
TL;DR - "bad implementations are bad"
- plainOldText 13y ago
  If you really want to tl;dr it, I would use this last paragraph: "Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour."
Spearchucker 13y ago
It's why, in this agile world, I prefer noobs to use cases to user stories. Most templates I found explicitly prompt you to think about fail conditions. If you're wondering what constitutes being a noob, it's when someone is too inexperienced to think about fail conditions and work them through without being prompted by someone or something, like your app crashing out. Note that I have nothing against inexperience - I used to be too. And noobs are malleable - in that sense they're easier to work with than experienced people.
glenjamin 13y ago
Based on my experience with RabbitMQ, failed messages (ie. unacked due to consumer crash) have a retry count property that is maintained by the broker. If the consumer simply checked if the retry count was above a threshold, and then redirected the message to a dead-letter-exchange, this would solve the "stuck message" problem. As for tracking build up of messages, this is just a case of adding something like a nagios check on size of queue. It seems like the solution outlined in the post is just implementing by hand the features of a decent message broker they chose not to use. Sounds like they managed to get the job done regardless.
- twic 13y ago
  I don't think RabbitMQ has a retry count. It has a retry flag, which tells you if this is the first time a message has gone out for delivery, or an attempt at a redelivery. You can use that to requeue after the first failure, but to reject (and send to a dead-letter queue) after the second. But you can't do anything more than that without manually managing a count. At least, that's the conclusion i came to when i looked at redelivery for an application i work on that uses RabbitMQ. Please do correct me if i'm wrong.
  - glenjamin 13y ago
    You are correct, I was mistaken. https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.deliver https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.del... The spec has a SHOULD for aborting after a retry count, but it doesn't appear to be exposed to clients.
georgemcbay 13y ago
I don't like sniping people about things that are unrelated to the actual point they are trying to get across; but in this case the problem caused me to just quit reading the post after the first paragraph, so.... I reckon the blog author should really reconsider the black on dark-grey color scheme of the blog; contrast is not your enemy when it comes to a simple, text-focused blog. This might look pretty decent on whatever system it is that he is writing it on, but on my PC monitor (which is quite a good one, carefully color calibrated and brightness reduced for editing photos), it is a complete disaster.
- userbinator 13y ago
  View -> CSS -> Off (What, your browser doesn't let you turn off stylesheets? :-)
brassybadger 13y ago
This is the old debate whether applications should be tightly or loosely coupled. As the article concludes, there are tradeoffs to consider - the broker is a SPOF indeed, but on the other hand it'll take care of all the queueing logic (which is not that simple as it seems - acknowledgements, backpressure, redelivery, timeouts etc etc). It's also much easier to scale the system out if applications are loosely coupled. Furthermore, a good broker supports some kind of failover. We run a RabbitMQ cluster behind a load balancer, and it works great - once e.g. we had a hardware failure, and the applications did not notice anything from it. Of course we also have to monitor the queues on the broker (one of the few ways to bring down Rabbit is to fill up its disk). But having a central broker for connecting apps also means you have a central point you can keep an eye on to check where your bottlenecks are (which queues build up). Loosely coupled applications are also much more unix-like in the sense that you can make them do one thing instead of baking in your custom restart logic (over a network? please don't).
- HeyImAlex 13y ago
  I feel like it's an endless battle, trying to move away from a SPOF; even in your rabbitMQ cluster your load balancer is a SPOF. Who watches the watchmen? At a certain point you just have to accept that you're not gaining a whole lot by adding more complexity. Process supervision over a network (a la something like fleet or flynn or mesos) is a pretty neat solution. I've been playing around with this stuff the past week and I'm having a lot of trouble coming up with the perfect setup haha
  - vidarh 13y ago
    A load balancer need (should) not be SPOF: You run two (or more) of them, with IP takeover (e.g. via keepalived or similar, that continuously vote on the master), or you make the clients try to connect to two or more IP's. Or both. It's really quite trivial to ensure your load balancer is not SPOF if you're ok with clients having to reconnect on failure, and only slightly more hassle if you want to allow connections to survive one of your load balancer boxes failing.
twic 13y ago
There's a whole other axis of pain around message queues that isn't mentioned, and that's regulating delivery. If you're pushing messages to workers using HTTP or whatever, then you can regulate delivery at the sender, or at a load balancer in between. If you're pulling messages via an MQ protocol, you can only do it at the receiver. I'm not aware of any message broker that lets you throttle messages on a per-consumer basis, much less one that does this automatically based on load or health checks. Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself. Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick. Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem. Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load. You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work. It seems to be considerably more involved than doing it with HTTP. Basically, using a message queue pushes a concern which should be external to the worker into it, and that creates work that needs to be done. I am coming to think that whilst it's okay for apps to send messages to a queue, they should never receive consume messages from one. Rather, i suspect there should be some small infrastructural component which pulls messages off a queue and sends them to workers with HTTP, via a load balancer. I think this component can follow some simple, generic rules, and leave the smarts in the load balancer as normal. I haven't fully fleshed this idea out yet, though.
- jallmann 13y ago
  It sounds the real problem is there might be a lack of production caliber load balancers for $MQ, rather than HTTP possessing any intrinsic advantage. How is load balancing anything but an implementation detail, since one can usually be written as a service utilizing the same protocol in question? That's how HTTP reverse proxies usually work (notwithstanding other methods of balancing load such as DNS). Other monitoring aspects such as feedback are already built into well-designed MQs. If that feedback isn't being acted upon properly, then that's again a problem with the implementation -- the same could be said for a HTTP-based task queue that simply ignores errors. It's not "extra work"; it's just different.
Gigablah 13y ago
> We’d ignored and subsequently embraced three good design principles for reliable systems: Fail fast, Process Supervision, and the End-to-End principle. Can't you have a message broker AND embrace those design principles... without pushing responsibility onto your crawler? What happened to the Single Responsibility Principle?
joelberman 13y ago
These reckons you mention. Over the years I have come across people suggesting the same approach to a problem. It makes sense in theory and it should work. Yet it doesn't. And I cannot explain why it does not work. Queueing/throttling/steering/pub-sub, caching, and synchronization seem to be three areas that spawn unintended consequences like crazy. If some young'un suggests something you have learned will fail, let them try it if you have the time. And then ask them to explain why it did not work. It's fun watching people learn.
fallous 13y ago
"Used message queue as replacement for RPC service without actually implementing RPC pattern." Key learned from inevitable failure? Message queues are BAD! Except for the part where you weren't actually testing an MQ, you were trying to use an MQ as an RPC service without actually solving the problem and instead insisted on building an RPC service with none of the failure-tolerance of a persistent broker. I don't disagree with the OP that this is a common mistake, but the solution offered is more of a "we also made this mistake, so clearly it is a problem with MQs and not us!" which isn't actually true. Additionally, MQs are really (IMO) a better fit for systems that either have a measurable level of complexity on writes or are lightly written but heavily read. You see the same general pattern failure with people who insist that RDBMS is The Answer(tm) even though the data they write/read isn't relational and often read-heavy.
- fallous 13y ago
  I want to make clear that I'm not intentionally faulting the OP in this situation, because the decision with regards to message brokers isn't easy in a complex system. Often you have to mix "lossy" messages (pubsub) with simple ack messages (receipt confirmation) as well as full RPC-style messaging (receipt ack, result response) and that is usually a non-trivial endeavor from both an architectural as well as implementation standpoint.
csense 13y ago
Every time I've looked at message brokers, I've been struck by how over-engineered they seem to be. For e.g. the distributed crawler example he gives, I'd rather write a database-backed RESTful service myself than use something like RabbitMQ, because with the former at least I have a nonzero chance of understanding what the hell it's doing and what the potential failure modes are. Whereas every message queue system I've ever seen has incredibly complex documentation, and you have to study it for quite a while to even figure out what the message broker's model of the problem domain actually is. I don't know enough about message brokers to say for certain they wouldn't be right for certain really complicated commercial sites with extremely high load, where you can afford a full operations team with one or two people who specialize in understanding message broker technology, but for anything smaller than that, to me it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology.
- userbinator 13y ago
  They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them. Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.
  - bananas 13y ago
    I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems. Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.
vidarh 13y ago
So here's an alternative view of how he could've achieved what he wanted: - Use a broker that supports a timeout on the messages. Many do. E.g. beanstalkd lets you reserve a message for a specified time period, otherwise it goes back on the queue automatically. You can let the server knows if you need longer to finish. But a locked process will not hold on to a job. - Use a timeout on the crawlers. A hard timeout like alarm() that causes the process to die if it fails. Run them under a process monitor that'll respawn them. - Have the crawler or a separate system monitor queue depth, and slow down / alert/ throw errors if the queue goes above a certain depth. All of this is trivial. If introducing a broker takes the whole app with it if its fails, then you're doing it wrong: Unless you need in-order guaranteed delivery of messages, making a message queue resilient to failures is trivial: Just add more of them, and let the client talk to all of them. If you need in-order delivery, then a active-passive setup is easy enough too. Put the queue in a synchronously replicated database if you can't afford to lose any of them. So his argument about failures, is an argument against letting "inexperienced programmers" set up systems with single points of failure, not against message brokers. The irony of this is that his solution is that it's great when the process generating the messages to queue can be run on a single, reliable system, by someone who will notice when that system fails and recover from it. But you've now stuffed your crawler full of state, and created a central point of failure. Just what he complained about. The broker solution is great when you want scale - you can multiply the number of brokers. You can multiply the number of machines the workers run on. You can multiply the number of machines processing the crawled documents. It is also great when you want resilience for the same reason: By decoupling, it makes it easier. If you don't need that scale, then go for the simpler solution. But if you do need that scale, then his solution isn't a solution at all. If you introduce a broker, and your queues start growing out of control. Guess what? You have a scaling problem you would likely have without the broker too. Only in a decoupled system you have a much easier way out: Add more machines/VMs with workers; add more brokers. Don't feel you have a way of controlling it? This is what your monitoring and orchestration systems should be doing. Queues over a certain threshold? Trigger an alert, and optionally spin up an exra VM.
- angrybits 13y ago
  "All of this is trivial." No, describing it in English on HN is trivial. Getting all of this to work in the really real world is incredibly difficult and time-consuming. Components and infrastructure will inevitably find novel ways of failing that your three-bullet architecture didn't take into consideration. That said, I agree that this is a problem that can be solved with a message broker. I just don't think it is easy. If you can make a system work with fewer moving parts, then you probably should.

63 points by tonyg 13y ago

What the author is alluding to is that centralized messages queues can cause catastrophic failure over[1]. Industry knowledge of how to distribute loads has improved significantly in the past 10 years. I wish the author would touch on some other options like using a load balancer with consistent hashing[2]. [1] http://martinfowler.com/bliki/CatastrophicFailover.html http://martinfowler.com/bliki/CatastrophicFailover.html [2] https://en.wikipedia.org/wiki/Consistent_hashing https://en.wikipedia.org/wiki/Consistent_hashing

grumblestumble 13y ago

TL;DR - "bad implementations are bad"

plainOldText 13y ago
If you really want to tl;dr it, I would use this last paragraph: "Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour."

Spearchucker 13y ago

It's why, in this agile world, I prefer noobs to use cases to user stories. Most templates I found explicitly prompt you to think about fail conditions. If you're wondering what constitutes being a noob, it's when someone is too inexperienced to think about fail conditions and work them through without being prompted by someone or something, like your app crashing out. Note that I have nothing against inexperience - I used to be too. And noobs are malleable - in that sense they're easier to work with than experienced people.

glenjamin 13y ago

Based on my experience with RabbitMQ, failed messages (ie. unacked due to consumer crash) have a retry count property that is maintained by the broker. If the consumer simply checked if the retry count was above a threshold, and then redirected the message to a dead-letter-exchange, this would solve the "stuck message" problem. As for tracking build up of messages, this is just a case of adding something like a nagios check on size of queue. It seems like the solution outlined in the post is just implementing by hand the features of a decent message broker they chose not to use. Sounds like they managed to get the job done regardless.

twic 13y ago
I don't think RabbitMQ has a retry count. It has a retry flag, which tells you if this is the first time a message has gone out for delivery, or an attempt at a redelivery. You can use that to requeue after the first failure, but to reject (and send to a dead-letter queue) after the second. But you can't do anything more than that without manually managing a count. At least, that's the conclusion i came to when i looked at redelivery for an application i work on that uses RabbitMQ. Please do correct me if i'm wrong.
- glenjamin 13y ago
  You are correct, I was mistaken. https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.deliver https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.del... The spec has a SHOULD for aborting after a retry count, but it doesn't appear to be exposed to clients.

georgemcbay 13y ago

I don't like sniping people about things that are unrelated to the actual point they are trying to get across; but in this case the problem caused me to just quit reading the post after the first paragraph, so.... I reckon the blog author should really reconsider the black on dark-grey color scheme of the blog; contrast is not your enemy when it comes to a simple, text-focused blog. This might look pretty decent on whatever system it is that he is writing it on, but on my PC monitor (which is quite a good one, carefully color calibrated and brightness reduced for editing photos), it is a complete disaster.

userbinator 13y ago
View -> CSS -> Off (What, your browser doesn't let you turn off stylesheets? :-)

brassybadger 13y ago

This is the old debate whether applications should be tightly or loosely coupled. As the article concludes, there are tradeoffs to consider - the broker is a SPOF indeed, but on the other hand it'll take care of all the queueing logic (which is not that simple as it seems - acknowledgements, backpressure, redelivery, timeouts etc etc). It's also much easier to scale the system out if applications are loosely coupled. Furthermore, a good broker supports some kind of failover. We run a RabbitMQ cluster behind a load balancer, and it works great - once e.g. we had a hardware failure, and the applications did not notice anything from it. Of course we also have to monitor the queues on the broker (one of the few ways to bring down Rabbit is to fill up its disk). But having a central broker for connecting apps also means you have a central point you can keep an eye on to check where your bottlenecks are (which queues build up). Loosely coupled applications are also much more unix-like in the sense that you can make them do one thing instead of baking in your custom restart logic (over a network? please don't).

HeyImAlex 13y ago
I feel like it's an endless battle, trying to move away from a SPOF; even in your rabbitMQ cluster your load balancer is a SPOF. Who watches the watchmen? At a certain point you just have to accept that you're not gaining a whole lot by adding more complexity. Process supervision over a network (a la something like fleet or flynn or mesos) is a pretty neat solution. I've been playing around with this stuff the past week and I'm having a lot of trouble coming up with the perfect setup haha
- vidarh 13y ago
  A load balancer need (should) not be SPOF: You run two (or more) of them, with IP takeover (e.g. via keepalived or similar, that continuously vote on the master), or you make the clients try to connect to two or more IP's. Or both. It's really quite trivial to ensure your load balancer is not SPOF if you're ok with clients having to reconnect on failure, and only slightly more hassle if you want to allow connections to survive one of your load balancer boxes failing.

twic 13y ago

There's a whole other axis of pain around message queues that isn't mentioned, and that's regulating delivery. If you're pushing messages to workers using HTTP or whatever, then you can regulate delivery at the sender, or at a load balancer in between. If you're pulling messages via an MQ protocol, you can only do it at the receiver. I'm not aware of any message broker that lets you throttle messages on a per-consumer basis, much less one that does this automatically based on load or health checks. Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself. Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick. Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem. Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load. You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work. It seems to be considerably more involved than doing it with HTTP. Basically, using a message queue pushes a concern which should be external to the worker into it, and that creates work that needs to be done. I am coming to think that whilst it's okay for apps to send messages to a queue, they should never receive consume messages from one. Rather, i suspect there should be some small infrastructural component which pulls messages off a queue and sends them to workers with HTTP, via a load balancer. I think this component can follow some simple, generic rules, and leave the smarts in the load balancer as normal. I haven't fully fleshed this idea out yet, though.

jallmann 13y ago
It sounds the real problem is there might be a lack of production caliber load balancers for $MQ, rather than HTTP possessing any intrinsic advantage. How is load balancing anything but an implementation detail, since one can usually be written as a service utilizing the same protocol in question? That's how HTTP reverse proxies usually work (notwithstanding other methods of balancing load such as DNS). Other monitoring aspects such as feedback are already built into well-designed MQs. If that feedback isn't being acted upon properly, then that's again a problem with the implementation -- the same could be said for a HTTP-based task queue that simply ignores errors. It's not "extra work"; it's just different.

Gigablah 13y ago

> We’d ignored and subsequently embraced three good design principles for reliable systems: Fail fast, Process Supervision, and the End-to-End principle. Can't you have a message broker AND embrace those design principles... without pushing responsibility onto your crawler? What happened to the Single Responsibility Principle?

joelberman 13y ago

These reckons you mention. Over the years I have come across people suggesting the same approach to a problem. It makes sense in theory and it should work. Yet it doesn't. And I cannot explain why it does not work. Queueing/throttling/steering/pub-sub, caching, and synchronization seem to be three areas that spawn unintended consequences like crazy. If some young'un suggests something you have learned will fail, let them try it if you have the time. And then ask them to explain why it did not work. It's fun watching people learn.

fallous 13y ago

"Used message queue as replacement for RPC service without actually implementing RPC pattern." Key learned from inevitable failure? Message queues are BAD! Except for the part where you weren't actually testing an MQ, you were trying to use an MQ as an RPC service without actually solving the problem and instead insisted on building an RPC service with none of the failure-tolerance of a persistent broker. I don't disagree with the OP that this is a common mistake, but the solution offered is more of a "we also made this mistake, so clearly it is a problem with MQs and not us!" which isn't actually true. Additionally, MQs are really (IMO) a better fit for systems that either have a measurable level of complexity on writes or are lightly written but heavily read. You see the same general pattern failure with people who insist that RDBMS is The Answer(tm) even though the data they write/read isn't relational and often read-heavy.

fallous 13y ago
I want to make clear that I'm not intentionally faulting the OP in this situation, because the decision with regards to message brokers isn't easy in a complex system. Often you have to mix "lossy" messages (pubsub) with simple ack messages (receipt confirmation) as well as full RPC-style messaging (receipt ack, result response) and that is usually a non-trivial endeavor from both an architectural as well as implementation standpoint.

csense 13y ago

Every time I've looked at message brokers, I've been struck by how over-engineered they seem to be. For e.g. the distributed crawler example he gives, I'd rather write a database-backed RESTful service myself than use something like RabbitMQ, because with the former at least I have a nonzero chance of understanding what the hell it's doing and what the potential failure modes are. Whereas every message queue system I've ever seen has incredibly complex documentation, and you have to study it for quite a while to even figure out what the message broker's model of the problem domain actually is. I don't know enough about message brokers to say for certain they wouldn't be right for certain really complicated commercial sites with extremely high load, where you can afford a full operations team with one or two people who specialize in understanding message broker technology, but for anything smaller than that, to me it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology.

userbinator 13y ago
They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them. Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.
- bananas 13y ago
  I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems. Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.

vidarh 13y ago

So here's an alternative view of how he could've achieved what he wanted: - Use a broker that supports a timeout on the messages. Many do. E.g. beanstalkd lets you reserve a message for a specified time period, otherwise it goes back on the queue automatically. You can let the server knows if you need longer to finish. But a locked process will not hold on to a job. - Use a timeout on the crawlers. A hard timeout like alarm() that causes the process to die if it fails. Run them under a process monitor that'll respawn them. - Have the crawler or a separate system monitor queue depth, and slow down / alert/ throw errors if the queue goes above a certain depth. All of this is trivial. If introducing a broker takes the whole app with it if its fails, then you're doing it wrong: Unless you need in-order guaranteed delivery of messages, making a message queue resilient to failures is trivial: Just add more of them, and let the client talk to all of them. If you need in-order delivery, then a active-passive setup is easy enough too. Put the queue in a synchronously replicated database if you can't afford to lose any of them. So his argument about failures, is an argument against letting "inexperienced programmers" set up systems with single points of failure, not against message brokers. The irony of this is that his solution is that it's great when the process generating the messages to queue can be run on a single, reliable system, by someone who will notice when that system fails and recover from it. But you've now stuffed your crawler full of state, and created a central point of failure. Just what he complained about. The broker solution is great when you want scale - you can multiply the number of brokers. You can multiply the number of machines the workers run on. You can multiply the number of machines processing the crawled documents. It is also great when you want resilience for the same reason: By decoupling, it makes it easier. If you don't need that scale, then go for the simpler solution. But if you do need that scale, then his solution isn't a solution at all. If you introduce a broker, and your queues start growing out of control. Guess what? You have a scaling problem you would likely have without the broker too. Only in a decoupled system you have a much easier way out: Add more machines/VMs with workers; add more brokers. Don't feel you have a way of controlling it? This is what your monitoring and orchestration systems should be doing. Queues over a certain threshold? Trigger an alert, and optionally spin up an exra VM.

angrybits 13y ago
"All of this is trivial." No, describing it in English on HN is trivial. Getting all of this to work in the really real world is incredibly difficult and time-consuming. Components and infrastructure will inevitably find novel ways of failing that your three-bullet architecture didn't take into consideration. That said, I agree that this is a problem that can be solved with a message broker. I just don't think it is easy. If you can make a system work with fewer moving parts, then you probably should.