4 ms·

Every time I've looked at message brokers, I've been struck by how over-engineered they seem to be. For e.g. the distributed crawler example he gives, I'd rath

by csense 13y ago

Every time I've looked at message brokers, I've been struck by how over-engineered they seem to be. For e.g. the distributed crawler example he gives, I'd rather write a database-backed RESTful service myself than use something like RabbitMQ, because with the former at least I have a nonzero chance of understanding what the hell it's doing and what the potential failure modes are. Whereas every message queue system I've ever seen has incredibly complex documentation, and you have to study it for quite a while to even figure out what the message broker's model of the problem domain actually is. I don't know enough about message brokers to say for certain they wouldn't be right for certain really complicated commercial sites with extremely high load, where you can afford a full operations team with one or two people who specialize in understanding message broker technology, but for anything smaller than that, to me it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology.

userbinator 13y ago
They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them. Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.
- bananas 13y ago
  I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems. Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.
  - jdaley 13y ago
    Do you find that the message broker makes that implementation easier/better than just using a database table to store the incoming messages until they can be processed? I've worked on enterprise products, off-the-shelf and in-house, which use message brokers, and every time I could not fathom how the extra complexity was justified. If the messages were just queued in a simple database table instead, the implementations would have been much easier to work with: easier setup and admin (it's just part of the existing database), easier to check and monitor (just select counts from the table), easier to inspect the pending messages and fix problematic messages. In all cases I get the feeling the architects were just looking for an excuse to use a new technology. I can only see high-load scenarios as a possible justification, where the volume of incoming message data is physically too much for the RDMS.
    - vidarh 13y ago
      Many message brokers do use or support using a database table to store the messages. What they tend to add is ease of scalability and failover, and higher level functionality such as some of what the author of the article complained about, like automatically marking the message as available again if the worker that was processed it did not give any liveness sign within X seconds to deal with hung workers etc.
    - bananas 13y ago
      A database table is just a storage medium. The programming model is what is important. Some tasks don't fit the imperative model very well, some tasks have transactions open for days, some tasks need correlation across hundreds of different integrations, some tasks need to happen in guaranteed windows after initiated, some tasks need to be parallelised across lots of systems. These are a few cases I can think of. From a current position perspective, we use MQs for quoting. Someone raises a quote, this is parallelised across 20 providers all with different integration methods, transformed back to a common object model, inserted as a single transaction into a database table, handed over to another system as an integration task which calculates risk, posted back from that to our system, a risk model is applied, the data is updated, the user is notified. This is all plugged into a 2 million line java and c# jumble that evolved over 15 years from a C++/COM nightmare. We rewrote it with full test coverage in a month. We can add a new provider in an hour. Imperative/table integration. No thanks! People who distrust this methodology need to grab a copy of Enterprise Integration Patterns. There's a huge amount of wisdom in that book that puts Joe blogg's average dynamic language task queue to shame. Without wishing to use the phrase, but enterprise software is about a hell of a lot of stuff people don't understand. This is probably 1% of it.
uniclaude 13y ago
> it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology This is not because you do not understand how message broker works that they are not simple. Message brokers implement "reliable, well-understood" protocols, and you do not need to have engineers who "specialize in understanding message broker technology". This is downright FUD. The little complexity of those protocols is here for a reason, and, seriously, we're not talking Enterprise Java complexity. It's just like saying that you could use bash scripts to automate your infrastructure unless you have Google scale, because Puppet, Chef, or Ansible are too complex and over-engineered. Writing a database-backed RESTful service for a message queue is not without reminding me of those who reimplement RDBMS over MongoDB because RDBMSes are "over engineered". If the AMQP protocol is too complex for you, there are other, simple protocols that message brokers implement.
vidarh 13y ago
Message brokers can be very simple. My first production Ruby app was a message broker that took all of about 700 lines of code. An indeed I wrote it because I'd rather deal with something I knew intimately than deal with one of the over-engineered brokers. But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it. Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)

by csense 13y ago

userbinator 13y ago

They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them. Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.

bananas 13y ago
I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems. Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.
- jdaley 13y ago
  Do you find that the message broker makes that implementation easier/better than just using a database table to store the incoming messages until they can be processed? I've worked on enterprise products, off-the-shelf and in-house, which use message brokers, and every time I could not fathom how the extra complexity was justified. If the messages were just queued in a simple database table instead, the implementations would have been much easier to work with: easier setup and admin (it's just part of the existing database), easier to check and monitor (just select counts from the table), easier to inspect the pending messages and fix problematic messages. In all cases I get the feeling the architects were just looking for an excuse to use a new technology. I can only see high-load scenarios as a possible justification, where the volume of incoming message data is physically too much for the RDMS.
  - vidarh 13y ago
    Many message brokers do use or support using a database table to store the messages. What they tend to add is ease of scalability and failover, and higher level functionality such as some of what the author of the article complained about, like automatically marking the message as available again if the worker that was processed it did not give any liveness sign within X seconds to deal with hung workers etc.
  - bananas 13y ago
    A database table is just a storage medium. The programming model is what is important. Some tasks don't fit the imperative model very well, some tasks have transactions open for days, some tasks need correlation across hundreds of different integrations, some tasks need to happen in guaranteed windows after initiated, some tasks need to be parallelised across lots of systems. These are a few cases I can think of. From a current position perspective, we use MQs for quoting. Someone raises a quote, this is parallelised across 20 providers all with different integration methods, transformed back to a common object model, inserted as a single transaction into a database table, handed over to another system as an integration task which calculates risk, posted back from that to our system, a risk model is applied, the data is updated, the user is notified. This is all plugged into a 2 million line java and c# jumble that evolved over 15 years from a C++/COM nightmare. We rewrote it with full test coverage in a month. We can add a new provider in an hour. Imperative/table integration. No thanks! People who distrust this methodology need to grab a copy of Enterprise Integration Patterns. There's a huge amount of wisdom in that book that puts Joe blogg's average dynamic language task queue to shame. Without wishing to use the phrase, but enterprise software is about a hell of a lot of stuff people don't understand. This is probably 1% of it.

uniclaude 13y ago

> it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology This is not because you do not understand how message broker works that they are not simple. Message brokers implement "reliable, well-understood" protocols, and you do not need to have engineers who "specialize in understanding message broker technology". This is downright FUD. The little complexity of those protocols is here for a reason, and, seriously, we're not talking Enterprise Java complexity. It's just like saying that you could use bash scripts to automate your infrastructure unless you have Google scale, because Puppet, Chef, or Ansible are too complex and over-engineered. Writing a database-backed RESTful service for a message queue is not without reminding me of those who reimplement RDBMS over MongoDB because RDBMSes are "over engineered". If the AMQP protocol is too complex for you, there are other, simple protocols that message brokers implement.

vidarh 13y ago

Message brokers can be very simple. My first production Ruby app was a message broker that took all of about 700 lines of code. An indeed I wrote it because I'd rather deal with something I knew intimately than deal with one of the over-engineered brokers. But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it. Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)