8 ms·

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Night.pdf https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-not-the-right-tool-for-a-queue-based-system.html https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/351449/message-queue-database-vs-dedicated-mq https://softwareengineering.stackexchange.com/questions/3514... I wonder if the designers of this system disagree, or if they did anything in particular to mitigate these issues?

bradstewart 7y ago
It depends heavily on what exactly you need out of a message queue. E.g. if extremely (extremely) high throughput is required and you don't mind dropped or lost messages, SQL may be a bad choice. But I've had a lot of success using Postgres-backed messaging. Also your links are (mostly) from message queue companies who have a vested interest in your not using Postgres, or are from a time before LISTEN/NOTIFY and JSON(B) columns.
mumblemumble 7y ago
There is definitely a performance ceiling if you use the RDMBS as your event store. I used to worry about that. Now that I've got a few years working at a very successful high frequency trading firm that did message queues in an RDBMS behind me, I don't worry about it so much anymore. They didn't use it for everything; there was also a blindingly efficient homegrown messaging system for the stuff that needed to be really fast. But that was for fairly well-defined use cases. (Latency and concurrency were the key limiting factors.) By default, everything went into the RDBMS-based system. It was inherently more robust, it replaced a whole quagmire of design questions you have to make when using other messaging systems with ACID guarantees, and it made it a lot quicker & easier to do post-mortem analysis of any surprising things that happened in production.
ethangarofolo 7y ago
In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints (https://martinfowler.com/articles/microservices.html#SmartEndpointsAndDumbPipes https://martinfowler.com/articles/microservices.html#SmartEn...). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe" side of the divide. Message DB is a message store, a database optimized for storing message data. The database doesn't track the status of anything---that responsibility falls to consumers of the data. Message queues are fine pieces of technology when what you need is a message queue. Message DB, on the other hand, is used for event-sourced systems. "Event-sourced" in turn differs from merely "event-based." Since I've started building event-sourced systems, I haven't run across the need for message queues, but ymmv, and of course, like all humans, I too have my own hammer/nail biases.
- SideburnsOfDoom 7y ago
  > Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement there? or are you suggesting that such a microservice's first action on receiving a message from a queue would be to store it a local, private Message DB? That might make sense, I've seen enough tables that store a JSON blob already.
  - mumblemumble 7y ago
    Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that. It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.
sbellware 7y ago
Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be pointed out that any message queue with persistent messaging is built on a database. Even RabbitMQ's persistent messaging is built on a database (Mnesia). In the end, a message store or event store can be used to model message queues, but the opposite is rarely true. One of the creators of the Event Store database once said to me something like, "What's a queue but a degenerate form of a stream". A message store provides for patterns that are above and beyond message queues, like event sourcing, for example. Like Ethan mentioned, it's an application of "dumb pipes". It's in the same vein as Event Store or Kafka, rather than RabbitMQ, ActiveMQ, etc. The critical difference is durability of messages. If you have an application that doesn't require durability of messages, then a plain old message queue or message broker technology may be a better choice for the situation. In the end, polling is totally fine and totally manageable. What matters is that it's not done naively; that batching is intelligent and polling is only optionally triggered in the right circumstances and tuned based on batch processing cycle time and message arrival time. Polling doesn't mean that the database will be "hammered" unless it's implemented that way.
- SideburnsOfDoom 7y ago
  > Sooner or later, in anything built with a computer, there's going to be polling. ... But under the hood, there's polling. Is that true? I am reminded of this essay that goes into a deep dive of what happens under the hood with http and the underlying network I/O https://blog.stephencleary.com/2013/11/there-is-no-thread.html https://blog.stephencleary.com/2013/11/there-is-no-thread.ht... And at the lowest level, the network card interrupts the CPU because it has finished reading or writing data. > Some time after the write request started, the device finishes writing. It notifies the CPU via an interrupt. Is that polling? It seems more like a push.
  - sbellware 7y ago

bradstewart 7y ago

It depends heavily on what exactly you need out of a message queue. E.g. if extremely (extremely) high throughput is required and you don't mind dropped or lost messages, SQL may be a bad choice. But I've had a lot of success using Postgres-backed messaging. Also your links are (mostly) from message queue companies who have a vested interest in your not using Postgres, or are from a time before LISTEN/NOTIFY and JSON(B) columns.

mumblemumble 7y ago

There is definitely a performance ceiling if you use the RDMBS as your event store. I used to worry about that. Now that I've got a few years working at a very successful high frequency trading firm that did message queues in an RDBMS behind me, I don't worry about it so much anymore. They didn't use it for everything; there was also a blindingly efficient homegrown messaging system for the stuff that needed to be really fast. But that was for fairly well-defined use cases. (Latency and concurrency were the key limiting factors.) By default, everything went into the RDBMS-based system. It was inherently more robust, it replaced a whole quagmire of design questions you have to make when using other messaging systems with ACID guarantees, and it made it a lot quicker & easier to do post-mortem analysis of any surprising things that happened in production.

ethangarofolo 7y ago

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints (https://martinfowler.com/articles/microservices.html#SmartEndpointsAndDumbPipes https://martinfowler.com/articles/microservices.html#SmartEn...). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe" side of the divide. Message DB is a message store, a database optimized for storing message data. The database doesn't track the status of anything---that responsibility falls to consumers of the data. Message queues are fine pieces of technology when what you need is a message queue. Message DB, on the other hand, is used for event-sourced systems. "Event-sourced" in turn differs from merely "event-based." Since I've started building event-sourced systems, I haven't run across the need for message queues, but ymmv, and of course, like all humans, I too have my own hammer/nail biases.

SideburnsOfDoom 7y ago
> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement there? or are you suggesting that such a microservice's first action on receiving a message from a queue would be to store it a local, private Message DB? That might make sense, I've seen enough tables that store a JSON blob already.
- mumblemumble 7y ago
  Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that. It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.

sbellware 7y ago

Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be pointed out that any message queue with persistent messaging is built on a database. Even RabbitMQ's persistent messaging is built on a database (Mnesia). In the end, a message store or event store can be used to model message queues, but the opposite is rarely true. One of the creators of the Event Store database once said to me something like, "What's a queue but a degenerate form of a stream". A message store provides for patterns that are above and beyond message queues, like event sourcing, for example. Like Ethan mentioned, it's an application of "dumb pipes". It's in the same vein as Event Store or Kafka, rather than RabbitMQ, ActiveMQ, etc. The critical difference is durability of messages. If you have an application that doesn't require durability of messages, then a plain old message queue or message broker technology may be a better choice for the situation. In the end, polling is totally fine and totally manageable. What matters is that it's not done naively; that batching is intelligent and polling is only optionally triggered in the right circumstances and tuned based on batch processing cycle time and message arrival time. Polling doesn't mean that the database will be "hammered" unless it's implemented that way.

SideburnsOfDoom 7y ago
> Sooner or later, in anything built with a computer, there's going to be polling. ... But under the hood, there's polling. Is that true? I am reminded of this essay that goes into a deep dive of what happens under the hood with http and the underlying network I/O https://blog.stephencleary.com/2013/11/there-is-no-thread.html https://blog.stephencleary.com/2013/11/there-is-no-thread.ht... And at the lowest level, the network card interrupts the CPU because it has finished reading or writing data. > Some time after the write request started, the device finishes writing. It notifies the CPU via an interrupt. Is that polling? It seems more like a push.
- sbellware 7y ago