4 ms·

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microser

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints (https://martinfowler.com/articles/microservices.html#SmartEndpointsAndDumbPipes https://martinfowler.com/articles/microservices.html#SmartEn...). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe" side of the divide. Message DB is a message store, a database optimized for storing message data. The database doesn't track the status of anything---that responsibility falls to consumers of the data. Message queues are fine pieces of technology when what you need is a message queue. Message DB, on the other hand, is used for event-sourced systems. "Event-sourced" in turn differs from merely "event-based." Since I've started building event-sourced systems, I haven't run across the need for message queues, but ymmv, and of course, like all humans, I too have my own hammer/nail biases.

SideburnsOfDoom 7y ago
> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement there? or are you suggesting that such a microservice's first action on receiving a message from a queue would be to store it a local, private Message DB? That might make sense, I've seen enough tables that store a JSON blob already.
- mumblemumble 7y ago
  Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that. It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.
  - sbellware 7y ago
    > The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. Indeed. Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/ https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu... At the risk of beating the point to death, the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. In the microservices era, the basic presumption is that the transport has no knowledge of the state of the message processors' progress through a queue.
    - SideburnsOfDoom 7y ago
      > the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. That doesn't to seem to be characteristic of Microservices at all. It's more about small bounded context and independent deployment of the service. All of which is completely possible over the "smart pipes" that message queues that I mentioned will give you. Insisting that this wheel must be re-invented or worked around to make it a microservice seems very odd.
      - hmeh 7y ago
        > That doesn't to seem to be characteristic of Microservices at all. Many have their own "unique" definition, which is why "microservices" is bordering on no longer meaning anything. Some people say it and mean SOA—the characteristics you mention are pretty much characteristics of SOA when done right, though independent deployment isn't a necessity. Martin Fowler attempted to codify a definition, and this is inline with the definition that sbellware is referring to. He (and others) refer specifically to smart endpoints and dumb pipes: https://martinfowler.com/articles/microservices.html#SmartEndpointsAndDumbPipes https://martinfowler.com/articles/microservices.html#SmartEn...
        7y ago
- ethangarofolo 7y ago
  100% with you that messages are a better way for microservices to receive their input. I don't see how a service could receive its input over HTTP and still retain its autonomy. A message store like Message DB can at the same time serve as record of system state and communication channel. Writing a message to the store is the same as publishing it for consumers to pick up. Messages are written to streams, and interested parties subscribe to those streams by polling them for new messages. The store itself isn't aware of which components are subscribing or managing their read positions. So to your question, the first action of a component receiving a message is do whatever it does to handle the message and then write new messages (specifically events) to record and signal what happened in response to the original message.

SideburnsOfDoom 7y ago

> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement there? or are you suggesting that such a microservice's first action on receiving a message from a queue would be to store it a local, private Message DB? That might make sense, I've seen enough tables that store a JSON blob already.

mumblemumble 7y ago
Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that. It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.
- sbellware 7y ago
  > The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. Indeed. Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/ https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu... At the risk of beating the point to death, the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. In the microservices era, the basic presumption is that the transport has no knowledge of the state of the message processors' progress through a queue.
  - SideburnsOfDoom 7y ago
    > the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. That doesn't to seem to be characteristic of Microservices at all. It's more about small bounded context and independent deployment of the service. All of which is completely possible over the "smart pipes" that message queues that I mentioned will give you. Insisting that this wheel must be re-invented or worked around to make it a microservice seems very odd.
    - hmeh 7y ago
      > That doesn't to seem to be characteristic of Microservices at all. Many have their own "unique" definition, which is why "microservices" is bordering on no longer meaning anything. Some people say it and mean SOA—the characteristics you mention are pretty much characteristics of SOA when done right, though independent deployment isn't a necessity. Martin Fowler attempted to codify a definition, and this is inline with the definition that sbellware is referring to. He (and others) refer specifically to smart endpoints and dumb pipes: https://martinfowler.com/articles/microservices.html#SmartEndpointsAndDumbPipes https://martinfowler.com/articles/microservices.html#SmartEn...
      - 7y ago
ethangarofolo 7y ago
100% with you that messages are a better way for microservices to receive their input. I don't see how a service could receive its input over HTTP and still retain its autonomy. A message store like Message DB can at the same time serve as record of system state and communication channel. Writing a message to the store is the same as publishing it for consumers to pick up. Messages are written to streams, and interested parties subscribe to those streams by polling them for new messages. The store itself isn't aware of which components are subscribing or managing their read positions. So to your question, the first action of a component receiving a message is do whatever it does to handle the message and then write new messages (specifically events) to record and signal what happened in response to the original message.