6 ms·

After reading Enterprise Integration Patterns [1] a few years ago, I've been under the impression that message queues are the holy grail for decoupling systems

After reading Enterprise Integration Patterns [1] a few years ago, I've been under the impression that message queues are the holy grail for decoupling systems that you want to pull apart, but this offers a really nice perspective on where they might not be a panacea. Thank you! There's a couple of nice ideas in here (the "message pump" for one at least) that I'm going to steal and use at work. It's also comforting to know that we're not completely crazy for using a DB to store tasks/processes that need to be run and retried if necessary instead of a message queue. [1] https://en.wikipedia.org/wiki/Enterprise_Integration_Patterns https://en.wikipedia.org/wiki/Enterprise_Integration_Pattern...

balfirevic 9y ago
I know what you meant by saying you use DB _instead_ of message queue but I think it kind of points to a broader point - message queue is a concept and the technology used to implement it is just implementation detail. It just turns out that databases have all kinds of useful and powerful operations that make implementing message queues relatively easy.
- beat 9y ago
  The reason to use message queues rather than RDBMS for queue behaviors isn't because queues do stuff databases don't, but rather because of connection cost. There's a lot of complicated handling noise around the database connection, whereas queues tend to be very inexpensive.
  - balfirevic 9y ago
    Do you mean development/complexity cost or performance cost? If it is the former, then I don't really see it. If it is the latter then yes - RDBMS backed queue is probably going to hit scaling limits earlier then dedicated product.
  - jsight 9y ago
    As an alternate anecdote, I have had far more issues with MQ connections than with RDBMS connections. I don't think there is a clear advantage to messaging from that standpoint.
- coredog64 9y ago
  Having been on the receiving end of more than a few apps where the devs used a database as a queue, I say "please don't". Yes, Oracle's AQ product is built on top of an RDBMS. And yes, early versions of Microsoft's MSMQ used SQL Server. But most dev teams use the database as a shortcut and don't invest the effort in making their hacktastic DB queue work like a real product.
  - balfirevic 9y ago
    I have used DB as queues very successfully in multiple projects (as well as "real" queues in others), so your "please don't" is not very persuasive. Some actual argument, instead of content-free "hacktastic" vs. "real product" designations would be appreciated. And you use "shortcut" like some kind of negative attribute. Of course they use DB as a shortcut, and they should - it works well and even has additional features (transactions with your primary store) that you wouldn't get with separate queue product.
    - FLUX-YOU 9y ago
      Monitoring and administration are more difficult until you write them yourself, especially if you are splitting monolithic code. Many of the message queue products you download include these. Can your DB message queue also support different queue concepts and message routing?
      - balfirevic 9y ago
        In my case it supported multiple queues (also, a concept of hierarchical queues), but no message routing. As for administration and monitoring - you are correct, you don't get it out of the box. Well, you do get some basic administration for free from whoever administers your database :) On the other hand, you can grow solution for those organically over time to fit your needs.
  - rangersanger 9y ago
    In a previous life I was on a team that moved a ecomm backend-Email Service Provider(ESP) from a postgres based queue to rabbitMQ. The motivation was voiced in language similar to yours, "hacktasic" was actually used more than once as a reason to replace the DB with rabbitMQ. In the end, I'd say they were equally reliable, but from a data analyst's perspective I very much missed the postgres based queue. I think the mechanics inherent to a database solution prompted the original implementers to keep messages around over time vs the mechanics inherent to an MQ solution where they became ephemeral (on the senders side.) Having access to those messages was a treasure trove for troubleshooting and for analytics. That and from a pure cost benefit stance, sending 10-20k messages a day to the ESP definitely didn't necessitate the expensive rearchitecture, as the DB solution was more than capable of handling the load on cheap hardware.
- vkjv 9y ago
  If you don't need high throughput and otherwise don't have a reason to add a separate technology to the stack just to have a queue, SQL can make a great, simple and reliable queue! * Postgresql and Oracle with skip locked * MS SQL Server with readpast * DB2 with skip locked data If you use MySQL, it's going to be a bit more difficult.
  - burkemw3 9y ago
    I hadn't heard of skip locked before, and very much enjoyed this article walking through it for postgres: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/ https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...
    - swah 9y ago
      Love the headline "Most work queue implementations in SQL are wrong". That kind of stuff makes me read an article... This short screencast was also very helpful https://www.pgcasts.com/episodes/7/skip-locked/ https://www.pgcasts.com/episodes/7/skip-locked/
  - monksy 9y ago
    Using the database as a queue is an anti-pattern. There used to be an entire blog series about why this is the case. However, I can't find it at the moment.
  - eternalvision 9y ago
    With MySQL, there's a fairly simple trick you can do to build a queue. Set a temp var to null, run an update with a select subquery to update state of next row(s), then select that var to get your queue row pk's.
swah 9y ago
> message queues are the holy grail for decoupling systems I also like the idea, but the counterpoint is: "now you have to monitor the queue". (I only work in small apps for my own use so its more like Postgres and forget...)
- mattwad 9y ago
  you mean now you CAN monitor the queue... it's been very helpful for us to just throw alerts on queue size in AWS Simple Queue Service.. but we have some apps keeping state in Mongo and you have to write a custom script each time to query and send data to Cloudwatch
hvidgaard 9y ago
Using message queues to decouple the components suffer the same problem as micro services. You start to get a lot of implicit dependencies that you must document.
- i_cant_speel 9y ago
  What kind of implicit dependencies?
  - hvidgaard 9y ago
    Service A sends a message to Service B, so Service A is dependent on Service B to function properly. With careful management, it can be kept to a minimum, but I haven't seen that work for systems with a large numbers of developers working on it yet.
    - i_cant_speel 9y ago
      But I was under the impression that the point of message brokering is that it doesn't have to be service A that puts it there; only that some service puts the message there. I feel like the article is attacking message brokering by discussing the disadvantages of bad use cases for them. A good use case for message brokers is when work needs to be done on an item, but not immediately. My company uses them in a way I believe is quite effective. We pull in data from an external source, and send the ID of the item to about five different queues to do different tasks. Each time one of the queues finishes the work, it send a message to a validator that checks to see if all the work is done. If it isn't, it waits to get that message again from another worker. If it is, it marks that item as ready for the end user. These are the kinds of use cases I think message brokers should be used for. Not to send a message and wait to get an answer back. Why not just use an HTTP request for that?
      - hvidgaard 9y ago
        > But I was under the impression that the point of message brokering is that it doesn't have to be service A that puts it there; only that some service puts the message there. It doesn't matter, but it's more about debugging. If Service A does not work correctly, where is the bug? Is i Service A? Service B? The network? The use at your company, is idiomatic to the paradigm. You have n different units of work, that can run seperately, so you do that and communicate with messages.
deleted 9y ago
[deleted]

balfirevic 9y ago

I know what you meant by saying you use DB _instead_ of message queue but I think it kind of points to a broader point - message queue is a concept and the technology used to implement it is just implementation detail. It just turns out that databases have all kinds of useful and powerful operations that make implementing message queues relatively easy.

beat 9y ago
The reason to use message queues rather than RDBMS for queue behaviors isn't because queues do stuff databases don't, but rather because of connection cost. There's a lot of complicated handling noise around the database connection, whereas queues tend to be very inexpensive.
- balfirevic 9y ago
  Do you mean development/complexity cost or performance cost? If it is the former, then I don't really see it. If it is the latter then yes - RDBMS backed queue is probably going to hit scaling limits earlier then dedicated product.
- jsight 9y ago
  As an alternate anecdote, I have had far more issues with MQ connections than with RDBMS connections. I don't think there is a clear advantage to messaging from that standpoint.
coredog64 9y ago
Having been on the receiving end of more than a few apps where the devs used a database as a queue, I say "please don't". Yes, Oracle's AQ product is built on top of an RDBMS. And yes, early versions of Microsoft's MSMQ used SQL Server. But most dev teams use the database as a shortcut and don't invest the effort in making their hacktastic DB queue work like a real product.
- balfirevic 9y ago
  I have used DB as queues very successfully in multiple projects (as well as "real" queues in others), so your "please don't" is not very persuasive. Some actual argument, instead of content-free "hacktastic" vs. "real product" designations would be appreciated. And you use "shortcut" like some kind of negative attribute. Of course they use DB as a shortcut, and they should - it works well and even has additional features (transactions with your primary store) that you wouldn't get with separate queue product.
  - FLUX-YOU 9y ago
    Monitoring and administration are more difficult until you write them yourself, especially if you are splitting monolithic code. Many of the message queue products you download include these. Can your DB message queue also support different queue concepts and message routing?
    - balfirevic 9y ago
      In my case it supported multiple queues (also, a concept of hierarchical queues), but no message routing. As for administration and monitoring - you are correct, you don't get it out of the box. Well, you do get some basic administration for free from whoever administers your database :) On the other hand, you can grow solution for those organically over time to fit your needs.
- rangersanger 9y ago
  In a previous life I was on a team that moved a ecomm backend-Email Service Provider(ESP) from a postgres based queue to rabbitMQ. The motivation was voiced in language similar to yours, "hacktasic" was actually used more than once as a reason to replace the DB with rabbitMQ. In the end, I'd say they were equally reliable, but from a data analyst's perspective I very much missed the postgres based queue. I think the mechanics inherent to a database solution prompted the original implementers to keep messages around over time vs the mechanics inherent to an MQ solution where they became ephemeral (on the senders side.) Having access to those messages was a treasure trove for troubleshooting and for analytics. That and from a pure cost benefit stance, sending 10-20k messages a day to the ESP definitely didn't necessitate the expensive rearchitecture, as the DB solution was more than capable of handling the load on cheap hardware.
vkjv 9y ago
If you don't need high throughput and otherwise don't have a reason to add a separate technology to the stack just to have a queue, SQL can make a great, simple and reliable queue! * Postgresql and Oracle with skip locked * MS SQL Server with readpast * DB2 with skip locked data If you use MySQL, it's going to be a bit more difficult.
- burkemw3 9y ago
  I hadn't heard of skip locked before, and very much enjoyed this article walking through it for postgres: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/ https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...
  - swah 9y ago
    Love the headline "Most work queue implementations in SQL are wrong". That kind of stuff makes me read an article... This short screencast was also very helpful https://www.pgcasts.com/episodes/7/skip-locked/ https://www.pgcasts.com/episodes/7/skip-locked/
- monksy 9y ago
  Using the database as a queue is an anti-pattern. There used to be an entire blog series about why this is the case. However, I can't find it at the moment.
- eternalvision 9y ago
  With MySQL, there's a fairly simple trick you can do to build a queue. Set a temp var to null, run an update with a select subquery to update state of next row(s), then select that var to get your queue row pk's.

swah 9y ago

> message queues are the holy grail for decoupling systems I also like the idea, but the counterpoint is: "now you have to monitor the queue". (I only work in small apps for my own use so its more like Postgres and forget...)

mattwad 9y ago
you mean now you CAN monitor the queue... it's been very helpful for us to just throw alerts on queue size in AWS Simple Queue Service.. but we have some apps keeping state in Mongo and you have to write a custom script each time to query and send data to Cloudwatch

hvidgaard 9y ago

Using message queues to decouple the components suffer the same problem as micro services. You start to get a lot of implicit dependencies that you must document.

i_cant_speel 9y ago
What kind of implicit dependencies?
- hvidgaard 9y ago
  Service A sends a message to Service B, so Service A is dependent on Service B to function properly. With careful management, it can be kept to a minimum, but I haven't seen that work for systems with a large numbers of developers working on it yet.
  - i_cant_speel 9y ago
    But I was under the impression that the point of message brokering is that it doesn't have to be service A that puts it there; only that some service puts the message there. I feel like the article is attacking message brokering by discussing the disadvantages of bad use cases for them. A good use case for message brokers is when work needs to be done on an item, but not immediately. My company uses them in a way I believe is quite effective. We pull in data from an external source, and send the ID of the item to about five different queues to do different tasks. Each time one of the queues finishes the work, it send a message to a validator that checks to see if all the work is done. If it isn't, it waits to get that message again from another worker. If it is, it marks that item as ready for the end user. These are the kinds of use cases I think message brokers should be used for. Not to send a message and wait to get an answer back. Why not just use an HTTP request for that?
    - hvidgaard 9y ago
      > But I was under the impression that the point of message brokering is that it doesn't have to be service A that puts it there; only that some service puts the message there. It doesn't matter, but it's more about debugging. If Service A does not work correctly, where is the bug? Is i Service A? Service B? The network? The use at your company, is idiomatic to the paradigm. You have n different units of work, that can run seperately, so you do that and communicate with messages.

deleted 9y ago

[deleted]