5 ms·

> Why use a database as a queue? Already have a central, configured and monitored server and need "just a small queue" for something. This is not per se a bad

by dom0 9y ago

> Why use a database as a queue? Already have a central, configured and monitored server and need "just a small queue" for something. This is not per se a bad decision. For the same reason it doesn't have to be a bad idea to cache things in the main database, instead of using a dedicated cache like Redis.

IgorPartola 9y ago
This. Also, things like RabbitMQ are complex and their durability properties can be different than those provided by your RDBMS. This can get problematic if you are mixing tasks in the queue that have different priorities. For example, emailing the invoice to a client should not fail silently and should happen at most once. Same with a notification from your doctor that you need to call to discuss your test results. Tossing that into an ephemeral queue is probably not the best solution. Having said that, RabbitMQ does have a ton of settings where you can turn durability up/down as much as you want.
- wbl 9y ago
  Email is unreliable. You should number invoices, and tell recipients to ignore duplicates.
BurningFrog 9y ago
In my experience, the cost of adding and maintaining another storage subsystem to a project is often hugely underestimated. It's easy to see the benefits and ignore the costs. If I can solve a problem reasonably well by adding a table to my Postgres DB, that will always beat out adding the specialized ScrewdriverDB that does it perfectly.
- mbell 9y ago
  Depends on what you're adding. Running Redis is dead simple and easy to have visibility into. RabbitMQ / Kafka are much larger undertakings.
  - brianwawok 9y ago
    RabbitMQ is on docker, its more or less the same work to launch as Redis these days. (It can take a bit more tuning so I think it is unfair to say it is the SAME work, but it is seriously not a huge deal to run RabbitMQ in the post docker world)
    - true_religion 9y ago
      Is Docker really necessary for this? I've never had any trouble just installing RabbitMQ from the package manager, and running it with only a tiny bit of initial configuration.
      - brianwawok 9y ago
        Well, if for example your entire infrastructure is in K8s, then yes you kind of need docker. Redis vs RabbitMQ stateful-set.yaml would be me or less the same, and something like 20 lines of yaml. I was never a big fan of docker for running some thing on a server, but once you start to go down the K8s path, it finally makes sense.
  - LoSboccacc 9y ago
    Depends on the scale mostly. People often forget how not every project is Google/Facebook/Twitter.
  - derefr 9y ago
    If you already had a need for a durable database, and so you properly implemented Postgres streaming-archiving + disaster recovery, at much personal effort for your tiny startup... and you now need a durable queue as well... then "just installing Redis" won't get you there. You'll need to that whole ops setup all over again for Redis. Whereas, if your queue is in Postgres, the ops problem is already solved. If you have a full-time ops staff, throwing another marginal piece of infrastructure on the pile isn't much of an issue. If you're your own "ops staff of one", each moving part of your infrastructure—that you need to ensure works, to guarantee you won't lose client data—is another thing slowing down your iteration speed.
- jpalomaki 9y ago
  I agree. If it sounds simple, then you are probably not thinking hard enough. Think for example how to do backups. If you have a database and completely separate queue system your application state is distributed into two places. This means taking consistent backups is not straightforward. You can of course try to work around this on the application level (database as single source of truth, queues as "nice to have" -thing, but this makes things more complicated for the app).
  - dom0 9y ago
    > This means taking consistent backups is not straightforward. And they're not even straightforward to begin with...
emmelaich 9y ago
Yep. Better asked as "Why use a conventional|relational database as a queue?" Because all queues have to be databases i.e. have the same qualities / assurances as databases.
tyingq 9y ago
It also makes it easier to delete or update the queue entry in an atomic transaction that spans other tables. If that has value for the specific use case.
- Smudge 9y ago
  Very common in the web context -- you perform some form of relational persistence while also inserting a job to schedule background work (like sending an email). Having those both in the same transaction gets rid of a lot of tricky failure cases.
  - ruslan_talpa 9y ago
    you can still have the notion of a transaction while integrating rabbitmq like this you do your mutations in a transaction and also in that transaction you execute a NOTIFY command. If transaction is successful, the notify will go through at the end of the transaction. the notify events and be "bridged" to a messaging server like rabbitmq (see my other comment)
    - balfirevic 9y ago
      What if your network has a glitch when you try to write to RabbitMQ after you receive NOTIFY? You will lose that message, correct?
      - ruslan_talpa 9y ago
        see my comment above
    - Smudge 9y ago
      Nice -- pg-amqp-bridge seems like a clever solution for coordinating the two. It still puts some amount of the queuing load on the DB, but I'd be comfortable with that trade-off if I really wanted to use rabbitmq as the primary queue. Question though -- Does it guarantee at-least-once or at-most-once on the NOTIFY? (Like, if there is a network blip, will it retry the NOTIFY?) And if it is at-least-once, I assume that consumer apps will have to handle deduplication/idempotency.
brianwawok 9y ago
> Already have a central, configured and monitored server and need "just a small queue" for something. Fine. So use SQS or cloud pub sub. Both take 0 "server configuration work", and you aren't adding load to likely the single most expensive part of your infrastructure (RDBMS). (The exception to where a RDBMS is not the most expensive part of your infastructure is where you have very large data with either nosql something, or a machine learning GPU array.. but not sure that is super relevant here)
- dom0 9y ago
  That's an entirely valid line of reasoning, but it only applies to a certain set of applications. Further, SQS or whatever have the same drawbacks as other external queues compared to an in-DB queue; see all the sibling comments in this big thread.

by dom0 9y ago

IgorPartola 9y ago

This. Also, things like RabbitMQ are complex and their durability properties can be different than those provided by your RDBMS. This can get problematic if you are mixing tasks in the queue that have different priorities. For example, emailing the invoice to a client should not fail silently and should happen at most once. Same with a notification from your doctor that you need to call to discuss your test results. Tossing that into an ephemeral queue is probably not the best solution. Having said that, RabbitMQ does have a ton of settings where you can turn durability up/down as much as you want.

wbl 9y ago
Email is unreliable. You should number invoices, and tell recipients to ignore duplicates.

BurningFrog 9y ago

In my experience, the cost of adding and maintaining another storage subsystem to a project is often hugely underestimated. It's easy to see the benefits and ignore the costs. If I can solve a problem reasonably well by adding a table to my Postgres DB, that will always beat out adding the specialized ScrewdriverDB that does it perfectly.

mbell 9y ago
Depends on what you're adding. Running Redis is dead simple and easy to have visibility into. RabbitMQ / Kafka are much larger undertakings.
- brianwawok 9y ago
  RabbitMQ is on docker, its more or less the same work to launch as Redis these days. (It can take a bit more tuning so I think it is unfair to say it is the SAME work, but it is seriously not a huge deal to run RabbitMQ in the post docker world)
  - true_religion 9y ago
    Is Docker really necessary for this? I've never had any trouble just installing RabbitMQ from the package manager, and running it with only a tiny bit of initial configuration.
    - brianwawok 9y ago
      Well, if for example your entire infrastructure is in K8s, then yes you kind of need docker. Redis vs RabbitMQ stateful-set.yaml would be me or less the same, and something like 20 lines of yaml. I was never a big fan of docker for running some thing on a server, but once you start to go down the K8s path, it finally makes sense.
- LoSboccacc 9y ago
  Depends on the scale mostly. People often forget how not every project is Google/Facebook/Twitter.
- derefr 9y ago
  If you already had a need for a durable database, and so you properly implemented Postgres streaming-archiving + disaster recovery, at much personal effort for your tiny startup... and you now need a durable queue as well... then "just installing Redis" won't get you there. You'll need to that whole ops setup all over again for Redis. Whereas, if your queue is in Postgres, the ops problem is already solved. If you have a full-time ops staff, throwing another marginal piece of infrastructure on the pile isn't much of an issue. If you're your own "ops staff of one", each moving part of your infrastructure—that you need to ensure works, to guarantee you won't lose client data—is another thing slowing down your iteration speed.
jpalomaki 9y ago
I agree. If it sounds simple, then you are probably not thinking hard enough. Think for example how to do backups. If you have a database and completely separate queue system your application state is distributed into two places. This means taking consistent backups is not straightforward. You can of course try to work around this on the application level (database as single source of truth, queues as "nice to have" -thing, but this makes things more complicated for the app).
- dom0 9y ago
  > This means taking consistent backups is not straightforward. And they're not even straightforward to begin with...

emmelaich 9y ago

Yep. Better asked as "Why use a conventional|relational database as a queue?" Because all queues have to be databases i.e. have the same qualities / assurances as databases.

tyingq 9y ago

It also makes it easier to delete or update the queue entry in an atomic transaction that spans other tables. If that has value for the specific use case.

Smudge 9y ago
Very common in the web context -- you perform some form of relational persistence while also inserting a job to schedule background work (like sending an email). Having those both in the same transaction gets rid of a lot of tricky failure cases.
- ruslan_talpa 9y ago
  you can still have the notion of a transaction while integrating rabbitmq like this you do your mutations in a transaction and also in that transaction you execute a NOTIFY command. If transaction is successful, the notify will go through at the end of the transaction. the notify events and be "bridged" to a messaging server like rabbitmq (see my other comment)
  - balfirevic 9y ago
    What if your network has a glitch when you try to write to RabbitMQ after you receive NOTIFY? You will lose that message, correct?
    - ruslan_talpa 9y ago
      see my comment above
  - Smudge 9y ago
    Nice -- pg-amqp-bridge seems like a clever solution for coordinating the two. It still puts some amount of the queuing load on the DB, but I'd be comfortable with that trade-off if I really wanted to use rabbitmq as the primary queue. Question though -- Does it guarantee at-least-once or at-most-once on the NOTIFY? (Like, if there is a network blip, will it retry the NOTIFY?) And if it is at-least-once, I assume that consumer apps will have to handle deduplication/idempotency.

brianwawok 9y ago

> Already have a central, configured and monitored server and need "just a small queue" for something. Fine. So use SQS or cloud pub sub. Both take 0 "server configuration work", and you aren't adding load to likely the single most expensive part of your infrastructure (RDBMS). (The exception to where a RDBMS is not the most expensive part of your infastructure is where you have very large data with either nosql something, or a machine learning GPU array.. but not sure that is super relevant here)

dom0 9y ago
That's an entirely valid line of reasoning, but it only applies to a certain set of applications. Further, SQS or whatever have the same drawbacks as other external queues compared to an in-DB queue; see all the sibling comments in this big thread.