8 ms·

You’re dealing with survivorship bias - those companies whose products were successful enough that they needed your services. You don’t speak to the teams who w

You’re dealing with survivorship bias - those companies whose products were successful enough that they needed your services. You don’t speak to the teams who went bankrupt because they spent too much time fiddling with RabbitMQ, when Postgres would have been fine. As always, there are trade offs, no silver bullets.

2muchcoffeeman 5y ago
In my experience proper abstraction is the key. If you have a clear abstraction and structure you can replace the implementation when you need to without too much disruption. If it’s too leaky you’re screwed.
- goodoldneon 5y ago
  100%! Create a good enough solution and put it behind an abstraction that remains stable when you later create an ideal solution
- hughrr 5y ago
  Yes. Use a queue abstraction for which there are numerous queues available off the shelf for!
- aparsons 5y ago
  In my experience, great abstractions without a good data migration blueprint is as about as useful as no abstraction at all. In this example - great, you have a “queue” abstraction. When you switch to RabbitMQ or Postgres, how do you move your existing data without a quality of service or data loss? It can be difficult with production datastores even if the abstraction within your service is beautiful.
  - lolinder 5y ago
    Isn't migrating a queue super simple? Have a window in which you only add to the new system and you listen to both. When the old queue is empty, delete it. If you need to keep track of old data, then yes, migration is hard. But queues don't save old data—that's the point of a queue.
alunchbox 5y ago
You're not wrong, but if anyone on the team has already worked with a queue system that they need to manage then utilize something that scales. Otherwise there's no reason to use the cloud solution you're hosting with which hand hold you to make it work quickly with your integration.
skytreader 5y ago
> You’re dealing with survivorship bias - those companies whose products were successful enough that they needed your services In fairness, Redis has only been available for 12 years so someone who has been in the industry longer has probably encountered systems using DBs as queues for no better reason than an alternative was not available when the system was made. (Rabbit just a couple years older I think.) But in this day and age, you have better options to start with c'mon. > they spent too much time fiddling with RabbitMQ, when Postgres would have been fine. Rather ironic claim in a thread full of tips, tricks, and gotchas just to make PG behave like a queue no? I can't speak for RabbitMQ personally but Redis has got to be among the simplest drop-in dependencies I've ever encountered. For a simple queue, the defaults do out-of-the-box. Gut feel also says, Redis defaults for queuing would also serve more than PG defaults for queuing so you could leave it untouched for far longer to focus on your product, get to profitability, etc. Feel free to debunk with data.
- hughrr 5y ago
  I've got about 6 years operational experience with Redis. I would not use it for a queue. I struggle to consider using it as a data store these days as well with other options on the table. But that's a whole new thread so I'm not going into details here. If you want a queue, get a queue. If you want a relational database, get a relational database. If you want a key-value store, get a key-value store. Don't cross the streams.
  - mikeklaas 5y ago
    What would you use redis for, if anything?
    - yawaramin 5y ago
      I assume a key-value store.
      - hughrr 5y ago
        Correct
paxys 5y ago
You are wayyy overestimating how complex something like RabbitMQ or Redis is. You don’t need to hire entire teams of engineers or go bankrupt setting up an instance. It is less work than implementing a production-level queue in Postgres for sure.
- zomgwat 5y ago
  I agree. I’ll also add that Redis is a level easier to operate than RabbitMQ.
- aetherson 5y ago
  Have worked at multiple companies that successfully used redis or rabbit with teams of less than five engineers. It's a little insane that a highly rated thread on HN is telling people to use postgres as their queuing solution. The world is wide, I'm sure that somewhere out there there is a situation where using postgres of a queue makes sense, but in 99% of all cases, this is a terrible idea. Also, SQS and nsq are simple.
  - politician 5y ago
    If your data storage pattern involves a database write followed by a message queue insert, then the ability to wrap those in a transaction can be a good trade-off to avoid consistency failures between the writes. Avoid consuming that queue directly though -- which is probably what you're thinking when saying this is a dumb idea and I tend to agree. Typically, you want to have a worker that loads entries into a more optimal queue for your application. Bottom line though, there is not a single best "queue" product. There are lots of queues offering wildly different semantics that directly impact use cases.
  - GordonS 5y ago
    For years I've been using RabbitMQ in small teams, and me as a one-man team too. As long as you don't need clustering (even a single node can handle some pretty heavy load), it's actually really simple to setup and use - way easier than Postgres itself, for example. My biggest beefs with it have historically been the Erlang-style config files (which are basically unreadable), and the ridiculous spiel of error messages you get if you have an invalid configuration. But thankfully RabbitMQ switched to a much simpler config file format one or two years back, and I understand the Erlang OTP is working on better error messages and stack traces.
robwwilliams 5y ago
Great comment: Been reading Taleb’s Black Swan this morning on ascertainment bias, so this resonated well.
dorfsmay 5y ago
There is a lot of space between Postres and RabbitMQ. If performance is a factor it takes alot of work to make a Postgres table fast enough. These days, and for the past at least 5 years, you're better off using a service like SQS, or setup Redis yourself.
cortesoft 5y ago
Sure, but I also feel like this comment is based on an assumption that everyone works at startups. A lot of us work for larger, already established companies that are past that will we or won’t we survival stage. We are trying to design systems to solve needs for already validated products. It might seem like everyone works for a startup, but big companies have a lot of engineers, and we still have to solve new problems there even though the company is not new.
ikiris 5y ago
You say that as if rabbitmq takes more than an 8th grader to set up in a few hours?
- GordonS 5y ago
  To be fair, you can pull and run the Docker image in a few minutes, and the defaults will work out-of-the-box for a lot of folks. If you don't need clustering, RabbitMQ really is pretty simple to setup.
rantwasp 5y ago
no. he/she actually has a point. a database is not a queue https://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html?m=1 https://mikehadlow.blogspot.com/2012/04/database-as-queue-an...
- geggo98 5y ago
  The blog post you mentions brings two arguments: first the database would require looking. Second the database would need tradeoffs between reading (polling) and writing (adding and removing to the queue). The original article handles the first argument: Postgres doesn't need polling. Instead it provides a notify mechanism, that informs the application when the table changed (something was added or removed from the queue) via the SQL NOTIFY statement. For the second point it also provides a solution: since optimization for reading is not needed anymore with the NOTIFY statement, the trade-off like different: we now need an efficient way to write. For this the article provides an efficient update statement with special lock behavior. This helps to make writes efficient, too. It looks like both points from the blog post you linked are handled in the original article.
piokoch 5y ago
Not sure. All messaging solution brokers walked away from relational database based storage long time ago (I've seen how badly JBoss 4 messaging with persistence sucked) and for a good reason as either database was occasionally killed by messaging traffic, or messaging traffic was crawling because database was too slow. In these days with Docker, ready cloud solutions getting the message broker is so simple that there is no reason to use database as a broker. If we know that our traffic will be so small, just install DB and message broker on the same VM, to avoid spending more on hardware or hosting.
pyuser583 5y ago
I’ve watched companies maintain their non-scaling DBs all the way to the grave. It’s a killer. If it’s “just a cache” OK. Fine. If you need it for something important like queuing or certain caching scenarios - be aware it won’t scale well.
rkk3 5y ago
This. So much this.

2muchcoffeeman 5y ago

In my experience proper abstraction is the key. If you have a clear abstraction and structure you can replace the implementation when you need to without too much disruption. If it’s too leaky you’re screwed.

goodoldneon 5y ago
100%! Create a good enough solution and put it behind an abstraction that remains stable when you later create an ideal solution
hughrr 5y ago
Yes. Use a queue abstraction for which there are numerous queues available off the shelf for!
aparsons 5y ago
In my experience, great abstractions without a good data migration blueprint is as about as useful as no abstraction at all. In this example - great, you have a “queue” abstraction. When you switch to RabbitMQ or Postgres, how do you move your existing data without a quality of service or data loss? It can be difficult with production datastores even if the abstraction within your service is beautiful.
- lolinder 5y ago
  Isn't migrating a queue super simple? Have a window in which you only add to the new system and you listen to both. When the old queue is empty, delete it. If you need to keep track of old data, then yes, migration is hard. But queues don't save old data—that's the point of a queue.

alunchbox 5y ago

You're not wrong, but if anyone on the team has already worked with a queue system that they need to manage then utilize something that scales. Otherwise there's no reason to use the cloud solution you're hosting with which hand hold you to make it work quickly with your integration.

skytreader 5y ago

> You’re dealing with survivorship bias - those companies whose products were successful enough that they needed your services In fairness, Redis has only been available for 12 years so someone who has been in the industry longer has probably encountered systems using DBs as queues for no better reason than an alternative was not available when the system was made. (Rabbit just a couple years older I think.) But in this day and age, you have better options to start with c'mon. > they spent too much time fiddling with RabbitMQ, when Postgres would have been fine. Rather ironic claim in a thread full of tips, tricks, and gotchas just to make PG behave like a queue no? I can't speak for RabbitMQ personally but Redis has got to be among the simplest drop-in dependencies I've ever encountered. For a simple queue, the defaults do out-of-the-box. Gut feel also says, Redis defaults for queuing would also serve more than PG defaults for queuing so you could leave it untouched for far longer to focus on your product, get to profitability, etc. Feel free to debunk with data.

hughrr 5y ago
I've got about 6 years operational experience with Redis. I would not use it for a queue. I struggle to consider using it as a data store these days as well with other options on the table. But that's a whole new thread so I'm not going into details here. If you want a queue, get a queue. If you want a relational database, get a relational database. If you want a key-value store, get a key-value store. Don't cross the streams.
- mikeklaas 5y ago
  What would you use redis for, if anything?
  - yawaramin 5y ago
    I assume a key-value store.
    - hughrr 5y ago
      Correct

paxys 5y ago

You are wayyy overestimating how complex something like RabbitMQ or Redis is. You don’t need to hire entire teams of engineers or go bankrupt setting up an instance. It is less work than implementing a production-level queue in Postgres for sure.

zomgwat 5y ago
I agree. I’ll also add that Redis is a level easier to operate than RabbitMQ.
aetherson 5y ago
Have worked at multiple companies that successfully used redis or rabbit with teams of less than five engineers. It's a little insane that a highly rated thread on HN is telling people to use postgres as their queuing solution. The world is wide, I'm sure that somewhere out there there is a situation where using postgres of a queue makes sense, but in 99% of all cases, this is a terrible idea. Also, SQS and nsq are simple.
- politician 5y ago
  If your data storage pattern involves a database write followed by a message queue insert, then the ability to wrap those in a transaction can be a good trade-off to avoid consistency failures between the writes. Avoid consuming that queue directly though -- which is probably what you're thinking when saying this is a dumb idea and I tend to agree. Typically, you want to have a worker that loads entries into a more optimal queue for your application. Bottom line though, there is not a single best "queue" product. There are lots of queues offering wildly different semantics that directly impact use cases.
- GordonS 5y ago
  For years I've been using RabbitMQ in small teams, and me as a one-man team too. As long as you don't need clustering (even a single node can handle some pretty heavy load), it's actually really simple to setup and use - way easier than Postgres itself, for example. My biggest beefs with it have historically been the Erlang-style config files (which are basically unreadable), and the ridiculous spiel of error messages you get if you have an invalid configuration. But thankfully RabbitMQ switched to a much simpler config file format one or two years back, and I understand the Erlang OTP is working on better error messages and stack traces.

robwwilliams 5y ago

Great comment: Been reading Taleb’s Black Swan this morning on ascertainment bias, so this resonated well.

dorfsmay 5y ago

There is a lot of space between Postres and RabbitMQ. If performance is a factor it takes alot of work to make a Postgres table fast enough. These days, and for the past at least 5 years, you're better off using a service like SQS, or setup Redis yourself.

cortesoft 5y ago

Sure, but I also feel like this comment is based on an assumption that everyone works at startups. A lot of us work for larger, already established companies that are past that will we or won’t we survival stage. We are trying to design systems to solve needs for already validated products. It might seem like everyone works for a startup, but big companies have a lot of engineers, and we still have to solve new problems there even though the company is not new.

ikiris 5y ago

You say that as if rabbitmq takes more than an 8th grader to set up in a few hours?

GordonS 5y ago
To be fair, you can pull and run the Docker image in a few minutes, and the defaults will work out-of-the-box for a lot of folks. If you don't need clustering, RabbitMQ really is pretty simple to setup.

rantwasp 5y ago

no. he/she actually has a point. a database is not a queue https://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html?m=1 https://mikehadlow.blogspot.com/2012/04/database-as-queue-an...

geggo98 5y ago
The blog post you mentions brings two arguments: first the database would require looking. Second the database would need tradeoffs between reading (polling) and writing (adding and removing to the queue). The original article handles the first argument: Postgres doesn't need polling. Instead it provides a notify mechanism, that informs the application when the table changed (something was added or removed from the queue) via the SQL NOTIFY statement. For the second point it also provides a solution: since optimization for reading is not needed anymore with the NOTIFY statement, the trade-off like different: we now need an efficient way to write. For this the article provides an efficient update statement with special lock behavior. This helps to make writes efficient, too. It looks like both points from the blog post you linked are handled in the original article.

piokoch 5y ago

Not sure. All messaging solution brokers walked away from relational database based storage long time ago (I've seen how badly JBoss 4 messaging with persistence sucked) and for a good reason as either database was occasionally killed by messaging traffic, or messaging traffic was crawling because database was too slow. In these days with Docker, ready cloud solutions getting the message broker is so simple that there is no reason to use database as a broker. If we know that our traffic will be so small, just install DB and message broker on the same VM, to avoid spending more on hardware or hosting.

pyuser583 5y ago

I’ve watched companies maintain their non-scaling DBs all the way to the grave. It’s a killer. If it’s “just a cache” OK. Fine. If you need it for something important like queuing or certain caching scenarios - be aware it won’t scale well.

rkk3 5y ago

This. So much this.