7 ms·

How does this compare to Redis? Seems like Redis would handily beat it.

by dpedu 7y ago

dickeytk 7y ago
The author addresses Redis: > So, many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your precious relational data safe and consistent. Inevitably, a machine is going to fail, and you or someone else is going to be manually picking through your data, trying to figure out what it should look like. I disagree though. BLPOP is far easier to grok than any Postgres solution and Redis is rock-solid. Either using the new Redis streams or a good queueing library is going to guarantee you don’t miss any jobs and have a need for transactions across both systems either. I would also be very hesitant to add work to my database. That’s often a sensitive part of systems.
- shandor 7y ago
  I was also wondering about the "safe and consistent" part. Doesn't Redis Streams with persistence solve those pretty well? Edit: oh, someone mentioned this being from 2013. No Redis Streams back then.
  - ukd1 7y ago
    Ya, it doesn't mean streams - even if it's consistent in isolation, you're using it with Postgres - how do you make actions in one consistent with actions in the other? (TLDR: you won't/don't)
- ukd1 7y ago
  You can disagree, but you're still at least partially wrong; if you're using postgres already, adding redis WILL make your system less reliable as you now have another point of failure to deal with (something else to patch, keep up to date, restart, etc, etc). Unless you have super high traffic, low latency requirements - using postgres adds some load, plus saves you having to write code to deal with rollback / race conditions with postgres vs a-redis-queue.
- erichocean 7y ago
  > Redis is rock-solid I've lost gigabytes of data with Redis; nothing with Postgres. I run both in production. Redis has a lot of obscure failure modes, lacks transactions (with rollback), and has no query language. For simple stuff where it's okay to lose data regularly, it's fantastic (that's when we use it). That said, we've been doing more and more with Postgres over time, and less and less with Redis. The benefits of having all of your data in the same storage, transactionally consistent, with a query planner for ad hoc visualizations and reporting is just too great.
danharaj 7y ago
Postgres is not best in class in many things it does well but it suits the needs most people will actually encounter in the wild while being built on a rock solid foundation. It truly is a fantastic hammer for every nail involving data integrity or synchronization. It's well designed so that when you do venture into territory where you need a specialized solution it's not difficult to replace. But you'll definitely miss ACID and relational data if you're leveraging them. There is a great operational economy to making Postgres your default solution for these sorts of problems.
- deleted 7y ago
  [deleted]
tracer4201 7y ago
If your requirements don’t necessitate such scale, I can see Postgres being a viable alternative. I’d rather not introduce another dependency unless it’s absolutely needed.
- dickeytk 7y ago
  I’ve never in my life worked on something where we didn’t keep having to upgrade our database resources
  - oftenwrong 7y ago
    When you scale your db, now you're scaling your queue, too. One thing to scale, one thing to monitor.
n4r9 7y ago
You may be right in terms of performance (I've no idea) but article states: > many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your precious relational data safe and consistent
- dimino 7y ago
  I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a specific problem. Using a RDBMS for queueing is not what it was built to do, and you will run into issues doing so (I have). By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve. It's bad engineering to try and force Postgres into the role of queue when vastly superior technologies exist. You're actively hurting the engineers you work with, and the company you work for, if you force the same tech into use cases it isn't optimal for. I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group.
  - mlyle 7y ago
    Fewer pieces in the technology stack is better, because each piece is operationally expensive (it is its own set of work in maintenance, upgrades, migrations, etc). There are also the mentioned costs (you can't atomically do things across divergent pieces of infrastructure, at least not very easily at all). Fighting the tendency of everyone wanting to draw in each new interesting bit of technology is important. Every piece of complexity you incur should be proven as required, and you should build things to just somewhat larger than near term projected scale... and not expect that you are a temporarily embarrassed unicorn with loads of unexpected demand showing up. If you run out of performance, you have some headroom from tuning; from that point, you can then choose whether it makes sense to retrofit to a higher scale technology or throw larger hardware at the problem (or both).
Exuma 7y ago
Because Que has transactional/ACID guarantees, and Redis does not.
- adverbly 7y ago
  Actually, redis makes several guarantees. Just not as acid as rds. Source: https://redis.io/topics/transactions https://redis.io/topics/transactions
  - balfirevic 7y ago
    Enqueueing jobs in Redis is not transactional with regard to your primary data store.
    - adverbly 7y ago
      Oh I see what you mean. So ya its not transactional across systems. Ya good point. I usually manage that by just updating the primary data store in the job, but I can see what you mean, and how that might not always be an option.
truth_seeker 7y ago
Redis would be faster. In PG you can use UNLOGGED table which is basically a in-memory table.
- ukd1 7y ago
  UNLOGGED: You could, but if you get some types of crash, you'd be in trouble. More on non-durable settings here - https://www.postgresql.org/docs/current/non-durability.html https://www.postgresql.org/docs/current/non-durability.html
mobilemidget 7y ago
I'm still waiting for a work project to come by where I can replace Redis with https://keydb.dev https://keydb.dev and test that in action.
- wjd2030 7y ago
  I have tried this in our dev environment several times and failed b/c it didnt support all RESP2 commands (Using stackexchange.redis as the lib)
echlebek 7y ago
With postgres, you have a chance to not lose data.

How does this compare to Redis? Seems like Redis would handily beat it.

by dpedu 7y ago

How does this compare to Redis? Seems like Redis would handily beat it.

The author addresses Redis: > So, many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your precious relational data safe and consistent. Inevitably, a machine is going to fail, and you or someone else is going to be manually picking through your data, trying to figure out what it should look like. I disagree though. BLPOP is far easier to grok than any Postgres solution and Redis is rock-solid. Either using the new Redis streams or a good queueing library is going to guarantee you don’t miss any jobs and have a need for transactions across both systems either. I would also be very hesitant to add work to my database. That’s often a sensitive part of systems.

shandor 7y ago
I was also wondering about the "safe and consistent" part. Doesn't Redis Streams with persistence solve those pretty well? Edit: oh, someone mentioned this being from 2013. No Redis Streams back then.
- ukd1 7y ago
  Ya, it doesn't mean streams - even if it's consistent in isolation, you're using it with Postgres - how do you make actions in one consistent with actions in the other? (TLDR: you won't/don't)
ukd1 7y ago
You can disagree, but you're still at least partially wrong; if you're using postgres already, adding redis WILL make your system less reliable as you now have another point of failure to deal with (something else to patch, keep up to date, restart, etc, etc). Unless you have super high traffic, low latency requirements - using postgres adds some load, plus saves you having to write code to deal with rollback / race conditions with postgres vs a-redis-queue.
erichocean 7y ago
> Redis is rock-solid I've lost gigabytes of data with Redis; nothing with Postgres. I run both in production. Redis has a lot of obscure failure modes, lacks transactions (with rollback), and has no query language. For simple stuff where it's okay to lose data regularly, it's fantastic (that's when we use it). That said, we've been doing more and more with Postgres over time, and less and less with Redis. The benefits of having all of your data in the same storage, transactionally consistent, with a query planner for ad hoc visualizations and reporting is just too great.

danharaj 7y ago

Postgres is not best in class in many things it does well but it suits the needs most people will actually encounter in the wild while being built on a rock solid foundation. It truly is a fantastic hammer for every nail involving data integrity or synchronization. It's well designed so that when you do venture into territory where you need a specialized solution it's not difficult to replace. But you'll definitely miss ACID and relational data if you're leveraging them. There is a great operational economy to making Postgres your default solution for these sorts of problems.

deleted 7y ago
[deleted]

tracer4201 7y ago

If your requirements don’t necessitate such scale, I can see Postgres being a viable alternative. I’d rather not introduce another dependency unless it’s absolutely needed.

dickeytk 7y ago
I’ve never in my life worked on something where we didn’t keep having to upgrade our database resources
- oftenwrong 7y ago
  When you scale your db, now you're scaling your queue, too. One thing to scale, one thing to monitor.

n4r9 7y ago

You may be right in terms of performance (I've no idea) but article states: > many developers have started going straight to Redis-backed queues (Resque, Sidekiq) or dedicated queues (beanstalkd, ZeroMQ...), but I see these as suboptimal solutions - they're each another moving part that can fail, and the jobs that you queue with them aren't protected by the same transactions and atomic backups that are keeping your precious relational data safe and consistent

dimino 7y ago
I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a specific problem. Using a RDBMS for queueing is not what it was built to do, and you will run into issues doing so (I have). By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve. It's bad engineering to try and force Postgres into the role of queue when vastly superior technologies exist. You're actively hurting the engineers you work with, and the company you work for, if you force the same tech into use cases it isn't optimal for. I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group.
- mlyle 7y ago
  Fewer pieces in the technology stack is better, because each piece is operationally expensive (it is its own set of work in maintenance, upgrades, migrations, etc). There are also the mentioned costs (you can't atomically do things across divergent pieces of infrastructure, at least not very easily at all). Fighting the tendency of everyone wanting to draw in each new interesting bit of technology is important. Every piece of complexity you incur should be proven as required, and you should build things to just somewhat larger than near term projected scale... and not expect that you are a temporarily embarrassed unicorn with loads of unexpected demand showing up. If you run out of performance, you have some headroom from tuning; from that point, you can then choose whether it makes sense to retrofit to a higher scale technology or throw larger hardware at the problem (or both).

Exuma 7y ago

Because Que has transactional/ACID guarantees, and Redis does not.

adverbly 7y ago
Actually, redis makes several guarantees. Just not as acid as rds. Source: https://redis.io/topics/transactions https://redis.io/topics/transactions
- balfirevic 7y ago
  Enqueueing jobs in Redis is not transactional with regard to your primary data store.
  - adverbly 7y ago
    Oh I see what you mean. So ya its not transactional across systems. Ya good point. I usually manage that by just updating the primary data store in the job, but I can see what you mean, and how that might not always be an option.

truth_seeker 7y ago

Redis would be faster. In PG you can use UNLOGGED table which is basically a in-memory table.

ukd1 7y ago
UNLOGGED: You could, but if you get some types of crash, you'd be in trouble. More on non-durable settings here - https://www.postgresql.org/docs/current/non-durability.html https://www.postgresql.org/docs/current/non-durability.html

mobilemidget 7y ago

I'm still waiting for a work project to come by where I can replace Redis with https://keydb.dev https://keydb.dev and test that in action.

wjd2030 7y ago
I have tried this in our dev environment several times and failed b/c it didnt support all RESP2 commands (Using stackexchange.redis as the lib)

echlebek 7y ago

With postgres, you have a chance to not lose data.