4 ms·

Can you explain how using Postgres for a job queue, ideally using something like pgmq[0], is in "you better know what you're doing" territory, while using Redis

by strken 1y ago

Can you explain how using Postgres for a job queue, ideally using something like pgmq[0], is in "you better know what you're doing" territory, while using Redis isn't, and without talking about performance? Redis is fine as-is and I'm not suggesting anyone should rebuild the whole thing in Postgres, only that if they want a specific thing that Redis can do and they already use Postgres, then they might be able to do the same thing in Postgres with less effort and thus avoid losing atomicity and consistency and any other properties of Postgres that they find desirable. [0] Which was mentioned by the original "You Don't Need Kafka" article

threatofrain 1y ago
If you want something simple then you could use Redis Streams, and if you want a more mature solution you could use BullMQ, which is pretty solid and well liked. Knowing what you're doing in BullMQ is reading some docs. Do I really need to discuss why BullMQ is non-trivial software, or Redis Streams with consumer groups? And I asked but you didn't answer. How am I supposed to discover the ergonomics of your custom Redis or BullMQ replacement? Do I read your hand-written docs?
- strken 1y ago
  You could go read the pgmq docs, look at the fifty or so lines of code that make up the thin wrapper around it, and understand it just as well as you understand BullMQ. Or you could use one of the extensive set of pgmq libraries if you aren't comfortable calling Postgres functions (for some reason). I'm not suggesting that BullMQ and Redis are trivial and that you should rewrite them yourself, I'm telling you that Postgres has multiple existing implementations of a message queue already. And frankly, the fact that something is non-trivial does not automatically mean you can't or shouldn't rewrite it, only that you need a good reason. Even if there was no existing Postgres message queue (there are many), atomicity is a good enough reason on its own. Imagine if you could avoid idempotency issues when calling third party APIs until you had more than ten engineering teams! Well, you can! Just don't add more than one method of storing online data to your architecture.

by strken 1y ago

threatofrain 1y ago

If you want something simple then you could use Redis Streams, and if you want a more mature solution you could use BullMQ, which is pretty solid and well liked. Knowing what you're doing in BullMQ is reading some docs. Do I really need to discuss why BullMQ is non-trivial software, or Redis Streams with consumer groups? And I asked but you didn't answer. How am I supposed to discover the ergonomics of your custom Redis or BullMQ replacement? Do I read your hand-written docs?

strken 1y ago
You could go read the pgmq docs, look at the fifty or so lines of code that make up the thin wrapper around it, and understand it just as well as you understand BullMQ. Or you could use one of the extensive set of pgmq libraries if you aren't comfortable calling Postgres functions (for some reason). I'm not suggesting that BullMQ and Redis are trivial and that you should rewrite them yourself, I'm telling you that Postgres has multiple existing implementations of a message queue already. And frankly, the fact that something is non-trivial does not automatically mean you can't or shouldn't rewrite it, only that you need a good reason. Even if there was no existing Postgres message queue (there are many), atomicity is a good enough reason on its own. Imagine if you could avoid idempotency issues when calling third party APIs until you had more than ten engineering teams! Well, you can! Just don't add more than one method of storing online data to your architecture.