4 ms·

The problem is that now you use Postgres for 95% of your system, and also Redis or NATS, which means you lose the ability to atomically commit changes to your d

by strken 1y ago

The problem is that now you use Postgres for 95% of your system, and also Redis or NATS, which means you lose the ability to atomically commit changes to your database and send a message in one transaction. You can work around this, but to the best of my knowledge you can't have consistency (between your existing Postgres database and your separate queue or event log) and simplicity.

hbogert 1y ago
Indeed, only eventual consistency. The article approaches this subject and mentions the use of the outbox pattern and/or using tools like Debezium.
threatofrain 1y ago
One of the secret traps of Kafka is that it tried to be a collection of nice primitives without that much built on top. So the Fortune 500 story of Kafka is actually one of everyone building on top of Kafka internally with a bigger team. If you weren't prepared to do a little building on top of Kafka you might find the ecosystem a little empty. But with PSQL you have even less built on top for a use case it wasn't meant for. Now you are in the "you better know what you're doing" territory.
- strken 1y ago
  My experience has been that a SQL database is an easier foundation to build on top of than a distributed event streaming log until you run into performance issues that can't be solved by understanding your database better and/or scaling vertically, and that companies which run into performance issues can afford to migrate to other technologies and/or pay for a good ops team, while companies which started using message queues for systems handling 5 requests per second often struggle to make their system sound and get features out the door.
  - threatofrain 1y ago
    You're still talking about performance. I'm talking about how much you have to build. When people use Redis/whatever, are you surprised that speed is just a side benefit and ergonomics is what they're looking for? Those are the same ergonomics you'd have to rebuild, as opposed to not build at all because it's already there. If you want to rebuild Redis in PSQL you are very much in the "you better know what you're doing" territory, and you're also very much in the "are you sure this is your core business value" territory as you rediscover how much nice stuff was packed into the Redis ecosystem. And how am I supposed to uncover the surface or ergonomics of your Redis replacement?
    - strken 1y ago
      Can you explain how using Postgres for a job queue, ideally using something like pgmq[0], is in "you better know what you're doing" territory, while using Redis isn't, and without talking about performance? Redis is fine as-is and I'm not suggesting anyone should rebuild the whole thing in Postgres, only that if they want a specific thing that Redis can do and they already use Postgres, then they might be able to do the same thing in Postgres with less effort and thus avoid losing atomicity and consistency and any other properties of Postgres that they find desirable. [0] Which was mentioned by the original "You Don't Need Kafka" article
      - threatofrain 1y ago
        If you want something simple then you could use Redis Streams, and if you want a more mature solution you could use BullMQ, which is pretty solid and well liked. Knowing what you're doing in BullMQ is reading some docs. Do I really need to discuss why BullMQ is non-trivial software, or Redis Streams with consumer groups? And I asked but you didn't answer. How am I supposed to discover the ergonomics of your custom Redis or BullMQ replacement? Do I read your hand-written docs?
        strken 1y ago
        You could go read the pgmq docs, look at the fifty or so lines of code that make up the thin wrapper around it, and understand it just as well as you understand BullMQ. Or you could use one of the extensive set of pgmq libraries if you aren't comfortable calling Postgres functions (for some reason). I'm not suggesting that BullMQ and Redis are trivial and that you should rewrite them yourself, I'm telling you that Postgres has multiple existing implementations of a message queue already. And frankly, the fact that something is non-trivial does not automatically mean you can't or shouldn't rewrite it, only that you need a good reason. Even if there was no existing Postgres message queue (there are many), atomicity is a good enough reason on its own. Imagine if you could avoid idempotency issues when calling third party APIs until you had more than ten engineering teams! Well, you can! Just don't add more than one method of storing online data to your architecture.

by strken 1y ago

hbogert 1y ago

Indeed, only eventual consistency. The article approaches this subject and mentions the use of the outbox pattern and/or using tools like Debezium.

threatofrain 1y ago

One of the secret traps of Kafka is that it tried to be a collection of nice primitives without that much built on top. So the Fortune 500 story of Kafka is actually one of everyone building on top of Kafka internally with a bigger team. If you weren't prepared to do a little building on top of Kafka you might find the ecosystem a little empty. But with PSQL you have even less built on top for a use case it wasn't meant for. Now you are in the "you better know what you're doing" territory.

strken 1y ago
My experience has been that a SQL database is an easier foundation to build on top of than a distributed event streaming log until you run into performance issues that can't be solved by understanding your database better and/or scaling vertically, and that companies which run into performance issues can afford to migrate to other technologies and/or pay for a good ops team, while companies which started using message queues for systems handling 5 requests per second often struggle to make their system sound and get features out the door.
- threatofrain 1y ago
  You're still talking about performance. I'm talking about how much you have to build. When people use Redis/whatever, are you surprised that speed is just a side benefit and ergonomics is what they're looking for? Those are the same ergonomics you'd have to rebuild, as opposed to not build at all because it's already there. If you want to rebuild Redis in PSQL you are very much in the "you better know what you're doing" territory, and you're also very much in the "are you sure this is your core business value" territory as you rediscover how much nice stuff was packed into the Redis ecosystem. And how am I supposed to uncover the surface or ergonomics of your Redis replacement?
  - strken 1y ago
    Can you explain how using Postgres for a job queue, ideally using something like pgmq[0], is in "you better know what you're doing" territory, while using Redis isn't, and without talking about performance? Redis is fine as-is and I'm not suggesting anyone should rebuild the whole thing in Postgres, only that if they want a specific thing that Redis can do and they already use Postgres, then they might be able to do the same thing in Postgres with less effort and thus avoid losing atomicity and consistency and any other properties of Postgres that they find desirable. [0] Which was mentioned by the original "You Don't Need Kafka" article
    - threatofrain 1y ago
      If you want something simple then you could use Redis Streams, and if you want a more mature solution you could use BullMQ, which is pretty solid and well liked. Knowing what you're doing in BullMQ is reading some docs. Do I really need to discuss why BullMQ is non-trivial software, or Redis Streams with consumer groups? And I asked but you didn't answer. How am I supposed to discover the ergonomics of your custom Redis or BullMQ replacement? Do I read your hand-written docs?
      - strken 1y ago
        You could go read the pgmq docs, look at the fifty or so lines of code that make up the thin wrapper around it, and understand it just as well as you understand BullMQ. Or you could use one of the extensive set of pgmq libraries if you aren't comfortable calling Postgres functions (for some reason). I'm not suggesting that BullMQ and Redis are trivial and that you should rewrite them yourself, I'm telling you that Postgres has multiple existing implementations of a message queue already. And frankly, the fact that something is non-trivial does not automatically mean you can't or shouldn't rewrite it, only that you need a good reason. Even if there was no existing Postgres message queue (there are many), atomicity is a good enough reason on its own. Imagine if you could avoid idempotency issues when calling third party APIs until you had more than ten engineering teams! Well, you can! Just don't add more than one method of storing online data to your architecture.