5 ms·
Postgres as a queue is one of the worst decisions I've seen made and I and others I've worked with have spent years of our lives unwinding the consequences.
by dharmab 3y ago
Postgres as a queue is one of the worst decisions I've seen made and I and others I've worked with have spent years of our lives unwinding the consequences.
- mikercampbell 3y agoIt’s almost as if we were in an ordered line, all waiting to learn the same thing, and had the lesson delivered no more than once.
- dralley 3y agoCompletely disagree for our use case. If your messages aren't send-and-forget but rather represent work that needs to be tracked, it is incredibly difficult to manage the state when your database and queue are separate. Using postgresql as the queue and leveraging the same transactions as your business logic solves many many issues.
- nostrebored 3y agoHow is it incredibly difficult? I’ve seen this work very well. Is your code handling different varieties of event states stored in your database? Why implement this as a monolithic data source? Was it not possible to separate different states by additional queues? Curious to hear the decision making process here. Maybe I can see it for low volume?
- jvans 3y agodralley is saying your queue push and database write aren't transactional. You have to be ok suffering some small % of message loss or messages sent without a database commit
- nostrebored 3y agoDepends on your queue semantics, ie is it just an at least once delivery or is there an ack back to the queue service. If there is a two phase process then you just need idempotence and can safely transact with the db.
- jvans 3y agoIt doesn't matter what your queue semantics are. If it's not part of the transaction, delivery and push cannot be guaranteed to be atomic.
- dralley 3y agoAdmittedly we're not using it for anything high volume.
- arcbyte 3y agoAt the end of the day, the ONLY way to reliably do this is to hook into the databases native journal/log anyway. Postgres gives you better primitives to work with than installing Qlik for example.
- nostrebored 3y agoWhy do you think so?
- nurettin 3y agoAs a person who has implemented worker farms at scale, I don't understand which part is "incredibly difficult", or what having a dedicated queue server prevents you from updating database state within "the same transactions". If your worker process has to update some kind of task state and calculate some business logic, it can still do so whether if you use rmq, redis or whatever.
- canadiantim 3y agoyears seems like hyperbole
- mistrial9 3y agosaying so without context or requirements at all reduces the effectiveness of the insight
- dharmab 3y agoSadly my work is subject to regulatory requirements that prevent me from discussing specific details :(
- sbuttgereit 3y agoSurely if your complaint is general enough to give such advice to everyone here, there must be a way to express generalized technical details of that complaint without disclosing anything more privileged than you already have. If that kind of abstraction and generalization isn't possible, then it sounds to me like something specific to your work's implementation of the ideas and not something that applies broadly.
- cqqxo4zV46cp 3y agoOh, okay. It’s great that you’ve let us all know that you hold The Secret. Thank you.
- wavemode 3y agoCare to elaborate?