5 ms·
> storing jobs in a database allows a program to take advantage of its transactional consistency; when an operation fails and rolls back, an injected job rolls
by CodeWriter23 9y ago
> storing jobs in a database allows a program to take advantage of its transactional consistency; when an operation fails and rolls back, an injected job rolls back with it
This is a false premise for using a database as a queue. The same goal can be simply achieved with a different queuing system, say SQS, by only enqueuing jobs AFTER successful transaction commit.
- IgorPartola 9y agoWhat happens when the enqueuing fails after a successful commit?
- CodeWriter23 9y agoRetries.
- IgorPartola 9y agoSo you retry indefinitely? How do you keep track of your retries? Do you have a queue to get the items into the real queue? One solution to this is two phase commits, but I am not aware of any message broker that supports that. So basically you are stuck assuming that your message broker is never going to fail, or at least not going to fail for long enough for the users to notice.
- ruslan_talpa 9y agoHow do you queue that job only after the transaction committed?
- sorkin2 9y agoThe same way the new built-in logical replication works: logical decoding.
- ruslan_talpa 9y agothis means it does not work on RDS
- CodeWriter23 9y agoCheck the return code from libpq.
- elmigranto 9y agoIIRC, with Postgres, you can write to Redis directly from SQL and as part of transaction. (See `redis fdw` on google.) Perhaps, there is writable FDW driver for your queue system. (Or maybe you can just dumpt stuff to sockets by using Postgres function written in Python, JS, Perl or w/ever you prefer.)