5 ms·
What if your listeners crashed/were down at the time of the `PUB` message? Does this mean the message falls into oblivion (since it will never receive a reply/A
by MuffinFlavored 7y ago
What if your listeners crashed/were down at the time of the `PUB` message? Does this mean the message falls into oblivion (since it will never receive a reply/ACK/get worked on)?
- colinchartier 7y agoThe pattern I follow is that workers immediately poll once when they start - this means that when your workers restart after crashing, they'll pick up any missing jobs.
- pm90 7y agoI'm not seeing how this recovers from the failure scenario where a worker grabbed an event and then immediately crashed.
- tjungblut 7y agoYou need some timeout and/or leasing mechanism to make it work in this scenario.
- Rapzid 7y agoThe comment you are replying to wasn't addressing that scenario. Typically you would implement visibility timeouts and other such stuff. Depending on the use case you could make specific optimizations or keep it generic and have SQS like semantics or something.
- disiplus 7y agoat this point why not use something like rabbitmq and not reinvent it in postgres.
- rapind 7y agoBecause then you're running and maintaining another service. Might make sense, but depends.
- stickfigure 7y agoBecause it's trivially easy. The dequeue operation updates the last_lease time; the query makes sure you don't dequeue something that was recently leased. It's far, far easier than setting up a separate job queue, especially if you are already using an RDBMS.
- privateSFacct 7y agoIs the activity wrapped in a transaction? If so if worker crashes you get a connection timeout on postgresql and it rolls back the transaction and releases row locks - then next select SKIP picks it up. If you want fast response with safety listen and on restart do a poll always to pick up anything that's ready that you might have missed a pub for. If you want lots of coverage do a poll every 10s - postgresql can handle it, you still get immediate response by listening to notify. If you need fancier approaches you can do a last_updated time on the job and or watch waiting on pg_stat_activity or similar etc - but all totally overkill I think - the simple approach get's pretty far (not an expert though at all).