3 ms·
This is a really great summary of the problems with queues in Postgres. I ended up working around this by moving jobs to an "archived" table when they were com
by kevinburke 9y ago
This is a really great summary of the problems with queues in Postgres.
I ended up working around this by moving jobs to an "archived" table when they were complete, so the "hot" table never gets too large, and by avoiding long running transactions - we update jobs to in-progress, and then the worker is responsible for marking them as "complete" or "failed". If the worker crashes, a "stuck jobs" background worker will mark jobs as failed after a timeout.
https://github.com/Shyp/rickover https://github.com/Shyp/rickover
- elmigranto 9y agoHere's the one using similar approach, but for Node.js. https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss
- codesnik 9y agoIn one of my projects I was going to use something like that: "update ... skip locked" was used in isolated short transaction, which would just update that job with "taken" flag and timestamp and commit that. "stuck jobs" worker was meant to fire as rare as possible for cases of unhandled failures, but was still there.