5 ms·
Running this exact implementation with 47M jobs processed and counting. SKIP LOCKED is great for VACUUM, and having durable storage with indexes make otherwise
by levkk 3y ago
Running this exact implementation with 47M jobs processed and counting. SKIP LOCKED is great for VACUUM, and having durable storage with indexes make otherwise expensive patterns like delayed jobs, retries, status updates, "at least once", etc. really easy to implement.
- dlisboa 3y agoDo you have some idea of how many jobs per minute or hour do you have? Just want to compare with what we have on Redis at work. Do you also have any idea on the concurrency? How many workers you have pulling from Postgres. I’ve used this approach before (ages ago) when Redis wasn’t even a thing, though not at high throughout requirements.
- cle 3y agoI recently did the same thing but without LISTEN/NOTIFY and with a partial index. Pushed about 700 jobs/sec with 1000 workers (via pgbouncer).
- devoutsalsa 3y agoI'm sure Redis is much faster than an RDBMS w/ all the ACID features turned on. The biggest concern I always have with Redis is simply overwhelming the in-memory storage limits when someone wants to do process a large number of good-sized messages at an inconvenient time. #tradeoffs
- dilyevsky 3y agoI’ve seen it used for up to 1000 jobs per second with concurrency of 3-12