4 ms·
Bloat due to long running transactions and batch writes is a significant operational pain in PostgreSQL. For extreme cases where vacuum isn't doing its job, che
by smilliken 9y ago
Bloat due to long running transactions and batch writes is a significant operational pain in PostgreSQL. For extreme cases where vacuum isn't doing its job, check out pg_repack.
There is another approach to queuing in postgresql that doesn't suffer this problem though:
* Give each of your workers an id from 0..n.
* Open a connection to read from the queue, ordered by your priority index, and partition for your worker id (eg primary key % n = worker id). Make it a cursor.
* Read from the cursor in batches. You'll never see the same dead row twice because the cursor is stateful and is doing a single scan. You generally won't see dead rows anyway since this worker is the only one assigned to update these queue items.
* Perform your writes in a separate postgresql connection to avoid invalidating the cursor. Think about it like how a Unix process has a stdin and stdout pipe, except your pipes are postgresql connections.
This approach is great for throughout because you request queued items in batch, and there's no contention between workers. It's admittedly not as good for latency because a new queue item won't be dequeued until the worker it's assigned to finishes its run and restarts with a new cursor. For low latency queues, use FOR UPDATE SKIP LOCKED.