5 ms·
As I understand, with SKIP LOCKED rows would no longer be processed in-order?
by orangepanda 3y ago
As I understand, with SKIP LOCKED rows would no longer be processed in-order?
- riku_iki 3y agoarticle says he also uses "order by" clause, but I am wondering if it will severely limit throughput since all messages will need to be sorted on each lookup, but this probably can be solved by introducing index.
- vore 3y agoIt seems strictly worse to use ORDER BY in this case, since if you're using SKIP LOCKED you should be doing parallel processing anyway, and if you're doing parallel processing, ordering is already going out the window.
- nsonha 3y agoParallel or not, the order is of importance in any queue system.
- vore 3y agoYou have no ordering guarantees, so how can order be important? If 4 work items are scheduled on 4 independent workers, you have no guarantee which will start first or finish first.
- nsonha 3y agoThe order matters in the sense that the 5th jobs should not be atempted before those 4.
- riku_iki 3y agoI think the order matter at least because you want to have some FIFO approximation, otherwise some tasks can forever stuck in queue and never be picked up.
- vore 3y agoThen I think what you actually care about is scheduling fairness, and a strict ordering of execution of job 5 after job 4 is unimportant.
- sarchertech 3y agoUnless you can guarantee that the processing time of each job is exactly the same, if you have multiple workers processing the same queue, you can’t order anything except the start time. You can use locks to effectively break the queue into sub queues so that each sub queue is only being processed by 1 worker. Then you can order that sub queue.
- nsonha 3y agojob should be attempted inthe same order/priority they are enqueued, that's the meaning of the word "queue". That they take varrying amounts of time is another matter.
- worthless-trash 3y agoQueue can clearly mean "work that needs to be completed" not necessarily 'work completed in order'. Your definition is much stricter than it needs to be for most use cases.
- nsonha 3y agoThere is clearly a conceptual difference between a set of things from which you pull things out randomly, and a queue. A queue always has intrinsic criteria to select the next item to be pulled out.
- sarchertech 3y agoThere are many times when the start order doesn’t really matter, and the additional sorting overhead isn’t worth it. In those cases people will still tend to refer to the entity holding the jobs to be processed as a queue despite the fact that it doesn’t strictly follow FIFO order. If they are being technically precise, queue isn’t the correct term, but language changes with context and time. Either way the implementation isn’t wrong if strict start order has been considered and isn’t important.
- 3y ago
- klysm 3y agoDepends on how many consumers you have. If you need order guarantees, then something like the outbox pattern is probably a better fit.
- LennyWhiteJr 3y agoNothing about the outbox pattern guarantees ordering.
- klysm 3y agoIf you use Postgres logical replication, that is not true.
- vore 3y agoYes, but if you're going through the queue with multiple workers in parallel, you lose ordering guarantees anyway.