3 ms·
w > t / p .... works fine. as long as nothing ever goes wrong ;) Then you have a rabbit queue that has hundreds of rescheds piled up and I have to manually unc
by crucialfelix 10y ago
w > t / p .... works fine. as long as nothing ever goes wrong ;) Then you have a rabbit queue that has hundreds of rescheds piled up and I have to manually unclog the queue.
Also increasing workers isn't so simple. There is only so much memory on a machine, and I don't need or want to add machines. A more efficient loop was the better solution. The machine is now full of work all the time and stable in memory, CPU usage and db access.
This is no longer a problem now that I don't use periodic tasks. "periodic" doesn't describe the nature of the work to be done.
In any case the main problem I needed to solve is that I don't have single tasks that need to happen every p seconds. I have 40 tasks that have different min-max-acceptable-backlog-queue-length, different min-max time periods that they could run in. Different day and night cycle ranges. Some really need to go every 2 minutes, but only if there are emails to send. Some are things to do whenever there is spare time. Some image processing queues need to be done very quickly, for others it can chew on it all night when there are spare cycles.