4 ms·
Durability is more important to this use case than performance. I want the option of picking up a jobs within, say, a minute, but I am willing to accept a much
by maxbond 2y ago
Durability is more important to this use case than performance. I want the option of picking up a jobs within, say, a minute, but I am willing to accept a much worse worst case so long as I never lose track of a job.
Another scenario to consider; a job fails, because of a transient error local to the worker (for instance, it may have exceeded an IP based rate limit on an API it consumes). If we drew another worker, it would succeed without a hitch. I want to put the job back into the queue and pick it up again quickly. But I can't just keep going on the worker I have.
- owl57 2y ago> within, say, a minute With this definition of real-time, you can get pretty far just polling for pending jobs.
- maxbond 2y agoAutoscaling workers polling in a tight loop seems like a way to bring down production to me. Database is getting too many queries, causing the job queue to grow, causing more workers to spin up, causing a higher load of queries, ....
- jhgg 2y agoSet a reasonable maximum for your auto-scaler to avoid this issue.
- owl57 2y ago> in a tight loop Here's the bug. If there are no tasks pending at the moment — you sleep, for example for 10 seconds. The code in the presentation had this handled correctly.
- maxbond 2y agoI expressed myself in a confusing way, I just meant "in a loop" and understood that to be a 60s sleep.
- owl57 2y agoThen the database load from transactions that didn't match a job won't be noticeable until there are thousands of workers. And when (and if!) there are, I don't have experience with that kind of scale but I suspect that all the bottlenecks will still be in the other branch — transactions that modify the database.