3 ms·
> within, say, a minute With this definition of real-time, you can get pretty far just polling for pending jobs.
by owl57 2y ago
> within, say, a minute
With this definition of real-time, you can get pretty far just polling for pending jobs.
- maxbond 2y agoAutoscaling workers polling in a tight loop seems like a way to bring down production to me. Database is getting too many queries, causing the job queue to grow, causing more workers to spin up, causing a higher load of queries, ....
- jhgg 2y agoSet a reasonable maximum for your auto-scaler to avoid this issue.
- owl57 2y ago> in a tight loop Here's the bug. If there are no tasks pending at the moment — you sleep, for example for 10 seconds. The code in the presentation had this handled correctly.
- maxbond 2y agoI expressed myself in a confusing way, I just meant "in a loop" and understood that to be a 60s sleep.
- owl57 2y agoThen the database load from transactions that didn't match a job won't be noticeable until there are thousands of workers. And when (and if!) there are, I don't have experience with that kind of scale but I suspect that all the bottlenecks will still be in the other branch — transactions that modify the database.