4 ms·
> distributing the work elsewhere, rather than getting concurrency right locally It's entirely orthogonal concerns. Job queues are for "fire and forget" tasks.
by byroot 4y ago
> distributing the work elsewhere, rather than getting concurrency right locally
It's entirely orthogonal concerns. Job queues are for "fire and forget" tasks. Using Sidekiq and co offer tons of advantage over queuing this in a local thread pool / coroutine or similar.
It provides efficient backoff retries, work distribution, durability, backpressure, and a tons of other goodies.
If you were to just queue these things in-process, you may queue more work than your process can handle which may hurt latency.
It also make deploy of web application awkward. If you know jobs are externally queue, you can stop sending traffic to that process and once all in-flight requests are completed you know it's safe to stop the process. If that process may contains queued work, well who knows when it's safe to restart it.
In process concurrency (or parallelism) is useful for other things (e.g. parallel queries to a service), and that's were you use threads / fibers / ractors / async.