2 ms·
For a small side project, instead of queuing all the tasks or writing a custom fair multi-tenant queuing system, I added all the "tasks" to the db, then with a
by ceritium 3y ago
For a small side project, instead of queuing all the tasks or writing a custom fair multi-tenant queuing system, I added all the "tasks" to the db, then with a periodic job I query with SKIP LOCKED for pending tasks and enqueue them on my queue system (sidekiq, Goodjob, etc). I only enqueue them if there are less than some amount of jobs pending, it is hardcoded, but I could do it dynamically.
The query logic take into consideration the tenants so, a big tenant (or misbehaving) can't block small ones.
The periodic job is executed every few second, but it's also enqueued when added new tasks to the db (unless it's already enqueued) or when the query logic detect that there is more jobs available, so the lag is minimal.
At the moment I am doing this with only one type of jobs, but I want to expand it to most of the jobs, and let the "query logic" balance the queues.