4 ms·
I elaborated a bit on this elsewhere (https://news.ycombinator.com/item?id=21537414 https://news.ycombinator.com/item?id=21537414) but Que’s design has changed
by bgentry 7y ago
I elaborated a bit on this elsewhere (https://news.ycombinator.com/item?id=21537414 https://news.ycombinator.com/item?id=21537414) but Que’s design has changed significantly and no longer holds open a connection or transaction for the duration of working a job. It holds one database connection per worker process. Each worker process handles all job locking and assignment to the individual worker threads in that process, each of which only use connections that they themselves decide to open.
Otherwise, I completely agree on the benefits of transactional job enqueueing. This lets you push off so much complexity until you actually need it (when you’ve scaled such that a single database doesn’t handle your needs well). At that point, you get to solve the same challenges you will have been solving all along to deal with jobs that might run which depend on transactions that may not have committed (yet or ever). I believe this model is almost always the right starting point for a web application, barring some unusual job requirements or massive initial scale.
- ianai 7y agoIs this what you’re talking about? https://github.com/que-rb/que https://github.com/que-rb/que
- s0l1dsnak3123 7y agoYep!
- deleted 7y ago[deleted]
- zahreeley 7y agoGood job
- Diggsey 7y agoThat is a very neat solution. The one downside I see is that you need to be much more careful about gracefully handling errors within workers. For example: if a single worker thread crashes but the process doesn't realise it, it may be possible for jobs to become stuck, because other workers rely on the process releasing the job back to the queue. It's definitely solvable but something to be aware of.
- grumpy8 7y ago> I believe this model is almost always the right starting point for a web application, barring some unusual job requirements or massive initial scale. Would you run this job queue on the same postgres database as the rest of the application or rather use a different one specific for workers?
- bgentry 7y agoOne of the main benefits of this model is the ability to transactionally enqueue any jobs that are being triggered at the same time as other changes your app is making. Also the ability to transactionally apply changes to your database at the same time as marking the job completed. You only get this benefit if you’re doing everything in a single Postgres database. By the time you outgrow that setup, you may as well move to a more dedicated job queue or something built on Redis, because you’ve already lost the benefits of having a single system. I suppose you could maintain the queue in a separate Postgres database, but without sharing ACID with the app I’m not sure What you’d gain vs another system.