5 ms·
People considering this project should also probably consider Graphile Worker[1] I've scaled Graphile Worker to 10m daily jobs just fine The behavior of this l
by seveibar 2y ago
People considering this project should also probably consider Graphile Worker[1] I've scaled Graphile Worker to 10m daily jobs just fine
The behavior of this library is a bit different and in some ways a bit lower level. If you are using something like this, expect to get very intimate with it as you scale- a lot of times your custom workload would really benefit from a custom index and it's handy to understand how the underlying system works.
[1] https://worker.graphile.org/ https://worker.graphile.org/
- philefstat 2y agohave also used/introduced this to several places I've worked and it's been great each time. My only qualm is it's not particularly easy to modify the exponential back off timing without hacky solutions. Have you ever found a good way to do that?
- valenterry 2y agoIs there something like that in the jvm world?
- RedShift1 2y agoYou don't need anything specific. SELECT ... FROM queue FOR UPDATE SKIP LOCKED is the secret sauce.
- valenterry 2y agoWould be nice to get some goodies for free, like overview, pausing, state, statistics etc. :-)
- chuckhend 2y agoThis is exactly how pgmq is implemented, + the usage of VT.
- wmfiv 2y agohttps://www.jobrunr.io//en/ https://www.jobrunr.io//en/ seems to be popular at the moment.
- valenterry 2y agoThat looks pretty good! Too bad that the OSS version limits the number of jobs quite a bit.
- deleted 2y ago[deleted]
- brycelarkin 2y agoDo you add jobs inside a transaction? Or with a root connection? There doesn't seem to be an out of the box way to add jobs in a transaction.
- philefstat 2y agoyes you can add jobs with sql in a transaction, e.g. SELECT graphile_worker.add_job( 'send_email', json_build_object( 'to', 'someone@example.com', 'subject', 'graphile-worker test' ) ); (can also use the PERFORM statement instead of SELECT)