4 ms·
If you're working with Ruby I have had good experiences with Que[1], which implements a pattern similar to the OP using advisory locks. [1]: https://github.com
by zrail 7y ago
If you're working with Ruby I have had good experiences with Que[1], which implements a pattern similar to the OP using advisory locks.
[1]: https://github.com/que-rb/que https://github.com/que-rb/que
- theptip 7y agoIs there a similar library in Python? I really like the look of this approach, but don't want to build it myself if I can avoid it.
- zrail 7y agohttps://pypi.org/project/pq/ https://pypi.org/project/pq/ looks similar, although it looks like they use `SKIP LOCKED` instead of advisory locks. I'm not sure what the tradeoff is.
- fjp 7y agoSKIP LOCKED is much simpler and doesn't really have any gotchas. We had an internal library that used advisory locks which had all sorts of strange behavior we couldn't figure out until we just moved to SKIP LOCKED
- faizshah 7y agoI was looking into python task queuing recently, here's some other choices from my notes: https://github.com/malthe/pq https://github.com/malthe/pq - postgres queue based on rq and ruby queue_classic https://github.com/coleifer/huey https://github.com/coleifer/huey - sqlite, redis and in memory by coleifer (peewee creator). Possible to implement postgres storage layer simply. https://github.com/closeio/tasktiger https://github.com/closeio/tasktiger - flexible redis-based python task queue alternative to celery https://dramatiq.io/ https://dramatiq.io/ - actor based python job queue on redis/rabbitMQ https://github.com/GoogleCloudPlatform/psq https://github.com/GoogleCloudPlatform/psq - gcp pub/sub based task queue
- theptip 7y agoI just remembered that Celery has a DB backend too, but I've not looked deeply; I can't see it doing anything particularly fancy in the implementation: https://github.com/celery/celery/blob/master/celery/backends/database/__init__.py https://github.com/celery/celery/blob/master/celery/backends...
- 3pt14159 7y agoI'll second this recommendation. It's great because it just snaps onto your backups for your regular database and you can query on it if you need to.
- floatboth 7y agohttps://github.com/timgit/pg-boss https://github.com/timgit/pg-boss for node, https://github.com/mbuhot/ecto_job https://github.com/mbuhot/ecto_job for elixir. I wish all these libraries agreed on a common schema >_<
- cobbzilla 7y agoAnyone know of a good Java library for PostgreSQL-backed queues using the "SKIP LOCKED" technique? I've scoured GitHub but can't seem to find anything.