5 ms·
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https:
by cklee 2y ago
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban https://github.com/sorentwo/oban
Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while maintaining compatibility.
Alternatively, we could consider developing a core job queue library in Rust, with language-specific bindings. This would provide a robust, cross-language solution while leveraging the performance and safety benefits of Rust.
- deleted 2y ago[deleted]
- memset 2y agoI am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq https://github.com/poundifdef/smoothmq It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)
- GordonS 2y agoDoes SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)
- memset 2y agoNot today. It's a work in progress! There are several iterations that I'm working on: 1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently. Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)
- justinclift 2y agoSounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
- memset 2y agoIt does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull. The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.
- agrothberg 2y agoSQS does offer long polling, which looks closer to "push" semantics.
- memset 2y agoFair enough. I do implement long polling!
- earthnail 2y agoThis would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.
- mind-blight 2y agoThis is the exact use case I'm running into right now. I've been looking at BullMQ some it has good typescript support, and is working towards a 1.0 for python. But, I have tried it out in a production stack yet
- jmvoodoo 2y agoWe have been using bullmq in production for just over a year. It is a piece of technology that our team doesn't have to think about, which is pretty much all I could ask for. We did end up adding some additional generics which allows us to get strong typing between producers and consumers. That I think has been a key piece of making it easy to use and avoiding dumb mistakes.
- bgentry 2y agoRiver ( https://riverqueue.com https://riverqueue.com ) is a Postgres background job engine written in Go, which also has insert only clients in other languages. Currently we have these for Ruby and Python: https://github.com/riverqueue/riverqueue-ruby https://github.com/riverqueue/riverqueue-ruby https://github.com/riverqueue/riverqueue-python https://github.com/riverqueue/riverqueue-python
- stephenr 2y agoQless "solves" this problem (in redis) by having all core logic written as lua and executed in redis. You could take a similar approach for pg: define a series of procedures that provide all the required functionality, and then language bindings are all just thin wrappers (to handle language native stuff) around calls to execute a given procedure with the correct arguments.
- rileymichael 2y agoIf you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq.
- tmountain 2y agoAlso, pgmq can run as a TLE (trusted language extension), so you can install it into cloud hosted Postgres solutions like Supabase. We're using pgmq, and it's solid so far.
- vantiro 2y agoWhat I like about https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq is that it can be used with any programming language and does not require any background workers or running a binary in addition to postgres.
- AlphaSite 2y agoRiver is my go to in Golang, it’s really handy to have transactional queuing with a nice little ui.
- kraihx 2y agoFor JavaScript and Perl there is already Minion, which relies on listen/notify + FOR UPDATE SKIP LOCKED. https://github.com/mojolicious/minion.js https://github.com/mojolicious/minion.js https://github.com/mojolicious/minion https://github.com/mojolicious/minion
- deleted 2y ago[deleted]
- ukd1 2y agoA common schema is one nice thing, but imho the win of these db backed queues is being able to do things, including enqueue background jobs in a single transaction. e.g. create user, enqueue welcome email - both get done, or not - with redid-based, this is ... not usually a thing; if you fail to do one, it's left half done, leading to more code etc p.s. I maintain a ruby equivalent called QueueClassic