8 ms·

River: A fast, robust job queue for Go and Postgres

360 points by bo0tzz 3y ago

hipadev23 3y ago
What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
- zackkitzmiller 3y ago
  I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
  - eximius 3y ago
    No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data. We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
    - iskela 3y ago
      So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
      - eximius 3y ago
        Basically. For our particular use case, I think we're actually not using notify events. We just insert rows into the outbox table and the poller re-emits as kafka events and deletes successfully emitted events from the table.
      - youerbt 3y ago
        Either business data and job are committed or none of them. Then as you write, either polling or listening to an even worker, can pick it up. Bonus stuff, from implementation perspective, is that if worker selects row FOR UPDATE (locking the job from others to pick up) and dies, Postgres will release the lock after some time, making the job available for other workers.
- brandur 3y ago
  Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far superior user experience. Example: User sign-up where you want to send a verification email. Talking to a foreign API like Mailgun might be a 100 ms to multisecond (worst case scenario) operation — why make the user wait on that? Instead, send it to the background, and give them a tight < 100 ms sign up experience that's so fast that for all intents and purposes, it feels instant.
  - stouset 3y ago
    GP isn’t taking umbrage with the concept of needing to offload work to a background process.
  - hipadev23 3y ago
    > Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.
    - toolz 3y ago
      as far as I'm aware the most popular job queue library in elixir depends on postgres and has performance characteristics that cover the vast majority of background processing needs I've come across. I wonder maybe if you've limited yourself by assuming relational DBs only have features for relational data. That isn't the case now and really hasn't been the case for quite some time now.
    - qaq 3y ago
      Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.
- teraflop 3y ago
  It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where things can get out of sync.
  - deleted 3y ago
    [deleted]
  - hipadev23 3y ago
    > should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.
    - j45 3y ago
      Maybe it’s not designed for that or all use cases and that can make sense. Personally, I need long running jobs.
    - teraflop 3y ago
      I don't understand what you mean. The job is "created" as part of the transaction, so it only becomes visible (and hence eligible to be executed) when the transaction commits.
    - Chris911 3y ago
      The job is queued as part of the transaction. It is executed by a worker outside the scope of the transaction.
    - eximius 3y ago
      That part is somewhat poorly explained. That is a motivating example of why having your job queue system be separate from your system of record can be bad. e.g., 1. Application starts transaction 2. Application updates DB state (business details) 3. Application enqueues job in Redis 4. Redis jobworkers pick up job 5. Redis jobworkers error out 6. Application commits transaction This motivates placing the jobworker state in the same transaction whereas non-DB based job queues have issues like this.
- qaq 3y ago
  Job is not dependent on extant transaction. The bookkeeping of job state runs in the same transaction as your domain state manipulation so you will never get into situation where job domain mutation commited but job state failed to update to complete.
- maherbeg 3y ago
  I think you may be misunderstanding the design here. The transaction for initiating the job is only for queuing. The dequeue and execution of the job happens in a separate process. The example on the home page makes this clear where a user is created and a job is created at the same time. This ensures that the job is queued up with the user creation. If any parts of that initial transaction fails, then the job queuing doesn't actually happen.
latchkey 3y ago
If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.) [0] https://cloud.google.com/tasks https://cloud.google.com/tasks
- brandur 3y ago
  There's a lot to be said about the correctness benefits of a transactional model. The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for. In practice, this leads to a lot of uncertainty around the edges, and operators having to reconcile things manually. There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.
  - latchkey 3y ago
    > what if it fails, or what if you're not sure about whether it failed? This is covered in the GCP Tasks documentation. > There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice. My post was talking about what I would implement if I was doing my own queue, as the authors were. Not about using GCP Tasks.
    - politician 3y ago
      Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack. https://brandur.org/job-drain https://brandur.org/job-drain https://news.ycombinator.com/item?id=15294722 https://news.ycombinator.com/item?id=15294722
victorbjorklund 3y ago
Looks great. For people wondering about wether postgres really is a good choice for a job queue I can recommend checking out Oban in Elixir that has been running in production for many years: https://github.com/sorentwo/oban https://github.com/sorentwo/oban Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps. https://getoban.pro/articles/one-million-jobs-a-minute-with-oban https://getoban.pro/articles/one-million-jobs-a-minute-with-...
- bgentry 3y ago
  Oban is fantastic and has been a huge source of inspiration for us, showing what is possible in this space. In fact I think during my time at Distru we were one of Parker's first customers with Oban Web / Pro :) We've also had a lot of experience with with other libraries like Que ( https://github.com/que-rb/que https://github.com/que-rb/que ) and Sidekiq (https://sidekiq.org/ https://sidekiq.org/) which have certainly influenced us over the years.
  - sorentwo 3y ago
    The very first paying Pro customer, as a matter of fact =) You said back then that you planned on pursuing a Go client; now, four years later, here we are. River looks excellent, and the blog post does a fantastic job explaining all the benefits of job queues in Postgres.
bgentry 3y ago
Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these same problems. I wrote a bit about them here in our docs: https://riverqueue.com/docs/transactional-enqueueing https://riverqueue.com/docs/transactional-enqueueing Ultimately I want to help engineers be able to focus their time on building a reliable product, not chasing down distributed systems edge cases. I think most people underestimate just how far you can get with this model—most systems will never outgrow the scaling constraints and the rest are generally better off not worrying about these problems until they truly need to. Please check out the website and docs for more info. We have a lot more coming but first we want to iron out the API design with the community and get some feedback on what features people are most excited for. https://riverqueue.com/ https://riverqueue.com/
- tombh 3y ago
  At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?
  - mosen 3y ago
    Looks like it’s underway: > We're hard at work on more advanced features including a self-hosted web interface. Sign up to get updates on our progress.
  - bgentry 3y ago
    The UI isn't quite ready for outside consumption yet but it is being worked on. I would love to hear more about what you'd like to see in it if you want to share.
    - fithisux 3y ago
      An Airflow for Gophers?
surprisetalk 3y ago
I love PG job queues! They’re surprisingly easy to implement in plain SQL: [1] https://taylor.town/pg-task https://taylor.town/pg-task The nice thing about this implementation is that you can query within the same transaction window
- rockwotj 3y ago
  Agreed. Shortwave [1] is built completely on this, but with the added layer of having a leasing system that is per user on top of the tasks. So you only need to `SKIP LOCKED` to grab a lease, then you can grab as many tasks as you want and process them in bulk. It allows higher throughput of tasks, and also was required for the use case as the leases where tied to a user and tasks for a single user must be processed in order. [1]: https://www.shortwave.com/ https://www.shortwave.com/
RedShift1 3y ago
All these job queue implementations do the same thing right, SELECT ... FOR UPDATE SKIP LOCKED? Why does every programming language need its own variant?
- rockwotj 3y ago
  To work with each language's drivers
  - RedShift1 3y ago
    But it's the same thing every time. Turn autocommit off, run the SELECT, commit, repeat? Or am I missing something?
    - ddorian43 3y ago
      You just do the update with skip locked and finish the job until time runs out or update the job "liveness". (can be autocommit on & off)
bennyp101 3y ago
Nice, I've been using graphile-worker [0] for a while now, and it handles our needs perfectly, so I can totally see why you want something in the go world. Just skimming the docs, can you add a job directly via the DB? So a native trigger could add a job in? Or does it have to go via a client? [0] https://worker.graphile.org/ https://worker.graphile.org/
rubenfiszel 3y ago
Looks cool and thanks for sharing. Founder of windmill.dev, an open-source, extremely fast workflow engine to run jobs in ts,py,gosh whose most important piece, the queue, is also just rust + postgresql (and mostly the FOR UPDATE SKIP LOCKED). I'd be curious to compare performances once you guys are comfortable with that, we do them openly and everyday on: https://github.com/windmill-labs/windmill/tree/benchmarks https://github.com/windmill-labs/windmill/tree/benchmarks I wasn't aware of the skip B-tree splits and the REINDEX CONCURRENTLY tricks. But curious what do you index in your jobs that use those. We mostly rely on the tag/queue_name (which has a small cardinality), scheduled_for, and running boolean which don't seem good fit for b-trees.
- troft 3y ago
  [dead]
JoshGlazebrook 3y ago
I didn't really see this feature, but I think another good one would be a way to schedule a future job that is not periodic. ie: "schedule job in 1 hr" to where it's either not enqueued or not available to be consumed until (at least) the schedule time.
- bgentry 3y ago
  You've found an underdocumented feature, but in fact River does already do what you're asking for! Check out `ScheduledAt` on the `InsertOpts`: https://pkg.go.dev/github.com/riverqueue/river#InsertOpts https://pkg.go.dev/github.com/riverqueue/river#InsertOpts I'll try to work this into the higher level docs website later today with an example :)
andrewstuart 3y ago
The thing is there are now vast numbers of queue solutions. What’s the goal for the project? Is it to be commercial? If so you face massive headwind because it’s so incredibly easy to implement a queue now.
youerbt 3y ago
Job queues in RDBMS are always so controversial here on HN, which is kinda sad. Not everybody needs insane scale or whatever else a dedicated solutions offer. Not to mention if you already have RDBMS laying around, you don't have to pay for extra complexity.
- deleted 3y ago
  [deleted]
sotraw 3y ago
If you are on Kafka already, there is an alternative to schedule a job without PG [0] [0] https://www.wgtwo.com/blog/kafka-timers/ https://www.wgtwo.com/blog/kafka-timers/
- deleted 3y ago
  [deleted]
gregwebs 3y ago
We are looking right now to use a stable PG job queue built in Go. We have found 2 already existing ones: * neoq: https://github.com/acaloiaro/neoq https://github.com/acaloiaro/neoq * gue: https://github.com/vgarvardt/gue https://github.com/vgarvardt/gue Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out. Gue has been around for a while and is probably serving its users well. Looking forward to trying out River now. I do wonder if neoq and river might be better off joining forces.
- acaloiar 3y ago
  Hey thanks for the plug! River's professional looking website makes me think there are more commercial ambitions behind River than neoq, but maybe I'm wrong. I do think they're very similar options, but neoq has no commercial ambitions. I simply set out to create a good, open source, queue-agnostic solution for Go users. I'm actually switching its license to BSD or MIT this week to highlight that fact.
sorentwo 3y ago
> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level. Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poolers like pgbouncer, which prevent NOTIFY ever triggering, and the extra db load of trigger handling wasn't worth it. [1]: https://github.com/sorentwo/oban/commit/7688651446a76d766f394a9a904fca54d5fd788b https://github.com/sorentwo/oban/commit/7688651446a76d766f39...
- bgentry 3y ago
  That makes a lot of sense, I've had the thought a few times that the NOTIFY overhead could get overwhelming in a high-throughput queue but haven't yet had an opportunity to verify this or experiment with a mechanism for reducing this overhead.
  - joker666 3y ago
    Also pgbouncer.
bojanz 3y ago
This looks like a great effort and I am looking forward to trying it out. I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed previously by the community in posts such as these [1][2][3] I am assuming that bgentry and brandur have strong thoughts on the topic since they avoided the default Go license choice of BSD/MIT, so I'd love to hear more. [1] https://www.makeworld.space/2021/01/lgpl_go.html https://www.makeworld.space/2021/01/lgpl_go.html [2] https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-linking-pitfalls https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-... [3] https://softwareengineering.stackexchange.com/questions/179084/is-there-a-modified-lgpl-license-that-allows-static-linking https://softwareengineering.stackexchange.com/questions/1790...
- deleted 3y ago
  [deleted]
- bgentry 3y ago
  Hi bojanz, to be honest we were not well informed enough on the licensing nuances. I appreciate you sharing these links, please tune into this GitHub issue where we'll give updates soon and make sure any ambiguity is resolved. https://github.com/riverqueue/river/issues/47 https://github.com/riverqueue/river/issues/47
sorentwo 3y ago
The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an attempted_by column - Storing a list of errors inline on the job - The same check constraints and the same compound indexes - Almost the entire table schema, really - Unique jobs with the exact same option names - Table-backed leadership election Please give some credit where it's due. [1]: https://github.com/sorentwo/oban https://github.com/sorentwo/oban
- kamikaz1k 3y ago
  …how else do you spell cancelled? With one l? Wow, learning this on my keyboard as I type this…
- bgentry 3y ago
  Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX. Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different implementations before ultimately apparently landing in a similar place. I'm not quite sure what to say on the spelling of "cancelled" though, I've always written it that way and can't help but read "canceled" like "concealed" in my head :) As I think I mentioned when we first chatted years ago this has been a hobby interest of mine for many years so when a new database queue library pops up I tend to go see how it works. We've been in a bit of a mad dash trying to get this ready for release and didn't even think about crediting the projects that inspired us, but I'll sync with Brandur and make sure we can figure out the right way to do that. I really appreciate you raising your concerns here and would love to talk further if you'd like. I just sent you an email to reconnect.
  - throwawaymaths 3y ago
sa46 3y ago
I wrote our own little Go and Postgres job queue similar in spirit. Some tricks we used: - Use FOR NO KEY UPDATE instead of FOR UPDATE so you don't block inserts into tables with a foreign key relationship with the job table. [1] - We parallelize worker by tenant_id but process a single tenant sequentially. I didn't see anything in the docs about that use case; might be worth some design time. [1]: https://www.migops.com/blog/select-for-update-and-its-behavior-with-foreign-keys-in-postgresql/ https://www.migops.com/blog/select-for-update-and-its-behavi...
molszanski 3y ago
Would love to see an SQLite driver
chuckhend 3y ago
Awesome! Seems like this would be a lot easier to work with and perhaps more performant than Skye's pg-queue? Queue workload is a lot like OLTP, which IMO, makes Postgres great for it (but does require some extra tuning). Unlike https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq a project we've been working on at Tembo, many queue projects still require you to run and manage a process external to the database, like a background worker. Or they ship as a client library and live in your application, which will limit the languages you can chose to work with. PGMQ is a pure SQL API, so any language that can connect to Postgres can use it.
throwawaaarrgh 3y ago
Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)
- whalesalad 3y ago
  A lot has changed in 15 years.
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - throwawaaarrgh 3y ago
    Like?
kromem 3y ago
Nice! A few years ago I wrote my own in house distributed job queue and scheduler in Go on top of Postgres and would have been very happy if a library like this had existed before. The two really are a great pair for this usecase for most small to medium scaled applications, and it's awesome to see someone putting a library out there publicly doing it - great job!!
orefalo 3y ago
Interesting, I would have though a solution like https://temporal.io/ https://temporal.io/ would be more appropriate for these use cases. a job queue might just be the tip of the use cases iceberg... isn't it? in the end it's a pub/sub - I use nats.io workers for this. arf, just read a few comments on this same line down bellow.
- meowtimemania 3y ago
  The main benefit of something like River is simplicity. With River your application might consist of two components, a database, and a code server. Such an architecture is really easy to test, develop, debug and deploy. Adding temporal.io means introducing a third component. More components usually means more complexity. More complexity means more difficult to test, develop, debug and deploy. As with everything, it's all about tradeoffs.
m-a-r-c-e-l 3y ago
Nice problem observation. One solution is the outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html https://microservices.io/patterns/data/transactional-outbox....

360 points by bo0tzz 3y ago

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

zackkitzmiller 3y ago
I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
- eximius 3y ago
  No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data. We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
  - iskela 3y ago
    So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
    - eximius 3y ago
      Basically. For our particular use case, I think we're actually not using notify events. We just insert rows into the outbox table and the poller re-emits as kafka events and deletes successfully emitted events from the table.
    - youerbt 3y ago
      Either business data and job are committed or none of them. Then as you write, either polling or listening to an even worker, can pick it up. Bonus stuff, from implementation perspective, is that if worker selects row FOR UPDATE (locking the job from others to pick up) and dies, Postgres will release the lock after some time, making the job available for other workers.
brandur 3y ago
Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far superior user experience. Example: User sign-up where you want to send a verification email. Talking to a foreign API like Mailgun might be a 100 ms to multisecond (worst case scenario) operation — why make the user wait on that? Instead, send it to the background, and give them a tight < 100 ms sign up experience that's so fast that for all intents and purposes, it feels instant.
- stouset 3y ago
  GP isn’t taking umbrage with the concept of needing to offload work to a background process.
- hipadev23 3y ago
  > Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.
  - toolz 3y ago
    as far as I'm aware the most popular job queue library in elixir depends on postgres and has performance characteristics that cover the vast majority of background processing needs I've come across. I wonder maybe if you've limited yourself by assuming relational DBs only have features for relational data. That isn't the case now and really hasn't been the case for quite some time now.
  - qaq 3y ago
    Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.
teraflop 3y ago
It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where things can get out of sync.
- deleted 3y ago
  [deleted]
- hipadev23 3y ago
  > should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.
  - j45 3y ago
    Maybe it’s not designed for that or all use cases and that can make sense. Personally, I need long running jobs.
  - teraflop 3y ago
    I don't understand what you mean. The job is "created" as part of the transaction, so it only becomes visible (and hence eligible to be executed) when the transaction commits.
  - Chris911 3y ago
    The job is queued as part of the transaction. It is executed by a worker outside the scope of the transaction.
  - eximius 3y ago
    That part is somewhat poorly explained. That is a motivating example of why having your job queue system be separate from your system of record can be bad. e.g., 1. Application starts transaction 2. Application updates DB state (business details) 3. Application enqueues job in Redis 4. Redis jobworkers pick up job 5. Redis jobworkers error out 6. Application commits transaction This motivates placing the jobworker state in the same transaction whereas non-DB based job queues have issues like this.
qaq 3y ago
Job is not dependent on extant transaction. The bookkeeping of job state runs in the same transaction as your domain state manipulation so you will never get into situation where job domain mutation commited but job state failed to update to complete.
maherbeg 3y ago
I think you may be misunderstanding the design here. The transaction for initiating the job is only for queuing. The dequeue and execution of the job happens in a separate process. The example on the home page makes this clear where a user is created and a job is created at the same time. This ensures that the job is queued up with the user creation. If any parts of that initial transaction fails, then the job queuing doesn't actually happen.

latchkey 3y ago

If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.) [0] https://cloud.google.com/tasks https://cloud.google.com/tasks

brandur 3y ago
There's a lot to be said about the correctness benefits of a transactional model. The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for. In practice, this leads to a lot of uncertainty around the edges, and operators having to reconcile things manually. There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.
- latchkey 3y ago
  > what if it fails, or what if you're not sure about whether it failed? This is covered in the GCP Tasks documentation. > There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice. My post was talking about what I would implement if I was doing my own queue, as the authors were. Not about using GCP Tasks.
  - politician 3y ago
    Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack. https://brandur.org/job-drain https://brandur.org/job-drain https://news.ycombinator.com/item?id=15294722 https://news.ycombinator.com/item?id=15294722

victorbjorklund 3y ago

Looks great. For people wondering about wether postgres really is a good choice for a job queue I can recommend checking out Oban in Elixir that has been running in production for many years: https://github.com/sorentwo/oban https://github.com/sorentwo/oban Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps. https://getoban.pro/articles/one-million-jobs-a-minute-with-oban https://getoban.pro/articles/one-million-jobs-a-minute-with-...

bgentry 3y ago
Oban is fantastic and has been a huge source of inspiration for us, showing what is possible in this space. In fact I think during my time at Distru we were one of Parker's first customers with Oban Web / Pro :) We've also had a lot of experience with with other libraries like Que ( https://github.com/que-rb/que https://github.com/que-rb/que ) and Sidekiq (https://sidekiq.org/ https://sidekiq.org/) which have certainly influenced us over the years.
- sorentwo 3y ago
  The very first paying Pro customer, as a matter of fact =) You said back then that you planned on pursuing a Go client; now, four years later, here we are. River looks excellent, and the blog post does a fantastic job explaining all the benefits of job queues in Postgres.

bgentry 3y ago

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these same problems. I wrote a bit about them here in our docs: https://riverqueue.com/docs/transactional-enqueueing https://riverqueue.com/docs/transactional-enqueueing Ultimately I want to help engineers be able to focus their time on building a reliable product, not chasing down distributed systems edge cases. I think most people underestimate just how far you can get with this model—most systems will never outgrow the scaling constraints and the rest are generally better off not worrying about these problems until they truly need to. Please check out the website and docs for more info. We have a lot more coming but first we want to iron out the API design with the community and get some feedback on what features people are most excited for. https://riverqueue.com/ https://riverqueue.com/

tombh 3y ago
At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?
- mosen 3y ago
  Looks like it’s underway: > We're hard at work on more advanced features including a self-hosted web interface. Sign up to get updates on our progress.
- bgentry 3y ago
  The UI isn't quite ready for outside consumption yet but it is being worked on. I would love to hear more about what you'd like to see in it if you want to share.
  - fithisux 3y ago
    An Airflow for Gophers?

surprisetalk 3y ago

I love PG job queues! They’re surprisingly easy to implement in plain SQL: [1] https://taylor.town/pg-task https://taylor.town/pg-task The nice thing about this implementation is that you can query within the same transaction window

rockwotj 3y ago
Agreed. Shortwave [1] is built completely on this, but with the added layer of having a leasing system that is per user on top of the tasks. So you only need to `SKIP LOCKED` to grab a lease, then you can grab as many tasks as you want and process them in bulk. It allows higher throughput of tasks, and also was required for the use case as the leases where tied to a user and tasks for a single user must be processed in order. [1]: https://www.shortwave.com/ https://www.shortwave.com/

RedShift1 3y ago

All these job queue implementations do the same thing right, SELECT ... FOR UPDATE SKIP LOCKED? Why does every programming language need its own variant?

rockwotj 3y ago
To work with each language's drivers
- RedShift1 3y ago
  But it's the same thing every time. Turn autocommit off, run the SELECT, commit, repeat? Or am I missing something?
  - ddorian43 3y ago
    You just do the update with skip locked and finish the job until time runs out or update the job "liveness". (can be autocommit on & off)

bennyp101 3y ago

Nice, I've been using graphile-worker [0] for a while now, and it handles our needs perfectly, so I can totally see why you want something in the go world. Just skimming the docs, can you add a job directly via the DB? So a native trigger could add a job in? Or does it have to go via a client? [0] https://worker.graphile.org/ https://worker.graphile.org/

rubenfiszel 3y ago

Looks cool and thanks for sharing. Founder of windmill.dev, an open-source, extremely fast workflow engine to run jobs in ts,py,gosh whose most important piece, the queue, is also just rust + postgresql (and mostly the FOR UPDATE SKIP LOCKED). I'd be curious to compare performances once you guys are comfortable with that, we do them openly and everyday on: https://github.com/windmill-labs/windmill/tree/benchmarks https://github.com/windmill-labs/windmill/tree/benchmarks I wasn't aware of the skip B-tree splits and the REINDEX CONCURRENTLY tricks. But curious what do you index in your jobs that use those. We mostly rely on the tag/queue_name (which has a small cardinality), scheduled_for, and running boolean which don't seem good fit for b-trees.

troft 3y ago
[dead]

JoshGlazebrook 3y ago

I didn't really see this feature, but I think another good one would be a way to schedule a future job that is not periodic. ie: "schedule job in 1 hr" to where it's either not enqueued or not available to be consumed until (at least) the schedule time.

bgentry 3y ago
You've found an underdocumented feature, but in fact River does already do what you're asking for! Check out `ScheduledAt` on the `InsertOpts`: https://pkg.go.dev/github.com/riverqueue/river#InsertOpts https://pkg.go.dev/github.com/riverqueue/river#InsertOpts I'll try to work this into the higher level docs website later today with an example :)

andrewstuart 3y ago

The thing is there are now vast numbers of queue solutions. What’s the goal for the project? Is it to be commercial? If so you face massive headwind because it’s so incredibly easy to implement a queue now.

youerbt 3y ago

Job queues in RDBMS are always so controversial here on HN, which is kinda sad. Not everybody needs insane scale or whatever else a dedicated solutions offer. Not to mention if you already have RDBMS laying around, you don't have to pay for extra complexity.

deleted 3y ago
[deleted]

sotraw 3y ago

If you are on Kafka already, there is an alternative to schedule a job without PG [0] [0] https://www.wgtwo.com/blog/kafka-timers/ https://www.wgtwo.com/blog/kafka-timers/

deleted 3y ago
[deleted]

gregwebs 3y ago

We are looking right now to use a stable PG job queue built in Go. We have found 2 already existing ones: * neoq: https://github.com/acaloiaro/neoq https://github.com/acaloiaro/neoq * gue: https://github.com/vgarvardt/gue https://github.com/vgarvardt/gue Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out. Gue has been around for a while and is probably serving its users well. Looking forward to trying out River now. I do wonder if neoq and river might be better off joining forces.

acaloiar 3y ago
Hey thanks for the plug! River's professional looking website makes me think there are more commercial ambitions behind River than neoq, but maybe I'm wrong. I do think they're very similar options, but neoq has no commercial ambitions. I simply set out to create a good, open source, queue-agnostic solution for Go users. I'm actually switching its license to BSD or MIT this week to highlight that fact.

sorentwo 3y ago

> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level. Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poolers like pgbouncer, which prevent NOTIFY ever triggering, and the extra db load of trigger handling wasn't worth it. [1]: https://github.com/sorentwo/oban/commit/7688651446a76d766f394a9a904fca54d5fd788b https://github.com/sorentwo/oban/commit/7688651446a76d766f39...

bgentry 3y ago
That makes a lot of sense, I've had the thought a few times that the NOTIFY overhead could get overwhelming in a high-throughput queue but haven't yet had an opportunity to verify this or experiment with a mechanism for reducing this overhead.
- joker666 3y ago
  Also pgbouncer.

bojanz 3y ago

This looks like a great effort and I am looking forward to trying it out. I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed previously by the community in posts such as these [1][2][3] I am assuming that bgentry and brandur have strong thoughts on the topic since they avoided the default Go license choice of BSD/MIT, so I'd love to hear more. [1] https://www.makeworld.space/2021/01/lgpl_go.html https://www.makeworld.space/2021/01/lgpl_go.html [2] https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-linking-pitfalls https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-... [3] https://softwareengineering.stackexchange.com/questions/179084/is-there-a-modified-lgpl-license-that-allows-static-linking https://softwareengineering.stackexchange.com/questions/1790...

deleted 3y ago
[deleted]
bgentry 3y ago
Hi bojanz, to be honest we were not well informed enough on the licensing nuances. I appreciate you sharing these links, please tune into this GitHub issue where we'll give updates soon and make sure any ambiguity is resolved. https://github.com/riverqueue/river/issues/47 https://github.com/riverqueue/river/issues/47

sorentwo 3y ago

The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an attempted_by column - Storing a list of errors inline on the job - The same check constraints and the same compound indexes - Almost the entire table schema, really - Unique jobs with the exact same option names - Table-backed leadership election Please give some credit where it's due. [1]: https://github.com/sorentwo/oban https://github.com/sorentwo/oban

kamikaz1k 3y ago
…how else do you spell cancelled? With one l? Wow, learning this on my keyboard as I type this…
bgentry 3y ago
Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX. Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different implementations before ultimately apparently landing in a similar place. I'm not quite sure what to say on the spelling of "cancelled" though, I've always written it that way and can't help but read "canceled" like "concealed" in my head :) As I think I mentioned when we first chatted years ago this has been a hobby interest of mine for many years so when a new database queue library pops up I tend to go see how it works. We've been in a bit of a mad dash trying to get this ready for release and didn't even think about crediting the projects that inspired us, but I'll sync with Brandur and make sure we can figure out the right way to do that. I really appreciate you raising your concerns here and would love to talk further if you'd like. I just sent you an email to reconnect.
- throwawaymaths 3y ago

sa46 3y ago

I wrote our own little Go and Postgres job queue similar in spirit. Some tricks we used: - Use FOR NO KEY UPDATE instead of FOR UPDATE so you don't block inserts into tables with a foreign key relationship with the job table. [1] - We parallelize worker by tenant_id but process a single tenant sequentially. I didn't see anything in the docs about that use case; might be worth some design time. [1]: https://www.migops.com/blog/select-for-update-and-its-behavior-with-foreign-keys-in-postgresql/ https://www.migops.com/blog/select-for-update-and-its-behavi...

molszanski 3y ago

Would love to see an SQLite driver

chuckhend 3y ago

Awesome! Seems like this would be a lot easier to work with and perhaps more performant than Skye's pg-queue? Queue workload is a lot like OLTP, which IMO, makes Postgres great for it (but does require some extra tuning). Unlike https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq a project we've been working on at Tembo, many queue projects still require you to run and manage a process external to the database, like a background worker. Or they ship as a client library and live in your application, which will limit the languages you can chose to work with. PGMQ is a pure SQL API, so any language that can connect to Postgres can use it.

throwawaaarrgh 3y ago

Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)

whalesalad 3y ago
A lot has changed in 15 years.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- throwawaaarrgh 3y ago
  Like?

kromem 3y ago

Nice! A few years ago I wrote my own in house distributed job queue and scheduler in Go on top of Postgres and would have been very happy if a library like this had existed before. The two really are a great pair for this usecase for most small to medium scaled applications, and it's awesome to see someone putting a library out there publicly doing it - great job!!

orefalo 3y ago

Interesting, I would have though a solution like https://temporal.io/ https://temporal.io/ would be more appropriate for these use cases. a job queue might just be the tip of the use cases iceberg... isn't it? in the end it's a pub/sub - I use nats.io workers for this. arf, just read a few comments on this same line down bellow.

meowtimemania 3y ago
The main benefit of something like River is simplicity. With River your application might consist of two components, a database, and a code server. Such an architecture is really easy to test, develop, debug and deploy. Adding temporal.io means introducing a third component. More components usually means more complexity. More complexity means more difficult to test, develop, debug and deploy. As with everything, it's all about tradeoffs.

m-a-r-c-e-l 3y ago

Nice problem observation. One solution is the outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html https://microservices.io/patterns/data/transactional-outbox....