25 ms·

GoodJob – a Postgres-based ActiveJob back end for Ruby on Rails

178 points by another-dave 6y ago

abc_321 6y ago
this is a test comment
simplify 6y ago
This is the exact sort of thing I love about the Rails community. Always innovating to keep things simple.
another-dave 6y ago
It would be nice to have one less dependency in the tech stack but honestly have no complaints with Sidekiq so far. I'm quite new to Ruby/Rails though — would be interesting to hear from others how they think it stacks up (unfortunately the author doesn't compare in the blog post)
- dyeje 6y ago
  Yea I agree, a comparison to sidekiq would be helpful in evaluating and was what I was expecting to be covered on the page.
- simplify 6y ago
  Agreed, Sidekiq is great and I don't imagine anyone switching away from it for existing projects. But I can see GoodJob being better for spinning up new projects, as there are fewer moving parts to worry about, and it's easy to upgrade to Sidekiq, etc. when you need to (thanks to ActiveJob's standard abstraction).
- cactus2093 6y ago
  The standard advice used to be that neither Redis or Postgres are very good for using as a queue. The creator and former maintainer of Redis was up until a few years ago discouraging its use as a queue, I think mainly because of its lack of durability and high availability at the time. He built a prototype Disque[0] to address the issues but it never became production ready. The other downside is that Redis is in-memory which means the queues have less capacity/are more expensive for the same capacity than an on-disk solution, but as memory gets cheaper over the years this becomes less and less of an issue. The upside is the throughput of Redis is very high. I have personally worked on rails apps using redis-based queues like resque (and to a lesser extent sidekiq), and actually haven't run into any redis crashes or downtime in years of runtime, redis is very solid in general. You can also snapshot the redis instance periodically, to limit the number of jobs you would lose if it did crash. In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects. The issue is that eventually your queues will back up, whether it's due to a bug, surge of traffic, or just complex interaction of behavior in your app that cascades a ton of jobs at once when you run a backfill or something. When your app starts to get overloaded it's pretty trivial to increase the number of web instances running, so your bottleneck in these situations is going to be the db performance. As your queues get backed up, your queue workers are running at full speed processing jobs nonstop, which puts strain on your DB. Additionally, the act of enqueueing and dequeuing a job itself also puts strain on the db, so you can easily get into an unstable situation where each job that gets added to the queue makes every other job take longer. If you allocate a separate DB instance that is only running your queue, that is much safer. Still, a DB like postgres is not great at doing constant writes and deletes, it creates additional auto-vacuum pressure for instance. But this will manifest as just getting worse throughput on the same hardware than you could get from a dedicated queue like rabbit mq, so if you're not at large scale it's a fine option. Edit: And one other thing to add, for a lot of web apps the scope of what is needed from a queue these days is a lot less now than it was in the past. It used to be, and in large enterprise systems it often still is, the case that when people talked about a message queue they wanted something to facilitate passing messages between many completely separate apps. Now most apps just use a rest api for that (or perhaps protobufs or graphql or something but still over http). So I think historically an additional reason against using a simple datastore as a queue was that it didn't have enough features so you'd end up re-inventing the wheel with things like brokers, fan out and broadcast patterns, at-most-once vs at-least-once semantics, etc. But here I'm just considering the very limited usecase of a sidekiq-like queue, for processing jobs in the background for a single web app. tl;dr: Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong. [0] https://github.com/antirez/disque https://github.com/antirez/disque
  - learc83 6y ago
    That completely depends on the amount of items you plan on adding to the queue. I’ve used the primary DB as a queue often over the last 10 years for things like sending notifications. Sure there’s a point where this doesn’t make sense, but for the majority of cases there are ways to mitigate the positive reinforcement loop you’re taking about. The easiest is to limit the number of workers to some number that won’t impact DB performance if they are running full tilt. You can even use an enum on existing records to determine the background job status if you have few enough rows (we do this for a table with a few hundred thousand job applicants). I’ve found that in most cases we want a record that the background job was performed, so we were often updating the database anyway when a job was complete. Sure if you are firing off so many events that PG can’t keep up with writing them, then PG isn’t a good option.
  - pselbert 6y ago
    > Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong. That is a broad generalization that assumes most applications are operating at mega scale. The benefits of simplified dependencies (a single database instance), transactional guarantees (a single database instance) and persistence (not using Redis) far outweigh the eventual possibility that the queue will place too large a load on your database. As the author of Oban[0] (an PG backed persistent queue in Elixir) I'm definitely biased. However, the level of adoption in the Elixir community seems to signal that a lot of companies favor simplicity and safety over a possible scale issue down the road. The primary application I work on processes ~500k-1m jobs a day and the queue overhead is virtually invisible. [0] https://github.com/sorentwo/oban https://github.com/sorentwo/oban
    - cactus2093 6y ago
      You actually bring up another point that in my mind is yet another argument against using your primary DB as a queue, so I'm sure I am also biased from being burned in the past :) But in a past company I worked at, the company started out thinking sure just throw the queue in the primary db for simplicity. Eventually our slow query logs and db performance monitoring tools were showing that ~40% of the db load was due to the queue inserts and queries. It may be that it was doing something incredibly inefficient and unnecessary in the particular library we were using but we did look into it pretty thoroughly, this was a few years ago though so I don't recall all of the details. And that was at normal operation, then we ran into an issue that basically brought our site down when queues backed up. At that point it was definitely time to split out the queue, and when we did it we realized that we had implicitly been depending on transactional consistency between the queue and the app data in a few places, which was then extra work to track down and fix these types of issues. This is IMO a code smell as well in general - your data and your infrastructure should ideally not be so tightly coupled. Managed databases are so easy to set up these days, I would still definitely recommend a separate instance for the queue vs the primary db from day 1 in any new app I build. If you do want to combine them to save money on infra, use a separate logical db and separate connection pool and everything so that it's easy to split out in the future.
- chriszhu12 6y ago
  Another small thing to consider: Adding redis to your stack adds a new failure mode. A service I built a while ago was running on Heroku where Redis add-ons aren't cheap, and our Redis would always running against memory limits, which would cause our entire service to fail. Meanwhile, 250GB Postgres server has been churning along no problem. Also, conceptually, architecture where state = DB, and business logic = web server is generally easier to reason about during service migrations and such. If you don't use ActionCable which I think is the other core feature that relies on Redis, being able to remove it would've saved me a lot of early morning firefighting.
  - debaserab2 6y ago
    This is exactly why I stuck with DelayedJob after all of these years. The community seemed to flock to Resque or Sidekiq, but I always felt my needs were met just fine with Delayed Job. I never was interested in adding more unnecessary complexity to our dev and staging environments for a trade-off I didn't need (there wouldn't be a significant performance increase for my needs). I've seen this pattern happen with a lot of Ruby projects: there's a popular Gem that people use that grows over time, then someone writes a blog post about why that package isn't suitable for their needs due to a design trade-off (often introducing a new Gem that is advertised as superior to the previous one) and then suddenly the old Gem stagnates, causing the maintainer to lose interest and updates stop pacing Rails versions. Sometimes the old Gem gets deprecated altogether and maintainers stick a big DO NOT USE sign at the top of the github README, all but guaranteeing there is no further community traction or organization for the project. Meanwhile those of us using the old project chug along with a forked Gem that we cobble a bunch of patches into to meet our needs because there's no longer a centralized place to contribute to anymore. In some ways it's the double-edged sword of OSS. Maintainers aren't obligated to stay as maintainers of course, but it does get frustrating how quickly the community is willing to drop support of things that a lot people are using in production systems. It definitely makes me reconsider the "don't reinvent the wheel" advice that is so adamantly thought of as a common sense convention in our craft.
    - learc83 6y ago
      That’s exactly why I implemented my own queue system. If it’s something that will only take a few hours to build, and I’ll need to maintain it long term, I’ll always favor rolling it myself. There are very few “feature gems” that I’ve used in the past 15 years doing Rails development that I’ve been happy I used a year later. Development gems on the other hand like byebug are usually fine.
- mberning 6y ago
  Sidekiq is one of the best pieces of software I have ever used.
bmn__ 6y ago
Also see: https://news.ycombinator.com/item?id=9576864 https://news.ycombinator.com/item?id=9576864 Postgres Job Queues and Failure by MVCC
block_dagger 6y ago
Keeping your jobs in Redis (like Sidekiq does) seems to be a good feature to me. I’m curious how GoodJob compares when it comes to UI and queue management.
shay_ker 6y ago
I'm excited about ActiveJob because, finally, people will be able to build background jobs on things that aren't Redis. No hate on Redis - I'm a huge fan of it - but sometimes you have stronger consistency reqs that Redis does not have, and stronger availability reqs that SQS or PG don't have (aka multi-region jobs). If the concurrency in ActiveJob become as good or better than Sidekiq, we can really get a lot of interesting stuff out there.
- scruple 6y ago
  I used Que [0] in a number of projects in the last few years that had stronger consistency demands. It can plug into ActiveJob. Those projects are still humming along without trouble today. [0]: https://github.com/que-rb/que https://github.com/que-rb/que
  - nthj 6y ago
    Que also has a Golang port [1], which can be a tool to get towards The Citadel [2] for performance or dependency-heavy jobs without going full SOA [1] https://github.com/bgentry/que-go https://github.com/bgentry/que-go [2] https://m.signalvnoise.com/the-majestic-monolith-can-become-the-citadel/ https://m.signalvnoise.com/the-majestic-monolith-can-become-...
  - arcticfox 6y ago
    As a fellow Que user, I'm happy to see this line in the README: > For example, GoodJob is currently ~600 lines of code, whereas Que is ~1,200 lines, I've tried to dig into the Que internals before (looking into deadlocks) and they were gnarly; skimming through the GoodJob internals, it looks a lot more like what I'd hope for.
  - Exuma 6y ago
    Unfortunately its no longer maintained and his 2.0 branch is completely broken for me. I've been very stressed trying to find a que replacement, as almost all of my needs for job need ACID guarantees, and Sidekiq/Redis "best effort" is nowhere near what I need. I also refuse to pay $xxxx for Sidekiq pro just to get those guarantees. Good job looks pretty awesome, I'll have to give it a shot at some point.
dorianmariefr 6y ago
i'm not sure why i should use this instead of delayed job for instance
- pqdbr 6y ago
  If you're using delayedjob, I strongly suggest you check out the Performance table in Sidekiq's readme. 465 seconds for DelayedJob against 14 seconds for Sidekiq for the same workload. It's 7100 jobs/sec x 215 jobs/sec. I wonder how GoodJob will compare. https://github.com/mperham/sidekiq#performance https://github.com/mperham/sidekiq#performance
  - mbell 6y ago
    That benchmark is just showing job overhead. That Sidekiq can do nothing 7100 times a second only matters when overhead is significant, i.e. when your jobs do almost nothing.
- thibaut_barrere 6y ago
  delayed_job hasn't seen a release in 1 year, if that matters to you https://rubygems.org/gems/delayed_job/ https://rubygems.org/gems/delayed_job/
joemclarke 6y ago
This is awesome, thanks for sharing! Sidekiq works well for me but it’s good to have an option that doesn’t require Redis.
jbverschoor 6y ago
Hm. Got falsely excited. No listen/notify etc? This really is just delayed_job
- sideofbacon 6y ago
  GoodJob author here. I'm curious why you're specifically looking for listen/notify? I'm trying to prioritize that in the backlog.
  - juandazapata 6y ago
    Hi, sorry to hijack this thread, but since you mentioned you're the author, I just wanted to ask you something. Do you have any plans to use a different locking mechanism other than the advisory lock? I'm asking because eventually every postgres-backed app that reaches a certain scale, requires a connection pooler (pgbouncer being the de-facto standard nowadays). We usually run pgbouncer in transaction mode, which doesn't support advisory locks. Any thoughts?
    - sideofbacon 6y ago
      That was mentioned in another thread. I have two thoughts about it: - Going off of my own experience, I've never used PgBouncer and I've worked on some moderately sized applications (1s of millions of users). So I could say "not in scope for the target use-case". - It would be possible to add an argument option to use a transaction-based Advisory Lock instead. I can imagine that would have downsides (long-running transactions, yikes). I'm imagining that the lifecycle of GoodJob is that people starting new projects might choose it, and then the feature set would grow with their needs.
bdcravens 6y ago
In this post, I noticed comparison with other job libraries, but not Sidekiq, the most popular one. Why?
- geoffharcourt 6y ago
  I think they specifically wanted to compare themselves to queue systems that use the database as the storage for queued jobs.
- sideofbacon 6y ago
  GoodJob author here. The reason is that I don't have any production experience with Sidekiq. My target right now is someone, like myself, who would typically spin up a new project with delayed_job or que, and provide a better alternative with GoodJob. Maybe in the future I'll do some more research on understanding why someone might switch from a Sidekiq/Redis to not-Redis.
mhoad 6y ago
I feel like Rails is really starting to see some new life that has felt a bit absent from it for a while now. It almost seems like there is a new generation of people extremely burned out from the constant churn of the “JS-all-the-things” movement that just took over most of web development for the past several years. There is a lot of exciting stuff landing in Rails and it’s surrounding ecosystems at the moment and I think it’s still probably the best choice for many new SAAS / web app based companies and startups.
- WrtCdEvrydy 6y ago
  Rails was always great, JS just paid a lot more... It felt great to get back into it having people pay to migrate their Rails apps to React and Node.
  - gav 6y ago
    It's funny you say that, I've seen people pick a tech stack based on developer cost. Maybe five years back I had a client in NYC who had a predominantly Java stack, but due to internal politics, wouldn't pay more than 135k for a senior developer when the market was wanting at least 25-30k more than that. Everyone they hired was pretty terrible. At the time, Node developers were cheaper, so we split the application in two, wrapped all the old Java code with APIs, which was then able to be maintained by a small team while the front-end and any new features were all re-written with Node. It was architectural a better system--but the driving factor was really developer cost (the business hated to make any investments in technology). The problem was that Node got popular and it got harder and harder to hire good Node developers at what the company wanted to pay. They had a big re-org and fired everyone I knew, so I don't know what they are doing now, but I often wonder if they just repeated the process with whatever up-and-coming language was more hip.
    - WrtCdEvrydy 6y ago
      > I've seen people pick a tech stack based on developer cost You'd be surprised how often cost of developers comes into the overall conversation especially when you have tech sprawl and have to spend to get qualified people. I've seen projects green-light just on the premise that removing legacy code will lead to a payroll reduction.
dwheeler 6y ago
This looks very promising! I'm currently using active job, so I will need to take a look at this. Thanks for letting me know.
rubyn00bie 6y ago
Despite my username, I haven't written any serious Ruby is like 5+ years; that is to say, I'm fairly removed from a lot of the changes in the Rails world. Now, pardon me for my ignorance, but... isn't this just like Resque back in the day? Or like doesn't ActiveJob just support any ol' backend outta the box? What's the real use case for this? I'm genuinely curious not trying to be rude or diminish the work.
- techscruggs 6y ago
  You can implement background processing with less operational complexity. This allows you to leverage activejob to get something up and running quickly and easily. Granted, if you start having higher throughput, you want to move on from using postgres as your message bus.
- why-el 6y ago
  I didn't look into this one yet besides the home page, but I always thought Postgres would make a killer backend for ActiveJob, leveraging advisory locks for synchronizing jobs and such. Resque is Redis backed, but if you don't want to add that as a dependency and already have Postgres setup, then this would be a great fit.
  - paulryanrogers 6y ago
    Because of the MVCC implementation (needing VACUUM) it's not ideal without tuning.
- thibaut_barrere 6y ago
  I believe Resque was fork based (at least initially, it may have changed, I haven't looked at it in a while!), while GoodJob appears to be multithreaded (like Sidekiq), so this is quite different in terms of performance & throughput.
- sideofbacon 6y ago
  GoodJob author here. The core reason I wrote it was to see how lean/simple I could write a database-backed ActiveJob adapter today. And hope In practice any adapter should _just work_, but also all the other adapters pre-date ActiveJob (Rails 4.2) and Concurrent::Ruby (adopted in Rails 5.0 I think). The assumption is that building GoodJob on top of what already exists in Rails today, it can be performant (enough) and simple (easy to understand, maintain, keep compatible with new versions of Rails, etc). That's not a big claim, but maybe it's compelling.
techscruggs 6y ago
This looks great. I'd love to remove Redis from my tech stack on applications with low throughput background jobs. Sadly, it doesn't appear to have a UI. Resque and Sidekiq are both good enough solutions, but what makes them my defacto choice is the robust web UIs that comes with them.
samokhvalov 6y ago
To make it scale well, two things are required: 1. TRUNCATE if cleanup is needed (no DELETEs) + partitioning. 2. SELECT .. FOR UPDATE SKIP LOCKED or advisory locks. It's worth learning from Skype's PgQ developed in 2000-s. Update: found the work with advisory locks in the code, great.
SkyLinx 6y ago
Would this work with pgbouncer? I'm wondering because of the advisory locks.
- sideofbacon 6y ago
  GoodJob author here. Excellent observation. It should work if PgBouncer is using session-level pooling, but not transaction-level pooling. I made a ticket to document this: https://github.com/bensheldon/good_job/issues/52 https://github.com/bensheldon/good_job/issues/52 I don't have much experience with PgBouncer. What's the scale/need when a team would use PgBouncer? That would help me prioritize supporting it better.
  - Exuma 6y ago
    The main reason I can think of would be when you have hundreds of workers accessing a DB but the DB only has a small connection limit, say 200 or something. You can have the workers access the connection through pgBouncer to bring that up into the thousands.
    - sideofbacon 6y ago
      Thanks! That's really helpful context.
cpursley 6y ago
For a very nice library in the Elixir/Phoenix ecosystem have a look at https://github.com/sorentwo/oban https://github.com/sorentwo/oban
- regulation_d 6y ago
  I think one of my biggest annoyances, from a development perspective, with Sidekiq is that it typically moves you in the direction of running foreman or something like that, which makes using the pry a pain. One thing I really like about Phoenix development is that I don't have to change my flow to use a REPL while still being able to run background jobs.
Jack_uio 6y ago
I was just assigned to a project based on zend 2.x php framework (2015 version) and angularjs(version1) (not angular.io). Just looking at the code, I just want to scream, because I was told to make changes and add features. I could re-implement everything in Rails in about a week's time but manager's opinions just drown me.
BilalBudhani 6y ago
This looks like a great library on the surface. I would very much like to have a simpler alternative to Sidekiq for projects which doesn't require that level of complexity. Also, not having to add Redis just to run background processing makes life so much easier. I would like to thank the author of the library for all their efforts.

178 points by another-dave 6y ago

abc_321 6y ago

this is a test comment

simplify 6y ago

This is the exact sort of thing I love about the Rails community. Always innovating to keep things simple.

another-dave 6y ago

It would be nice to have one less dependency in the tech stack but honestly have no complaints with Sidekiq so far. I'm quite new to Ruby/Rails though — would be interesting to hear from others how they think it stacks up (unfortunately the author doesn't compare in the blog post)

dyeje 6y ago
Yea I agree, a comparison to sidekiq would be helpful in evaluating and was what I was expecting to be covered on the page.
simplify 6y ago
Agreed, Sidekiq is great and I don't imagine anyone switching away from it for existing projects. But I can see GoodJob being better for spinning up new projects, as there are fewer moving parts to worry about, and it's easy to upgrade to Sidekiq, etc. when you need to (thanks to ActiveJob's standard abstraction).
cactus2093 6y ago
The standard advice used to be that neither Redis or Postgres are very good for using as a queue. The creator and former maintainer of Redis was up until a few years ago discouraging its use as a queue, I think mainly because of its lack of durability and high availability at the time. He built a prototype Disque[0] to address the issues but it never became production ready. The other downside is that Redis is in-memory which means the queues have less capacity/are more expensive for the same capacity than an on-disk solution, but as memory gets cheaper over the years this becomes less and less of an issue. The upside is the throughput of Redis is very high. I have personally worked on rails apps using redis-based queues like resque (and to a lesser extent sidekiq), and actually haven't run into any redis crashes or downtime in years of runtime, redis is very solid in general. You can also snapshot the redis instance periodically, to limit the number of jobs you would lose if it did crash. In terms of using a primary db like postgres or mysql as a queue, I have personally run into issues with this multiple times. I would recommend never to do it, except on the smallest of side projects. The issue is that eventually your queues will back up, whether it's due to a bug, surge of traffic, or just complex interaction of behavior in your app that cascades a ton of jobs at once when you run a backfill or something. When your app starts to get overloaded it's pretty trivial to increase the number of web instances running, so your bottleneck in these situations is going to be the db performance. As your queues get backed up, your queue workers are running at full speed processing jobs nonstop, which puts strain on your DB. Additionally, the act of enqueueing and dequeuing a job itself also puts strain on the db, so you can easily get into an unstable situation where each job that gets added to the queue makes every other job take longer. If you allocate a separate DB instance that is only running your queue, that is much safer. Still, a DB like postgres is not great at doing constant writes and deletes, it creates additional auto-vacuum pressure for instance. But this will manifest as just getting worse throughput on the same hardware than you could get from a dedicated queue like rabbit mq, so if you're not at large scale it's a fine option. Edit: And one other thing to add, for a lot of web apps the scope of what is needed from a queue these days is a lot less now than it was in the past. It used to be, and in large enterprise systems it often still is, the case that when people talked about a message queue they wanted something to facilitate passing messages between many completely separate apps. Now most apps just use a rest api for that (or perhaps protobufs or graphql or something but still over http). So I think historically an additional reason against using a simple datastore as a queue was that it didn't have enough features so you'd end up re-inventing the wheel with things like brokers, fan out and broadcast patterns, at-most-once vs at-least-once semantics, etc. But here I'm just considering the very limited usecase of a sidekiq-like queue, for processing jobs in the background for a single web app. tl;dr: Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong. [0] https://github.com/antirez/disque https://github.com/antirez/disque
- learc83 6y ago
  That completely depends on the amount of items you plan on adding to the queue. I’ve used the primary DB as a queue often over the last 10 years for things like sending notifications. Sure there’s a point where this doesn’t make sense, but for the majority of cases there are ways to mitigate the positive reinforcement loop you’re taking about. The easiest is to limit the number of workers to some number that won’t impact DB performance if they are running full tilt. You can even use an enum on existing records to determine the background job status if you have few enough rows (we do this for a table with a few hundred thousand job applicants). I’ve found that in most cases we want a record that the background job was performed, so we were often updating the database anyway when a job was complete. Sure if you are firing off so many events that PG can’t keep up with writing them, then PG isn’t a good option.
- pselbert 6y ago
  > Never use your primary DB as a queue. Using a separate Postgres instance can work if you over-provision capacity and don't need to maximize throughput, and a Redis-based solution can work if you don't need high availability and can tolerate some messages lost if something goes wrong. That is a broad generalization that assumes most applications are operating at mega scale. The benefits of simplified dependencies (a single database instance), transactional guarantees (a single database instance) and persistence (not using Redis) far outweigh the eventual possibility that the queue will place too large a load on your database. As the author of Oban[0] (an PG backed persistent queue in Elixir) I'm definitely biased. However, the level of adoption in the Elixir community seems to signal that a lot of companies favor simplicity and safety over a possible scale issue down the road. The primary application I work on processes ~500k-1m jobs a day and the queue overhead is virtually invisible. [0] https://github.com/sorentwo/oban https://github.com/sorentwo/oban
  - cactus2093 6y ago
    You actually bring up another point that in my mind is yet another argument against using your primary DB as a queue, so I'm sure I am also biased from being burned in the past :) But in a past company I worked at, the company started out thinking sure just throw the queue in the primary db for simplicity. Eventually our slow query logs and db performance monitoring tools were showing that ~40% of the db load was due to the queue inserts and queries. It may be that it was doing something incredibly inefficient and unnecessary in the particular library we were using but we did look into it pretty thoroughly, this was a few years ago though so I don't recall all of the details. And that was at normal operation, then we ran into an issue that basically brought our site down when queues backed up. At that point it was definitely time to split out the queue, and when we did it we realized that we had implicitly been depending on transactional consistency between the queue and the app data in a few places, which was then extra work to track down and fix these types of issues. This is IMO a code smell as well in general - your data and your infrastructure should ideally not be so tightly coupled. Managed databases are so easy to set up these days, I would still definitely recommend a separate instance for the queue vs the primary db from day 1 in any new app I build. If you do want to combine them to save money on infra, use a separate logical db and separate connection pool and everything so that it's easy to split out in the future.
chriszhu12 6y ago
Another small thing to consider: Adding redis to your stack adds a new failure mode. A service I built a while ago was running on Heroku where Redis add-ons aren't cheap, and our Redis would always running against memory limits, which would cause our entire service to fail. Meanwhile, 250GB Postgres server has been churning along no problem. Also, conceptually, architecture where state = DB, and business logic = web server is generally easier to reason about during service migrations and such. If you don't use ActionCable which I think is the other core feature that relies on Redis, being able to remove it would've saved me a lot of early morning firefighting.
- debaserab2 6y ago
  This is exactly why I stuck with DelayedJob after all of these years. The community seemed to flock to Resque or Sidekiq, but I always felt my needs were met just fine with Delayed Job. I never was interested in adding more unnecessary complexity to our dev and staging environments for a trade-off I didn't need (there wouldn't be a significant performance increase for my needs). I've seen this pattern happen with a lot of Ruby projects: there's a popular Gem that people use that grows over time, then someone writes a blog post about why that package isn't suitable for their needs due to a design trade-off (often introducing a new Gem that is advertised as superior to the previous one) and then suddenly the old Gem stagnates, causing the maintainer to lose interest and updates stop pacing Rails versions. Sometimes the old Gem gets deprecated altogether and maintainers stick a big DO NOT USE sign at the top of the github README, all but guaranteeing there is no further community traction or organization for the project. Meanwhile those of us using the old project chug along with a forked Gem that we cobble a bunch of patches into to meet our needs because there's no longer a centralized place to contribute to anymore. In some ways it's the double-edged sword of OSS. Maintainers aren't obligated to stay as maintainers of course, but it does get frustrating how quickly the community is willing to drop support of things that a lot people are using in production systems. It definitely makes me reconsider the "don't reinvent the wheel" advice that is so adamantly thought of as a common sense convention in our craft.
  - learc83 6y ago
    That’s exactly why I implemented my own queue system. If it’s something that will only take a few hours to build, and I’ll need to maintain it long term, I’ll always favor rolling it myself. There are very few “feature gems” that I’ve used in the past 15 years doing Rails development that I’ve been happy I used a year later. Development gems on the other hand like byebug are usually fine.
mberning 6y ago
Sidekiq is one of the best pieces of software I have ever used.

bmn__ 6y ago

Also see: https://news.ycombinator.com/item?id=9576864 https://news.ycombinator.com/item?id=9576864 Postgres Job Queues and Failure by MVCC

block_dagger 6y ago

Keeping your jobs in Redis (like Sidekiq does) seems to be a good feature to me. I’m curious how GoodJob compares when it comes to UI and queue management.

shay_ker 6y ago

I'm excited about ActiveJob because, finally, people will be able to build background jobs on things that aren't Redis. No hate on Redis - I'm a huge fan of it - but sometimes you have stronger consistency reqs that Redis does not have, and stronger availability reqs that SQS or PG don't have (aka multi-region jobs). If the concurrency in ActiveJob become as good or better than Sidekiq, we can really get a lot of interesting stuff out there.

scruple 6y ago
I used Que [0] in a number of projects in the last few years that had stronger consistency demands. It can plug into ActiveJob. Those projects are still humming along without trouble today. [0]: https://github.com/que-rb/que https://github.com/que-rb/que
- nthj 6y ago
  Que also has a Golang port [1], which can be a tool to get towards The Citadel [2] for performance or dependency-heavy jobs without going full SOA [1] https://github.com/bgentry/que-go https://github.com/bgentry/que-go [2] https://m.signalvnoise.com/the-majestic-monolith-can-become-the-citadel/ https://m.signalvnoise.com/the-majestic-monolith-can-become-...
- arcticfox 6y ago
  As a fellow Que user, I'm happy to see this line in the README: > For example, GoodJob is currently ~600 lines of code, whereas Que is ~1,200 lines, I've tried to dig into the Que internals before (looking into deadlocks) and they were gnarly; skimming through the GoodJob internals, it looks a lot more like what I'd hope for.
- Exuma 6y ago
  Unfortunately its no longer maintained and his 2.0 branch is completely broken for me. I've been very stressed trying to find a que replacement, as almost all of my needs for job need ACID guarantees, and Sidekiq/Redis "best effort" is nowhere near what I need. I also refuse to pay $xxxx for Sidekiq pro just to get those guarantees. Good job looks pretty awesome, I'll have to give it a shot at some point.

dorianmariefr 6y ago

i'm not sure why i should use this instead of delayed job for instance

pqdbr 6y ago
If you're using delayedjob, I strongly suggest you check out the Performance table in Sidekiq's readme. 465 seconds for DelayedJob against 14 seconds for Sidekiq for the same workload. It's 7100 jobs/sec x 215 jobs/sec. I wonder how GoodJob will compare. https://github.com/mperham/sidekiq#performance https://github.com/mperham/sidekiq#performance
- mbell 6y ago
  That benchmark is just showing job overhead. That Sidekiq can do nothing 7100 times a second only matters when overhead is significant, i.e. when your jobs do almost nothing.
thibaut_barrere 6y ago
delayed_job hasn't seen a release in 1 year, if that matters to you https://rubygems.org/gems/delayed_job/ https://rubygems.org/gems/delayed_job/

joemclarke 6y ago

This is awesome, thanks for sharing! Sidekiq works well for me but it’s good to have an option that doesn’t require Redis.

jbverschoor 6y ago

Hm. Got falsely excited. No listen/notify etc? This really is just delayed_job

sideofbacon 6y ago
GoodJob author here. I'm curious why you're specifically looking for listen/notify? I'm trying to prioritize that in the backlog.
- juandazapata 6y ago
  Hi, sorry to hijack this thread, but since you mentioned you're the author, I just wanted to ask you something. Do you have any plans to use a different locking mechanism other than the advisory lock? I'm asking because eventually every postgres-backed app that reaches a certain scale, requires a connection pooler (pgbouncer being the de-facto standard nowadays). We usually run pgbouncer in transaction mode, which doesn't support advisory locks. Any thoughts?
  - sideofbacon 6y ago
    That was mentioned in another thread. I have two thoughts about it: - Going off of my own experience, I've never used PgBouncer and I've worked on some moderately sized applications (1s of millions of users). So I could say "not in scope for the target use-case". - It would be possible to add an argument option to use a transaction-based Advisory Lock instead. I can imagine that would have downsides (long-running transactions, yikes). I'm imagining that the lifecycle of GoodJob is that people starting new projects might choose it, and then the feature set would grow with their needs.

bdcravens 6y ago

In this post, I noticed comparison with other job libraries, but not Sidekiq, the most popular one. Why?

geoffharcourt 6y ago
I think they specifically wanted to compare themselves to queue systems that use the database as the storage for queued jobs.
sideofbacon 6y ago
GoodJob author here. The reason is that I don't have any production experience with Sidekiq. My target right now is someone, like myself, who would typically spin up a new project with delayed_job or que, and provide a better alternative with GoodJob. Maybe in the future I'll do some more research on understanding why someone might switch from a Sidekiq/Redis to not-Redis.

mhoad 6y ago

I feel like Rails is really starting to see some new life that has felt a bit absent from it for a while now. It almost seems like there is a new generation of people extremely burned out from the constant churn of the “JS-all-the-things” movement that just took over most of web development for the past several years. There is a lot of exciting stuff landing in Rails and it’s surrounding ecosystems at the moment and I think it’s still probably the best choice for many new SAAS / web app based companies and startups.

WrtCdEvrydy 6y ago
Rails was always great, JS just paid a lot more... It felt great to get back into it having people pay to migrate their Rails apps to React and Node.
- gav 6y ago
  It's funny you say that, I've seen people pick a tech stack based on developer cost. Maybe five years back I had a client in NYC who had a predominantly Java stack, but due to internal politics, wouldn't pay more than 135k for a senior developer when the market was wanting at least 25-30k more than that. Everyone they hired was pretty terrible. At the time, Node developers were cheaper, so we split the application in two, wrapped all the old Java code with APIs, which was then able to be maintained by a small team while the front-end and any new features were all re-written with Node. It was architectural a better system--but the driving factor was really developer cost (the business hated to make any investments in technology). The problem was that Node got popular and it got harder and harder to hire good Node developers at what the company wanted to pay. They had a big re-org and fired everyone I knew, so I don't know what they are doing now, but I often wonder if they just repeated the process with whatever up-and-coming language was more hip.
  - WrtCdEvrydy 6y ago
    > I've seen people pick a tech stack based on developer cost You'd be surprised how often cost of developers comes into the overall conversation especially when you have tech sprawl and have to spend to get qualified people. I've seen projects green-light just on the premise that removing legacy code will lead to a payroll reduction.

dwheeler 6y ago

This looks very promising! I'm currently using active job, so I will need to take a look at this. Thanks for letting me know.

rubyn00bie 6y ago

Despite my username, I haven't written any serious Ruby is like 5+ years; that is to say, I'm fairly removed from a lot of the changes in the Rails world. Now, pardon me for my ignorance, but... isn't this just like Resque back in the day? Or like doesn't ActiveJob just support any ol' backend outta the box? What's the real use case for this? I'm genuinely curious not trying to be rude or diminish the work.

techscruggs 6y ago
You can implement background processing with less operational complexity. This allows you to leverage activejob to get something up and running quickly and easily. Granted, if you start having higher throughput, you want to move on from using postgres as your message bus.
why-el 6y ago
I didn't look into this one yet besides the home page, but I always thought Postgres would make a killer backend for ActiveJob, leveraging advisory locks for synchronizing jobs and such. Resque is Redis backed, but if you don't want to add that as a dependency and already have Postgres setup, then this would be a great fit.
- paulryanrogers 6y ago
  Because of the MVCC implementation (needing VACUUM) it's not ideal without tuning.
thibaut_barrere 6y ago
I believe Resque was fork based (at least initially, it may have changed, I haven't looked at it in a while!), while GoodJob appears to be multithreaded (like Sidekiq), so this is quite different in terms of performance & throughput.
sideofbacon 6y ago
GoodJob author here. The core reason I wrote it was to see how lean/simple I could write a database-backed ActiveJob adapter today. And hope In practice any adapter should _just work_, but also all the other adapters pre-date ActiveJob (Rails 4.2) and Concurrent::Ruby (adopted in Rails 5.0 I think). The assumption is that building GoodJob on top of what already exists in Rails today, it can be performant (enough) and simple (easy to understand, maintain, keep compatible with new versions of Rails, etc). That's not a big claim, but maybe it's compelling.

techscruggs 6y ago

This looks great. I'd love to remove Redis from my tech stack on applications with low throughput background jobs. Sadly, it doesn't appear to have a UI. Resque and Sidekiq are both good enough solutions, but what makes them my defacto choice is the robust web UIs that comes with them.

samokhvalov 6y ago

To make it scale well, two things are required: 1. TRUNCATE if cleanup is needed (no DELETEs) + partitioning. 2. SELECT .. FOR UPDATE SKIP LOCKED or advisory locks. It's worth learning from Skype's PgQ developed in 2000-s. Update: found the work with advisory locks in the code, great.

SkyLinx 6y ago

Would this work with pgbouncer? I'm wondering because of the advisory locks.

sideofbacon 6y ago
GoodJob author here. Excellent observation. It should work if PgBouncer is using session-level pooling, but not transaction-level pooling. I made a ticket to document this: https://github.com/bensheldon/good_job/issues/52 https://github.com/bensheldon/good_job/issues/52 I don't have much experience with PgBouncer. What's the scale/need when a team would use PgBouncer? That would help me prioritize supporting it better.
- Exuma 6y ago
  The main reason I can think of would be when you have hundreds of workers accessing a DB but the DB only has a small connection limit, say 200 or something. You can have the workers access the connection through pgBouncer to bring that up into the thousands.
  - sideofbacon 6y ago
    Thanks! That's really helpful context.

cpursley 6y ago

For a very nice library in the Elixir/Phoenix ecosystem have a look at https://github.com/sorentwo/oban https://github.com/sorentwo/oban

regulation_d 6y ago
I think one of my biggest annoyances, from a development perspective, with Sidekiq is that it typically moves you in the direction of running foreman or something like that, which makes using the pry a pain. One thing I really like about Phoenix development is that I don't have to change my flow to use a REPL while still being able to run background jobs.

Jack_uio 6y ago

I was just assigned to a project based on zend 2.x php framework (2015 version) and angularjs(version1) (not angular.io). Just looking at the code, I just want to scream, because I was told to make changes and add features. I could re-implement everything in Rails in about a week's time but manager's opinions just drown me.

BilalBudhani 6y ago

This looks like a great library on the surface. I would very much like to have a simpler alternative to Sidekiq for projects which doesn't require that level of complexity. Also, not having to add Redis just to run background processing makes life so much easier. I would like to thank the author of the library for all their efforts.