3 ms·
I currently use RQ. Here is the logic that lead me to choosing it, then wishing I had just used Postgres. I need a queue to handle long running jobs. I looked
by elamje 7y ago
I currently use RQ. Here is the logic that lead me to choosing it, then wishing I had just used Postgres.
I need a queue to handle long running jobs. I looked around the Python ecosystem (bc Flask app) and found RQ. So now I add code to run RQ. Then I add Redis to act as the Queue. Then I realized I needed to track jobs in the queue, so I put them into the DB to track their state. Now I effectively have a circular dependency between my app, Redis, and my Postgres DB. If Redis goes down, I’m not really sure what happens. If the DB goes down I’m not really sure what’s going on in Redis. This added undue complexity to my small app. Since I’m trying to keep it simple, I recently found that you can use Postgres as a pub/sub queue which would have completely solved my needs while making the app much easier to reason about. Using Postgres will have plenty of room to grow and buy you time to figure out a more durable solution.
- samsbox 7y agoMay I ask, what (python) solution you ended up going with?
- elamje 7y agoI haven’t implemented the Postgres solution yet. It’s on my todo, but I will likely implement it myself, i.e. without a package.
- elcomet 7y agoCan't you track state in a "stateless" way ? For example having the job write it's progress in a file, so that if your app goes down, you can just re-read the file and know the state of your job?
- rcfox 7y agoRelying on the filesystem has its own drawbacks. If you scale horizontally, the local filesystem might not have the file you created because it's running on a different server. Mounting storage across multiple servers will introduce bottlenecks, and now you have to worry about the storage going down. If you're running within a Docker container, you need to make sure the file doesn't live inside the container or it won't be persisted.