6 ms·
I meant using PostgreSQL as the backend. Benefits: * when you back up your main data store, your jobs are also backed up. If you use sidekiq, are you remember
by joevandyk 13y ago
I meant using PostgreSQL as the backend.
Benefits:
* when you back up your main data store, your jobs are also backed up. If you use sidekiq, are you remembering to do frequent backups of redis?
* only one data store, not multiple. Simpler architecture.
* jobs can be inserted/updated in the same transactions as the rest of your data
* PostgreSQL supports listen/notify, don't have to poll for new jobs
* can use SQL to query jobs
* jobs can have foreign key constraints to the rest of your data
- jondot 13y agoI understand. You're completely right about Sidekiq. For RabbitMQ - you can have jobs persisted to disk on all nodes in the cluster which is like a backup. And of course transactional messaging is a problem that needs special attention.
- zimbatm 13y agoIt doesn't hurt to keep a higher-level job representation in DB and sync the state at each step of the processing. I don't know about sneakers but you might want to have more states than the usual new/processing/success|fail. Obviously it really depends on what you're doing, if the jobs are re-entrant, if they're the fire-and-forget kind of types, ... Still in most of the cases I would just store the jobs as a table entry and pass the table+id to the job processor. Even if PostgreSQL was the primary data store for the job processor there would still be issues with external state that needs to be healed when restoring from backup.