3 ms·
> There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.
by blueplanet200 5y ago
> There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.
If it happens though is it a big deal? I’ve seen Sidekiq jobs used to move money, for instance. This can result is Bad Times. Right tool for the job, if consistency is a requirement Sidekiq I think gets incorrectly applied in a lot of applications.
- prescriptivist 5y agoI don't move money, but if I did I would do it like we do for most objects in our system (like interacting with S3 objects for example), I'd represent that transaction (in the money sense) in Postgres with a state and the job would be expected to do work and update the state. The transacting job itself could be queued immediately for convenience but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency. In this sense the job acts on data, it doesn't hold data, and you can build in backstops in the event of any kind of failure, not just Redis. You'd want to build this into your architecture regardless of what your background job tooling and infrastructure you use.
- Fire-Dragon-DoL 5y agoWe worked on the same problem, and what you described works, however it also happen to be exactly what delayed job does. Which bring me to the next point: Sidekiq should never be the default choice. It brings distributed systems complexity to an audience that's trained to work on a monolithic database. Delayed job or similar should be the default, with sidekiq relegated tp jobs ready to be lost
- blueplanet200 5y agoI agree with you that people often overlook the complexity of tying changes to their database, with enqueuing jobs in a completely separate system. https://www.betterment.com/engineering/delayed-resilient-background-jobs-on-rails https://www.betterment.com/engineering/delayed-resilient-bac... does a great job explaining this.
- blueplanet200 5y ago> but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency. You've re-implemented a job queue in the database.
- deleted 5y ago[deleted]
- prescriptivist 5y agoNo I didn't.
- blueplanet200 5y agoYou kinda did though. Your models in states they shouldn’t be are jobs. These kind of headaches are avoided using a job queue that provides strong consistency. Everything is a trade off.