4 ms·
Am I missing something, or is this solution only going to prevent running jobs multiple times in case Resque is being shut down in an orderly fashion with TERM?
by pfg 12y ago
Am I missing something, or is this solution only going to prevent running jobs multiple times in case Resque is being shut down in an orderly fashion with TERM? What if your instance simply dies (which could happen for any number of reasons)?
Solving this problem can be rather complex whenever Third-Party services are involved, but somehow this feels like you've only lowered the likelihood of multiple job executions, which isn't something I'd be comfortable with when it comes to things like credit card charges.
- steveklabnik 12y agoSaid maintainer who hasn't merged this PR yet here. The reason I haven't is because I looked at this, went, uhhhhhh I'm not sure, and haven't had the time to figure out if it's a good change yet. I don't want to change signal handling and then break things for other people.
- ejlangev 12y agoSolid point. My purpose here was more to start a discussion around this problem and what we did about it rather than to say this is the only or best solution. Also I was sure other people must be having this issue on Heroku and couldn't find any other solutions that would work despite spending a good amount of time searching for one.
- steveklabnik 12y agoAbsolutely! I'm also not saying you're wrong, I'm just saying please don't take my failures as a maintainer as a signal either way. I appreciate the patch.
- ejlangev 12y agoI'm not sure I follow what you're saying re: preventing running jobs multiple times. If Resque isn't shut down in an orderly fashion then no retry logic would be invoked. Processes would likely be terminated via KILL in this case and thus given no chance to clean themselves up or re-enqueue themselves.
- pfg 12y agoSomehow I was under the impression Resque provides durability in case a worker process crashes (like [afaik] Gearman and Sidekiq Pro), which isn't the case, so my comment doesn't really make sense for your scenario.
- steveklabnik 12y agoResque maintainer here. Ruby + Redis = imperfect durability, as far as I can tell. At least, for the whole system. That said, we've been considering moving to RPOPLPUSH for a long time, but the patch hasn't landed yet. You're right that Resque could be better with regards to durability. Discussion: https://github.com/resque/resque/issues/758 https://github.com/resque/resque/issues/758
- antirez 12y ago> but somehow this feels like you've only lowered the likelihood of multiple job executions This is basically the best you can do. Imagine the context of a strongly consistent store you use for queues, and N workers. The store is strictly CP. One worker gets task "A", executes it, but does not provide the acknowledge, because it crashes before to ack. In order to guarantee at-least-once-execution property the system has to re-issue the task to another worker after a timeout, causing multiple job execution. If you want to avoid this, you have as a side effect a failure mode where it is possible that jobs are lost forever, which is a lot worse, since the original problem can be solved by making all jobs idempotent. Basically Redis-based queues should try to provide guarantees about durability of jobs, and try to provide just best-effort mechanisms to avoid duplication of jobs when possible.