3 ms·

I think calling it a "major problem" is overblown for the 90/95/98th percentile use cases. It certainly exists, and given a long enough time horizon it will hap

I think calling it a "major problem" is overblown for the 90/95/98th percentile use cases. It certainly exists, and given a long enough time horizon it will happen, but I have never worked on a system where datastore non-transactionality has caused issues that need to be resolved with more than, as you say, a one-off fix. I tend to implement what you suggest with absolutely critical jobs, but the level of juggling it requires is a little much for, again, the 90/95/98th percentile jobs. Few things are so critical that you can only do them once and don't have significant visibility into them that retry logic can't bail you out--though, I hasten to note, I do explicitly advise the use of idempotent job logic with this or any other task-running solution.

ryanworl 8y ago
99.99% of the time at small scale it will work fine. That is completely accurate. But random network hiccups are so common in today’s cloud environments that, even at medium scale, a few minutes of partition between your app server and Redis, while still allowing access to the database, will result in a huge number of dropped jobs. The fix is so tiny I don’t see why it would be an issue for a team to implement. It uses one new database table and a long lived process for writing to Redis, which could be just running a script on the same node as Redis polling the database. The rest of the code can be the same outside of that.
- eropple 8y ago
  Yeah, that's a fair criticism. For me, why I'm not comfortable with that approach for everything is that--if I'm understanding you correctly--means storing a lot of state with the job (state which can then go stale).
  - ryanworl 8y ago
    That would depend on the application’s needs, but isn’t a part of the approach itself. It is only guaranteeing delivery of the job to a worker at some point in the future, on average half the duration of the polling interval reading from the database. You can still use the strategy of only encoding object IDs in the job to avoid stale data. If the worker can’t reach the database, the job will just fail as it would at any other time and retry later. This is changing an at-most once delivery of the job into the queue into at-least once. Combine with idempotency you can get “exactly once semantics.”
ex3ndr 8y ago
Sure, but then our database will be a bottleneck and we actually can just use SQL for everything and not to forward information to another storage (like Redis) since there are no benefits. If you will deploy live system several times a day, eventually you will lose some data. Also, implementing monitoring for this is not an easy task and you will probably will never know if job wasn't scheduled.

ryanworl 8y ago

99.99% of the time at small scale it will work fine. That is completely accurate. But random network hiccups are so common in today’s cloud environments that, even at medium scale, a few minutes of partition between your app server and Redis, while still allowing access to the database, will result in a huge number of dropped jobs. The fix is so tiny I don’t see why it would be an issue for a team to implement. It uses one new database table and a long lived process for writing to Redis, which could be just running a script on the same node as Redis polling the database. The rest of the code can be the same outside of that.

eropple 8y ago
Yeah, that's a fair criticism. For me, why I'm not comfortable with that approach for everything is that--if I'm understanding you correctly--means storing a lot of state with the job (state which can then go stale).
- ryanworl 8y ago
  That would depend on the application’s needs, but isn’t a part of the approach itself. It is only guaranteeing delivery of the job to a worker at some point in the future, on average half the duration of the polling interval reading from the database. You can still use the strategy of only encoding object IDs in the job to avoid stale data. If the worker can’t reach the database, the job will just fail as it would at any other time and retry later. This is changing an at-most once delivery of the job into the queue into at-least once. Combine with idempotency you can get “exactly once semantics.”

ex3ndr 8y ago

Sure, but then our database will be a bottleneck and we actually can just use SQL for everything and not to forward information to another storage (like Redis) since there are no benefits. If you will deploy live system several times a day, eventually you will lose some data. Also, implementing monitoring for this is not an easy task and you will probably will never know if job wasn't scheduled.