4 ms·

Yes, this is a major problem most people simply ignore and deal with in a one-off manner when it fails. The only way to do this safely that is practical is to c

Yes, this is a major problem most people simply ignore and deal with in a one-off manner when it fails. The only way to do this safely that is practical is to create a jobs table which is written to in a transaction with the other writes. Another process reads the table and writes the job data into the queueing system. For example, the process starts up, reads the highest ID it has written from Redis, then starts reading the table for jobs with IDs higher than that. The job itself is written to Redis in a transaction along with the ID from the database. Ignoring the fact that Redis can lose data during failover, you’re pretty safe and duplicates will be minimal. You can add some idempotency on top if desired. Attempting to write to both a queue system and a relational database at the same time will fail at some point and you will be left to pick up the pieces of inconsistent state. If the write to the queue is to send an email, maybe you don’t care care. If it is to calculate a very important value that happens to be expensive to compute and needs to be done outside the request... good luck.

eropple 8y ago
I think calling it a "major problem" is overblown for the 90/95/98th percentile use cases. It certainly exists, and given a long enough time horizon it will happen, but I have never worked on a system where datastore non-transactionality has caused issues that need to be resolved with more than, as you say, a one-off fix. I tend to implement what you suggest with absolutely critical jobs, but the level of juggling it requires is a little much for, again, the 90/95/98th percentile jobs. Few things are so critical that you can only do them once and don't have significant visibility into them that retry logic can't bail you out--though, I hasten to note, I do explicitly advise the use of idempotent job logic with this or any other task-running solution.
- ryanworl 8y ago
  99.99% of the time at small scale it will work fine. That is completely accurate. But random network hiccups are so common in today’s cloud environments that, even at medium scale, a few minutes of partition between your app server and Redis, while still allowing access to the database, will result in a huge number of dropped jobs. The fix is so tiny I don’t see why it would be an issue for a team to implement. It uses one new database table and a long lived process for writing to Redis, which could be just running a script on the same node as Redis polling the database. The rest of the code can be the same outside of that.
  - eropple 8y ago
    Yeah, that's a fair criticism. For me, why I'm not comfortable with that approach for everything is that--if I'm understanding you correctly--means storing a lot of state with the job (state which can then go stale).
    - ryanworl 8y ago
      That would depend on the application’s needs, but isn’t a part of the approach itself. It is only guaranteeing delivery of the job to a worker at some point in the future, on average half the duration of the polling interval reading from the database. You can still use the strategy of only encoding object IDs in the job to avoid stale data. If the worker can’t reach the database, the job will just fail as it would at any other time and retry later. This is changing an at-most once delivery of the job into the queue into at-least once. Combine with idempotency you can get “exactly once semantics.”
- ex3ndr 8y ago
  Sure, but then our database will be a bottleneck and we actually can just use SQL for everything and not to forward information to another storage (like Redis) since there are no benefits. If you will deploy live system several times a day, eventually you will lose some data. Also, implementing monitoring for this is not an easy task and you will probably will never know if job wasn't scheduled.
deleted 8y ago
[deleted]

eropple 8y ago

I think calling it a "major problem" is overblown for the 90/95/98th percentile use cases. It certainly exists, and given a long enough time horizon it will happen, but I have never worked on a system where datastore non-transactionality has caused issues that need to be resolved with more than, as you say, a one-off fix. I tend to implement what you suggest with absolutely critical jobs, but the level of juggling it requires is a little much for, again, the 90/95/98th percentile jobs. Few things are so critical that you can only do them once and don't have significant visibility into them that retry logic can't bail you out--though, I hasten to note, I do explicitly advise the use of idempotent job logic with this or any other task-running solution.

ryanworl 8y ago
99.99% of the time at small scale it will work fine. That is completely accurate. But random network hiccups are so common in today’s cloud environments that, even at medium scale, a few minutes of partition between your app server and Redis, while still allowing access to the database, will result in a huge number of dropped jobs. The fix is so tiny I don’t see why it would be an issue for a team to implement. It uses one new database table and a long lived process for writing to Redis, which could be just running a script on the same node as Redis polling the database. The rest of the code can be the same outside of that.
- eropple 8y ago
  Yeah, that's a fair criticism. For me, why I'm not comfortable with that approach for everything is that--if I'm understanding you correctly--means storing a lot of state with the job (state which can then go stale).
  - ryanworl 8y ago
    That would depend on the application’s needs, but isn’t a part of the approach itself. It is only guaranteeing delivery of the job to a worker at some point in the future, on average half the duration of the polling interval reading from the database. You can still use the strategy of only encoding object IDs in the job to avoid stale data. If the worker can’t reach the database, the job will just fail as it would at any other time and retry later. This is changing an at-most once delivery of the job into the queue into at-least once. Combine with idempotency you can get “exactly once semantics.”
ex3ndr 8y ago
Sure, but then our database will be a bottleneck and we actually can just use SQL for everything and not to forward information to another storage (like Redis) since there are no benefits. If you will deploy live system several times a day, eventually you will lose some data. Also, implementing monitoring for this is not an easy task and you will probably will never know if job wasn't scheduled.

deleted 8y ago

[deleted]