4 ms·

Great! So I learned this technique. But one thing that's not clear to me - how a "expired" datetime field can allow easy retries? Anyone give me an example? Tha

Great! So I learned this technique. But one thing that's not clear to me - how a "expired" datetime field can allow easy retries? Anyone give me an example? Thanks!

chx 5y ago
Successful items are deleted from the queue before they expire. So an item with expired > current time needs to be retried because it failed processing. So, a cronjob resets these to expire 0 and the next worker can pick them up.
- edwinyzh 5y ago
  Thanks for the explanation and time is needed in order to digest it... I guess one might start with the simple "status" field approach ;)
  - DangitBobby 5y ago
    The status doesn't help if the worker died during processing. If you've got something that's been "in progress" for 20 minutes of a job that takes 2, your job might not be getting done. That's what expired is for.
    - edwinyzh 5y ago
      I see now! So the 'expired' field is for checking died/incompleted workers.
- layoutIfNeeded 5y ago
  Doesn’t sound like a robust solution. What if the job takes longer than your estimated expire time? Does the worker keep updating the expiry date on a background thread while doing the processing?
  - cbo100 5y ago
    Same for any queuing system. You need to set the expiry time long enough for the expected task duration. In SQS, for example, you use a visibility timeout set high enough that you will have time to finish the job and delete the message before SQS hand it off to another reader. You won’t always finish in time though, so ideally jobs are idempotent.
- ptx 5y ago
  You mean expired < current time, right?
- johbjo 5y ago
  Could the process not delete the row, and not commit until it has finished processing? Otherwise, the transaction is rolled back when the db connection ends. This would not require timeout or clean-up. One would need the appropriate transaction isolation, but shouldn't it work?
  - Seattle3503 5y ago
    That's requires maintaining more db locks.

Great! So I learned this technique. But one thing that's not clear to me - how a "expired" datetime field can allow easy retries? Anyone give me an example? Thanks!

chx 5y ago

Successful items are deleted from the queue before they expire. So an item with expired > current time needs to be retried because it failed processing. So, a cronjob resets these to expire 0 and the next worker can pick them up.

edwinyzh 5y ago
Thanks for the explanation and time is needed in order to digest it... I guess one might start with the simple "status" field approach ;)
- DangitBobby 5y ago
  The status doesn't help if the worker died during processing. If you've got something that's been "in progress" for 20 minutes of a job that takes 2, your job might not be getting done. That's what expired is for.
  - edwinyzh 5y ago
    I see now! So the 'expired' field is for checking died/incompleted workers.
layoutIfNeeded 5y ago
Doesn’t sound like a robust solution. What if the job takes longer than your estimated expire time? Does the worker keep updating the expiry date on a background thread while doing the processing?
- cbo100 5y ago
  Same for any queuing system. You need to set the expiry time long enough for the expected task duration. In SQS, for example, you use a visibility timeout set high enough that you will have time to finish the job and delete the message before SQS hand it off to another reader. You won’t always finish in time though, so ideally jobs are idempotent.
ptx 5y ago
You mean expired < current time, right?
johbjo 5y ago
Could the process not delete the row, and not commit until it has finished processing? Otherwise, the transaction is rolled back when the db connection ends. This would not require timeout or clean-up. One would need the appropriate transaction isolation, but shouldn't it work?
- Seattle3503 5y ago
  That's requires maintaining more db locks.