Retries become dangerous when workflows aren't designed to be idempotent from the beginning. What helped us was treating every job execution as replayable and

Retries become dangerous when workflows aren't designed to be idempotent from the beginning. What helped us was treating every job execution as replayable and attaching a unique operation key instead of relying on execution state alone. Otherwise retries silently create data inconsistencies that only appear much later.