All distributed locking systems have a liveness problem: what should you do when a participant fails? You can block forever, which is always correct but not sup

All distributed locking systems have a liveness problem: what should you do when a participant fails? You can block forever, which is always correct but not super helpful. You can assume after some time that the process is broken, which preserves liveness. But what if it comes back? What if it was healthy all along and you just couldn't talk to it? The classic solution is leases: assume bounded clock drift, and make lock holders promise to stop work some time after taking the lock. This is only correct if all clients play by the rules, and your clock drift hypothesis is right. The other solution is to validate that the lock holder hasn't changed on every call. For example, with a lock generation epoch number. This needs to be enforced by the callee, or by a middle layer, which might seem like you've just pushed the fault tolerance problem to somebody else. In practice, pushing it to somebody else, like a DB is super useful! Finally, you can change call semantics to offer idempotency (or other race-safe semantics). Nice if you can get it.