4 ms·
How are you blocking multiple director processes from writing to the JobDB instances? I.e. node A gets partitioned away, a new node B picks up in its place, but
by ryanworl 8y ago
How are you blocking multiple director processes from writing to the JobDB instances? I.e. node A gets partitioned away, a new node B picks up in its place, but then node A comes back and still thinks it owns the lock for the JobDB instance.
I would imagine you're taking the database version from Consul and using that as a fencing key for writes within a transaction in the database, but I didn't see that mentioned in the Go code snippet.
- rbranson 8y agoIn short... atypically long timeouts on these locks. We can afford to inject a few minutes of delay on a small number of in-flight messages to preserve this safety property. I’d agree that we should probably have a second layer of fencing.