4 ms·

etcd would be a fine choice, but Consul provides a lock feature[1] out of the box. 1. https://www.consul.io/docs/guides/leader-election.html https://www.consul

etcd would be a fine choice, but Consul provides a lock feature[1] out of the box. 1. https://www.consul.io/docs/guides/leader-election.html https://www.consul.io/docs/guides/leader-election.html

teraflop 11y ago
As the article explains at length, using Zookeeper or etcd as a "locking service" in this way is not safe, even if the service is perfect and failure-free. There is a fundamental race condition between finding out that you've obtained a lock, and doing some other operation that assumes mutual exclusion.
- bkeroack 11y ago
  It's not really "fundamental". It's simply that the process that acquires the lock can fail (or be paused, or partitioned away from everything else, etc), and if it does, but then comes back later with a valid lock, bad things may happen. The author's solution is to push serialization logic into the resource/storage layer (by checking fencing tokens). But what if the resource is itself distributed? Then it needs it's own synchronization mechanism? It's locks all the way down.
  - bkeroack 11y ago
    Thinking more about it, this is a fundamental weakness of having self-policing processes, which I suppose is the OP's main point. It can be mitigated by having infinite lock TTLs, at the cost of risking system deadlock on process failure. Thank you to GPP for spurring me to think more deeply about this. As I stated, though, if the resource being protected is either a distributed system itself, or a system that cannot support fencing logic, this failure mode is difficult or impossible to prevent. The frequency of failure should be kept in mind here: most services can probably guarantee 99.99% uptime against the likelihood of 5 minute GC pauses.
takeda 11y ago
Doesn't it have the same issue that redis does?
- harlowja 11y ago
  Nope, zookeeper nodes have ids that always increment and can be used for fencing, compare-and-set and more...
  - takeda 11y ago
    I was talking about consul and etcd.
    - syscomet 11y ago
      In the linked consul documentation, we see a `LockIndex` field in the lock state; https://www.consul.io/docs/agent/http/kv.html https://www.consul.io/docs/agent/http/kv.html confirms that "LockIndex is the number of times this key has successfully been acquired in a lock. If the lock is held, the Session key provides the session that owns the lock." So you can do fencing with consul.
    - syscomet 11y ago
      For etcd: https://coreos.com/etcd/docs/0.4.7/etcd-api/ https://coreos.com/etcd/docs/0.4.7/etcd-api/ node.modifiedIndex appears sensible for fencing.

etcd would be a fine choice, but Consul provides a lock feature[1] out of the box. 1. https://www.consul.io/docs/guides/leader-election.html https://www.consul.io/docs/guides/leader-election.html

teraflop 11y ago

As the article explains at length, using Zookeeper or etcd as a "locking service" in this way is not safe, even if the service is perfect and failure-free. There is a fundamental race condition between finding out that you've obtained a lock, and doing some other operation that assumes mutual exclusion.

bkeroack 11y ago
It's not really "fundamental". It's simply that the process that acquires the lock can fail (or be paused, or partitioned away from everything else, etc), and if it does, but then comes back later with a valid lock, bad things may happen. The author's solution is to push serialization logic into the resource/storage layer (by checking fencing tokens). But what if the resource is itself distributed? Then it needs it's own synchronization mechanism? It's locks all the way down.
- bkeroack 11y ago
  Thinking more about it, this is a fundamental weakness of having self-policing processes, which I suppose is the OP's main point. It can be mitigated by having infinite lock TTLs, at the cost of risking system deadlock on process failure. Thank you to GPP for spurring me to think more deeply about this. As I stated, though, if the resource being protected is either a distributed system itself, or a system that cannot support fencing logic, this failure mode is difficult or impossible to prevent. The frequency of failure should be kept in mind here: most services can probably guarantee 99.99% uptime against the likelihood of 5 minute GC pauses.

takeda 11y ago

Doesn't it have the same issue that redis does?

harlowja 11y ago
Nope, zookeeper nodes have ids that always increment and can be used for fencing, compare-and-set and more...
- takeda 11y ago
  I was talking about consul and etcd.
  - syscomet 11y ago
    In the linked consul documentation, we see a `LockIndex` field in the lock state; https://www.consul.io/docs/agent/http/kv.html https://www.consul.io/docs/agent/http/kv.html confirms that "LockIndex is the number of times this key has successfully been acquired in a lock. If the lock is held, the Session key provides the session that owns the lock." So you can do fencing with consul.
  - syscomet 11y ago
    For etcd: https://coreos.com/etcd/docs/0.4.7/etcd-api/ https://coreos.com/etcd/docs/0.4.7/etcd-api/ node.modifiedIndex appears sensible for fencing.