3 ms·

You don't need distributed consensus to have a cluster of reliable services. We've had large-scale clusters of applications for what, 30 years? And distributed

You don't need distributed consensus to have a cluster of reliable services. We've had large-scale clusters of applications for what, 30 years? And distributed consensus being generally used by a few players for about 10 of those. Etcd is neat, but in no way mandatory for what it was even invented for. 99% of people just don't need it.

jrockway 6y ago
I don't know of any clustering system that works without some sort of leader. In many cases, the leader is you. If your computer dies, you can walk to another one and still deploy software. If you disappear, that's a Big Problem, however. No modern clustering system is much different. Services aren't killed off if a leader can't be elected. It's just that having all your servers alive but uncontrollable is almost as scary as them being down.
- wahern 6y ago
  Distributed consensus doesn't require a leader. The same process used to elect a leader (i.e. change leader state) can simply be used to change state generally. See, e.g., There Is More Consensus in Egalitarian Parliaments: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf In fact, for something like what etcd is used for in k8s, as the root and lynchpin of all state, but not direct application I/O (pods use their own data stores, and even for cluster management etcd usually just contains pointers to external data), a leaderless architecture makes the most sense. Protocols to choose a leader are latency and throughput optimizations, but the core state for a cluster (members, pod metadata, etc) shouldn't require very much state and therefore should be relatively low traffic as compared to most database applications. And as shown in the above paper, for certain scenarios (including, arguably, the k8s scenario) a leaderless architecture can have better latency and throughput, not to mention availability.
  - smarterclayton 6y ago
    If you run etcd as a singleton you already get this behavior - consensus is shortcircuited and you only have to pay the cost to write to durable storage (which is already heavily batched). And you need durable writes so you can crash recover (kube has a gaping hole today in that a restore from an earlier time point breaks many controllers until a reconcile is performed). Note that Kubernetes requires a total ordering of writes (which simplifies how hard it is for us puny humans to reason about) AND requires strong consistency in order to provide guarantees like “this pod only runs on one machine at a time” and “PVs aren’t released until the pods are really stopped”. Leaderless is a simple tradeoff - singleton or three instances. That’s the best possible choice in the world and it’s etcd and it’s relative simplicity that make it possible. I’ve never seen a production Kube system with HA etcd go down due to non-human error, so I don’t believe single instance is going to give you better availability when single machine faults happen (they happen frequently; about 0.5-1% of the machines in the OpenShift fleet - cloud and on-premise - are down at any one time due to power, software, or hardware issues). Almost all of those clusters tick along fine when they lose that machine.
    - peterwwillis 6y ago
      > I’ve never seen a production Kube system with HA etcd go down due to non-human error I have, with OpenShift, several times. The worst one took hours to get it partially back online, days to fully recover. At least I got to have one free drink before I had to leave our holiday party to start fixing it. (Ok, technically it was non-prod; luckily prod was ECS... long story) > I don’t believe single instance is going to give you better availability when single machine faults happen Depends on your definition and context of availability. It is possibly the simplest thing to destroy any existing nodes (and in the event that a node can't be destroyed, null-route it and/or disable its network port) and bring up a new node. So single instance is fairly easy to recover. But with multiple nodes, for each function of each instance that is required to reach consensus, the likelihood of consensus failure increases; you actually need more nodes and variety in the cluster to resist consensus failure. Yet ironically, the more nodes you have, the higher the probability of failure. In the end, the real-world reliability of the system is based on additional factors besides the network model.

jrockway 6y ago

I don't know of any clustering system that works without some sort of leader. In many cases, the leader is you. If your computer dies, you can walk to another one and still deploy software. If you disappear, that's a Big Problem, however. No modern clustering system is much different. Services aren't killed off if a leader can't be elected. It's just that having all your servers alive but uncontrollable is almost as scary as them being down.

wahern 6y ago
Distributed consensus doesn't require a leader. The same process used to elect a leader (i.e. change leader state) can simply be used to change state generally. See, e.g., There Is More Consensus in Egalitarian Parliaments: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf In fact, for something like what etcd is used for in k8s, as the root and lynchpin of all state, but not direct application I/O (pods use their own data stores, and even for cluster management etcd usually just contains pointers to external data), a leaderless architecture makes the most sense. Protocols to choose a leader are latency and throughput optimizations, but the core state for a cluster (members, pod metadata, etc) shouldn't require very much state and therefore should be relatively low traffic as compared to most database applications. And as shown in the above paper, for certain scenarios (including, arguably, the k8s scenario) a leaderless architecture can have better latency and throughput, not to mention availability.
- smarterclayton 6y ago
  If you run etcd as a singleton you already get this behavior - consensus is shortcircuited and you only have to pay the cost to write to durable storage (which is already heavily batched). And you need durable writes so you can crash recover (kube has a gaping hole today in that a restore from an earlier time point breaks many controllers until a reconcile is performed). Note that Kubernetes requires a total ordering of writes (which simplifies how hard it is for us puny humans to reason about) AND requires strong consistency in order to provide guarantees like “this pod only runs on one machine at a time” and “PVs aren’t released until the pods are really stopped”. Leaderless is a simple tradeoff - singleton or three instances. That’s the best possible choice in the world and it’s etcd and it’s relative simplicity that make it possible. I’ve never seen a production Kube system with HA etcd go down due to non-human error, so I don’t believe single instance is going to give you better availability when single machine faults happen (they happen frequently; about 0.5-1% of the machines in the OpenShift fleet - cloud and on-premise - are down at any one time due to power, software, or hardware issues). Almost all of those clusters tick along fine when they lose that machine.
  - peterwwillis 6y ago
    > I’ve never seen a production Kube system with HA etcd go down due to non-human error I have, with OpenShift, several times. The worst one took hours to get it partially back online, days to fully recover. At least I got to have one free drink before I had to leave our holiday party to start fixing it. (Ok, technically it was non-prod; luckily prod was ECS... long story) > I don’t believe single instance is going to give you better availability when single machine faults happen Depends on your definition and context of availability. It is possibly the simplest thing to destroy any existing nodes (and in the event that a node can't be destroyed, null-route it and/or disable its network port) and bring up a new node. So single instance is fairly easy to recover. But with multiple nodes, for each function of each instance that is required to reach consensus, the likelihood of consensus failure increases; you actually need more nodes and variety in the cluster to resist consensus failure. Yet ironically, the more nodes you have, the higher the probability of failure. In the end, the real-world reliability of the system is based on additional factors besides the network model.