5 ms·

In both paxos and raft you only need a quorum to sign on it. In the case of raft/consul it's the majority. Which means, you need 3 of 5 nodes to sign on it. Al

In both paxos and raft you only need a quorum to sign on it. In the case of raft/consul it's the majority. Which means, you need 3 of 5 nodes to sign on it. Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent. Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group.

takeda 9y ago
I'm aware that Paxos and Raft require odd number of nodes. My point was that the more nodes you have the slower the cluster is, because increasing number of the nodes doesn't increase performance but resilience. With 3 nodes you can afford to lose at most 1 node, with 5 -> 2, with 7 -> 3 etc. > Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent. That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them. > Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group. Also that's not what the documentation says. Gossip is used for adding/removing nodes and the overall health of the node (whether host is reachable) is also done through gossip, but the actual health checks (whether a service is running) is done through consensus protocol. I still want to emphasize that using consensus protocol for SD is not a good idea. Let say you're running in AWS and follow the best practices which is spreading your servers over multiple AZ. Let say you use us-east-1 which has 6 AZ. An AWS starts having network issues (like few months ago) and 2 AZs have network issues and can't communicate with the rest. The machines are still running, just can't communicate. It just happens that on 2 of those AZs you run 2 out of your 3 consul servers. Even though all the machines are healthy and running your Consul cluster is down, even for the remaining 4 AZs that don't have any problems.
- scarface74 9y ago
  That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them. How do you come to the conclusion that you're communicating with the server? When I'm doing any type of communications with consul via the API I'm always communicating with 127.0.0.1/....
  - hamandcheese 9y ago
    Local agents forward to your central consul servers.
- hamandcheese 9y ago
  You should probably have a Consul server in each availability zone if you want to protect against this type of failure. Consul uses one raft cluster per "datacenter", so you don't have to worry about write performance seriously degrading as you scale, it will only be up to the max of the number of availability zones you'd like to operate in per region. Plus you shouldn't be doing write heavy KV operations with consul anyway. Just because you are using strong consistency doesn't mean you can't get reasonably high availability in practice.

takeda 9y ago

I'm aware that Paxos and Raft require odd number of nodes. My point was that the more nodes you have the slower the cluster is, because increasing number of the nodes doesn't increase performance but resilience. With 3 nodes you can afford to lose at most 1 node, with 5 -> 2, with 7 -> 3 etc. > Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent. That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them. > Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group. Also that's not what the documentation says. Gossip is used for adding/removing nodes and the overall health of the node (whether host is reachable) is also done through gossip, but the actual health checks (whether a service is running) is done through consensus protocol. I still want to emphasize that using consensus protocol for SD is not a good idea. Let say you're running in AWS and follow the best practices which is spreading your servers over multiple AZ. Let say you use us-east-1 which has 6 AZ. An AWS starts having network issues (like few months ago) and 2 AZs have network issues and can't communicate with the rest. The machines are still running, just can't communicate. It just happens that on 2 of those AZs you run 2 out of your 3 consul servers. Even though all the machines are healthy and running your Consul cluster is down, even for the remaining 4 AZs that don't have any problems.

scarface74 9y ago
That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them. How do you come to the conclusion that you're communicating with the server? When I'm doing any type of communications with consul via the API I'm always communicating with 127.0.0.1/....
- hamandcheese 9y ago
  Local agents forward to your central consul servers.
hamandcheese 9y ago
You should probably have a Consul server in each availability zone if you want to protect against this type of failure. Consul uses one raft cluster per "datacenter", so you don't have to worry about write performance seriously degrading as you scale, it will only be up to the max of the number of availability zones you'd like to operate in per region. Plus you shouldn't be doing write heavy KV operations with consul anyway. Just because you are using strong consistency doesn't mean you can't get reasonably high availability in practice.