3 ms·

You can use the Go binding to bind to the first private adapter in bind_addr: "bind_addr": "{{ GetPrivateInterfaces | include \"network\" \"172.31.0.0/24\"

You can use the Go binding to bind to the first private adapter in bind_addr: "bind_addr": "{{ GetPrivateInterfaces | include \"network\" \"172.31.0.0/24\" | attr \"address\" }}", It sounds like your team had hard coded IPs and when the consensus was lost, they had to manually edit the peers.json file to remove the failed instance. Also, how many consul servers were running? If it's an even number, the odds of getting into a split-brain scenario is high.

Kinrany 5y ago
Surely the number becomes odd every time a Consul server goes down?
- toolz 5y ago
  You need an odd number of max nodes, not an odd number of live nodes. Raft nodes require knowing they are in the majority to be healthy, if you have a network partition and half are on each side you get split brain because neither side can know for sure which side should be considered authoritative/healthy.
  - NavinF 5y ago
    I think what he meant to ask is “Are 4 nodes less reliable than 3?” and I’m pretty sure the answer is “no” despite the other comment implying otherwise. AFAIK, even nodes can hurt latency, but not reliability.
    - tick_tock_tick 5y ago
      “Are 4 nodes less reliable than 3?” The answer is 3 is no less reliable then 4.
    - eklavya 5y ago
      It depends on the network topology (likelihood not theoretical) but with the amount of split brains I have witnessed, I would say definitely yes, 4 is less reliable than 3.
    - mirekrusin 5y ago
      Counter-intuitively 4 nodes are less reliable than 3 nodes for majority vote systems. Both can tolerate single failed node. However probability of 2nd node failure is 50% higher for 4 node setup than for 3 node setup. That's because 2nd node failure probablity for 4 node system is P(a) + P(b) + P(c) = 3P and for 3 node system is P(a) + P(b) = 2P; it's 3P vs 2P probablity. It's more than 50% overall because similar probability happens for first node failure - it's more likely that one out of 4 nodes will fail than one out of 3. In other words you have to guarantee one more node running without any reliability gains and that guarantee costs reliability.
    - Kinrany 5y ago
      Gah, I meant even, not odd! That is, if having an even number of nodes causes problems, you'll have problems every time one of your odd number of nodes goes down!
atonse 5y ago
3 were running and our consul config file does have the server’s IP in it, and is generated at boot time, so joy really hard coded. But we ended up just wiping everything out to re-bootstrap it. If you know any devops consultants that have solid experience with consul/other hashicorp tools, we’d love to talk to them! (US only)
- Intermernet 5y ago
  The current leader of consul cluster can be found with with a GET call to `/status/leader`, but that should be lower level than you need to do manually. Consul should be tracking this and updating the cluster. You should not be hardcoding IP addresses at all. Instead you can use Consul's inbuilt DNS load-balancing, or more advanced techniques. Have a look at https://www.hashicorp.com/blog/load-balancing-strategies-for-consul https://www.hashicorp.com/blog/load-balancing-strategies-for... for more details.

Kinrany 5y ago

Surely the number becomes odd every time a Consul server goes down?

toolz 5y ago
You need an odd number of max nodes, not an odd number of live nodes. Raft nodes require knowing they are in the majority to be healthy, if you have a network partition and half are on each side you get split brain because neither side can know for sure which side should be considered authoritative/healthy.
- NavinF 5y ago
  I think what he meant to ask is “Are 4 nodes less reliable than 3?” and I’m pretty sure the answer is “no” despite the other comment implying otherwise. AFAIK, even nodes can hurt latency, but not reliability.
  - tick_tock_tick 5y ago
    “Are 4 nodes less reliable than 3?” The answer is 3 is no less reliable then 4.
  - eklavya 5y ago
    It depends on the network topology (likelihood not theoretical) but with the amount of split brains I have witnessed, I would say definitely yes, 4 is less reliable than 3.
  - mirekrusin 5y ago
    Counter-intuitively 4 nodes are less reliable than 3 nodes for majority vote systems. Both can tolerate single failed node. However probability of 2nd node failure is 50% higher for 4 node setup than for 3 node setup. That's because 2nd node failure probablity for 4 node system is P(a) + P(b) + P(c) = 3P and for 3 node system is P(a) + P(b) = 2P; it's 3P vs 2P probablity. It's more than 50% overall because similar probability happens for first node failure - it's more likely that one out of 4 nodes will fail than one out of 3. In other words you have to guarantee one more node running without any reliability gains and that guarantee costs reliability.
  - Kinrany 5y ago
    Gah, I meant even, not odd! That is, if having an even number of nodes causes problems, you'll have problems every time one of your odd number of nodes goes down!

atonse 5y ago

3 were running and our consul config file does have the server’s IP in it, and is generated at boot time, so joy really hard coded. But we ended up just wiping everything out to re-bootstrap it. If you know any devops consultants that have solid experience with consul/other hashicorp tools, we’d love to talk to them! (US only)

Intermernet 5y ago
The current leader of consul cluster can be found with with a GET call to `/status/leader`, but that should be lower level than you need to do manually. Consul should be tracking this and updating the cluster. You should not be hardcoding IP addresses at all. Instead you can use Consul's inbuilt DNS load-balancing, or more advanced techniques. Have a look at https://www.hashicorp.com/blog/load-balancing-strategies-for-consul https://www.hashicorp.com/blog/load-balancing-strategies-for... for more details.