3 ms·

I’m going to ask folks here a question. Recently our consul and nomad clusters both blew up within a day of each other. In some crazy twist of luck, Amazon re

by atonse 5y ago

I’m going to ask folks here a question. Recently our consul and nomad clusters both blew up within a day of each other. In some crazy twist of luck, Amazon removed the underlying instances that happened to run the leaders to both clusters. This was in our QA environment so they were all t2.nanos. In this situation, shouldn’t we just expect the other two nodes to hold an election and elect a new leader? Isn’t this the most plain vanilla use case there is? In both situations, the clusters got stuck without leaders indefinitely because they got stuck trying to ping the leader to see if they could hold an election (how could they?? the whole point of the election is because the leader disappeared). And the only way to recover was to do some insane dance of building a JSON file and hard coding IPs. Based on some research, it seems like we need to hard code IPs to avoid this in the future. This seems like a huge smell and goes against everything I’ve ever read about raft and the idea of self healing clusters. What am I missing here? I don’t remember ever having to deal with this with mongodb in 2014.

sriram_malhar 5y ago
That's a bogus implementation of leader election and consensus. You aren't missing anything. They are.
cyberge99 5y ago
You can use the Go binding to bind to the first private adapter in bind_addr: "bind_addr": "{{ GetPrivateInterfaces | include \"network\" \"172.31.0.0/24\" | attr \"address\" }}", It sounds like your team had hard coded IPs and when the consensus was lost, they had to manually edit the peers.json file to remove the failed instance. Also, how many consul servers were running? If it's an even number, the odds of getting into a split-brain scenario is high.
- Kinrany 5y ago
  Surely the number becomes odd every time a Consul server goes down?
  - toolz 5y ago
    You need an odd number of max nodes, not an odd number of live nodes. Raft nodes require knowing they are in the majority to be healthy, if you have a network partition and half are on each side you get split brain because neither side can know for sure which side should be considered authoritative/healthy.
    - NavinF 5y ago
      I think what he meant to ask is “Are 4 nodes less reliable than 3?” and I’m pretty sure the answer is “no” despite the other comment implying otherwise. AFAIK, even nodes can hurt latency, but not reliability.
      - tick_tock_tick 5y ago
        “Are 4 nodes less reliable than 3?” The answer is 3 is no less reliable then 4.
      - eklavya 5y ago
        It depends on the network topology (likelihood not theoretical) but with the amount of split brains I have witnessed, I would say definitely yes, 4 is less reliable than 3.
      - mirekrusin 5y ago
        Counter-intuitively 4 nodes are less reliable than 3 nodes for majority vote systems. Both can tolerate single failed node. However probability of 2nd node failure is 50% higher for 4 node setup than for 3 node setup. That's because 2nd node failure probablity for 4 node system is P(a) + P(b) + P(c) = 3P and for 3 node system is P(a) + P(b) = 2P; it's 3P vs 2P probablity. It's more than 50% overall because similar probability happens for first node failure - it's more likely that one out of 4 nodes will fail than one out of 3. In other words you have to guarantee one more node running without any reliability gains and that guarantee costs reliability.
      - Kinrany 5y ago
        Gah, I meant even, not odd! That is, if having an even number of nodes causes problems, you'll have problems every time one of your odd number of nodes goes down!
- atonse 5y ago
  3 were running and our consul config file does have the server’s IP in it, and is generated at boot time, so joy really hard coded. But we ended up just wiping everything out to re-bootstrap it. If you know any devops consultants that have solid experience with consul/other hashicorp tools, we’d love to talk to them! (US only)
  - Intermernet 5y ago
    The current leader of consul cluster can be found with with a GET call to `/status/leader`, but that should be lower level than you need to do manually. Consul should be tracking this and updating the cluster. You should not be hardcoding IP addresses at all. Instead you can use Consul's inbuilt DNS load-balancing, or more advanced techniques. Have a look at https://www.hashicorp.com/blog/load-balancing-strategies-for-consul https://www.hashicorp.com/blog/load-balancing-strategies-for... for more details.
kamikaz1k 5y ago
> A Raft cluster of 3 nodes can tolerate a single node failure while a cluster of 5 can tolerate 2 node failures. The document supports what you were expecting. So... I too am curious what happened to you.
deleted 5y ago
[deleted]

by atonse 5y ago

sriram_malhar 5y ago

That's a bogus implementation of leader election and consensus. You aren't missing anything. They are.

cyberge99 5y ago

You can use the Go binding to bind to the first private adapter in bind_addr: "bind_addr": "{{ GetPrivateInterfaces | include \"network\" \"172.31.0.0/24\" | attr \"address\" }}", It sounds like your team had hard coded IPs and when the consensus was lost, they had to manually edit the peers.json file to remove the failed instance. Also, how many consul servers were running? If it's an even number, the odds of getting into a split-brain scenario is high.

Kinrany 5y ago
Surely the number becomes odd every time a Consul server goes down?
- toolz 5y ago
  You need an odd number of max nodes, not an odd number of live nodes. Raft nodes require knowing they are in the majority to be healthy, if you have a network partition and half are on each side you get split brain because neither side can know for sure which side should be considered authoritative/healthy.
  - NavinF 5y ago
    I think what he meant to ask is “Are 4 nodes less reliable than 3?” and I’m pretty sure the answer is “no” despite the other comment implying otherwise. AFAIK, even nodes can hurt latency, but not reliability.
    - tick_tock_tick 5y ago
      “Are 4 nodes less reliable than 3?” The answer is 3 is no less reliable then 4.
    - eklavya 5y ago
      It depends on the network topology (likelihood not theoretical) but with the amount of split brains I have witnessed, I would say definitely yes, 4 is less reliable than 3.
    - mirekrusin 5y ago
      Counter-intuitively 4 nodes are less reliable than 3 nodes for majority vote systems. Both can tolerate single failed node. However probability of 2nd node failure is 50% higher for 4 node setup than for 3 node setup. That's because 2nd node failure probablity for 4 node system is P(a) + P(b) + P(c) = 3P and for 3 node system is P(a) + P(b) = 2P; it's 3P vs 2P probablity. It's more than 50% overall because similar probability happens for first node failure - it's more likely that one out of 4 nodes will fail than one out of 3. In other words you have to guarantee one more node running without any reliability gains and that guarantee costs reliability.
    - Kinrany 5y ago
      Gah, I meant even, not odd! That is, if having an even number of nodes causes problems, you'll have problems every time one of your odd number of nodes goes down!
atonse 5y ago
3 were running and our consul config file does have the server’s IP in it, and is generated at boot time, so joy really hard coded. But we ended up just wiping everything out to re-bootstrap it. If you know any devops consultants that have solid experience with consul/other hashicorp tools, we’d love to talk to them! (US only)
- Intermernet 5y ago
  The current leader of consul cluster can be found with with a GET call to `/status/leader`, but that should be lower level than you need to do manually. Consul should be tracking this and updating the cluster. You should not be hardcoding IP addresses at all. Instead you can use Consul's inbuilt DNS load-balancing, or more advanced techniques. Have a look at https://www.hashicorp.com/blog/load-balancing-strategies-for-consul https://www.hashicorp.com/blog/load-balancing-strategies-for... for more details.

kamikaz1k 5y ago

> A Raft cluster of 3 nodes can tolerate a single node failure while a cluster of 5 can tolerate 2 node failures. The document supports what you were expecting. So... I too am curious what happened to you.

deleted 5y ago

[deleted]