3 ms·

As a fairly regular consul cluster admin for the last 6 years or so, but not on that scale i can safely say that you generally have no idea if rolling back will

by hughrr 5y ago

As a fairly regular consul cluster admin for the last 6 years or so, but not on that scale i can safely say that you generally have no idea if rolling back will work. I’ve experienced everything up to complete cluster collapses before. I spent an entire night blasting and reseeding a 200 node cluster once after a well tested forward migration went into a leadership battle it never resolved. Even if you test it before that’s no guarantee it’ll be alright on the night. Quite frankly relying on consul scares the shit out of me. There are so few guarantees and so many pitfalls and traps that I don’t sleep well. At this point I consider it a mortal risk. That applies to vault as well.

darkwater 5y ago
> Quite frankly relying on consul scares the shit out of me. There are so few guarantees and so many pitfalls and traps that I don’t sleep well. At this point I consider it a mortal risk. Consul (and Vault) for sure a complex softwares that 99% of the time "just work", but when they fail they can fail big time, I concur. But calling it a mortal risk seems a bit far fetched in my opinion.
- hughrr 5y ago
  When you run 5 nines that’s a risk.
miyuru 5y ago
I also run 3 small clusters of consul and I went ahead and read the raft paper[1] so I can debug consul election problems if it occurs. Consul is awesome when it works, but when it breaks it can be hell to get it working again. thankfully it usually works fine. I only had 1 outage and it fixed itself after restarting the service. [1] https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf
- rfoo 5y ago
  > so I can debug consul election problems if it occurs Interestingly, reading this remind me of a HashiCorp Nomad marketing piece [1]: > "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself." I was always thinking "but what if something goes wrong? just call HashiCorp engs?" :p [1] https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox
- mrweasel 5y ago
  That seems to be a general problem with these types of solutions. You have the exact same issue with something like ZooKeeper. It awesome when it works, but good luck trying to figure out why it's broken. Just the author of the previous post relying on these types of services is something that can keep me up at night.

by hughrr 5y ago

darkwater 5y ago

> Quite frankly relying on consul scares the shit out of me. There are so few guarantees and so many pitfalls and traps that I don’t sleep well. At this point I consider it a mortal risk. Consul (and Vault) for sure a complex softwares that 99% of the time "just work", but when they fail they can fail big time, I concur. But calling it a mortal risk seems a bit far fetched in my opinion.

hughrr 5y ago
When you run 5 nines that’s a risk.

miyuru 5y ago

I also run 3 small clusters of consul and I went ahead and read the raft paper[1] so I can debug consul election problems if it occurs. Consul is awesome when it works, but when it breaks it can be hell to get it working again. thankfully it usually works fine. I only had 1 outage and it fixed itself after restarting the service. [1] https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf

rfoo 5y ago
> so I can debug consul election problems if it occurs Interestingly, reading this remind me of a HashiCorp Nomad marketing piece [1]: > "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself." I was always thinking "but what if something goes wrong? just call HashiCorp engs?" :p [1] https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox
mrweasel 5y ago
That seems to be a general problem with these types of solutions. You have the exact same issue with something like ZooKeeper. It awesome when it works, but good luck trying to figure out why it's broken. Just the author of the previous post relying on these types of services is something that can keep me up at night.