4 ms·
On the stability/reliability of etcd the team is very confident in how it behaves. Here is how we build that confidence: - We have a functional testing cluster
by philips 11y ago
On the stability/reliability of etcd the team is very confident in how it behaves. Here is how we build that confidence:
- We have a functional testing cluster that is constantly simulating various types of failures. You can find some information on that here: https://coreos.com/blog/new-functional-testing-in-etcd/ https://coreos.com/blog/new-functional-testing-in-etcd/
- The team is works to ensure the raft library that we use (and that we now share with cockroachdb) has unit tests that handle all of the edge cases described in the raft paper. See this paper as an example: https://github.com/coreos/etcd/blob/master/raft/raft_paper_test.go#L49 https://github.com/coreos/etcd/blob/master/raft/raft_paper_t...
- We have been running etcd in production for 1.5 years for a bootstrapping service called discovery.etcd.io and have focused on how to handle higher and higher read/write loads there. For example see the improvements we made in etcd 2.1 with async snapshots: https://twitter.com/BrandonPhilips/status/639843455141740544 https://twitter.com/BrandonPhilips/status/639843455141740544
- We have also focused on clear guides on how to operate etcd. All of this testing and performance work is of no use if people can't operate the system correctly: https://coreos.com/etcd/docs/latest/ https://coreos.com/etcd/docs/latest/
Of course if you have particular issues that you have encountered we would love to help: https://github.com/coreos/etcd#contact https://github.com/coreos/etcd#contact
- SEJeff 11y agoFantastic reply, thanks for this Brandon. I know that Kyle Kingsbury aka "aphyr, destroyer of distributed systems" broke etcd and at the time (before you wrote a new raft library), consul faired quite a bit better than consul. Are you using his jepesen testing tool as part of your testing for failure scenarios? If not, why?
- eis 11y agoYea I think I got the idea of Consul being more stable because of the jepsen results and some other project running into problems with the raft library that etcd used back then. If they rewrote that library and CockroachDB (whos engineers I respect) uses it, then I guess my fears are unwarranted.
- philips 11y agoIIRC, etcd and consul failed in the same way in aphyr's post. Essentially neither had quorum read. We don't have jepesen setup as a testing tool because it is really hard to make it a reliable false positive free system. Plus, the languages it is written in make it hard for us to hack on it. Instead what we have done is worked hard to build a functional testing suite to find non-algorithm issues (weird exhaustion issues, behavior under real disks, etc). And then have deep, fast, deterministic tests of the core raft algorithms.
- SEJeff 11y agoOh the stuff you've (you as in CoreOS) done with raft golang libraries (to test and fix it) is nothing short of incredible. The future with k8s with CoreOS as the base looks really great. Please do keep it up :)