3 ms·
Fantastic reply, thanks for this Brandon. I know that Kyle Kingsbury aka "aphyr, destroyer of distributed systems" broke etcd and at the time (before you wrote
by SEJeff 11y ago
Fantastic reply, thanks for this Brandon. I know that Kyle Kingsbury aka "aphyr, destroyer of distributed systems" broke etcd and at the time (before you wrote a new raft library), consul faired quite a bit better than consul.
Are you using his jepesen testing tool as part of your testing for failure scenarios? If not, why?
- eis 11y agoYea I think I got the idea of Consul being more stable because of the jepsen results and some other project running into problems with the raft library that etcd used back then. If they rewrote that library and CockroachDB (whos engineers I respect) uses it, then I guess my fears are unwarranted.
- philips 11y agoIIRC, etcd and consul failed in the same way in aphyr's post. Essentially neither had quorum read. We don't have jepesen setup as a testing tool because it is really hard to make it a reliable false positive free system. Plus, the languages it is written in make it hard for us to hack on it. Instead what we have done is worked hard to build a functional testing suite to find non-algorithm issues (weird exhaustion issues, behavior under real disks, etc). And then have deep, fast, deterministic tests of the core raft algorithms.
- SEJeff 11y agoOh the stuff you've (you as in CoreOS) done with raft golang libraries (to test and fix it) is nothing short of incredible. The future with k8s with CoreOS as the base looks really great. Please do keep it up :)