4 ms·
I give suggestions to my clients, and ask them what they care about testing most, and we... pretty much always agree on what direction the research goes next. F
by aphyr 7y ago
I give suggestions to my clients, and ask them what they care about testing most, and we... pretty much always agree on what direction the research goes next. FS/disk faults I don't have good tooling for yet, so I'd have to invest some time into developing those tools--makes sense to defer that work. Membership changes are infrequent and surprisingly difficult to write automation for, so I think it makes more sense to focus on the static-cluster case first.
- jinqueeny 7y agoRegarding FS/Disk faults, here is some information about what TiDB does: tl;dr: Chaos Engineering Some links: https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engineering%20to%20Build%20a%20Reliable%20TiDB.pdf https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engin... https://www.pingcap.com/blog/chaos-practice-in-tidb/ https://www.pingcap.com/blog/chaos-practice-in-tidb/ Regarding the Fault Injection tools we are using: - Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers. - SystemTap, a scripting language and tool diagnose of a performance or functional problem. - Fail, gofail for go and fail-rs for Rust - Namazu: a programmable fuzzy scheduler to test a distributed system. We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage