4 ms·

@aphyr: "We have not evaluated filesystem or disk faults with TiDB, and cannot speak to crash recovery. Nor have we tested dynamic membership changes. Both mig

@aphyr: "We have not evaluated filesystem or disk faults with TiDB, and cannot speak to crash recovery. Nor have we tested dynamic membership changes. Both might be fruitful avenues of investigation for future research." How do you choose what components to test once you get past obvious stuff like the highest claimed isolation level?

orthecreedence 7y ago
Probably depends what tests they pay for.
aphyr 7y ago
I give suggestions to my clients, and ask them what they care about testing most, and we... pretty much always agree on what direction the research goes next. FS/disk faults I don't have good tooling for yet, so I'd have to invest some time into developing those tools--makes sense to defer that work. Membership changes are infrequent and surprisingly difficult to write automation for, so I think it makes more sense to focus on the static-cluster case first.
- jinqueeny 7y ago
  Regarding FS/Disk faults, here is some information about what TiDB does: tl;dr: Chaos Engineering Some links: https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engineering%20to%20Build%20a%20Reliable%20TiDB.pdf https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engin... https://www.pingcap.com/blog/chaos-practice-in-tidb/ https://www.pingcap.com/blog/chaos-practice-in-tidb/ Regarding the Fault Injection tools we are using: - Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers. - SystemTap, a scripting language and tool diagnose of a performance or functional problem. - Fail, gofail for go and fail-rs for Rust - Namazu: a programmable fuzzy scheduler to test a distributed system. We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage

orthecreedence 7y ago

Probably depends what tests they pay for.

aphyr 7y ago

I give suggestions to my clients, and ask them what they care about testing most, and we... pretty much always agree on what direction the research goes next. FS/disk faults I don't have good tooling for yet, so I'd have to invest some time into developing those tools--makes sense to defer that work. Membership changes are infrequent and surprisingly difficult to write automation for, so I think it makes more sense to focus on the static-cluster case first.

jinqueeny 7y ago
Regarding FS/Disk faults, here is some information about what TiDB does: tl;dr: Chaos Engineering Some links: https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engineering%20to%20Build%20a%20Reliable%20TiDB.pdf https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engin... https://www.pingcap.com/blog/chaos-practice-in-tidb/ https://www.pingcap.com/blog/chaos-practice-in-tidb/ Regarding the Fault Injection tools we are using: - Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers. - SystemTap, a scripting language and tool diagnose of a performance or functional problem. - Fail, gofail for go and fail-rs for Rust - Namazu: a programmable fuzzy scheduler to test a distributed system. We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage