4 ms·

Makes me really happy to see how earnest y'all are being about these issues and how you're approaching them. FWIW, we had very similar issues at Neo4j a few ye

Makes me really happy to see how earnest y'all are being about these issues and how you're approaching them. FWIW, we had very similar issues at Neo4j a few years back; I kind of wish we'd have blogged about it, but like you insinuate, I guess there's a strong social pressure to keep your dirty laundry in the hamper. One of the interesting things we learned was that testing "worst case" was often not very useful - the first time we shipped a GA that'd gone through months-long soak testing in the worst possible load we could imagine, it took less than a few weeks for a customer with a seemingly benign load to come back with an unstable cluster. Today, stability testing is done both on the "max out write throughput, then pull the power cord, then pull the powercord again while it recovers, and do that a million times"-level and "run these four customer use cases, including their devops setup, for two months and fail if uptime is not 100%"-level. I'm not at Neo anymore, but I do wish someone who is would blog about the stability testing regimen, because at this point, it is brutal.

PetahNZ 10y ago
Unleash the chaos monkey!
- jakewins 10y ago
  Which reminds me - another thing we learned is that too much chaos can be easier on a distributed system than less chaos. One of the robustness suites tries as hard as it can to permanently destroy a Neo4j cluster, looking for things like distributed deadlocks, faults in the leader election, data inconsistencies between replicas and so on. It does that by applying a randomized load of all operations the system supports; reads, writes and schema changes. That's then combined with induced hardware faults and "regular" operational restarts and cluster configuration changes. The problem was that, early on, the test would actually create unresponsive clusters, but then the "chaos" would continue, stirring up enough dust to get the cluster going again before the "unresponsive" timeouts triggered, causing false green tests. Hence: Today this suite plays out a "chaos" scenario, but then it heals network partitions, turns unpowered instances back on and so forth, and sits back and waits for the cluster to recover "on its own".
  - buzzybee 10y ago
    That suggests that your original chaos was more "random" than "chaotic", as actually chaotic processes will go through indefinite periods of slack where they are apparently predictable. i.e. a better "chaos monkey" might employ some traversal of a fractal system rather than simply aim for high entropy noise.
  - brianwawok 10y ago
    This is considered multiple unrelated failures. You usually don't want to (at least to start) design a system to handle this. handle a machine power down? Yes. Handle a network cable unplug? Yes. Handle 1 machine power down and 1 network cable unplug at the same time? That may be impossible to handle, or it may be an order of magnitude harder to handle. Get all the single failure issues nailed down. Then perhaps you can do a little work on multiple unrelated failures, but it gets insanely complex fast.

PetahNZ 10y ago

Unleash the chaos monkey!

jakewins 10y ago
Which reminds me - another thing we learned is that too much chaos can be easier on a distributed system than less chaos. One of the robustness suites tries as hard as it can to permanently destroy a Neo4j cluster, looking for things like distributed deadlocks, faults in the leader election, data inconsistencies between replicas and so on. It does that by applying a randomized load of all operations the system supports; reads, writes and schema changes. That's then combined with induced hardware faults and "regular" operational restarts and cluster configuration changes. The problem was that, early on, the test would actually create unresponsive clusters, but then the "chaos" would continue, stirring up enough dust to get the cluster going again before the "unresponsive" timeouts triggered, causing false green tests. Hence: Today this suite plays out a "chaos" scenario, but then it heals network partitions, turns unpowered instances back on and so forth, and sits back and waits for the cluster to recover "on its own".
- buzzybee 10y ago
  That suggests that your original chaos was more "random" than "chaotic", as actually chaotic processes will go through indefinite periods of slack where they are apparently predictable. i.e. a better "chaos monkey" might employ some traversal of a fractal system rather than simply aim for high entropy noise.
- brianwawok 10y ago
  This is considered multiple unrelated failures. You usually don't want to (at least to start) design a system to handle this. handle a machine power down? Yes. Handle a network cable unplug? Yes. Handle 1 machine power down and 1 network cable unplug at the same time? That may be impossible to handle, or it may be an order of magnitude harder to handle. Get all the single failure issues nailed down. Then perhaps you can do a little work on multiple unrelated failures, but it gets insanely complex fast.