3 ms·

Oh, come on. Nothing short of formal verification is going to be sufficient for testing something like this (and formal verification is far, far harder for dis

Oh, come on. Nothing short of formal verification is going to be sufficient for testing something like this (and formal verification is far, far harder for distributed systems than it is for single-node ones), and even then it still can't validate your environment assumptions. DI isn't even a drop in the bucket. If anything, DI makes it more likely your tests will be off, especially if you're relying in any way on (even loose) global clock synchronization.

grizzles 10y ago
As the author mentioned in the original article TLA+, ATPs etc verify the correctness of systems, they don't apply to the type of stability problems which have plagued CockroachDB. That emergent behavior arose from it's implementation, optimizations, etc. Just because a system is unstable doesn't mean it's incorrect (or correct). That is why a simulator is useful. The entire point of building a sim is so you can run the exact code that comprises CockroachDB and control for any situation. For example, an optimization that works great at 10 nodes may be terrible for larger clusters. You need to find that out through testing because most humans can't look at a line of code and infer that. A side benefit of this approach is that if you change some parameter and suddenly experience a big increase or decrease in db throughput you gain a sense for which parameters are most sensitive to the overall system and will then be able to more easily diagnose problems in production.
- Jweb_Guru 10y ago
  What sort of cycle-accurate simulation are you proposing, exactly (I assume you're aiming for something pretty accurate if your simulation is going to figure out performance-related problems!), and how will it be improved by dependency injection? If you just want to simulate a network, why not use something like http://mininet.org/ http://mininet.org/? (Also: formal proofs of correctness for stuff like leader election are very much concerned with stability / liveness, not just safety; that's a common misconception).

grizzles 10y ago

As the author mentioned in the original article TLA+, ATPs etc verify the correctness of systems, they don't apply to the type of stability problems which have plagued CockroachDB. That emergent behavior arose from it's implementation, optimizations, etc. Just because a system is unstable doesn't mean it's incorrect (or correct). That is why a simulator is useful. The entire point of building a sim is so you can run the exact code that comprises CockroachDB and control for any situation. For example, an optimization that works great at 10 nodes may be terrible for larger clusters. You need to find that out through testing because most humans can't look at a line of code and infer that. A side benefit of this approach is that if you change some parameter and suddenly experience a big increase or decrease in db throughput you gain a sense for which parameters are most sensitive to the overall system and will then be able to more easily diagnose problems in production.

Jweb_Guru 10y ago
What sort of cycle-accurate simulation are you proposing, exactly (I assume you're aiming for something pretty accurate if your simulation is going to figure out performance-related problems!), and how will it be improved by dependency injection? If you just want to simulate a network, why not use something like http://mininet.org/ http://mininet.org/? (Also: formal proofs of correctness for stuff like leader election are very much concerned with stability / liveness, not just safety; that's a common misconception).