5 ms·

I found the conclusions for mitigating instability in future efforts very surprising. There was no mention of test suites, dependency injection, etc. If I was

I found the conclusions for mitigating instability in future efforts very surprising. There was no mention of test suites, dependency injection, etc. If I was building this type of system, I would want to build an exhaustive test suite that allowed me to make stability invariant forward progress on the code base without relying on engineers eye catching regressions. That would mean constructing a system to enumerate all the edge cases at different cluster sizes, automate testing, be able to simulate the network with DI and without docker, and be able configure different network delays and other salient parameters. The omission of a system like this is surprising because it's really not that hard to build and make it an integral part of core development. I may be wrong, but the last paragraph of the post certainly seems to imply that this isn't done or hasn't been considered.

qaq 10y ago
There is nothing simple in building or testing a system like this. How exactly would DI help you test multi-node system?
- grizzles 10y ago
  Everything's relative. It's easier to build a test framework for CockroachDB than to build CockroachDB itself. DI could help by simulating the network. It gives you more power to test out and record states around weird edge cases. If you were using docker, there are things that you either probably can't configure or it would be very difficult to. Eg. changing dns lookup latency for a subset of hosts.
  - qaq 10y ago
    It's not relative it's literally the hardest type of DB system you can try to build. I have my doubts that an entity with relatively modest resources can pull it off, but pretending there are trivial solutions for even testing a distributed system let alone a distributed system of this complexity is simply not grounded in reality.
  - Jweb_Guru 10y ago
    Oh, come on. Nothing short of formal verification is going to be sufficient for testing something like this (and formal verification is far, far harder for distributed systems than it is for single-node ones), and even then it still can't validate your environment assumptions. DI isn't even a drop in the bucket. If anything, DI makes it more likely your tests will be off, especially if you're relying in any way on (even loose) global clock synchronization.
    - grizzles 10y ago
      As the author mentioned in the original article TLA+, ATPs etc verify the correctness of systems, they don't apply to the type of stability problems which have plagued CockroachDB. That emergent behavior arose from it's implementation, optimizations, etc. Just because a system is unstable doesn't mean it's incorrect (or correct). That is why a simulator is useful. The entire point of building a sim is so you can run the exact code that comprises CockroachDB and control for any situation. For example, an optimization that works great at 10 nodes may be terrible for larger clusters. You need to find that out through testing because most humans can't look at a line of code and infer that. A side benefit of this approach is that if you change some parameter and suddenly experience a big increase or decrease in db throughput you gain a sense for which parameters are most sensitive to the overall system and will then be able to more easily diagnose problems in production.
      - Jweb_Guru 10y ago
        What sort of cycle-accurate simulation are you proposing, exactly (I assume you're aiming for something pretty accurate if your simulation is going to figure out performance-related problems!), and how will it be improved by dependency injection? If you just want to simulate a network, why not use something like http://mininet.org/ http://mininet.org/? (Also: formal proofs of correctness for stuff like leader election are very much concerned with stability / liveness, not just safety; that's a common misconception).
- draw_down 10y ago
  Religion.

qaq 10y ago

There is nothing simple in building or testing a system like this. How exactly would DI help you test multi-node system?

grizzles 10y ago
Everything's relative. It's easier to build a test framework for CockroachDB than to build CockroachDB itself. DI could help by simulating the network. It gives you more power to test out and record states around weird edge cases. If you were using docker, there are things that you either probably can't configure or it would be very difficult to. Eg. changing dns lookup latency for a subset of hosts.
- qaq 10y ago
  It's not relative it's literally the hardest type of DB system you can try to build. I have my doubts that an entity with relatively modest resources can pull it off, but pretending there are trivial solutions for even testing a distributed system let alone a distributed system of this complexity is simply not grounded in reality.
- Jweb_Guru 10y ago
  Oh, come on. Nothing short of formal verification is going to be sufficient for testing something like this (and formal verification is far, far harder for distributed systems than it is for single-node ones), and even then it still can't validate your environment assumptions. DI isn't even a drop in the bucket. If anything, DI makes it more likely your tests will be off, especially if you're relying in any way on (even loose) global clock synchronization.
  - grizzles 10y ago
    As the author mentioned in the original article TLA+, ATPs etc verify the correctness of systems, they don't apply to the type of stability problems which have plagued CockroachDB. That emergent behavior arose from it's implementation, optimizations, etc. Just because a system is unstable doesn't mean it's incorrect (or correct). That is why a simulator is useful. The entire point of building a sim is so you can run the exact code that comprises CockroachDB and control for any situation. For example, an optimization that works great at 10 nodes may be terrible for larger clusters. You need to find that out through testing because most humans can't look at a line of code and infer that. A side benefit of this approach is that if you change some parameter and suddenly experience a big increase or decrease in db throughput you gain a sense for which parameters are most sensitive to the overall system and will then be able to more easily diagnose problems in production.
    - Jweb_Guru 10y ago
      What sort of cycle-accurate simulation are you proposing, exactly (I assume you're aiming for something pretty accurate if your simulation is going to figure out performance-related problems!), and how will it be improved by dependency injection? If you just want to simulate a network, why not use something like http://mininet.org/ http://mininet.org/? (Also: formal proofs of correctness for stuff like leader election are very much concerned with stability / liveness, not just safety; that's a common misconception).
draw_down 10y ago
Religion.