3 ms·

I love that someone is actually testing database fault tolerance in public, instead of just assuming that anything with replication is fault tolerant... Statef

I love that someone is actually testing database fault tolerance in public, instead of just assuming that anything with replication is fault tolerant... Stateful distributed systems are really, really hard to get right. The only hope is to do a LOT of simulation. Not that testing is a substitute for designing things right, but no one is smart enough to be sure they've thought of all cases. At FoundationDB we do this sort of testing millions of times over: http://foundationdb.com/white-papers/testing/ http://foundationdb.com/white-papers/testing/ so that we can let people plug and unplug our servers at trade shows without fear (video): http://blog.foundationdb.com/post/50924738464/foundationdb-fault-tolerance-demo-video http://blog.foundationdb.com/post/50924738464/foundationdb-f... and ultimately so that applications on top of our system face a somewhat less scary world.

shin_lao 13y ago
Nice to see you are serious about testing, however this PR page has got something that hits me as strange: As a rough measure of the effectiveness of our simulation testing framework, Quicksand has only ever found one bug. To me, this means your test is useless, not that your software is reliable. Generally speaking, simulating hardware faults is not very useful because you're testing the software layer below you: you don't talk to the hardware, you talk to the kernel which talks to the hardware. (Unless your filesystem and network code is in kernel land in which case I will retract this comment and go die in a dark corner of Earth. ;) )
- voidmain 13y ago
  Quicksand (the physical fault testing environment) doesn't find our bugs because our simulation-based testing finds them first! Our deterministic simulations cover all of the failure cases that quicksand can test and many more. The main reason we built quicksand is precisely to find out if our assumptions about the layers below us are wrong, since the simulator can't test its own assumptions. For example, there are hardware/software combinations out there where fsync() can't be trusted...
  - malkia 13y ago
    Can this piece of software be used to test other systems/db's in combination with OS?

shin_lao 13y ago

Nice to see you are serious about testing, however this PR page has got something that hits me as strange: As a rough measure of the effectiveness of our simulation testing framework, Quicksand has only ever found one bug. To me, this means your test is useless, not that your software is reliable. Generally speaking, simulating hardware faults is not very useful because you're testing the software layer below you: you don't talk to the hardware, you talk to the kernel which talks to the hardware. (Unless your filesystem and network code is in kernel land in which case I will retract this comment and go die in a dark corner of Earth. ;) )

voidmain 13y ago
Quicksand (the physical fault testing environment) doesn't find our bugs because our simulation-based testing finds them first! Our deterministic simulations cover all of the failure cases that quicksand can test and many more. The main reason we built quicksand is precisely to find out if our assumptions about the layers below us are wrong, since the simulator can't test its own assumptions. For example, there are hardware/software combinations out there where fsync() can't be trusted...
- malkia 13y ago
  Can this piece of software be used to test other systems/db's in combination with OS?