3 ms·

I'm starting to come round to the belief that the only robust remedy is to release the chaos monkey: have the kernel delete all the state of the filesystem driv

by leoc 3y ago

I'm starting to come round to the belief that the only robust remedy is to release the chaos monkey: have the kernel delete all the state of the filesystem driver at randomly-chosen intervals, a few hours apart on average, and force the driver to recover itself.

bcrl 3y ago
That is pretty close to what those of us building robust storage systems in the real world have to do. At a previous employer, we had test suites that exercised all kinds of corner cases by triggering failover and system reboots in the middle of heavy persistent messaging workloads. Filesystems have other horrors that you learn about during testing. I had one test case where it would take ext4 80 seconds to write out an 8MB file after a fresh mount of an 8TB filesystem. Free space was fragmented in just the right way that we hit the single threaded reading of block groups and bitmaps in the kernel and it took forever to get that data off the disk array.
- Gibbon1 3y ago
  I've done the same with embedded drivers, randomly flip bits in the drivers state and see what happens. Does it recover, throw fault, hang, or explode like a bomb. I got that from a hardware designer talking about robust state machines. Any random illegal state should sequence back to a known good state.
  - zbentley 3y ago
    > Any random illegal state should sequence back to a known good state. What benefit does that property provide? Other than helping to deal with random memory corruption bit-flips, I'm having trouble understanding why state machines (in hardware, or any level of software) should ever be expected to handle arbitrary memory manipulation outside of the rules of the state machine.
    - fodkodrasz 3y ago
      You often don't have memory protection in embedded and coding errors, running over stack space could cause such problems. Usual remedy is to detect these with diagnostics and issue a reboot to clean state ASAP.

by leoc 3y ago

bcrl 3y ago

That is pretty close to what those of us building robust storage systems in the real world have to do. At a previous employer, we had test suites that exercised all kinds of corner cases by triggering failover and system reboots in the middle of heavy persistent messaging workloads. Filesystems have other horrors that you learn about during testing. I had one test case where it would take ext4 80 seconds to write out an 8MB file after a fresh mount of an 8TB filesystem. Free space was fragmented in just the right way that we hit the single threaded reading of block groups and bitmaps in the kernel and it took forever to get that data off the disk array.

Gibbon1 3y ago
I've done the same with embedded drivers, randomly flip bits in the drivers state and see what happens. Does it recover, throw fault, hang, or explode like a bomb. I got that from a hardware designer talking about robust state machines. Any random illegal state should sequence back to a known good state.
- zbentley 3y ago
  > Any random illegal state should sequence back to a known good state. What benefit does that property provide? Other than helping to deal with random memory corruption bit-flips, I'm having trouble understanding why state machines (in hardware, or any level of software) should ever be expected to handle arbitrary memory manipulation outside of the rules of the state machine.
  - fodkodrasz 3y ago
    You often don't have memory protection in embedded and coding errors, running over stack space could cause such problems. Usual remedy is to detect these with diagnostics and issue a reboot to clean state ASAP.