5 ms·

Causing ZFS corruption for fun, profit, and quality assurance

113 points by binwiederhier 7y ago

the8472 7y ago
Some device-mapper virtual block device types also provide ways to simulate error conditions. For bad block errors there is dm-dust. https://www.kernel.org/doc/Documentation/device-mapper/ https://www.kernel.org/doc/Documentation/device-mapper/
deleted 7y ago
[deleted]
williesleg 7y ago
Yeah ZFS is shit especially since Oracle purchased Sun and fired everyone. Too bad Butters FS isn't ready (btrfs) I wish there were a new filesystem, or a new OS that did away with this ancient way of filing data. It's almost 2020 and we're still using 1990's technology. WTF?
derekmhewitt 7y ago
This was an interesting read! I hope they do a follow up article on how they automatically manage the rest of their recovery process.
- pizza234 7y ago
  If with "recovery", you intend recovering that file, it's not possible, since all the copies available in the mirror are corrupted. In this case, the recovery (repair) of the system (that is, to bring it to a clean state) simply consists in deleting the file (see https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#gbctx https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#...). If there was an intact copy of the file available in the mirror, then one doesn't need to do anything, as ZFS is self healing: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.
  - kchamplewski 7y ago
    Presumably they're referring to Datto's recovery process (i.e. recovering automatically from secondary or tertiary backups), as opposed to just the ZFS mirror.
pizza234 7y ago
ZFS volumes can also be backed by files. In this case, the volumes can be corrupted by opening the backing file in a text editor, and perform a search/replace. This is simpler but of course less interesting :-) Another interesting article I've mentioned in another post, is a demonstration of ZFS' self-healing capabilities: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.
- hnlmorg 7y ago
  Being able to back volumes by files is an amazing feature for testing. I've lost count of the number of times I've done this to prove some documentation works before applying it to the actual storage array in question.

113 points by binwiederhier 7y ago

Some device-mapper virtual block device types also provide ways to simulate error conditions. For bad block errors there is dm-dust. https://www.kernel.org/doc/Documentation/device-mapper/ https://www.kernel.org/doc/Documentation/device-mapper/

deleted 7y ago

[deleted]

williesleg 7y ago

Yeah ZFS is shit especially since Oracle purchased Sun and fired everyone. Too bad Butters FS isn't ready (btrfs) I wish there were a new filesystem, or a new OS that did away with this ancient way of filing data. It's almost 2020 and we're still using 1990's technology. WTF?

derekmhewitt 7y ago

This was an interesting read! I hope they do a follow up article on how they automatically manage the rest of their recovery process.

pizza234 7y ago
If with "recovery", you intend recovering that file, it's not possible, since all the copies available in the mirror are corrupted. In this case, the recovery (repair) of the system (that is, to bring it to a clean state) simply consists in deleting the file (see https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#gbctx https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#...). If there was an intact copy of the file available in the mirror, then one doesn't need to do anything, as ZFS is self healing: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.
- kchamplewski 7y ago
  Presumably they're referring to Datto's recovery process (i.e. recovering automatically from secondary or tertiary backups), as opposed to just the ZFS mirror.

pizza234 7y ago

ZFS volumes can also be backed by files. In this case, the volumes can be corrupted by opening the backing file in a text editor, and perform a search/replace. This is simpler but of course less interesting :-) Another interesting article I've mentioned in another post, is a demonstration of ZFS' self-healing capabilities: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.

hnlmorg 7y ago
Being able to back volumes by files is an amazing feature for testing. I've lost count of the number of times I've done this to prove some documentation works before applying it to the actual storage array in question.