3 ms·
I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team? Seems like 5 Whys in that scenario wou
by jimbokun 8y ago
I don't understand, if the bug was written by your team, how could the ultimate cause be something outside of your team?
Seems like 5 Whys in that scenario would lead you to look at things like insufficient test coverage, no code review, code reviewers missed the bug because...
Those paths of inquiry lead to your team changing things under your direct control that could make it less likely for similar bugs to reach production in the future.
- tedunangst 8y agoTeam two promised us this service would never fail, so we didn't check for errors...
- espadrine 8y agoBroken expectations are a cause. For instance, when ext4 gained traction, some users found a number of zeroed-out files after a crash. Why? Because the programs managing those files updated them by truncating them and then writing them. Why? Because fopen() has no "replace file" mode, even though the "w" mode seems to do just that as long as the system doesn't crash (it does O_WRONLY|O_CREAT|O_TRUNC). Why? Because file systems used to not have delayed allocation when fopen() was created. ext4 introduced the concept to the ext family. Why? Because ext4 was designed to improve performance compared to ext3, and that change does that without breaking POSIX. Why? Because, even with ext3, the only way to correctly implement "replace file" on POSIX is to write the replacement file on the same directory, fsync() the file, close the file descriptor, fsync() the directory, and rename the file. At that point, the whys have shifted the blame through userland programmers, POSIX fopen(), ext4, and back to userland programmers. Depending on where you stop, the solution can be: 1. fix all userland programs, 2. change ext4 to force sync upon closing a truncated file's descriptor (which got implemented), 3. upgrade the POSIX standard to have a dedicated "replace file" function call. The truth is, userland is buggy, and the POSIX standard is arcane, so they both share blame. (But ext4, which was correctly implemented, is the only thing that got patched.) http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ http://dream.thunk.org/tytso/blog/2009/03/12/delayed-allocat...
- grok2 8y agoTo me it seems like ext4 was the problem here -- it tried to provide a feature that wasn't adequately supported by the underlying OS.
- espadrine 8y agoDelayed allocation was adequately supported. Implementing it in ext4 increased the risk of zeroing out a file from “having all files updated in the past 20ms be zeroed out” to “having all files updated in the past 4 seconds be zeroed out” in the event of a system crash.
- ben509 8y agoThe root cause may be something outside your team, but that doesn't mean you're saying you'll sit on your hands. "The service we depend on failed" can still mean, "and we need to implement better retries or redundancy" as a remedy.