3 ms·
Years ago, I worked in an investment bank and we had a programmer put a batch program into production that executed the following as a shell command: rm -rf
by ams6110 11y ago
Years ago, I worked in an investment bank and we had a programmer put a batch program into production that executed the following as a shell command:
rm -rf foo /
It was supposed to be:
rm -rf foo/
It didn't run as root, but still managed to wipe out all the business data files. What saved us was that the servers were configured with RAID 1 and before the start of the nightly batch cycle, the mirror was "split" and only one copy mounted.
So we just had to restore the missing files from the other half of the mirror to revert to the start of the batch window and rerun the entire night's jobs.
- ArkyBeagle 11y agoSo in other words, the acceptance protocol for this script was inadequate. In the minicomputer era, it was common for a programmer to be required to run it on this one poor donkey of a machine to make it caught nothing on fire before moving to the big machine.
- ams6110 11y agoIt was the 1990s, not that it's an excuse but TDD wasn't a "thing" yet, nor was version control (at least not in that shop). For every change to a program we printed the diff, attached a cover memo, and filed it in a cabinet. Yes, there were test systems and programers were supposed to test all their changes, but they also were the ones who deployed their own changes to production so there were ways for this to happen pretty easily.
- gnarbarian 11y agoWhat happened to the programmer?
- ams6110 11y agoNothing. She continued to work there until after I moved on.
- mfoy_ 11y agoAfter a scare like that I'll bet she never makes that mistake again and makes sure to triple check for typos in dangerous commands.
- organsnyder 11y agoIf the programmer followed proper review and change control procedures, nothing should happen. Everyone writes bad code once in a while; in this particular case, the bug happened to be more catastrophic, but that's more bad luck than anything else. We have code reviews and change controls not only to reduce the number of defects, but also to provide cover when mistakes inevitably slip through.
- ataylor32 11y agoReminds me of this blunder: https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/6cd6b2485668e8a87485cb34ca8a0a937e73f16d https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi... ...and its fix: https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi...