4 ms·

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desir

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.

mdaniel 5y ago
I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place
karmakaze 5y ago
This is what happens when DRY gets taken too far, deduplicating similarly named or conceptualized operations with vastly different use-cases.
kadoban 5y ago
It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.
- taspeotis 5y ago
  “A script was executed incorrectly” does not blame anyone. Saying a script was faulty obfuscates what happened.
  - kadoban 5y ago
    It is faulty though? There should be more that's required to accidentally delete data than a manually specified switch.
    - jonwest 5y ago
      I mean, if it’s got flags to represent what you’re asking it to do and you set the wrong flag that’s not a script issue. I can run rm -rf / and that’s not a fault of rm, that’s a behind the keyboard issue.
      - deleted 5y ago
        [deleted]
      - thawaya3113 5y ago
        No. That is a fault of rm that it makes it so easy to do something that in 9.99999% of the cases isn’t what the user wants to do. In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this. Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
        clhodapp 5y ago
        Adding a hardcoded extra check for / to rm feels like theater because it barely makes it more safe. It's nearly as damaging to most Linux systems to remove /bin or /usr or any number of other directories. If you wanted to actually solve this problem, it seems like you would need to add some kind of "dangerous-to-remove-this" metadata in the filesystem. Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
      - kadoban 5y ago
        It's a process issue that a script like that is ~commonly used on production data. Processes should be in place where that's essentially impossible to do wrong. "oops I passed the wrong flag" isn't user error, that's system error that it was set up like that.
        5y ago
terom 5y ago
It was faulty in the sense of allowing it to be executed with incorrect human inputs, and in such a manner that there was no recovery process in place for what happened. Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions. I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable. Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
yuliyp 5y ago
Yes. Having the safe thing and very dangerous thing just be a boolean flag apart is bad. It increases the odds of the wrong flag being used at some point. It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before. While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
TomSwirly 5y ago
Also this - that the script runs in one step! When I do big deletions in an automated fashion, I always make it a two-step process. The first step identifies exactly the items which will be deleted. This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?" In many tools, this is as simple as running it twice, the first time with `--dry-run`.
devjam 5y ago
There’s some assumption here that the script directly updated/deleted customer data. In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.
darth_avocado 5y ago
Okay, so the script was faulty and the data it was running on was faulty. But I see no mention of testing. In my entire I’ve never seen initiatives like this not have a staging environment with dry runs and validation. This is bad engineering practice.

mdaniel 5y ago

I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place

karmakaze 5y ago

This is what happens when DRY gets taken too far, deduplicating similarly named or conceptualized operations with vastly different use-cases.

kadoban 5y ago

It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.

taspeotis 5y ago
“A script was executed incorrectly” does not blame anyone. Saying a script was faulty obfuscates what happened.
- kadoban 5y ago
  It is faulty though? There should be more that's required to accidentally delete data than a manually specified switch.
  - jonwest 5y ago
    I mean, if it’s got flags to represent what you’re asking it to do and you set the wrong flag that’s not a script issue. I can run rm -rf / and that’s not a fault of rm, that’s a behind the keyboard issue.
    - deleted 5y ago
      [deleted]
    - thawaya3113 5y ago
      No. That is a fault of rm that it makes it so easy to do something that in 9.99999% of the cases isn’t what the user wants to do. In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this. Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
      - clhodapp 5y ago
        Adding a hardcoded extra check for / to rm feels like theater because it barely makes it more safe. It's nearly as damaging to most Linux systems to remove /bin or /usr or any number of other directories. If you wanted to actually solve this problem, it seems like you would need to add some kind of "dangerous-to-remove-this" metadata in the filesystem. Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
    - kadoban 5y ago
      It's a process issue that a script like that is ~commonly used on production data. Processes should be in place where that's essentially impossible to do wrong. "oops I passed the wrong flag" isn't user error, that's system error that it was set up like that.
      - 5y ago

terom 5y ago

It was faulty in the sense of allowing it to be executed with incorrect human inputs, and in such a manner that there was no recovery process in place for what happened. Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions. I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable. Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.

yuliyp 5y ago

Yes. Having the safe thing and very dangerous thing just be a boolean flag apart is bad. It increases the odds of the wrong flag being used at some point. It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before. While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.

TomSwirly 5y ago

Also this - that the script runs in one step! When I do big deletions in an automated fashion, I always make it a two-step process. The first step identifies exactly the items which will be deleted. This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?" In many tools, this is as simple as running it twice, the first time with `--dry-run`.

devjam 5y ago

There’s some assumption here that the script directly updated/deleted customer data. In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.

darth_avocado 5y ago

Okay, so the script was faulty and the data it was running on was faulty. But I see no mention of testing. In my entire I’ve never seen initiatives like this not have a staging environment with dry runs and validation. This is bad engineering practice.