10 ms·

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archi

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder. In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.

zeristor 4y ago
Yes, I love the idea of the Plan Apply.
tauwauwau 4y ago
Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple reviewers here didn't catch the mistake https://www.bloombergquint.com/markets/citi-s-900-million-misfire-happened-in-midst-of-software-switch https://www.bloombergquint.com/markets/citi-s-900-million-mi...
- dkersten 4y ago
  > Multiple reviewers here didn't catch the mistake Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.
- Danieru 4y ago
  That form had a couple weird checkboxes with odd wording. It is a famous mistake, but also rather understandable just because the form was cryptic.
- irrational 4y ago
  Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.
- HowardStark 4y ago
  While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone. [1] https://www.atlasobscura.com/articles/pointing-and-calling-japan-trains https://www.atlasobscura.com/articles/pointing-and-calling-j... [2] https://en.wikipedia.org/wiki/Pointing_and_calling https://en.wikipedia.org/wiki/Pointing_and_calling
  - emerged 4y ago
    “I’m removing that semicolon!” (Pointing)
    - b112 4y ago
      Parent meant this sort of pointing. https://t.co/TjfX5K54H7 https://t.co/TjfX5K54H7
  - samhw 4y ago
    Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.
    - Blackcatmaxy 4y ago
      There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.
- slaymaker1907 4y ago
  I'm a fan of doing things temporally so data is very rarely actually deleted from the database. Most of the time, you just update the "valid_to" field to the current time. Sometimes real deleted are required such as with privacy requests, but I think that sort of thing is pretty rare. If your application has space concerns, you can modify this approach to be like a recycle bin where you delete records which are no longer valid and have been invalid for over a month (or whatever time frame is appropriate for your application). However, I think this is unnecessary in most cases except for blob/file storage.
JadeNB 4y ago
> Then have multiple people inspect the logs. I think that this is the most important part of any check. Your parent refers to checking the log five times, but, at least in my experience, I won't catch any more errors on the fifth time than the first—if I once saw what I expected rather than what was there, I'll keep doing so. Of course everyone has their blind spots, but, as in the famous Swiss-cheese approach, we just hope that they don't line up!
dredmorbius 4y ago
mv then rm is another idiom. So long as you have the space. For database entries, flag for deletion, then delete. In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover. Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specific directory, I happened to be in root. However ... - I'd booted a live-Linux version. (This was back when those still ran from floppy). - I'd mounted all partitions other than the one I was performing surgery on '-ro' (read-only). So what I bought was a reboot, and an opportunity to see what a Linux system with an active shell, but no executables, looks like. Plan ahead. Make big changes in stages. Measure twice (or 3, or 10, or 20 times), cut once. Sit on your hands for a minute before running as root. Paste into an editor session (C-x C-e Readline command, as noted elsewhere in this thread). Have backups.
- marcosdumay 4y ago
  You mean cp then rm? And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail. Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable. Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.
  - andi999 4y ago
    I think mv then rm is probably meant as 'windows trash bin' style.
  - dredmorbius 4y ago
    No, mv. Example: cd datadir mkdir delete mv <list of files to be deleted> ./delete # test to see if anything looks broken. # This might take a few seconds, or months, though it's usually reasonably brief. rm -rf ./delete The reasons for mv: - It's atomic (on a single filesystem). There's no risk of ending up with a partial operation or an incomplete operation. - It doesn't copy the data, it renames the file. (mv and rename are largely synonyms.) - There's no duplication of space usage. Where you're dealing with large files, this is helpful. The process is similar to the staged deletion most desktop OS users are familiar with, of "drag to trash, then empty trash". Used in the manner I'm deploying it, it's a bit more like a staged warehouse purge or ordering a dumpster bin --- more structured / controlled staged deletion than a household or small office might use.
mmmm2 4y ago
Another good approach is do deletions slowly. Put sleeps between each operation, and log everything. That way if you realize something is broken, you have a chance of catching it before it's too late.
water8 4y ago
It never hurts to ask for another set of eyes to review. At the least if something goes awry, the blame isn't solely on you.
csours 4y ago
Make a plan, check the plan, [fix the plan, check the plan (loop)], do the plan See PDCA for more a more time critical decision loop. https://en.wikipedia.org/wiki/PDCA https://en.wikipedia.org/wiki/PDCA
crispyambulance 4y ago
> ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue... This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal that was processed much like a silicon IC. The process was for TWO of us to sit in front of computer and review the curves in this crazy old EDA layout tool called "L-edit" before it got sent to have the actual masks made (which were very expensive). It took HOURS to check everything. The first hour was tolerable but then boredom started to creep in and we got sloppy. The whole reason TWO people got tasked with this was because it was thought that we would keep each other focused-- 2 pairs of eyes are better than one, right?. Instead, it just underscored the tedium of it all. One day someone walked in and found us BOTH in DEEP SLEEP in front of the monitor. Having two people didn't decrease the waste caused by mistakes, it just bored the hell out of more people.
- foota 4y ago
  How many mistakes did you catch?
  - crispyambulance 4y ago
    ONE real one and some occasional nitpicks to show that we were busy (after being caught asleep). Was it worth it? No, I don't think so from an opportunity cost perspective-- even though we were the most junior folks there. A mind is a terrible thing to waste!
  - Freestyler_3 4y ago
    From his story I can tell he found one big mistake. The tedious work itself.

zeristor 4y ago

Yes, I love the idea of the Plan Apply.

tauwauwau 4y ago

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple reviewers here didn't catch the mistake https://www.bloombergquint.com/markets/citi-s-900-million-misfire-happened-in-midst-of-software-switch https://www.bloombergquint.com/markets/citi-s-900-million-mi...

dkersten 4y ago
> Multiple reviewers here didn't catch the mistake Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.
Danieru 4y ago
That form had a couple weird checkboxes with odd wording. It is a famous mistake, but also rather understandable just because the form was cryptic.
irrational 4y ago
Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.
HowardStark 4y ago
While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone. [1] https://www.atlasobscura.com/articles/pointing-and-calling-japan-trains https://www.atlasobscura.com/articles/pointing-and-calling-j... [2] https://en.wikipedia.org/wiki/Pointing_and_calling https://en.wikipedia.org/wiki/Pointing_and_calling
- emerged 4y ago
  “I’m removing that semicolon!” (Pointing)
  - b112 4y ago
    Parent meant this sort of pointing. https://t.co/TjfX5K54H7 https://t.co/TjfX5K54H7
- samhw 4y ago
  Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.
  - Blackcatmaxy 4y ago
    There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.
slaymaker1907 4y ago
I'm a fan of doing things temporally so data is very rarely actually deleted from the database. Most of the time, you just update the "valid_to" field to the current time. Sometimes real deleted are required such as with privacy requests, but I think that sort of thing is pretty rare. If your application has space concerns, you can modify this approach to be like a recycle bin where you delete records which are no longer valid and have been invalid for over a month (or whatever time frame is appropriate for your application). However, I think this is unnecessary in most cases except for blob/file storage.

JadeNB 4y ago

> Then have multiple people inspect the logs. I think that this is the most important part of any check. Your parent refers to checking the log five times, but, at least in my experience, I won't catch any more errors on the fifth time than the first—if I once saw what I expected rather than what was there, I'll keep doing so. Of course everyone has their blind spots, but, as in the famous Swiss-cheese approach, we just hope that they don't line up!

dredmorbius 4y ago

mv then rm is another idiom. So long as you have the space. For database entries, flag for deletion, then delete. In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover. Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specific directory, I happened to be in root. However ... - I'd booted a live-Linux version. (This was back when those still ran from floppy). - I'd mounted all partitions other than the one I was performing surgery on '-ro' (read-only). So what I bought was a reboot, and an opportunity to see what a Linux system with an active shell, but no executables, looks like. Plan ahead. Make big changes in stages. Measure twice (or 3, or 10, or 20 times), cut once. Sit on your hands for a minute before running as root. Paste into an editor session (C-x C-e Readline command, as noted elsewhere in this thread). Have backups.

marcosdumay 4y ago
You mean cp then rm? And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail. Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable. Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.
- andi999 4y ago
  I think mv then rm is probably meant as 'windows trash bin' style.
- dredmorbius 4y ago
  No, mv. Example: cd datadir mkdir delete mv <list of files to be deleted> ./delete # test to see if anything looks broken. # This might take a few seconds, or months, though it's usually reasonably brief. rm -rf ./delete The reasons for mv: - It's atomic (on a single filesystem). There's no risk of ending up with a partial operation or an incomplete operation. - It doesn't copy the data, it renames the file. (mv and rename are largely synonyms.) - There's no duplication of space usage. Where you're dealing with large files, this is helpful. The process is similar to the staged deletion most desktop OS users are familiar with, of "drag to trash, then empty trash". Used in the manner I'm deploying it, it's a bit more like a staged warehouse purge or ordering a dumpster bin --- more structured / controlled staged deletion than a household or small office might use.

mmmm2 4y ago

Another good approach is do deletions slowly. Put sleeps between each operation, and log everything. That way if you realize something is broken, you have a chance of catching it before it's too late.

water8 4y ago

It never hurts to ask for another set of eyes to review. At the least if something goes awry, the blame isn't solely on you.

csours 4y ago

Make a plan, check the plan, [fix the plan, check the plan (loop)], do the plan See PDCA for more a more time critical decision loop. https://en.wikipedia.org/wiki/PDCA https://en.wikipedia.org/wiki/PDCA

crispyambulance 4y ago

> ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue... This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal that was processed much like a silicon IC. The process was for TWO of us to sit in front of computer and review the curves in this crazy old EDA layout tool called "L-edit" before it got sent to have the actual masks made (which were very expensive). It took HOURS to check everything. The first hour was tolerable but then boredom started to creep in and we got sloppy. The whole reason TWO people got tasked with this was because it was thought that we would keep each other focused-- 2 pairs of eyes are better than one, right?. Instead, it just underscored the tedium of it all. One day someone walked in and found us BOTH in DEEP SLEEP in front of the monitor. Having two people didn't decrease the waste caused by mistakes, it just bored the hell out of more people.

foota 4y ago
How many mistakes did you catch?
- crispyambulance 4y ago
  ONE real one and some occasional nitpicks to show that we were busy (after being caught asleep). Was it worth it? No, I don't think so from an opportunity cost perspective-- even though we were the most junior folks there. A mind is a terrible thing to waste!
- Freestyler_3 4y ago
  From his story I can tell he found one big mistake. The tedious work itself.