4 ms·

I disagree. Most places I've worked, "I fucked up" is an acceptable response in an RCA as long as "I fucked up" isn't the end of the analysis. You shouldn't jus

I disagree. Most places I've worked, "I fucked up" is an acceptable response in an RCA as long as "I fucked up" isn't the end of the analysis. You shouldn't just be to throw your hands up and go, "well I guess nothing can be done then". People will fuck up, but you can do things to protect them from themselves, limit the blast radius, and make recovery smoother. From my own life, I worked at a start up early in my career and dropped the entire production database. I was troubleshooting a production bug and had the prod DB open in one terminal window and my local DB open in another. I meant to drop my local DB and instead typed it into the prod terminal and that was that. In the RCA we didn't just end at "I fucked up". The line of questioning went beyond that into what gaps allowed that fuck up to be possible which lead to action items around restricting access to production data to only those who need it as well as ensuring we had proper backups (thankfully, we did at the time) and ensuring we had runbooks on restoring data in the event that it happened again.

mh- 2y ago
> [..] had the prod DB open in one terminal window and my local DB open in another. I meant to drop my local DB [..] I feel like every single one of us who has worked in this role has this same story, haha. Mine was iTerm split panes, but still. Over a decade ago and I still have my own little system I (now) adhere to ensure that never happens again.
- alexjplant 2y ago
  I'm a big fan of shisa kanko [1] as practiced by Japanese rail workers. Every time I go to make a change to a critical system I deliberately say out loud what I'm connected to, what I'm about to do, and why I'm doing it as I confirm these assertions before taking action. If at all possible I have somebody else on the call watching. I'm sure it looks goofy to my colleagues but I'd like to think that it helps to avoid these types of mistakes. [1] https://www.atlasobscura.com/articles/pointing-and-calling-japan-trains https://www.atlasobscura.com/articles/pointing-and-calling-j...
- superfrank 2y ago
  Lol. Now that you say it, I'm thinking back and I'm pretty sure mine was also iTerm split panes
Filligree 2y ago
And changing the look of the prompts?
XorNot 2y ago
Agreed with this: the real question is always to ask how it got to that stage. Like a reasonable answer at all times to this problem is things like: why did you feel like you had to work this way? Why was it even possible to do so? Why was no one else aware this was happening? Why did no one else feel empowered to halt the process if they did know? etc. That sort of personal fuckup is just...never actually a personal fuckup, not when the collateral damage is so large. "I was logged into production as superuser" has the obvious follow up issues of "why was it easier to do that then have a correctly limited account?" (this one happens all the damn time - want a new account? Well here's 10 pages of justifications we need...also no versus "fix production now, just grab superuser from the password DB...) EDIT: And I don't say this as (1) someone who hasn't done this and (2) someone who hasn't ever been using superuser to debug production.
snackbroken 2y ago
In addition to the usual backup/restore policies, at $previous_workplace we disabled the "normal" prod DB admin accounts from running the usual suspect destructive queries (drop, alter, update, etc.) on the production DB. If you really needed to run something dangerous, we had a script named FuckUpProduction that 1) made you wait one minute before it would do anything to give you time to reflect on your life choices, 2) made sure to begin a transaction so you could roll it back before it was too late, 3) showed you the affected tables & rows, 4) made you type "FUCK MY SHIT UP" to confirm and 5) pinged everyone in our #DBA channel "<username> is about to fuck up production by running <query> affecting <rows> in <tables>. To stop them from doing something stupid, type STOP or NO" and aborted if anyone did so within the waiting period. Anyone having to use FuckUpProduction was treated as a process failure and we did the whole RCA song and dance to determine why the normal testing/deployment process was insufficient. When FuckUpProduction was designed, some team members expressed discomfort at the use of profanity. This was considered a feature. We had a "swear jar" in the office that you had to put a dollar in if you were ever saved from accidentally running a query in prod either by FuckUpProduction or by the restrictive DBA accounts. The jar had enough money to buy a pizza before I left the company.
- inglor_cz 2y ago
  4) made you type "FUCK MY SHIT UP" to confirm That is amazing, will inspire myself.
furyofantares 2y ago
I feel like you are in agreement with the post you replied to. > In the RCA we didn't just end at "I fucked up". You didn't end at "I fucked up" because "I fucked up" is not the root cause. It is not an acceptable answer to the question that root cause analysis is seeking to answer. Which doesn't mean you can't use the phrase to communicate, it's just not an acceptable answer.

mh- 2y ago

> [..] had the prod DB open in one terminal window and my local DB open in another. I meant to drop my local DB [..] I feel like every single one of us who has worked in this role has this same story, haha. Mine was iTerm split panes, but still. Over a decade ago and I still have my own little system I (now) adhere to ensure that never happens again.

alexjplant 2y ago
I'm a big fan of shisa kanko [1] as practiced by Japanese rail workers. Every time I go to make a change to a critical system I deliberately say out loud what I'm connected to, what I'm about to do, and why I'm doing it as I confirm these assertions before taking action. If at all possible I have somebody else on the call watching. I'm sure it looks goofy to my colleagues but I'd like to think that it helps to avoid these types of mistakes. [1] https://www.atlasobscura.com/articles/pointing-and-calling-japan-trains https://www.atlasobscura.com/articles/pointing-and-calling-j...
superfrank 2y ago
Lol. Now that you say it, I'm thinking back and I'm pretty sure mine was also iTerm split panes

Filligree 2y ago

And changing the look of the prompts?

XorNot 2y ago

Agreed with this: the real question is always to ask how it got to that stage. Like a reasonable answer at all times to this problem is things like: why did you feel like you had to work this way? Why was it even possible to do so? Why was no one else aware this was happening? Why did no one else feel empowered to halt the process if they did know? etc. That sort of personal fuckup is just...never actually a personal fuckup, not when the collateral damage is so large. "I was logged into production as superuser" has the obvious follow up issues of "why was it easier to do that then have a correctly limited account?" (this one happens all the damn time - want a new account? Well here's 10 pages of justifications we need...also no versus "fix production now, just grab superuser from the password DB...) EDIT: And I don't say this as (1) someone who hasn't done this and (2) someone who hasn't ever been using superuser to debug production.

snackbroken 2y ago

In addition to the usual backup/restore policies, at $previous_workplace we disabled the "normal" prod DB admin accounts from running the usual suspect destructive queries (drop, alter, update, etc.) on the production DB. If you really needed to run something dangerous, we had a script named FuckUpProduction that 1) made you wait one minute before it would do anything to give you time to reflect on your life choices, 2) made sure to begin a transaction so you could roll it back before it was too late, 3) showed you the affected tables & rows, 4) made you type "FUCK MY SHIT UP" to confirm and 5) pinged everyone in our #DBA channel "<username> is about to fuck up production by running <query> affecting <rows> in <tables>. To stop them from doing something stupid, type STOP or NO" and aborted if anyone did so within the waiting period. Anyone having to use FuckUpProduction was treated as a process failure and we did the whole RCA song and dance to determine why the normal testing/deployment process was insufficient. When FuckUpProduction was designed, some team members expressed discomfort at the use of profanity. This was considered a feature. We had a "swear jar" in the office that you had to put a dollar in if you were ever saved from accidentally running a query in prod either by FuckUpProduction or by the restrictive DBA accounts. The jar had enough money to buy a pizza before I left the company.

inglor_cz 2y ago
4) made you type "FUCK MY SHIT UP" to confirm That is amazing, will inspire myself.

furyofantares 2y ago

I feel like you are in agreement with the post you replied to. > In the RCA we didn't just end at "I fucked up". You didn't end at "I fucked up" because "I fucked up" is not the root cause. It is not an acceptable answer to the question that root cause analysis is seeking to answer. Which doesn't mean you can't use the phrase to communicate, it's just not an acceptable answer.