4 ms·

Accidentally deleting prod is a rite of passage.

davidthewatson 3y ago
Been there, done that. The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. But the production delete? You said,"drop", we said, "gone". Either production database deletes need to adopt alert fatigue or the ops are already so fatigued that it doesn't matter.
- ilyt 3y ago
  I'm pretty annoyed by those prompts. Yes, fuck you, do what I fucking said, my backups work fine. Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Then work can go fast and uninterrupted while still being resilient to mistakes. Obviously not possible everywhere but where it is it should be the default.
  - Xylakant 3y ago
    Cool that you have a working backup, but restores are what counts. And they’re not instantaneous in almost all cases. So asking before certain destructive actions is still a good practice.
  - darkwater 3y ago
    > Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Obviously. When undoing is not easy or possible automatically, there is the "Are you really really sure?" dialog.
- Aerroon 3y ago
  >The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. Pop-up warnings that are too common are a problem itself. People get so used to accepting them immediately that they stop serving their purpose and just become an annoyance. With one exception: they can prevent misclicks.
  - davidthewatson 3y ago
    Right! That's why I used the term alert fatigue: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue The misclick point is beyond truth, particularly given the storm of marketing that is mobile web.
- taneliv 3y ago
  In a company where we chose to do production SQL shenanigans in the spirit of "move fast, break things", I developed a habit of starting those with "BEGIN;" and paying careful attention to the number of records modified before calling it a "COMMIT;". In case of unexpected numbers, immediate "ROLLBACK;" so as to not hog the resources and make the applications suffer. Even fatigued, "DELETE 1182900" tends to stick out, if you're used to, and expecting to, see "DELETE 1". ("BEGIN" appears to be PostgreSQLism, "START TRANSACTION" is more standard.)
moralestapia 3y ago
Definitely! There's probably some mystical mechanism making sure newbies go through it. I consider myself an "experienced" developer, yet on the first day on my current gig I dropped a '--recursive' flag in an AWS command and deleted a bunch of stuff I shouldn't have. ¯\_(ツ)_/¯
- _boffin_ 3y ago
  I’m sure some processes and procedures were updated on that day and the next.
  - Zobat 3y ago
    At some time we learned about delete locks in Azure and promptly created those for all production resources. Have had/heard of enough actual/almost "killed production" stories to understand their value. A bit cumbersome when you actually need to delete something but so worth it for the mental health of the team.
    - bombcar 3y ago
      The key with delete locks and similar is it HAS to be something out-of-band; if the normal way to delete is via a website, the delete lock removal should be API or command line, etc A simple --max-really-delete added to a command line is subject to "replay attacks" on yourself when you use CTRL+R.
pjmlp 3y ago
Yeah, raises hand. Even though I take all care to differenciate environments, always cross check, we had to live drop a table while on a meeting, and due to ongoing conversation the wrong window was selected, and multitasking ended up causing a major distraction.
Moru 3y ago
I sort of did that at my first web-job. I thought it was a bit unhandy that the customers had to be mailed a link to the admin interface, why not just put a link to the login on the front page? A couple of days later I learned why. There's always a good reason somewhere, you just don't know why yet. The reason? Well, the admin interface was written in ASP. According to the guy that called me in panic said that ASP version was so very old it was extremely insecure. I didn't have access to the database so could not verify anything but the webserver logs had a lot of "Bobby tables" in them so I have a strong suspicion it wasn't all ASP's fault... Anyway, we ended up with the database disappearing in the night or containing random data until I got that phonecall asking why we were linking directly to the admin interface :-) They were very good at restoring the database though, just took an hour or so every morning until they found the reason.
briffle 3y ago
At a previous position a co-worker reached his rite of passage. Lots of connections, and not paying attention to which screen window/session he was in. After that, we then changed the bash prompts on all prod servers to be red. Never had another incident..
huehehue 3y ago
I dropped ~100 records from a join table thinking I was connected to the dev db. "No worries, we have backups", I thought. Turns out those were more like replication, copying over every change immediately (including deletes). In the end, fixed with a short script that cross-referenced tables and checked for missing relationships, but terrifying nonetheless.
nso 3y ago
Worked for a CRM SaaS. I once did a DELETE FROM users; WHERE... Open office. Boss rolled over on his chair and started saying "so..." and I dismissed him with a "deleted all users leave me alone". He rolled away, kinda like Homer in that GIF where he disappears into the hedge. Luckily it was back up in minutes, but a new phobia was unlocked.
- thedanbob 3y ago
  > and I dismissed him with a "deleted all users leave me alone" I think I'm going to start unplugging my desk phone when one of those moments happen to me. "Hey, there's something wrong with..." "YES I KNOW I WAS FIXING IT BEFORE YOU INTERRUPTED ME"

Been there, done that. The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. But the production delete? You said,"drop", we said, "gone". Either production database deletes need to adopt alert fatigue or the ops are already so fatigued that it doesn't matter.

ilyt 3y ago
I'm pretty annoyed by those prompts. Yes, fuck you, do what I fucking said, my backups work fine. Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Then work can go fast and uninterrupted while still being resilient to mistakes. Obviously not possible everywhere but where it is it should be the default.
- Xylakant 3y ago
  Cool that you have a working backup, but restores are what counts. And they’re not instantaneous in almost all cases. So asking before certain destructive actions is still a good practice.
- darkwater 3y ago
  > Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Obviously. When undoing is not easy or possible automatically, there is the "Are you really really sure?" dialog.
Aerroon 3y ago
>The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam. Pop-up warnings that are too common are a problem itself. People get so used to accepting them immediately that they stop serving their purpose and just become an annoyance. With one exception: they can prevent misclicks.
- davidthewatson 3y ago
  Right! That's why I used the term alert fatigue: https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue The misclick point is beyond truth, particularly given the storm of marketing that is mobile web.
taneliv 3y ago
In a company where we chose to do production SQL shenanigans in the spirit of "move fast, break things", I developed a habit of starting those with "BEGIN;" and paying careful attention to the number of records modified before calling it a "COMMIT;". In case of unexpected numbers, immediate "ROLLBACK;" so as to not hog the resources and make the applications suffer. Even fatigued, "DELETE 1182900" tends to stick out, if you're used to, and expecting to, see "DELETE 1". ("BEGIN" appears to be PostgreSQLism, "START TRANSACTION" is more standard.)

moralestapia 3y ago

Definitely! There's probably some mystical mechanism making sure newbies go through it. I consider myself an "experienced" developer, yet on the first day on my current gig I dropped a '--recursive' flag in an AWS command and deleted a bunch of stuff I shouldn't have. ¯\_(ツ)_/¯

_boffin_ 3y ago
I’m sure some processes and procedures were updated on that day and the next.
- Zobat 3y ago
  At some time we learned about delete locks in Azure and promptly created those for all production resources. Have had/heard of enough actual/almost "killed production" stories to understand their value. A bit cumbersome when you actually need to delete something but so worth it for the mental health of the team.
  - bombcar 3y ago
    The key with delete locks and similar is it HAS to be something out-of-band; if the normal way to delete is via a website, the delete lock removal should be API or command line, etc A simple --max-really-delete added to a command line is subject to "replay attacks" on yourself when you use CTRL+R.

pjmlp 3y ago

Yeah, raises hand. Even though I take all care to differenciate environments, always cross check, we had to live drop a table while on a meeting, and due to ongoing conversation the wrong window was selected, and multitasking ended up causing a major distraction.

Moru 3y ago

I sort of did that at my first web-job. I thought it was a bit unhandy that the customers had to be mailed a link to the admin interface, why not just put a link to the login on the front page? A couple of days later I learned why. There's always a good reason somewhere, you just don't know why yet. The reason? Well, the admin interface was written in ASP. According to the guy that called me in panic said that ASP version was so very old it was extremely insecure. I didn't have access to the database so could not verify anything but the webserver logs had a lot of "Bobby tables" in them so I have a strong suspicion it wasn't all ASP's fault... Anyway, we ended up with the database disappearing in the night or containing random data until I got that phonecall asking why we were linking directly to the admin interface :-) They were very good at restoring the database though, just took an hour or so every morning until they found the reason.

briffle 3y ago

At a previous position a co-worker reached his rite of passage. Lots of connections, and not paying attention to which screen window/session he was in. After that, we then changed the bash prompts on all prod servers to be red. Never had another incident..

huehehue 3y ago

I dropped ~100 records from a join table thinking I was connected to the dev db. "No worries, we have backups", I thought. Turns out those were more like replication, copying over every change immediately (including deletes). In the end, fixed with a short script that cross-referenced tables and checked for missing relationships, but terrifying nonetheless.

nso 3y ago

Worked for a CRM SaaS. I once did a DELETE FROM users; WHERE... Open office. Boss rolled over on his chair and started saying "so..." and I dismissed him with a "deleted all users leave me alone". He rolled away, kinda like Homer in that GIF where he disappears into the hedge. Luckily it was back up in minutes, but a new phobia was unlocked.

thedanbob 3y ago
> and I dismissed him with a "deleted all users leave me alone" I think I'm going to start unplugging my desk phone when one of those moments happen to me. "Hey, there's something wrong with..." "YES I KNOW I WAS FIXING IT BEFORE YOU INTERRUPTED ME"