8 ms·

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accide

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

brlewis 6y ago
>very experienced developers in their 40s I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.
- BalinKing 6y ago
  This may not have been their first time, though :-P
  - __d 6y ago
    Some time ago, it was common in Unix sites to have an NFS filesystem mounted on all machines that contained locally-built binaries to augment those provided by the operating system. At this site, we used a bunch of different platforms: OSF/1, Solaris, Linux, HP/UX, etc. So we had a large filesystem containing the source code, and built binaries for all the different platforms, and this included heaps of things, from Bash upwards. A colleague of mine accidentally ran rm -rf on this filesystem. It was taking a loooong time, so he realised and killed it, but not before it had removed a heap of stuff. Because this was something that could be rebuilt, it wasn't backed up, so we had to go through the process of downloading the tarballs, and recompiling everything for all the different platforms. It took a few days to recover most of it, and weeks to completely restore things. The day after the incident, when he arrived at work, he found his keyboard was missing a few keycaps. It took him a while to realise that there were four gone: 'R', 'M', '-', and 'F' ... Good times.
- PopeDotNinja 6y ago
  I once cloned a directory for standing up an environment via Terraform. I modified all of the environment variables and config and ran it. It worked perfectly. Except I’d forgotten to wipe out the Terraform state, which meant that in the process of creating a new environment, it completely deleted the environment I had cloned. That was my initiation into very experienced :)
mehrdadn 6y ago
Reminds me of when I accidentally deleted a virtual hard disk I had a few years ago, because I'd copied it earlier and I thought I still had the other copy left. Only afterward did I remember I'd done the exact same thing to the other copy earlier... thankfully the information on it wasn't critical, but it was kind of terrifying to realize it very well could have been.
contravariant 6y ago
Ironically there seems to be no time more prone to these kinds of mistakes than when you're trying to prevent or fix them.
- jrott 6y ago
  Most of the worst production issues I've been involved with have come from trying to fix a minor issue and then somebody making a mistake. The way our brains are wired to handle stress isn't really useful for debugging complicated problems.
  - Aeolun 6y ago
    One of the best pieces of advice I’ve ever gotten from a manager (in regards to production issues): First, calm down. I’m still amazed that he could be so calm when I’d just deleted a bunch of stuff on a clients production environment. May not have been the most lucrative company I’ve ever worked for, but it was definitely the best one.
jlmorton 6y ago
War story time. Long ago, I worked for an interesting company that insisted on running its entire business on Linux desktops, all the way back between 1999-2002. Imagine running StarOffice/OpenOffice, Thunderbird, Netscape Navigator, etc, for your entire business back in 2000, including your executive team, marketing teams, everyone, most of whom had never even heard of Linux before. Anyway, this being Linux, everyone's home directory was mounted on NFS. All our builds were standardized with a tool called SystemImager, which we could use to push out updates to everyone's desktop whenever we wanted. If there was a new version of KDE, we could pretty easily push that change out. Sometimes it was convenient for me to work on updates to these images by chrooting into a directory containing the "image," which was really just an rsync tree. And sometimes, when updating these images, it was convenient to mount our NFS home directories in this chroot environment, so I could access things like an archive I had just downloaded on my own desktop. And eventually we had lots of different images, and the old ones were using up a lot of disk space, so I decide to clean up some space removing the old images. And these are fairly large images, with lots of small files, and this was before SSDs were a thing, so it made sense that deleting them was taking a while, and I stepped out to grab something to eat. As I was eating lunch, I started getting the tech support escalations. But this wasn't that unusual, our users routinely had problems with the environment we had provided. They hated it, because it was in many ways terrible, and they made sure we knew it. So I wasn't terribly alarmed. I didn't think any major changes had been made, so I didn't hurry back. By the time I leisurely returned from lunch, half the NFS home directories for our users were gone, along with all their documents, emails, bookmarks, or whatever else. Suddenly it hit me what had happened: at some point, perhaps months earlier, I had left our NFS home directories mounted within one of these image chroots. And now I had sudo rm -rf'd it. We had backups, but they were on tape, and it took several days to restore, with about a day of data loss.
- stmw 6y ago
  That's a very sad worry story, hope it turned out OK. Sorry you and the users had to go through that.
- eljimmy 6y ago
  Oh man - this one is anxiety inducing. I feel like this would haunt me for years.
sgustard 6y ago
I have been that boss. Is that you, Wendel? In any case: the deletion even had a "type your app name to confirm" prompt, but I knew I wanted to act on production; the issue was deleting the wrong one of multiple production databases. The takeaway was to grab a second pair of eyes to review any dangerous operations.
aidenn0 6y ago
I deleted our production CRM database meaning to delete the test database. While my boss was running queries on the database for setting my quarterly bonus. Good news is that I was deleting the test database to ensure that the recovery from backups was properly automated, so it wasn't down too long.
davedx 6y ago
Yup. Senior dev here, my own devops config screw up wiped out all production sales order data earlier this year. Had to restore from multiple backups, took a while. Stressful experience. Consider network partitioning so dev/test/accept just has 0 contact with prod.

brlewis 6y ago

>very experienced developers in their 40s I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.

BalinKing 6y ago
This may not have been their first time, though :-P
- __d 6y ago
  Some time ago, it was common in Unix sites to have an NFS filesystem mounted on all machines that contained locally-built binaries to augment those provided by the operating system. At this site, we used a bunch of different platforms: OSF/1, Solaris, Linux, HP/UX, etc. So we had a large filesystem containing the source code, and built binaries for all the different platforms, and this included heaps of things, from Bash upwards. A colleague of mine accidentally ran rm -rf on this filesystem. It was taking a loooong time, so he realised and killed it, but not before it had removed a heap of stuff. Because this was something that could be rebuilt, it wasn't backed up, so we had to go through the process of downloading the tarballs, and recompiling everything for all the different platforms. It took a few days to recover most of it, and weeks to completely restore things. The day after the incident, when he arrived at work, he found his keyboard was missing a few keycaps. It took him a while to realise that there were four gone: 'R', 'M', '-', and 'F' ... Good times.
PopeDotNinja 6y ago
I once cloned a directory for standing up an environment via Terraform. I modified all of the environment variables and config and ran it. It worked perfectly. Except I’d forgotten to wipe out the Terraform state, which meant that in the process of creating a new environment, it completely deleted the environment I had cloned. That was my initiation into very experienced :)

mehrdadn 6y ago

Reminds me of when I accidentally deleted a virtual hard disk I had a few years ago, because I'd copied it earlier and I thought I still had the other copy left. Only afterward did I remember I'd done the exact same thing to the other copy earlier... thankfully the information on it wasn't critical, but it was kind of terrifying to realize it very well could have been.

contravariant 6y ago

Ironically there seems to be no time more prone to these kinds of mistakes than when you're trying to prevent or fix them.

jrott 6y ago
Most of the worst production issues I've been involved with have come from trying to fix a minor issue and then somebody making a mistake. The way our brains are wired to handle stress isn't really useful for debugging complicated problems.
- Aeolun 6y ago
  One of the best pieces of advice I’ve ever gotten from a manager (in regards to production issues): First, calm down. I’m still amazed that he could be so calm when I’d just deleted a bunch of stuff on a clients production environment. May not have been the most lucrative company I’ve ever worked for, but it was definitely the best one.

jlmorton 6y ago

War story time. Long ago, I worked for an interesting company that insisted on running its entire business on Linux desktops, all the way back between 1999-2002. Imagine running StarOffice/OpenOffice, Thunderbird, Netscape Navigator, etc, for your entire business back in 2000, including your executive team, marketing teams, everyone, most of whom had never even heard of Linux before. Anyway, this being Linux, everyone's home directory was mounted on NFS. All our builds were standardized with a tool called SystemImager, which we could use to push out updates to everyone's desktop whenever we wanted. If there was a new version of KDE, we could pretty easily push that change out. Sometimes it was convenient for me to work on updates to these images by chrooting into a directory containing the "image," which was really just an rsync tree. And sometimes, when updating these images, it was convenient to mount our NFS home directories in this chroot environment, so I could access things like an archive I had just downloaded on my own desktop. And eventually we had lots of different images, and the old ones were using up a lot of disk space, so I decide to clean up some space removing the old images. And these are fairly large images, with lots of small files, and this was before SSDs were a thing, so it made sense that deleting them was taking a while, and I stepped out to grab something to eat. As I was eating lunch, I started getting the tech support escalations. But this wasn't that unusual, our users routinely had problems with the environment we had provided. They hated it, because it was in many ways terrible, and they made sure we knew it. So I wasn't terribly alarmed. I didn't think any major changes had been made, so I didn't hurry back. By the time I leisurely returned from lunch, half the NFS home directories for our users were gone, along with all their documents, emails, bookmarks, or whatever else. Suddenly it hit me what had happened: at some point, perhaps months earlier, I had left our NFS home directories mounted within one of these image chroots. And now I had sudo rm -rf'd it. We had backups, but they were on tape, and it took several days to restore, with about a day of data loss.

stmw 6y ago
That's a very sad worry story, hope it turned out OK. Sorry you and the users had to go through that.
eljimmy 6y ago
Oh man - this one is anxiety inducing. I feel like this would haunt me for years.

sgustard 6y ago

I have been that boss. Is that you, Wendel? In any case: the deletion even had a "type your app name to confirm" prompt, but I knew I wanted to act on production; the issue was deleting the wrong one of multiple production databases. The takeaway was to grab a second pair of eyes to review any dangerous operations.

aidenn0 6y ago

I deleted our production CRM database meaning to delete the test database. While my boss was running queries on the database for setting my quarterly bonus. Good news is that I was deleting the test database to ensure that the recovery from backups was properly automated, so it wasn't down too long.

davedx 6y ago

Yup. Senior dev here, my own devops config screw up wiped out all production sales order data earlier this year. Had to restore from multiple backups, took a while. Stressful experience. Consider network partitioning so dev/test/accept just has 0 contact with prod.