3 ms·

I'm reminded of the time I was on the InfoSec team at a company you've all heard of where all software development was done on a handful of Unix servers with a

by raldi 4y ago

I'm reminded of the time I was on the InfoSec team at a company you've all heard of where all software development was done on a handful of Unix servers with a shared network file system. I reached out to the Systems team to suggest they look into mounting the home directories with the no-setuid option, and one of them invited me over to pair at their desk. They explained that there was a trick, where you could leave the existing mount point as is, with its existing mount options, and then mount the same device somewhere else with different options. So they got a root shell, went into some scratch directory, mounted the homedirs there, and we tested things out. I forget how that went, or what happened next, but it seems that we neglected to clean up after this experiment, and at some point, another Systems engineer was looking around the scratch directory and perhaps ran something like `du -ak` (forgetting to specify the option that disables crossing onto new devices), finding what appeared to be a gigantic copy of everyone's home directories. So they started `rm -rf`'ing it, and made a pretty huge dent before the phones started ringing and they realized that what they were erasing was not a copy on a local scratch space, but rather the real thing over on the NFS drive. Fortunately, this company did have great backups.

LanceH 4y ago
I was one ROLLBACK away from the Users table in production. Mid 90's, we fixed things in production all the time, and I left the where clause out. There were no soft deletes back then, we just deleted the row, and that would trigger the deletion of user resources elsewhere... I'm sure we had backups, but it would have been bad and resulted in a lot more fixes in production. I guess the upside of the "seat of the pants" days is that there was poor referential integrity, so it wouldn't have taken out anything else like billing history. edit: I never typed anything so carefully as I typed that rollback. I almost want to say there was an autocommit when closing the session, but my memory on that is fuzzy. MS SQL Server, if anyone knows. From then on, when called on for sketchy but necessary actions, I write the where clause first, run it as a select, see the number of results, wrap it in a transaction, etc...
- tomcam 4y ago
  Good story although there were indeed soft deletes in the 90s.
- knodi123 4y ago
  I was manually monitoring a slow query of mine with "SHOW FULL PROCESSLIST", and at one point I saw "delete from readings" pop up. Readings being a massive, massive table that drove most of our functionality. I quickly killed the query and went about trying to see who else was connected to the DB, but then I saw the query come up again. So I killed it again, and second later it came back. I had to shout one of the sales guys over, and tell him to go office to office and demand anybody who was deleting rows to stop. Meanwhile I stayed at my desk and kept killing the query. Shortly after, I heard our CTO yell "shit!", and the query went away. He was trying to clear out his local testing DB, and didn't realize he was in a production shell. I don't know if I saved the company, but I certainly saved us from a couple of days of downtime and maybe an equal or larger amount of lost data. But the CTO couldn't praise me publicly or he'd look like an idiot, so that was the end of that story!

by raldi 4y ago

LanceH 4y ago

I was one ROLLBACK away from the Users table in production. Mid 90's, we fixed things in production all the time, and I left the where clause out. There were no soft deletes back then, we just deleted the row, and that would trigger the deletion of user resources elsewhere... I'm sure we had backups, but it would have been bad and resulted in a lot more fixes in production. I guess the upside of the "seat of the pants" days is that there was poor referential integrity, so it wouldn't have taken out anything else like billing history. edit: I never typed anything so carefully as I typed that rollback. I almost want to say there was an autocommit when closing the session, but my memory on that is fuzzy. MS SQL Server, if anyone knows. From then on, when called on for sketchy but necessary actions, I write the where clause first, run it as a select, see the number of results, wrap it in a transaction, etc...

tomcam 4y ago
Good story although there were indeed soft deletes in the 90s.
knodi123 4y ago
I was manually monitoring a slow query of mine with "SHOW FULL PROCESSLIST", and at one point I saw "delete from readings" pop up. Readings being a massive, massive table that drove most of our functionality. I quickly killed the query and went about trying to see who else was connected to the DB, but then I saw the query come up again. So I killed it again, and second later it came back. I had to shout one of the sales guys over, and tell him to go office to office and demand anybody who was deleting rows to stop. Meanwhile I stayed at my desk and kept killing the query. Shortly after, I heard our CTO yell "shit!", and the query went away. He was trying to clear out his local testing DB, and didn't realize he was in a production shell. I don't know if I saved the company, but I certainly saved us from a couple of days of downtime and maybe an equal or larger amount of lost data. But the CTO couldn't praise me publicly or he'd look like an idiot, so that was the end of that story!