7 ms·

Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a

• alberth 3y ago
  Hope you bought that guy a beer. Great story, thanks for sharing.
• cdogl 3y ago
  Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or truncate them, as part of the test run. One day, a lead developer ran “make test” after he’d been doing some exploratory work in the prod database as part of a bug fix. The test code respected the environment variables and connected to prod instead of docker. Immediately, our tests dropped and recreated the production tables for that database a few dozen times.
  • cachvico 3y ago
    Verbatim from my current code: if strings.Contains(dbname, "prod") { panic("Refusing to wipe production database!") } Truncate(db)
    • amluto 3y ago
      I also have a table with one row in it indicating whether the database is prod.
    • toasted-subs 3y ago
      Yup, I have 3 prompts if you want to wipe anything. One of the reasons I put interactions between databases behind a cli.
    • devsda 3y ago
      Ours are not named with a common identifier and this also needs constant effort to maintain while refactoring and there's still scope for a mistake. *ideally* devs should not have prod access or their credentials should only have limited access without permissions for destructive actions like drop/truncate etc. But in reality, there's always that one helpful dba/dev who shares admin credentials for a quick prod fix with someone and then those credentials end up in a wiki somewhere as part of an SOP.
      • masklinn 3y ago
        That‘s why you do credentialing via ssh keys, and keys are explained and map to a user, and non-dba keys should expire. If you need access for a quick prod fix, your key gets added to the machine with that explanation and a week (or lees) lifetime.
    • csomar 3y ago
      This is bad because if someone forgot to add prod or for whatever reason the code executed beyond the panic, you’ll wipe out the db. There is no code that will protect your db/data. Only replication to a read-only storage will help in such situations.
• toasted-subs 3y ago
  Nightmare fuel
• justo-rivera 3y ago
  Seems this is very typical, first time launches usually lose some data. We never hear about first time launch deploys that wipe ALL data because whoever is so unlucky probably never got to browse hacker news
• askvictor 3y ago
  One place I worked (some 20 years ago) had a policy that any time you run a sudo command, another person has to check the command before you hit enter. Could apply the same kind of policy/convention for anything in production.
  • deleted 3y ago
    [deleted]
  • yafbum 3y ago
    I'm not sure this doesn't just lead to blind rubber-stamping unless this is done very very rarely
    • 0xDEAFBEAD 3y ago
      The trick is to have good access controls so confirmations happen often enough to be useful, but not so often to be rubber-stamped
    • j16sdiz 3y ago
      I guess most of the common task are scripted / automated. Running a "raw" sudo command should be very very rare.
  • HenriTEL 3y ago
    That's not a good advice IMO, as most sudo commands will mess-up just one host, and it's something you should generally be prepared for. You're more likely to develop a culture where engineers think about hosts as critical resources whereas they should be generally considered as instances that can be thrown away. It's better to identify hosts that are SPOF and be cautious on those only. I can think of a larger blast radius when deleting files on a shared mount point for example but it's not representative to the regular use of sudo.
• stickfigure 3y ago
  I have a rule when working on production databases: Always `start transaction` before doing any kind of update - and pay close attention to the # of rows affected.
  • jeremyjh 3y ago
    If you use postgres you can put \set AUTOCOMMIT off In your .psqlrc and then you can never forget the begin transaction; every statement is already in a transaction, its just the default behaviour to automatically commit the statements for some ungodly reason. Years ago I hired an experienced Oracle developer and put him to work right away on a SQL Server project. Oracle doesn't autocommit by default, and SQL Server does. You don't want to learn this when you type "rollback;". I took responsibility and we had all the data in an audit table and recovered quickly. I wonder if there are still people who call him "Rollback" though.
    • groestl 3y ago
      > its just the default behaviour That's good from the DBA perspective, but relying on that default as a user is risky in itself, when you deal with multiple hosts and not all are set up this way.
• totallywrong 3y ago
  As a young consultant, I was once one Enter away from causing a disaster, but something stopped me. I still shudder even though it didn't actually happen. Nothing of the sort in many years since, so a great lesson in retrospect I guess.
• schemescape 3y ago
  Anecdote: I ran a migration on a production database from inside Visual Studio. In retrospect, it was recoverable, but I nearly had a heart attack when all the tables started disappearing from the tree view in VS… …only to reappear a second later. It was just the view refreshing! Talk about awful UI!
• donatj 3y ago
  Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site. I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut. I write a quick little UPDATE query to update some marketing text on a product and when the query takes more than an instant I knew I had screwed up. Reading my query, I quickly realize I had ran the UPDATE entirely unbounded and changed the description of thousands and thousands of products. Our database admin with access to the database backups had gone home hours earlier as he worked in a different timezone. It took me many phone calls and well over an hour to get ahold of him and get the descriptions restored. The quick change on my way out the door ended up taking me multiple hours to resolve. My friend in marketing apologized profusely but it was my mistake, not theirs. As far as I remember we never heard anything from the client about it, I put that entirely down to it being 5pm on Friday of a holiday weekend.
  • kfrane 3y ago
    That's why I always write a BEGIN statement before executing updates and deletes. If they are not instant or don't return the expected number of modified rows I can just rollback the transaction.
    • deleted 3y ago
      [deleted]
    • groestl 3y ago
      That, and I start the line with /*, write the where clause first, and immediately before I execute the query I check the db host. Oh, and I absolutely refuse to do anything but the most critical stuff against prod on Fridays.
      • deleted 3y ago
        [deleted]
  • xattt 3y ago
• markmark 3y ago
  At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason). Luckily the customer sites each had a local db that synced to the central db (so the product could run with patchy connectivity), but the guy spent 3 or 4 days working looooong days rebuilding the master db from a combination of old backups and the client-site data.
  • Feathercrown 3y ago
    Isn't that almost exactly what happened at github too?
    • ThePowerOfFuet 3y ago
      Yup.
    • technimad 3y ago
      This happened to Gitlab.
  • dotsam 3y ago
    > logged into his dev db and the prod db at the same time in different windows I am very worried about doing the wrong thing in the wrong terminal, so for some machines I colour-code my ssh windows, red for prod, yellow for staging and green for dev. e.g. in my ~/.bashrc I have: echo -ne '\e]11;#907800\a' #yellow background
    • prox 3y ago
      This is a good idea, especially when tired, end of day, crunch time work is happening !
  • arethuza 3y ago
    About 10 years ago I literally saw the blood drain from a colleagues face as he realised he had dropped a production database because he thought he was in a dev environment. A DBA colleague sitting nearby laughed and had things restored back within a few minutes....
• jeffreygoesto 3y ago
  A friend once had to remotely do an OS update of a banking system. Being cautious, he thought he'd back up some central files, just in case and went "mv libc.so old_libc.so". Had to call some guy in that town to throw in the Solaris CD on prem at 2:30 in the morning...
  • xyzzy123 3y ago
    Never this simple and calling someone still probs right thing to but fixing stuff like this is what /sbin is for.
• T-A 3y ago
  That reminds me... https://news.ycombinator.com/item?id=14476421 https://news.ycombinator.com/item?id=14476421
• hannofcart 3y ago
  What strikes me as remarkable in all such stories is how almost always, the person committing the mistake is a junior who never deserves the blame. And how cavalier the handoff/onboarding by the 'seniors' working on the projects are. Having worked in enough of these though, I am aware that even they (the "seniors") are seldom entirely responsible for all the issues. It's mostly business constraints that forces cutting of corners and that ends up jeopardizing the business in the long run.
  • zimpenfish 3y ago
    As I said on Slack the other day in response to a similar story, "If, on your first day, you can destroy the prod database, it's not your fault." (One of my standard end-of-interview questions is "how easy is it for me to trash the production database?" Having done this previously[1] and had a few near misses, it's not something I want to do again.) [1] In my defence, I was young and didn't know that /tmp on Solaris was special. Not until someone rebooted the box, anyway.
    • lostlogin 3y ago
      > /tmp on Solaris was special. I’ve had a search but can’t work out why it’s special.
      • gosub100 3y ago
        it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!
        • sbierwagen 3y ago
          This is rude, but I'd like to reply a comment you deleted in a separate thread. "why didn't they have a hot-spare" They do! Flight spares are complete, flight-rated copies of spacecraft built for exactly this contingency: https://en.wikipedia.org/wiki/Flight_spare https://en.wikipedia.org/wiki/Flight_spare After launch the flight spares are used for terrain testing and troubleshooting. (The "mars yard" has flight spares for Curiosity and Perseverance https://www-robotics.jpl.nasa.gov/how-we-do-it/facilities/marsyard-iii/ https://www-robotics.jpl.nasa.gov/how-we-do-it/facilities/ma... which were used to test some wheels to destruction after Curiosity started showing some wear https://www.planetary.org/articles/08190630-curiosity-wheel-damage https://www.planetary.org/articles/08190630-curiosity-wheel-... ) The blog post lays it on a bit thick with the $500 million number and the "launch only two weeks away" given that the article itself is illustrated with a photo of the Sojourner flight spare. Spirit had the SSTB1 test rover. If he had actually blown out the entire electrical system, they could have launched it instead. Swapping out the entire vehicle right before launch would have been an awful job, but it's not flat out impossible.
• ThePowerOfFuet 3y ago
  >The very moment after I hit the Enter key, I realized I had made a mistake This brief moment in time has a name: an ohnosecond. https://en.wiktionary.org/wiki/ohnosecond https://en.wiktionary.org/wiki/ohnosecond
• NL807 3y ago
  Hats off to the backup guys.
• 3abiton 3y ago
  Backup saving the day.
• PH95VuimJjqBqy 3y ago
  to be fair, it's a rite of passage to do something like this. But you should definitely have bought that man a beer :)
• Tangurena2 3y ago
  One way I mistake-proof things in SQL Management Studio is to have different colors for production vs test databases. To do that, on the "connect to server" dialog, click "options". On the tab "connection properties" in the "connection" option group, check "use custom color". And I pick the reddest red there is. The bottom of the results window will have that color. edit: my horrible foul-up was restoring a database to production. The "there is trouble" pagers were all Iridium pagers since they loved climbing mountains (where there was no cell service back then). But then that place didn't use source control, so it was disasters all the way down.
• RHSman2 3y ago
  I’d love to know the long term physiological effect on the body of these events. Have had a few. Still feel shakey :)
• cloths 3y ago
  We used to have another engineer watch over your shoulder when you do Prod stuff, can be very helpful.