12 ms·

More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineeri

• liquidise 14y ago
  Exactly what you said. Rigor is truly the right word to use here. Cancelling your db backups is basically asking for a disaster. I'm not sure i have ever been at job that didnt require a db backup for some reason at some time.
  • mooreds 14y ago
    To say nothing of running development code against a production database. If there were only two junior folks, what were the senior folks doing?
    • liquidise 14y ago
      The author makes mention of using a UI to connect to their db. If i was in a position over there i can see myself writing a script to clear out the tables i wanted. This reduces errors, but not the risk.
      • Cthulhu_ 14y ago
        And yet, the net result is the same; right-click, clear table, or ./clearTable.sh. Both are human actions, and both are fallible. What if some prankster edited clearTable.sh to do the users table instead of the raids table? What if he did it himself to test something?
        • chris_wot 14y ago
          Forget a prankster! What happens if the full RAID array fails that holds the database? No backups: dead company.
          • amputect 14y ago
            Heck when you put it that way, this guy actually did them a FAVOR. He ONLY wiped out the User table. The company was able to learn the value of backups, and they had enough data left to be able to partially recover it from the remaining tables, which is much better than the worst case scenario.
            • dsuth 14y ago
              Do you think the company learned the value of backups, or do you think they learned to blame junior devs for fuckups? Sounds like they learned nothing, because no attempt was made to determine the root cause.
      • Aardwolf 14y ago
        Arrow up, enter Or ctrl+r <something> enter Can be very dangerous if something other than your routine is in the bash history.
        • podperson 14y ago
          Exactly. It's not like nothing bad ever happened from the command line ;-) rm * .doc
    • at-fates-hands 14y ago
      This was my question. Usually the "junior" folks are shown how to do things by the senior engineers. The fact they threw this guy under the bus while letting the rest of the senior guys skate is appalling. Part of your job as a senior developer is to ensure this very scenario doesn't happen, let alone to someone on your watch.
    • asveikau 14y ago
      The junior dev was 22 at the time and fresh out of school. I'm assuming the senior devs were 23 and had a year on the job. The principal devs were a ripe 24, otherwise known as old men. :-)
      • heathlilley 14y ago
        He and his coworkers might not have been "Senior" before and they still might not be fully senior, but they are MUCH closer now. >Ding! >Gratz!
• benjaminwootton 14y ago
  I came here to type that almost exactly word for word. This is a truly amazing story if this system was really supporting milions in revenue.
• potatolicious 14y ago
  More than that, it's telling that the company threw him under the bus when it happened. I've been through major fuckups before, and in all cases the team presents a united front - the company fucked up, not an individual. Which is, if you think about it, true, given that the series of events leading up to the disaster (the lack of a testing environment, working with prod databases, lack of safeties in the tools used to connect to database, etc...). The correct way to respond to disasters like this is "we fucked up", not "someone fucked up".
  • coverband 14y ago
    Exactly this.
    • sc0rb 14y ago
      These kind of comments need to be down voted for adding no value.
      • tomp 14y ago
        Now there are already three comments adding no value.
        • marcosploither 14y ago
          4.
          • joedev 14y ago
            make that 5
            • nalgas 14y ago
              me six
  • oseibonsu 14y ago
• pavel_lishin 14y ago
  > It's noble of the author to quit It's not just noble. Considering how everyone treated him, and the company's attitude in general, he had no future there, and neither should anyone else.
• x3ord 14y ago
  Exactly. I'm ashamed that my first reaction reading this was to blame OP. But in the 2 min it took to read the post I had come full circle to wondering what kind of terribly run company would allow this to happen--I guess the type that hires philosophy majors straight out of college without vetting their engineering skills.
  • pnathan 14y ago
    > the type that hires philosophy majors straight out of college without vetting their engineering skills. Or hires them and then adequately educating them in software development.
    • quanticle 14y ago
      Or one that doesn't have a process to ensure that backup and recovery procedures when something like this happens are as painless as possible.
• jcromartie 14y ago
  Indeed. If it had been a sporadic hardware failure, they would have been exactly as screwed. The fact that they gave an overworked junior dev direct read/write access to the production database is astounding.
  • gte910h 14y ago
    It's not that they gave him r/w at all that's so criminally stupid. It's that the required him to clear a table manually, using generic full access tools, over and over. In reality, this should have been re-factored to the dev db. If it couldn't be, the junior dev should have been given access to the raids table alone for writes. Lastly the developer who didn't back up this table is the MOST to blame. Money was paid for the state in that table. That means people TRUST you to keep it safe. I count tons of people to blame here. I don't really see the junior dev as one of them.
    • sskates 14y ago
      Yeah I think him manually clearing a table over and over again was the big problem here. The amount of entropy that had to be introduced into the process to turn a routine task into millions of dollars of loss was tiny. He just needed to click in a slightly different spot on the webpage to bring up USERS instead of RAIDS.
• indrax 14y ago
  Amazon is at the other end of the spectrum: randomly breaking things so that everything has to be fault tolerant. http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html http://www.codinghorror.com/blog/2011/04/working-with-the-ch... This only happened because nobody even asked "What happens if I press this button?"
  • ajross 14y ago
    This really needs to be more of a standard thing. I've been near (but as an engineer, never responsible for) production systems my whole career. None of these systems were as terribly maintained as the one in the linked article. Production data was isolated. Backups were done regularly. Systems were provisioned with fault tolerance in mind. Not once have I seen a full backup restore tested. Not once have I seen a network failure simulated (though I've seen several system failures due to "kicking out a cable" that sort of acts as a proxy for that technique). On multiple occasions I've seen systems taken down by single points of failure[1] that weren't forseen, but probably could have been. [1] My favorite: the whole closet went down once because everything was plugged into a single, very expensive, giant UPS that went poof. $40/system for consumer batteries from Office Depot would have been a much better bet. And the best part? Once the customer service engineer replaced whatever doodad failed and brought the thing back up? They plugged everything right back into it.
    • alexkus 14y ago
      > Not once have I seen a network failure simulated. Reminds me of the webserver UPS setup at a previous company. The router (for the incoming T1) and the webserver were plugged in to the UPS. UPS connected (via serial port) to webserver. Stuff running on webserver to poll whether UPS running from mains power or batteries and send panic emails if on batteries (for more than 60 seconds) and eventually shutdown the webserver cleanly if UPS power dropped below 25%. Thing not plugged in to UPS: DMZ Network switch (that provided the connectivity between webserver and router).
    • hga 14y ago
      Bingo. As I like to emphasize, people don't care about backups---this company certainly didn't---they care about restores. And almost no one is willing to put up the money to do compete testing of restore paths, let along statistically making sure they continue to work.
• j_baker 14y ago
  "Nobody cares about technical infrastructure. Our customers don't pay us for engineering rigor. We need to just ship!" Of course the person saying that is likely to care about technical infrastructure when it costs them money and/or customers due to being hacked-together.
  • chris_mahan 14y ago
    Nobody cares about the diameter of the cylinders in the engine, all they care about is going from point A to point B safely, affordably, in style.
• downandout 14y ago
  Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have been fired. I've had more than my fair share of failures in the start up world. It always drives me crazy to see internet companies that have absolutely no technical ability succeed, while competing services that are far superior technically never get any traction.
  • happycube 14y ago
    And worse, they had a backup service, and then dropped it to save money.
    • reinhardt 14y ago
      This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".
      • georgemcbay 14y ago
        Unfortunately it sounds perfectly plausible to me. "Why are we paying for backups? The database has never failed (yet)"!
        • whyaduck 14y ago
          Super important goal my boss gave me: reduce expenditures...CHECK! Can't wait for that bonus check!!
      • angersock 14y ago
        Having just helped a friend who was miffed at the idea of spending money on a new UPS for >$5k worth of networking equipment: do not be surprised. Penny-wise pound-foolish.
• sybhn 14y ago
  Absolutely. Your immediate technical management sucked, and you were made the scape goat for your management's failure. Welcome to the real world. Don't get me wrong, you should feel bad, very bad, bad enough so that never again you do that. But you shouldn't feel guilty nor rethink your career.
  • hga 14y ago
    A little bit of feeling guilty is in order; the author "didn't know that he didn't know", and I'm sure this motivated him to learn a lot more about proper engineering processes ... something I'll note aren't particularly a focus in CS degrees. Especially since I haven't come across anyone who's really dedicated to them who hadn't first gotten burned in one way or another. There's a big difference between being told "Do X, don't do Y" and that sinking feeling you get when you realize a big problem exists, regardless of the eventual outcome.
• fitandfunction 14y ago
  Exactly. Problems with production databases are inevitable. It's just a matter of time. The guy who should be falling on the sword, if anyone, is the person in charge of backups. Better yet, the CEO or CTO should have made this a learning opportunity and taken the blame for the oversight + praised the team for banding together and coming up with a solution + a private chat with the OP.
• gawker 14y ago
  Agree with the point that humans are fallible. We should always have back ups. A company with 1000s of paying customers should at least take steps to protect itself from this sort of catastrophe.
• jalayr 14y ago
  Completely agreed.
• mncolinlee 14y ago
  I cannot even begin to fathom how they functioned without a working development environment for testing, let alone let their backups lapse. The kind of table manipulations he mentions would be unspeakable in most companies. Someone changing the wrong table would be inevitable. If I were an auditor, I would rake them all over the coals.
• coopdog 14y ago
  That's the Agile way, people not processes. When the people screw up, there's no processes to hold them back, they can really, really screw it up. Once you hit millions in revenue it's probably wise to put a couple of fall back processes in place, reliability becomes as important as agility.
• cybernomad99 14y ago
  To the credit of the management, they did not fire him. He resigned. But the coworkers felt he was responsible personally. That makes a uneasy work environment.
  • ftay 14y ago
    That's not much to give credit for. They could certainly have done more to help him recover from this.
• gtirloni 14y ago
  When I was a junior analyst, I once deleted the main table that contained all 70k+ users, passwords, etc. The problem was fixed in 15min after the DBA was engaged to copy all data back from the QA environment that was synchronized every X minutes. Or we could have restored a back from a few hours ago. The whole company fucked this one up pretty badly. NO excuses.
• jliptzin 14y ago
  Unbelievable. Even at my 3 person startup, back in 2010, with thousands in revenue, not millions, we had development environments with test databases and automated daily database snapshotting. Sure I've accidentally truncated a few tables in my time, but luckily I wasn't dumb enough to be developing on a production server.
• SteveGerencser 14y ago
  I worked for a startup a few years back where the CEO deleted the entire 1+TB database/website when the server was hacked and being used as a spam server because he 1. didn't know who to disable the site short of deleting and 2. could reach anyone that did know how. The next morning he told us to restore the site from backups and fix the security hole. That's when we reminded him, again, that he had refused to pay for backup services for a site of that size. We all ended up looking fro new jobs withing a couple days.
• richo 14y ago
  You're forgetting the guy who didn't speak up to say "Hey, maybe we shouldn't do this in prod?". I feel for him, but at the same time there's a point at which you have to if testing guns by shooting them near (but not specifically at) your coworkers is actually a good idea.
• readme 14y ago
  I wish I could upvote this more. Production database? Seriously, they were one copy away from avoiding this whole outcome. CEO sounds incompetent as hell.
  • mehwoot 14y ago
    Hindsight is 20/20.
    • Aardwolf 14y ago
      If you have only one copy of data, especially if it is important, the chance of something happening to that copy, either hardware, software, or human error, is always big enough to justify a backup. No hindsight needed for that.
      • readme 14y ago
        Thinking about this just makes my mind go numb. All someone had to do was have the idea that they should back their database up. It would be done in 5 minutes, tops! I automated ours on S3 with email notifications in under an hour...