8 ms·

The tech world is rallying around a young developer who made a huge mistake

• ng12 9y ago
  This article clearly missed the point. He didn't make a huge mistake, he made a very tiny one. The company/CTO made the huge mistakes.
  • shirian 9y ago
    To be fair, the article states that the junior dev made one mistake, whereas the company made several. However, I agree with you in that it should emphasize that the mistake made by the junior dev is lesser in both size and quantity.
  • Avenger42 9y ago
    He made a tiny mistake that, due to the mistakes made by the company/CTO, had huge consequences. As a first-day developer, he shouldn't really have had the ability to make a mistake with any consequences beyond lost time.
  • protomyth 9y ago
    A old supervisor of mine told me that "You treat a new developer like a duckling near a pond with Northern Pike". There is no way you let them have access to production (or frankly, anything past a development environment) until they have been there a while and gotten the hang of it. On a tangent, I was paranoid enough in one position to have all my production terminal windows have a red background with yellow text. That brought the proper, serious attitude towards my work.
    • zaphod12 9y ago
      That's not paranoid! That's a fantastic thing to do and one that I highly recommend for anyone. It's terribly easy to forget which environment you're working in when you get distracted for a moment and that's how mistakes happen.
      • protomyth 9y ago
        I went the whole nine yards on that project. I had white (black font) for my own machine, green for development, blue for integration test, yellow for system test, and red for production. I liked the green / white text theme the most with blue being my second favorite to write code in so those got the ones I would use the most. I was never a lover of the old amber terminals and red is just plain painful to write on. I guess the more pain your terminal causes you, the more dangerous the environment.
        • wsinks 9y ago
          Was it automated? If so, how?
          • rocky1138 9y ago
            It doesn't help the terminal itself, but changing your ~/.vimrc to use different themes can help.
          • tbirrell 9y ago
            I have a similar setup, and it is relatively easy to set up using .bashrc and terminal color variables.
            • throwanem 9y ago
              I did something similar with a shell script for MySQL databases, both to support shorthand invocation (i.e. "mysql foo-bar-baz" to look up the named configuration, expand it into whatever set of arguments is defined therein, and invoke mysql(1) accordingly), and to allow for detailed and environment-specific prompts which warn me when I'm targeting production. It's hideous, of course. But it works very well for me, and I don't suppose I'd be averse, if interest becomes evident, to sharing it somewhere for others to benefit from and/or peruse in horrified fascination.
              • throwanem 9y ago
    • jt2190 9y ago
      Indeed. The difference between an experienced person a naive one is not that the experienced person doesn't make mistakes, but that they fully expect to make mistakes so they take precautions to mitigate their effects, take measures to prevent them, and minimize doing work that is mistake-prone.
      • protomyth 9y ago
        Inexperienced folks really haven't had that "oh hell" moment of contemplating what could go wrong and the experience to know how bad it can go wrong. I was at a site when a truck hit the power main and the UPS (diesel if I recall) for the server room failed to work. The new people were in the "It shouldn't take too long to reboot the servers" where the experienced people were "this is going to be a very long day". The later group was rewarded with a 36 hour day.
    • blackaspen 9y ago
      An old team did a similar thing! A terminal window with a skull and crossbones when you log into anything that's ".prod.X.com".
    • larrik 9y ago
      I definitely color my production terminals orange (and QA/staging get purple), and it's saved me a few times. I wouldn't consider NOT doing that.
    • Teknoman117 9y ago
      And here I was just put "(production)" in $PS1...
      • throwanem 9y ago
        Try this: export PS1=$(tput bold; tput setaf 1;)'(production)'$(tput sgr0)' \u@\h: \W $ ' and get something a little more lively! (For many more things you might be able to do with "tput", refer to "man 5 terminfo" on your local Linux machine.)
    • pfranz 9y ago
      I really like having a strong, visual indicator when you're doing something dangerous, as well. I had a job where we would often sudo as other users and ssh into other boxes. I kept trying and trying to find a way to change the terminal when sudoed as someone else because occasionally I would execute something in the first terminal I could grab. At the time I couldn't find anything or think of a way.
    • Zhenya 9y ago
      Apparently Northern Pike can eat ducklings https://www.youtube.com/watch?v=2_E4BUt1z8Y https://www.youtube.com/watch?v=2_E4BUt1z8Y
      • protomyth 9y ago
        Yep, fairly horrific. They will also attack a swimmer for no apparent reason other than they are an evil, foul-tempered fish. The bigger ones can make you bleed quite quickly. River barracuda is a name I heard once.
    • lgas 9y ago
      DBVisualizer has a setting that lets you designate different color schemes for different databases. It's saved me more than once.
      • reitanqild 9y ago
        I think TOAD has as well.
  • Florin_Andrei 9y ago
    It's really, really simple. Just restrict access to the production databases. Structure the access in such a way that this kind of honest mistake simply cannot happen. Add an extra hoop to jump through to access prod. Easy as that. Better yet, prod should be in a different "cloud universe" (root account, etc) from dev. > The company/CTO made the huge mistakes. Absolutely.
  • kafkaesq 9y ago
    The company/CTO made the huge mistakes. Indeed. To capitalize on a meme in current circulation, "One does not simply 'accidentally destroy' a production database." When things like this happen, they happen because someone who almost certainly did, in fact, know better -- at some point made a conscious decision to allow them to happen, by consciously cutting corners in certain matters of due diligence that we all know about. Things like backup systems, database roles, and hiring the right people to set up and maintain these things. Not because of something a junior dev did.
  • snambi 9y ago
    The CTO is incompetent. Don't blame the junior engineer.
• godzillabrennus 9y ago
  Glad to see more folks discussing this. I'm sure it's reassuring to the new developer that this wasn't their fault. It's still amazing how badly that company is being managed from a technical perspective.
  • runin2k1 9y ago
    Or... since no aspect of the story is verifiable, and much of it is nonsensical in the reactions of all parties involved, it is completely fake.
    • steveb0x 9y ago
      What would be the motivation behind fabricating a story like this? I see posts of this nature all the time on r/csquestions. This one just happened to go viral.
      • m0nty 9y ago
        > What would be the motivation behind fabricating a story like this? Lulz, getting one over on the Internet, being able to fabricate a plausible story and see people all over the world reacting to it. Some people get off on weird stuff.
      • runin2k1 9y ago
        Attention seeking behavior. Maybe a clever teacher asking students to create fake news to see how easy people are to fool. Boredom. See -- Programmer Automates His Job Away for 6 years and then forgets how to program. I don't think any individual aspect of the story is unbelievable, but when taken as a whole it stretches credulity.
• throwaway7645 9y ago
  Did anyone ever confirm it wasn't a hoax? Or find the actual company?
  • neuromantik8086 9y ago
    I mean, wasn't the crux of this story a plot point in Silicon Valley? Either its a hoax or Silicon Valley is truly uncannily realistic.
    • iaw 9y ago
      > Silicon Valley is truly uncannily realistic. Silicon Valley and Veep are both disturbingly prescient.
    • SideburnsOfDoom 9y ago
      > Either its a hoax or Silicon Valley is truly uncannily realistic. A rule of thumb is that the really ridiculous stuff in Silicon Valley like the "three commas" or the "blood boy", actually happened.
• fageyogurtspoon 9y ago
  This is why, when you see gaping holes in production systems, security or otherwise, and those holes aren't your fault, your best course of action is to do nothing. If you explore it, you might trigger cascading effects and you'll be blamed for it. If you expose it, you'll embarrass someone potentially much more senior/powerful than you. If you do nothing, someone else falls in and you plead ignorance.
• eddyg 9y ago
  Previous discussion (718 points, 262 comments): https://news.ycombinator.com/item?id=14476421 https://news.ycombinator.com/item?id=14476421
• pier25 9y ago
  I've said it before and I'll say it again. It doesn't make any sense that anyone would put a junior on his first day even close to a production environment, much less using a tool that can delete a database that btw you can't restore from a backup. IMO this is either a false story, or this was done on purpose. What if the company wanted to get rid of some data that would put the company in a difficult position in front of its shareholders or even the law? What if the lost data could prove that the CTO had made some other much bigger mistake that allowed hackers to get private information? We will never know of course.
  • throwanem 9y ago
    You assume a great deal of basic competence on the part of everyone at a relatively high level in our industry!
    • deleted 9y ago
      [deleted]
  • dlp211 9y ago
    Nah, this is pure Hanlon's Razor right here: "Never attribute to malice that which is adequately explained by stupidity."
    • amenod 9y ago
      That still only applies to the second option. The first one (false story) is very much possible and it seems very plausible to me. It could be a bet, a sociological research or one of those "why not" ideas... we'll probably never know.
  • scaryclam 9y ago
    One of the first things the contractors at my current company used to do was to make new guy (even a junior) a sudoers enabled account on production (we had no in house development before I arrived, so they had access to everything). Never ascribe to malice that which can be explained by incompetence.
  • 9y ago
• jwildeboer 9y ago
  Did anyone offer him/her a new job? Or pro bono legal help?
• cdubzzz 9y ago
  Fun story! A long time ago I interned for a very large catalog company. It was my first real internship in IT. One of the first things (so a couple days in, not exactly day one) the CTO had me do was "map" their server room (e.g. figure out where data and power cables were going to and coming from, physically and virtually). While working my way through the back of a massive UPS rack, I stepped on one power cord that happened to traverse over another 10 or 15 cords. I did it in such a way that all of those cords were pulled ever so slightly from their UPS connections. I didn't even realize it until one of the staff IT guys came in the room a few minutes later and asked me what happened. Turns out those cords powered the 300+ Citrix thin clients in use by the national call center downstairs. I thought for sure I was in huge trouble, but everything came back online fine, the staff IT guys poked some fun at me and the CTO helped turn it in to a valuable lesson. I learned a lot of great stuff during that interneship and to this day, 15 or so years later, I still think of that experience when doing risky work (or when I fuck it up).
  • djsumdog 9y ago
    Sounds like a cable fail: https://www.reddit.com/r/cablefail/ https://www.reddit.com/r/cablefail/
• spapas82 9y ago
  The original story (junior dev was given access to production / cleared the database while trying to setup his dev env) was funny to me. Actually, I can't really believe that what was described is possible. For me, there are two explanations: - It's a prank by the OP. He posted a story out of his mind for whatever reason. - It was all manipulated by a different employee of the company. He (by accident) broke the production db and decided to blame the junior dev: Most junior devs will just believe you (out of fear and inexperience) when you say to them that they broke the production. The 2nd one seems more possible to me.
• neoCrimeLabs 9y ago
  Unrelated to the story, but I've had a similar experience as to some of the stories described in the article - where a cascade of unseen problems landed on my lap and I took the blame even though little of it was my doing. (Note, this was many years ago and mildly traumatic, so some details might be fuzzy) In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients. One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less). So I come in. Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down. The client remotely shut down the servers in question but they didn't power off (missed clue). One by one I power them down, roll them out of the rack, insert memory. This was a pretty routine request, so it went smoothly - so I had thought. Switch the console to the database server, boot. The kernel starts without issue, then a slew of error messages. Too many to read. The Sun 420R's were known to have memory riser issues, so I power off reseat the riser, and all the memory. Same problem. I take out the new memory. Same problem. Before I could troubleshoot the client guesses it's a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate. Nothing comes up. Not One System So at this point the client starts to panic over the phone. They hang up on me. So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages. All the error messages are Bourne Shell (sh, not bash) errors. Realizing this doesn't appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system. Everything looked great. So I tried to run a script, and then it became obvious - everything LOOKED too great. All the init files had been reformatted. It said as much in a comment. A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server. The problem is, bourne shell didn't like some of the formatting changes - mainly space between variable, equals sign, and value declarations. Had he tested his changes, it would have been obvious what the problem was. Testing my theory I fixed the formatting of a single script, and it immediately works without error. Problem found, now I need to fix all the scripts on all servers. I call the client. No answer. I call the client again. Answer. "Did you fix our servers yet?" "No, I found..." phone disconnects. I call again, no answer. I call again, no answer. I call the NOC, asking they try to inform the client of the status. So I proceed to start fixing the first server. It took an hour, but the server came up. The second server came up a little faster, and I proceeded to work on the third and fourth. Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job. I find this out as one of the analysts comes up from the NOC and tells me I'm relieved, and was asked to go home with no further information. So I explain what had happened and what I was doing to fix the issue. The next day I am called into the office. Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client. Happy birthday. Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.
• heisenbit 9y ago
  Once I was put in charge of running regularly some DB scripts for a social site (non profit, not a payed job) with about 10K active users. It was all manual as the nobody had the time and skill to automate it. There was no development site, there was only production so adding any automation was also risky. And of course one fateful Sunday I pasted a script that was not properly adjusted and the DB was fubar. It took us more than a day to recover - thankfully we had a recently current backup. Later we all spoke about it and also did a write-up. The debriefing helped a lot processing the shock but for a long time I loathed to log into the DB. When I'm fully in control I engineer things in a way that make disasters unlikely and easily recoverable. But that is not always possible. I tend to avoid touching any production DB but again that is not always possible. I always believed I have a developer mindset more than an operational mindset and worked more in roles compatible with the former than the larger. When looking at the devops culture I always wonder whether the lessons of strict control of production environments are getting enough attention. There is value is closing the gap between the two world but please leave a gap.
• jzl 9y ago
  I'm going to quietly leave this obligatory Arrested Development reference right here ... https://www.youtube.com/watch?v=GwQW3KW3DCc https://www.youtube.com/watch?v=GwQW3KW3DCc
• SubuSS 9y ago
  I am surprised they even have their prod data cloned into test servers: Get to any reasonable size and this becomes a huge no-no. There were major measures in MS/Amazon (where I spent a bunch of time) and now at snap to ensure prod data isn't even visible without a bunch of hoops to jump through.