4 ms·

HA! This is my favorite interview question to ask candidates: "What is your all time biggest screw up, and how did you come back from it" - I then tell them th

HA! This is my favorite interview question to ask candidates: "What is your all time biggest screw up, and how did you come back from it" - I then tell them the story of me loosing several hundred thousand dollars and the funny things that happened around it to set the tone. If you have been in tech for any length of time you have one of these stories (if not a few). I have heard some great ones by simply asking and it gives great insight into a candidate (humor, stress response, the things you have seen).

stryk 9y ago
And how does that several hundred thousand dollar story go? (if you have time, and care to elaborate, that is) I'm curious.
- smudgymcscmudge 9y ago
  You have to apply for a job to find out
- jackhack 9y ago
  Me too. I just want to hear the "funny" part of losing that much money. I think the story would be a bit more somber, something along the lines of "...and that's how I ended up blacklisted from the Financial industry, and living in a van down by the river."
chasd00 9y ago
i was a college kid at my first "computer job" doing desktop support for a client in downtown Fort Worth in a fairly nice office suite. Remember those old AT power switches? I wired one backwards, turned it on, and it blew all the insulation off the wires in the case and caught on fire. All in a very nice office, fortunately no sprinkles or fire alarm but I was very embarrassed and nearly cost my boss the client. EDIT: I once got a call from junior developer who had issued an sql statement on an auto-commit database like "update <table> set x=y; where <some condition>; she fat fingered the semicolon and blew up the whole table. She was crying, I felt pretty bad we were able to get the table restored and back online in a few hours. She still tells me how sorry she was to this day ( about 15 years later ).
- wand3r 9y ago
  You can tell from your grammar you pronounce 'sql' in your head as S-Q-L and not sequel.
  - pm90 9y ago
    Wow. I had to read that statement a couple of time to understand exactly what you had inferred. That’s ... really neat.
    - bkmartin 9y ago
      It flowed just fine for me because I pronounce it S-Q-L in my head as well. I don't know why saying SEQUAL has always felt unnatural to me. Probably because I came across it independently and learned about it by myself instead of having someone else introduce it to me as SEQUAL before seeing the acronym.
brador 9y ago
What valuable skill or quality does this show?
- perl4ever 9y ago
  1. How are you at dealing with stress? 2. How are you at dealing with the instinctual desire to hide your mistakes, and instead immediately take steps to remediate them?
- squegles 9y ago
  Honesty and the ability to own up to your mistakes. It can also show arrogance depending on their response.
  - MiscIdeaMaker99 9y ago
    I always ask potential coworkers during an interview for a story like that for two reasons: 1) it'll tell me whether or not they have humility, honesty, etc; 2) and, because I love sharing war stories. One candidate we interviewed answered my question saying that he had never made a mistake like that. Then he went on to tell me a story about how a bad patch from Oracle which he applied brought down production this one time. The guy seemed arrogant, and I felt like he was lying or he is overstating his skills. Either way, he ended up getting hired by another team (knowing full well that we didn't like him), and he only stayed for about a month or two. Go figure.
- puzzle 9y ago
  In addition to what others have said about honesty and ownership of mistakes, it shows you how much they're capable of learning from the inevitable mistakes. Even more importantly, if they haven't made any major mistakes yet, there's a good chance they'll make it once you hire them, because they haven't been through the whole experience and have no scars to show for it. Or maybe they haven't had yet a job where they were responsible for anything that could result in a major incident. There's no lesson in paranoia as great as wiping the wrong storage array, shutting down services in an order other than the sanctioned one, etc.
- 9y ago
perl4ever 9y ago
I was swapping tables on a live production server,* and accidentally removed all rows on the production table rather than the auxiliary one. With autocommit on. I immediately confessed to my boss, and luckily, there was a backup. However, I never did what I saw other people do at another job (more than once), which was to run a delete on a live production table intentionally, but unintentionally leave off the "where" clause. And commit without thinking. I'm not aware of anything I've screwed up that had a dollar figure attached. *I am not and never have been a sysadmin per se.
- snuxoll 9y ago
  You didn't remove rows, you had an unscheduled test of your restore procedures. I've done something similar before, forgot to disable autocommit in DataGrip and ran an UPDATE without a where clause - thankfully we take weekly full backups, nightly incrementals and archive log segments so I shut down the database, performed a point-in-time-restore and the damage was undone. This was actually a good teaching moment, said database is used as a holding area for incoming data from hospitals before we load it into our billing system so our EDI team has limited write permissions on the database - it's much harder for them to run a "oops, time to grab a backup" query but it was a nice anecdote to use when telling them to run everything in a transaction if they're running anything other than SELECT statements.
jachee 9y ago
[13:45] core-router# debug all *** Session Terminated *** ...oops.
- jwbensley 9y ago
  You're not a NetEng until you've done this! I've done it more than once, so I must be great :)
  - willejs 9y ago
    I've done this once, and only once.

stryk 9y ago

And how does that several hundred thousand dollar story go? (if you have time, and care to elaborate, that is) I'm curious.

smudgymcscmudge 9y ago
You have to apply for a job to find out
jackhack 9y ago
Me too. I just want to hear the "funny" part of losing that much money. I think the story would be a bit more somber, something along the lines of "...and that's how I ended up blacklisted from the Financial industry, and living in a van down by the river."

chasd00 9y ago

i was a college kid at my first "computer job" doing desktop support for a client in downtown Fort Worth in a fairly nice office suite. Remember those old AT power switches? I wired one backwards, turned it on, and it blew all the insulation off the wires in the case and caught on fire. All in a very nice office, fortunately no sprinkles or fire alarm but I was very embarrassed and nearly cost my boss the client. EDIT: I once got a call from junior developer who had issued an sql statement on an auto-commit database like "update <table> set x=y; where <some condition>; she fat fingered the semicolon and blew up the whole table. She was crying, I felt pretty bad we were able to get the table restored and back online in a few hours. She still tells me how sorry she was to this day ( about 15 years later ).

wand3r 9y ago
You can tell from your grammar you pronounce 'sql' in your head as S-Q-L and not sequel.
- pm90 9y ago
  Wow. I had to read that statement a couple of time to understand exactly what you had inferred. That’s ... really neat.
  - bkmartin 9y ago
    It flowed just fine for me because I pronounce it S-Q-L in my head as well. I don't know why saying SEQUAL has always felt unnatural to me. Probably because I came across it independently and learned about it by myself instead of having someone else introduce it to me as SEQUAL before seeing the acronym.

brador 9y ago

What valuable skill or quality does this show?

perl4ever 9y ago
1. How are you at dealing with stress? 2. How are you at dealing with the instinctual desire to hide your mistakes, and instead immediately take steps to remediate them?
squegles 9y ago
Honesty and the ability to own up to your mistakes. It can also show arrogance depending on their response.
- MiscIdeaMaker99 9y ago
  I always ask potential coworkers during an interview for a story like that for two reasons: 1) it'll tell me whether or not they have humility, honesty, etc; 2) and, because I love sharing war stories. One candidate we interviewed answered my question saying that he had never made a mistake like that. Then he went on to tell me a story about how a bad patch from Oracle which he applied brought down production this one time. The guy seemed arrogant, and I felt like he was lying or he is overstating his skills. Either way, he ended up getting hired by another team (knowing full well that we didn't like him), and he only stayed for about a month or two. Go figure.
puzzle 9y ago
In addition to what others have said about honesty and ownership of mistakes, it shows you how much they're capable of learning from the inevitable mistakes. Even more importantly, if they haven't made any major mistakes yet, there's a good chance they'll make it once you hire them, because they haven't been through the whole experience and have no scars to show for it. Or maybe they haven't had yet a job where they were responsible for anything that could result in a major incident. There's no lesson in paranoia as great as wiping the wrong storage array, shutting down services in an order other than the sanctioned one, etc.
9y ago

perl4ever 9y ago

I was swapping tables on a live production server,* and accidentally removed all rows on the production table rather than the auxiliary one. With autocommit on. I immediately confessed to my boss, and luckily, there was a backup. However, I never did what I saw other people do at another job (more than once), which was to run a delete on a live production table intentionally, but unintentionally leave off the "where" clause. And commit without thinking. I'm not aware of anything I've screwed up that had a dollar figure attached. *I am not and never have been a sysadmin per se.

snuxoll 9y ago
You didn't remove rows, you had an unscheduled test of your restore procedures. I've done something similar before, forgot to disable autocommit in DataGrip and ran an UPDATE without a where clause - thankfully we take weekly full backups, nightly incrementals and archive log segments so I shut down the database, performed a point-in-time-restore and the damage was undone. This was actually a good teaching moment, said database is used as a holding area for incoming data from hospitals before we load it into our billing system so our EDI team has limited write permissions on the database - it's much harder for them to run a "oops, time to grab a backup" query but it was a nice anecdote to use when telling them to run everything in a transaction if they're running anything other than SELECT statements.

jachee 9y ago

[13:45] core-router# debug all *** Session Terminated *** ...oops.

jwbensley 9y ago
You're not a NetEng until you've done this! I've done it more than once, so I must be great :)
- willejs 9y ago
  I've done this once, and only once.