5 ms·

False negatives are a lot worse than most interviewers think. (False negative = you pass on a good candidate, false positive = you hire someone who turned out

by fsk 12y ago

False negatives are a lot worse than most interviewers think. (False negative = you pass on a good candidate, false positive = you hire someone who turned out to be unqualified) If a "good" candidate is a 1-in-100 find, then each false negative means you have to look at another 100 candidates. Also, if you decrease your false negative rate by more than you decrease your false positive rate, you're actually hiring MORE bad candidates (while spending more time interviewing). I.e., every time you pass on a good candidate, that gives you another chance to make a mistake and hire someone bad. Your odds of hiring one specific "bad" candidate may be small, but if most candidates are "bad", that actually makes it more likely to hire someone bad each time you pass on someone good.

geebee 12y ago
Arg! I'm on my phone and I meant to upvote! Apologies, this was a very insightful comment. I had never thought of it this way before but this is an instance of Bayes's rule. If the false negative rate goes too high and the percentage of good programmers is small then yes, the process could actually increase the odds of a bad hire.
- fsk 12y ago
  Actually, it happens as long as your "stricter hiring practices" increase your false negative percentage by a lot more than they decrease your false positive percentage. Try it out with some numbers. 10100 candidates, 100 are "good". Suppose you have 2% false positives and 1% false negatives. You hire 99 good candidates and 200 bad candidates. Suppose now you have 0.5% false positives and 90% false negatives. (You decreased your false positive rate by 4x but increased your false negative rate by 90x. This is typical for employers who look for every little excuse to reject someone.) You hire 10 good candidates and 50 bad candidates. Your "good hire" percentage went down, and you're churning through a lot more candidates to meet your hiring quota! So, "it is better to pass on a good candidate than hire a bad candidate" is FALSE if you wind up being too picky on passing on good candidates. Assuming you can identify losers and fire them after a year or two (with decent severance to be fair), you're actually better off hiring more leniently. It's also even worse when you realize that the candidate pool is more like: 10200 candidates, 100 are "good", 100 are "toxic", and the toxic people excel at pretending to be "good". Also, the rules for hiring are different for a famous employer and a no-name employer. Google and Facebook are going to have everyone competent applying. If you're a no-name startup, you'll be lucky to have 1 or 2 highly skilled candidates in your hiring pool.
  - fsk 12y ago
    Also, what makes this mistake common is the feedback you get. When you make a false negative, you never find out that you passed on someone amazing. When you make a false positive, it's professional embarrassment for the boss when he's forced to admit he made a mistake and fire them. So the incentive for the boss is to minimize false positives, even at the expense of too many false negatives. The boss is looking out for his personal interests, and not what's best for the business.
    - din-9 12y ago
      Reasons for not wanting to fire people also include: basic humanity.
      - fsk 12y ago
        But it's inhumane to your other employees to keep someone unqualified around.
  - frostmatthew 12y ago
    > Try it out with some numbers. 10100 candidates, 100 are "good". What you're attempting to do works well for hypothetical drug testing[1] or terrorists but not for hiring developers (or anyone else). With the numbers you used you're proposing that less than 1% of all candidates are "good" - nobody would reasonably set the "good" threshold to include only the top 1% of developers. [1] https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing
    - fsk 12y ago
      Even if you set the "good" percentage to 10%, too high a false positive rate will still ruin your results. Based on the people I've worked with over the years, I say that the actual skill distribution is: 5% toxic - These are the people who will ruin your business while deflecting blame to other people. 25% subtractors - These are the people who need more attention and help than the amount of work they get done. In the right environment, they can be useful. (Also, this is mostly independent of experience level. I know some really experienced people who were subtractors.) 60% average - These people are competent but not brilliant. These are solid performers. 9% above average - They can get 2x-5x the work done of someone average. 1% brilliant - These are the mythical 10x-100x programmers. These are the people who can write your MVP by themselves in 1-3 months and it'll be amazing. You first have to decide if you're targeting brilliant, above average, or average. For most businesses, average is good enough. If you incorrectly weed out the rare brilliant person, you might wind up instead with someone average, above average, or (even worse) toxic. Actually, when my employer was interviewing, I was surprised that the candidates were so strong. There was one brilliant guy and one above-average guy (My coworkers didn't like them; they failed the technical screening, which makes me distrust technical screening even more now). They wound up hiring one of the weakest candidates, a subtractor, and having worked with him for a couple of months my analysis of him hasn't changed.
sytelus 12y ago
No. Many companies have hiring rate of 10% [1] which means hiring 1 candidate may cost you 10 person-days assuming full day interviews for each candidate [2]. You can probably double that to cover the cost of phone screens, so let's say 20-person day of work per good hire. Assume you had 4 false negatives just because you are so aggressive and so a good hire ended up costing you 100 person days instead. This cost is still tiny compared to damages that a true negative would cause otherwise, i.e., (1) loss of entire person year or even two because annual reviews need to accumulate evidence for HR to fire (may be less time if you were startup without real "HR") (2) amount of cleanup other people have to do after that new bad hire (3) loss in moral for good people in your team who now perceives your hiring process at the company as "broken" (4) delays + bugs introduced in product because actual work probably didn't got done or badly done despite of you filling up your headcount (5) amount of money lost in salaries, signing bonuses, office space and benefits (typically > $200K) (6) amount of productivity lost because of wasted time by good people in the team trying to "ramp up" your true negative (7) emotional stress you caused to good people wondering them about their job stability and to managers who wasted their time in months of paper work and lot of explaining (8) emotional stress you caused to your true negative being fired who had moved across the country for you, bought a house on mortgage and had 3 school going children (9) Most likely, if you are big company, true negative didn't actually got fired because hiring manager never wanted to admit it. S/he was encouraged to join another team or role or even learned political tricks to get promoted contributing to ongoing bozo explosion[3] (10) I could go on and easily justify probably 3X-10X loss compared the case if true negative was avoided Footnotes: 1 - Many companies specify their hiring rate over total resume they received which is wrong. I'll use 10% as total number of full interviews that needed to be conducted which is average of 5%-15% at most companies. 2 - This is bad math. Assuming random trials, it would be actually 5 person-day on average but intuitive approach doesn't produce entirely bad results here so we will go with that. 3 - http://guykawasaki.com/how_to_prevent_/ http://guykawasaki.com/how_to_prevent_/
- fsk 12y ago
  Hiring rate of 10%? You mean you hire 10% of all people who submit a resume? Or 10% of all people who come in for an on-site interview? Those are two different things. Why should it take 2 years to fire someone? That sounds like a corporate bureaucracy problem.

by fsk 12y ago

geebee 12y ago

Arg! I'm on my phone and I meant to upvote! Apologies, this was a very insightful comment. I had never thought of it this way before but this is an instance of Bayes's rule. If the false negative rate goes too high and the percentage of good programmers is small then yes, the process could actually increase the odds of a bad hire.

fsk 12y ago
Actually, it happens as long as your "stricter hiring practices" increase your false negative percentage by a lot more than they decrease your false positive percentage. Try it out with some numbers. 10100 candidates, 100 are "good". Suppose you have 2% false positives and 1% false negatives. You hire 99 good candidates and 200 bad candidates. Suppose now you have 0.5% false positives and 90% false negatives. (You decreased your false positive rate by 4x but increased your false negative rate by 90x. This is typical for employers who look for every little excuse to reject someone.) You hire 10 good candidates and 50 bad candidates. Your "good hire" percentage went down, and you're churning through a lot more candidates to meet your hiring quota! So, "it is better to pass on a good candidate than hire a bad candidate" is FALSE if you wind up being too picky on passing on good candidates. Assuming you can identify losers and fire them after a year or two (with decent severance to be fair), you're actually better off hiring more leniently. It's also even worse when you realize that the candidate pool is more like: 10200 candidates, 100 are "good", 100 are "toxic", and the toxic people excel at pretending to be "good". Also, the rules for hiring are different for a famous employer and a no-name employer. Google and Facebook are going to have everyone competent applying. If you're a no-name startup, you'll be lucky to have 1 or 2 highly skilled candidates in your hiring pool.
- fsk 12y ago
  Also, what makes this mistake common is the feedback you get. When you make a false negative, you never find out that you passed on someone amazing. When you make a false positive, it's professional embarrassment for the boss when he's forced to admit he made a mistake and fire them. So the incentive for the boss is to minimize false positives, even at the expense of too many false negatives. The boss is looking out for his personal interests, and not what's best for the business.
  - din-9 12y ago
    Reasons for not wanting to fire people also include: basic humanity.
    - fsk 12y ago
      But it's inhumane to your other employees to keep someone unqualified around.
- frostmatthew 12y ago
  > Try it out with some numbers. 10100 candidates, 100 are "good". What you're attempting to do works well for hypothetical drug testing[1] or terrorists but not for hiring developers (or anyone else). With the numbers you used you're proposing that less than 1% of all candidates are "good" - nobody would reasonably set the "good" threshold to include only the top 1% of developers. [1] https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing https://en.wikipedia.org/wiki/Bayes%27_theorem#Drug_testing
  - fsk 12y ago
    Even if you set the "good" percentage to 10%, too high a false positive rate will still ruin your results. Based on the people I've worked with over the years, I say that the actual skill distribution is: 5% toxic - These are the people who will ruin your business while deflecting blame to other people. 25% subtractors - These are the people who need more attention and help than the amount of work they get done. In the right environment, they can be useful. (Also, this is mostly independent of experience level. I know some really experienced people who were subtractors.) 60% average - These people are competent but not brilliant. These are solid performers. 9% above average - They can get 2x-5x the work done of someone average. 1% brilliant - These are the mythical 10x-100x programmers. These are the people who can write your MVP by themselves in 1-3 months and it'll be amazing. You first have to decide if you're targeting brilliant, above average, or average. For most businesses, average is good enough. If you incorrectly weed out the rare brilliant person, you might wind up instead with someone average, above average, or (even worse) toxic. Actually, when my employer was interviewing, I was surprised that the candidates were so strong. There was one brilliant guy and one above-average guy (My coworkers didn't like them; they failed the technical screening, which makes me distrust technical screening even more now). They wound up hiring one of the weakest candidates, a subtractor, and having worked with him for a couple of months my analysis of him hasn't changed.

sytelus 12y ago

No. Many companies have hiring rate of 10% [1] which means hiring 1 candidate may cost you 10 person-days assuming full day interviews for each candidate [2]. You can probably double that to cover the cost of phone screens, so let's say 20-person day of work per good hire. Assume you had 4 false negatives just because you are so aggressive and so a good hire ended up costing you 100 person days instead. This cost is still tiny compared to damages that a true negative would cause otherwise, i.e., (1) loss of entire person year or even two because annual reviews need to accumulate evidence for HR to fire (may be less time if you were startup without real "HR") (2) amount of cleanup other people have to do after that new bad hire (3) loss in moral for good people in your team who now perceives your hiring process at the company as "broken" (4) delays + bugs introduced in product because actual work probably didn't got done or badly done despite of you filling up your headcount (5) amount of money lost in salaries, signing bonuses, office space and benefits (typically > $200K) (6) amount of productivity lost because of wasted time by good people in the team trying to "ramp up" your true negative (7) emotional stress you caused to good people wondering them about their job stability and to managers who wasted their time in months of paper work and lot of explaining (8) emotional stress you caused to your true negative being fired who had moved across the country for you, bought a house on mortgage and had 3 school going children (9) Most likely, if you are big company, true negative didn't actually got fired because hiring manager never wanted to admit it. S/he was encouraged to join another team or role or even learned political tricks to get promoted contributing to ongoing bozo explosion[3] (10) I could go on and easily justify probably 3X-10X loss compared the case if true negative was avoided Footnotes: 1 - Many companies specify their hiring rate over total resume they received which is wrong. I'll use 10% as total number of full interviews that needed to be conducted which is average of 5%-15% at most companies. 2 - This is bad math. Assuming random trials, it would be actually 5 person-day on average but intuitive approach doesn't produce entirely bad results here so we will go with that. 3 - http://guykawasaki.com/how_to_prevent_/ http://guykawasaki.com/how_to_prevent_/

fsk 12y ago
Hiring rate of 10%? You mean you hire 10% of all people who submit a resume? Or 10% of all people who come in for an on-site interview? Those are two different things. Why should it take 2 years to fire someone? That sounds like a corporate bureaucracy problem.