3 ms·

> Any good company will have a high false negative rate, because it's more expensive to hire a bad candidate than reject a good one. As I usually do in these d

> Any good company will have a high false negative rate, because it's more expensive to hire a bad candidate than reject a good one. As I usually do in these discussions, I would like to register that I think this is a load of crap. First, a good company will have a high F-measure. Optimizing your hiring process for precision and ignoring recall is just a sign of having a really deep, really talented applicant pool (or being deluded about the depth and talent of your pool). Second, when the costs of a bad hire come up lots of "cans" get thrown around without any grounding in reality. I view "optimiz[ing] for false negatives" the same way as I view security theater or overprotective parenting: attempts to stave off very low probability negative events that end up costing more than the events themselves would. Again, with a deep and talented pool this might not be the case, but if you are large enough to have that you are large enough that the damage a bad hire can cause is trivial to your business.

myblake 10y ago
I think it's probably much truer that big tech companies are optimizing for false negatives because they do have a really deep, talented applicant pool. Some start ups do but it's definitely more of a risk. I completely agree with you that in general you're better off optimizing the F-measure, but I'm not sure it's empirically true that lots of companies thought of as good do this.
- vonmoltke 10y ago
  Thinking about this a little more, your final point may be right. It may be that companies become good by optimizing for F-measure, develop a talented and deep pool, then transition to optimizing for precision to manage the applicant load. Thus, most companies considered good have already transitioned while the ones who haven't aren't widely known.
SomeCallMeTim 10y ago
So you're actually arguing for companies to hire employees who will likely produce negative output (their code will be so bad as to take not only their own time, but other programmers' time to fix) just to ...what, not hurt the feelings of the bottom 80 percent of developers? Your analogy is poor: Hiring a bad programmer is an extremely high probability event, even in a broad pool of candidates as would apply to a high-end company like Amazon, not a low priority event. There have been studies dating back to the 70's that show a 10-20x skill difference between average and great developers. If you're going to pay roughly the same for a 1x as a 10x developer, wouldn't you want the 10x? So they tune their hiring process to skew that number as high as possible. OP above may be an exception who fell through the cracks, not really ragging on them in particular. I did know at least one person who applied to Amazon and who was rejected -- and rightfully so, in my opinion. His skills were sub-par, and the interview process detected that. Most others that I worked with while I was at Amazon were also above average; people I've encountered at other jobs have been mediocre by comparison, with the exception of a number of game developers I know.
- vonmoltke 10y ago
  > So you're actually arguing for companies to hire employees who will likely produce negative output (their code will be so bad as to take not only their own time, but other programmers' time to fix) just to ...what, not hurt the feelings of the bottom 80 percent of developers? I did nothing of the sort. In fact, I am arguing that your hypothetical negative output employee is much rarer that some people think. > Your analogy is poor: Hiring a bad programmer is an extremely high probability event, even in a broad pool of candidates as would apply to a high-end company like Amazon, not a low priority event. I have seen no data to support this. My personal experience does not support this. > There have been studies dating back to the 70's that show a 10-20x skill difference between average and great developers. If you're going to pay roughly the same for a 1x as a 10x developer, wouldn't you want the 10x? So they tune their hiring process to skew that number as high as possible. Citation needed. I have certainly seen this bandied about the Internet quite a bit, but the best the citations only claim a 10x difference between great and terrible developers, and even those studies are likely flawed.[1] > I did know at least one person who applied to Amazon and who was rejected -- and rightfully so, in my opinion. His skills were sub-par, and the interview process detected that. Most others that I worked with while I was at Amazon were also above average; people I've encountered at other jobs have been mediocre by comparison, with the exception of a number of game developers I know. I'm not saying people shouldn't be rejected or that bad developers should have jobs at places like Amazon. I'm saying that optimizing for false negatives because you are paranoid of false positives is an anti-pattern and that there are better ways to deal with false positives. [1] http://blog.fogcreek.com/10x-programmer-and-other-myths-in-software-engineering-interview-with-laurent-bossavit/ http://blog.fogcreek.com/10x-programmer-and-other-myths-in-s...
  - eecc 10y ago
    About the 10x difference in programmer productivity: it's a "leprechaun", a false myth emerged from misquoting of previous articles, poor review processes of the claims made and the generalized tendency to ape a scientific process in the software engineering world. Quite some tall claims but I have read a book that makes some good arguments supporting this as well as debunking a couple of these legends turned truths: https://leanpub.com/leprechauns https://leanpub.com/leprechauns
pyb 10y ago
In my experience, there is a large hidden cost in rejecting good candidates. I've seen first hand companies go under and projects fail, because too few employees really had the experience or ability to tackle challenging projects.

myblake 10y ago

I think it's probably much truer that big tech companies are optimizing for false negatives because they do have a really deep, talented applicant pool. Some start ups do but it's definitely more of a risk. I completely agree with you that in general you're better off optimizing the F-measure, but I'm not sure it's empirically true that lots of companies thought of as good do this.

vonmoltke 10y ago
Thinking about this a little more, your final point may be right. It may be that companies become good by optimizing for F-measure, develop a talented and deep pool, then transition to optimizing for precision to manage the applicant load. Thus, most companies considered good have already transitioned while the ones who haven't aren't widely known.

SomeCallMeTim 10y ago

So you're actually arguing for companies to hire employees who will likely produce negative output (their code will be so bad as to take not only their own time, but other programmers' time to fix) just to ...what, not hurt the feelings of the bottom 80 percent of developers? Your analogy is poor: Hiring a bad programmer is an extremely high probability event, even in a broad pool of candidates as would apply to a high-end company like Amazon, not a low priority event. There have been studies dating back to the 70's that show a 10-20x skill difference between average and great developers. If you're going to pay roughly the same for a 1x as a 10x developer, wouldn't you want the 10x? So they tune their hiring process to skew that number as high as possible. OP above may be an exception who fell through the cracks, not really ragging on them in particular. I did know at least one person who applied to Amazon and who was rejected -- and rightfully so, in my opinion. His skills were sub-par, and the interview process detected that. Most others that I worked with while I was at Amazon were also above average; people I've encountered at other jobs have been mediocre by comparison, with the exception of a number of game developers I know.

vonmoltke 10y ago
> So you're actually arguing for companies to hire employees who will likely produce negative output (their code will be so bad as to take not only their own time, but other programmers' time to fix) just to ...what, not hurt the feelings of the bottom 80 percent of developers? I did nothing of the sort. In fact, I am arguing that your hypothetical negative output employee is much rarer that some people think. > Your analogy is poor: Hiring a bad programmer is an extremely high probability event, even in a broad pool of candidates as would apply to a high-end company like Amazon, not a low priority event. I have seen no data to support this. My personal experience does not support this. > There have been studies dating back to the 70's that show a 10-20x skill difference between average and great developers. If you're going to pay roughly the same for a 1x as a 10x developer, wouldn't you want the 10x? So they tune their hiring process to skew that number as high as possible. Citation needed. I have certainly seen this bandied about the Internet quite a bit, but the best the citations only claim a 10x difference between great and terrible developers, and even those studies are likely flawed.[1] > I did know at least one person who applied to Amazon and who was rejected -- and rightfully so, in my opinion. His skills were sub-par, and the interview process detected that. Most others that I worked with while I was at Amazon were also above average; people I've encountered at other jobs have been mediocre by comparison, with the exception of a number of game developers I know. I'm not saying people shouldn't be rejected or that bad developers should have jobs at places like Amazon. I'm saying that optimizing for false negatives because you are paranoid of false positives is an anti-pattern and that there are better ways to deal with false positives. [1] http://blog.fogcreek.com/10x-programmer-and-other-myths-in-software-engineering-interview-with-laurent-bossavit/ http://blog.fogcreek.com/10x-programmer-and-other-myths-in-s...
- eecc 10y ago
  About the 10x difference in programmer productivity: it's a "leprechaun", a false myth emerged from misquoting of previous articles, poor review processes of the claims made and the generalized tendency to ape a scientific process in the software engineering world. Quite some tall claims but I have read a book that makes some good arguments supporting this as well as debunking a couple of these legends turned truths: https://leanpub.com/leprechauns https://leanpub.com/leprechauns

pyb 10y ago

In my experience, there is a large hidden cost in rejecting good candidates. I've seen first hand companies go under and projects fail, because too few employees really had the experience or ability to tackle challenging projects.