5 ms·

I agree that most companies don't care about false negatives. The problem is that very few companies know how much false negatives are costing them, so it is v

I agree that most companies don't care about false negatives. The problem is that very few companies know how much false negatives are costing them, so it is very difficult to evaluate whether the false negatives are—cough—a negligible tradeoff against weeding out the false positives. False positives are right in your face. But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere. So people lean towards eliminating false positives at all costs. It's another variation on "Looking for the watch where the light is better."

ska 8y ago
It's another variation on "Looking for the watch where the light is better." This really doesn't fit. If there are dozens of watches scattered around, of course you first look where the light is good. These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position. Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).
- thirduncle 8y ago
  If there are dozens of watches scattered around, of course you first look where the light is good. At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
  - kirillseva 8y ago
    You have just equated hiring a coder for a job to finding a spouse.
    - thirduncle 8y ago
      Actually no, I didn't.
  - ska 8y ago
    Point is, it very often gets you far enough. So you try it first, and only adjust when you have real evidence that this is not working for you. Certainly in this scenario - the false positive rate is only important if the true positive rate is too low. It's not at all clear that is a factor.
    - thirduncle 8y ago
      So you try it first, and only adjust when you have real evidence that this is not working for you. That's the whole point -- in order to find evidence that it isn't working, you have to start looking at the false negatives -- or that is to say, where the "light is not".
- braythwayt 8y ago
  I was not equating this to a scenario where you are only looking for one watch. The expression (and anecdote behind it) is also used to describe managing according to metrics that are easy to collect, often at the expense of outcomes. That applies in the case of caring more about false positives than false negatives, since nobody (or close enough) even knows how many false negatives they have. Your statement that someone being successful elsewhere doesn't prove we have a false negative underscores the uncertainty involved. It doesn't invalidate what I'm saying in the least. But if we want to be rigorous, we could do some direct testing. Let's say we give a telephone screening and four interviews to each candidate. Simpkifying somewhat, let's say that our policy is that a fail on any one eliminates the candidate. If we hire lots of people, we can collect false positive and false negative stats by randomly passing someone who fails exactly one test. If a lot of people who fail a particular test end up succeeding, we can infer that the test provides false negatives. On the other hand, if everyone who gets the random "free pass" on one particular test ends up failing, we can presume it has low false negatives, and maybe abandon the random passes for that test.
  - deleted 8y ago
    [deleted]
  - ska 8y ago
    I understood the analogy you were trying to draw, I just disagree with it. The original joke/anectdote works because it leads to clearly suboptimal search - the guy knows his watch isn't near the streetlight, but he's looking there because he can see. This is a very different situation. To abuse the analogy further, the guys best bet to find a watch seems to be just looking near the streetlight. After all, not only can you see better there, but most of the watches are near there too.
    - pandaman 8y ago
      What your opponent is saying is that the cost of false negatives is unknown because it's hard to calculate. So people only calculate the cost of of false positive because it's easy to calculate then compare it to the unknown value and conclude that it's greater, not dissimilar to the gentleman who had been looking for the watch under a streetlight. Your saying "false negative is only a cost to you if you fail to hire well for that position" could be easily disproved with examples of people, who were rejected and started their own competitive business (e.g. [1]). IMHO, one of the goals of having 10Ks of engineers on staff is to keep them from the competition. But since nobody is tracking this and the extreme cases like WhatsApp can be hand-waved away it's easy to pretend that the false negative's cost is miniscule. 1. https://en.wikipedia.org/wiki/WhatsApp#History https://en.wikipedia.org/wiki/WhatsApp#History
gaius 8y ago
But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere. LinkedIn could offer this as a service, if there was sufficient demand that they could monetise it. Every company will routinely check out every candidate’s profile anyway. They just need an automated way to check again in 6 months and see if everyone they reject is going to a competitor who is beating them in the market.

ska 8y ago

It's another variation on "Looking for the watch where the light is better." This really doesn't fit. If there are dozens of watches scattered around, of course you first look where the light is good. These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position. Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).

thirduncle 8y ago
If there are dozens of watches scattered around, of course you first look where the light is good. At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
- kirillseva 8y ago
  You have just equated hiring a coder for a job to finding a spouse.
  - thirduncle 8y ago
    Actually no, I didn't.
- ska 8y ago
  Point is, it very often gets you far enough. So you try it first, and only adjust when you have real evidence that this is not working for you. Certainly in this scenario - the false positive rate is only important if the true positive rate is too low. It's not at all clear that is a factor.
  - thirduncle 8y ago
    So you try it first, and only adjust when you have real evidence that this is not working for you. That's the whole point -- in order to find evidence that it isn't working, you have to start looking at the false negatives -- or that is to say, where the "light is not".
braythwayt 8y ago
I was not equating this to a scenario where you are only looking for one watch. The expression (and anecdote behind it) is also used to describe managing according to metrics that are easy to collect, often at the expense of outcomes. That applies in the case of caring more about false positives than false negatives, since nobody (or close enough) even knows how many false negatives they have. Your statement that someone being successful elsewhere doesn't prove we have a false negative underscores the uncertainty involved. It doesn't invalidate what I'm saying in the least. But if we want to be rigorous, we could do some direct testing. Let's say we give a telephone screening and four interviews to each candidate. Simpkifying somewhat, let's say that our policy is that a fail on any one eliminates the candidate. If we hire lots of people, we can collect false positive and false negative stats by randomly passing someone who fails exactly one test. If a lot of people who fail a particular test end up succeeding, we can infer that the test provides false negatives. On the other hand, if everyone who gets the random "free pass" on one particular test ends up failing, we can presume it has low false negatives, and maybe abandon the random passes for that test.
- deleted 8y ago
  [deleted]
- ska 8y ago
  I understood the analogy you were trying to draw, I just disagree with it. The original joke/anectdote works because it leads to clearly suboptimal search - the guy knows his watch isn't near the streetlight, but he's looking there because he can see. This is a very different situation. To abuse the analogy further, the guys best bet to find a watch seems to be just looking near the streetlight. After all, not only can you see better there, but most of the watches are near there too.
  - pandaman 8y ago
    What your opponent is saying is that the cost of false negatives is unknown because it's hard to calculate. So people only calculate the cost of of false positive because it's easy to calculate then compare it to the unknown value and conclude that it's greater, not dissimilar to the gentleman who had been looking for the watch under a streetlight. Your saying "false negative is only a cost to you if you fail to hire well for that position" could be easily disproved with examples of people, who were rejected and started their own competitive business (e.g. [1]). IMHO, one of the goals of having 10Ks of engineers on staff is to keep them from the competition. But since nobody is tracking this and the extreme cases like WhatsApp can be hand-waved away it's easy to pretend that the false negative's cost is miniscule. 1. https://en.wikipedia.org/wiki/WhatsApp#History https://en.wikipedia.org/wiki/WhatsApp#History

gaius 8y ago

But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere. LinkedIn could offer this as a service, if there was sufficient demand that they could monetise it. Every company will routinely check out every candidate’s profile anyway. They just need an automated way to check again in 6 months and see if everyone they reject is going to a competitor who is beating them in the market.