7 ms·

An important thing to remember is that broken/not-broken is completely different for each side of that desk: candidate person & hiring person. This is always th

An important thing to remember is that broken/not-broken is completely different for each side of that desk: candidate person & hiring person. This is always the case for selection steps. The employer (or school, etc) wants to make successful hires and to avoid bad ones. False positives are always a big problem. False negatives are usually much smaller one. A process is good if it produces good hires, and no bad ones. In principle, a process that hires 3 good candidates from a pool containing 5 good candidates and lots of bad ones is good. Avoiding bad hires is paramount. Missing 2 good hires is not as bad as making one bad hire, not near as bad usually. Most employers don't care about false negatives at all. If they do, they probably care most about missing really good outlier hires, not mistakenly selecting the 5th best candidate instead of the 4th. The only reason to care is fairness. People instintively care about fairness so this affects practices but it's not nearly like false positives. Employers that need huge volume (eg, the army) are the only ones that care about false positives as much as candidates do. The candidate (student etc.) generally feels the process is "broken" when it is unfair to them. Unfair means that a person could have done the job well, but cannot get hired. Ie, if the candidate is a "false negative" then the process is broken. Anyway... When things become systemic, the "brokenness" becomes acute to a candidate. If whiteboards (your architecture version, Google's CS version or any other version) are prevelant and you suck at them... you can't get hired. There are people who suck at your test (whatever it is) but would be good at the job. Your proxy is a biased against some people, every proxy is. Like I said, a lot of things are like this. One side is working the numbers game, the other side isn't. Insurance markets, college entrance, executive appointments, PayPal's fraudster detection (bastards!), credit scores, investment (even startup investment)...

braythwayt 8y ago
I agree that most companies don't care about false negatives. The problem is that very few companies know how much false negatives are costing them, so it is very difficult to evaluate whether the false negatives are—cough—a negligible tradeoff against weeding out the false positives. False positives are right in your face. But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere. So people lean towards eliminating false positives at all costs. It's another variation on "Looking for the watch where the light is better."
- ska 8y ago
  It's another variation on "Looking for the watch where the light is better." This really doesn't fit. If there are dozens of watches scattered around, of course you first look where the light is good. These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position. Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).
  - thirduncle 8y ago
    If there are dozens of watches scattered around, of course you first look where the light is good. At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
    - kirillseva 8y ago
      You have just equated hiring a coder for a job to finding a spouse.
      - thirduncle 8y ago
        Actually no, I didn't.
    - ska 8y ago
      Point is, it very often gets you far enough. So you try it first, and only adjust when you have real evidence that this is not working for you. Certainly in this scenario - the false positive rate is only important if the true positive rate is too low. It's not at all clear that is a factor.
      - thirduncle 8y ago
        So you try it first, and only adjust when you have real evidence that this is not working for you. That's the whole point -- in order to find evidence that it isn't working, you have to start looking at the false negatives -- or that is to say, where the "light is not".
thomastjeffery 8y ago
False-negatives are only a small problem for employers because they don't experience them.
- kirillseva 8y ago
  This is exactly the point. A risk averse strategy is optimal for areas of a business that you do not have expertise in. If hiring isn't the thing you do best (supposedly your strengths are writing and selling software) then it makes sense to pick a hiring strategy that doesn't cause disruptions.
CodeMage 8y ago
While I agree with you, it's also noticeable that many companies do have a broken hiring process, but they don't realize it, because things aren't bad enough yet. However, whiteboarding is not the culprit. People complain whiteboarding because they've had bad multiple bad experiences with hiring processes that involve whiteboarding and they came to a conclusion that whiteboarding itself is the problem. Unfortunately, you can't fix a bad hiring process by simply changing the technique. Each technique has its own pros and cons, but no technique will guarantee success. These are the most common problems I've seen with hiring processes: 1) The people doing the hiring don't really know what they're looking for. Instead, they go by a vague, handwavy notion of whether the candidate is "a fit" or not. You have to define what "a fit" means in such terms that all of the people involve understand it, agree on it and know how to test for it. 2) The people doing the hiring don't know how to test for their criteria. Whether you're using whiteboard or a take-home project or some other testing technique, you have to design the test so that it helps you observe the traits you're looking for. [1] And you have to actually train your people to interview candidates. 3) Lack of transparency. This is a huge problem and it infests all aspects of hiring. Some recruiters won't even tell you which company they're hiring for. A lot of companies misrepresent what you would be doing for them, sometimes intentionally, but usually not. Some companies won't give you perfectly reasonable information about the job. [2] And, thanks to the litigious nature of our society, no company -- anywhere, ever -- will give the candidate honest, detailed feedback of why they were rejected. 4) The company is unreasonably demanding when it comes to candidate's personal time. The typical two-stage process -- a phone screen followed by on-site interview round -- is demanding enough, because it involves more than just the interviews. The candidate has to spend time and effort on other aspects, too: multiple phone calls with the recruiter, preparation for the interviews, travel, and juggling all of this with their personal and professional lives. Some companies don't understand this, so they think there's nothing wrong with a process that involves a take-home project, a debriefing phone call, a face-to-face conversation to test for cultural fit and a full-day on-site interview round. Most companies don't even know that their hiring process sucks. Of those who eventually realize something is wrong, many end up cargo-culting whatever they read or heard about "more successful" cases. As things are right now, the whole situation is eerily reminiscent of the fuss and hype the industry went through with software development processes. I wouldn't be surprised to see yet another wave of parasitic consultants pop up to prey on companies by flogging their hiring process improvements and best practices. ---------- [1] This is where a lot of people's complaints go wrong. The most frequent grumble is: why does this company's interview process focus on algorithms and data structures, if you'll just end up coding some services in Java? If the company is doing things right, they're using these topics as a tool to observe and measure whatever it is they're looking for; most often, it's the candidate's understanding of computational complexity. Just because you won't be writing your own implementations of common data structures and algorithms, it doesn't mean you shouldn't know how to pick the one that won't blow up your memory or execution time. [2] I had a case where I was interviewing with a company that gave me a print-out overview of their benefits, which included the price of each health plan, but not what it covered; when I inquired about it, they replied they wouldn't be able to give me that information until I signed the New Hire NDA.
Clubber 8y ago
>False positives are always a big problem. False negatives are usually much smaller one. I would agree with you if the supply and demand weren't so skewed towards top developers. Do you think you are choosing between A and D people, or are you really choosing between C and D people and A and B people never walk through the door?
- andrewvc 8y ago
  It's still the case! At the end of the day, the person making the call is a hiring manager. It's pretty hard for a hiring manager to be called out as too selective. However, if you let in a few bad apples then you look really bad. The incentives are for hiring managers to be too careful. It is, oddly enough, the case that if you are not hiring any bad engineers that you probably are being too selective.
  - Clubber 8y ago
    It's easy to fix a false positive, it's impossible to fix a false negative. You figure FOMO would kick in at some point.
    - cornellwright 8y ago
      Once you hire someone you have to give them a chance to be successful. As the threshold for firing someone is (rightly) much higher than simply not hiring them, fixing a false positive is much more expensive than missing out on a good candidate. Also you can fix some false negatives - allow people to interview again for other positions at the company. I've seen multiple people get hired on the 2nd or 3rd time around and ultimately be successful.
      - scarface74 8y ago
        Simple - contract to perm.
        gaius 8y ago
bradleyjg 8y ago
It may be understandable, but the prevailing attitude towards type I vs type II errors in tech industry hiring certainly makes me less sympathetic to claims that “we can’t find anyone to hire” and therefore governments should enact policies X,Y,&Z. If they are choosers they ought not to go begging.

braythwayt 8y ago

I agree that most companies don't care about false negatives. The problem is that very few companies know how much false negatives are costing them, so it is very difficult to evaluate whether the false negatives are—cough—a negligible tradeoff against weeding out the false positives. False positives are right in your face. But it is nearly impossible to systematically keep track of the candidates that aren't hired to see if they are successful elsewhere. So people lean towards eliminating false positives at all costs. It's another variation on "Looking for the watch where the light is better."

ska 8y ago
It's another variation on "Looking for the watch where the light is better." This really doesn't fit. If there are dozens of watches scattered around, of course you first look where the light is good. These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position. Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).
- thirduncle 8y ago
  If there are dozens of watches scattered around, of course you first look where the light is good. At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
  - kirillseva 8y ago
    You have just equated hiring a coder for a job to finding a spouse.
    - thirduncle 8y ago
      Actually no, I didn't.
  - ska 8y ago
    Point is, it very often gets you far enough. So you try it first, and only adjust when you have real evidence that this is not working for you. Certainly in this scenario - the false positive rate is only important if the true positive rate is too low. It's not at all clear that is a factor.
    - thirduncle 8y ago
      So you try it first, and only adjust when you have real evidence that this is not working for you. That's the whole point -- in order to find evidence that it isn't working, you have to start looking at the false negatives -- or that is to say, where the "light is not".

thomastjeffery 8y ago

False-negatives are only a small problem for employers because they don't experience them.

kirillseva 8y ago
This is exactly the point. A risk averse strategy is optimal for areas of a business that you do not have expertise in. If hiring isn't the thing you do best (supposedly your strengths are writing and selling software) then it makes sense to pick a hiring strategy that doesn't cause disruptions.

CodeMage 8y ago

While I agree with you, it's also noticeable that many companies do have a broken hiring process, but they don't realize it, because things aren't bad enough yet. However, whiteboarding is not the culprit. People complain whiteboarding because they've had bad multiple bad experiences with hiring processes that involve whiteboarding and they came to a conclusion that whiteboarding itself is the problem. Unfortunately, you can't fix a bad hiring process by simply changing the technique. Each technique has its own pros and cons, but no technique will guarantee success. These are the most common problems I've seen with hiring processes: 1) The people doing the hiring don't really know what they're looking for. Instead, they go by a vague, handwavy notion of whether the candidate is "a fit" or not. You have to define what "a fit" means in such terms that all of the people involve understand it, agree on it and know how to test for it. 2) The people doing the hiring don't know how to test for their criteria. Whether you're using whiteboard or a take-home project or some other testing technique, you have to design the test so that it helps you observe the traits you're looking for. [1] And you have to actually train your people to interview candidates. 3) Lack of transparency. This is a huge problem and it infests all aspects of hiring. Some recruiters won't even tell you which company they're hiring for. A lot of companies misrepresent what you would be doing for them, sometimes intentionally, but usually not. Some companies won't give you perfectly reasonable information about the job. [2] And, thanks to the litigious nature of our society, no company -- anywhere, ever -- will give the candidate honest, detailed feedback of why they were rejected. 4) The company is unreasonably demanding when it comes to candidate's personal time. The typical two-stage process -- a phone screen followed by on-site interview round -- is demanding enough, because it involves more than just the interviews. The candidate has to spend time and effort on other aspects, too: multiple phone calls with the recruiter, preparation for the interviews, travel, and juggling all of this with their personal and professional lives. Some companies don't understand this, so they think there's nothing wrong with a process that involves a take-home project, a debriefing phone call, a face-to-face conversation to test for cultural fit and a full-day on-site interview round. Most companies don't even know that their hiring process sucks. Of those who eventually realize something is wrong, many end up cargo-culting whatever they read or heard about "more successful" cases. As things are right now, the whole situation is eerily reminiscent of the fuss and hype the industry went through with software development processes. I wouldn't be surprised to see yet another wave of parasitic consultants pop up to prey on companies by flogging their hiring process improvements and best practices. ---------- [1] This is where a lot of people's complaints go wrong. The most frequent grumble is: why does this company's interview process focus on algorithms and data structures, if you'll just end up coding some services in Java? If the company is doing things right, they're using these topics as a tool to observe and measure whatever it is they're looking for; most often, it's the candidate's understanding of computational complexity. Just because you won't be writing your own implementations of common data structures and algorithms, it doesn't mean you shouldn't know how to pick the one that won't blow up your memory or execution time. [2] I had a case where I was interviewing with a company that gave me a print-out overview of their benefits, which included the price of each health plan, but not what it covered; when I inquired about it, they replied they wouldn't be able to give me that information until I signed the New Hire NDA.

Clubber 8y ago

>False positives are always a big problem. False negatives are usually much smaller one. I would agree with you if the supply and demand weren't so skewed towards top developers. Do you think you are choosing between A and D people, or are you really choosing between C and D people and A and B people never walk through the door?

andrewvc 8y ago
It's still the case! At the end of the day, the person making the call is a hiring manager. It's pretty hard for a hiring manager to be called out as too selective. However, if you let in a few bad apples then you look really bad. The incentives are for hiring managers to be too careful. It is, oddly enough, the case that if you are not hiring any bad engineers that you probably are being too selective.
- Clubber 8y ago
  It's easy to fix a false positive, it's impossible to fix a false negative. You figure FOMO would kick in at some point.
  - cornellwright 8y ago
    Once you hire someone you have to give them a chance to be successful. As the threshold for firing someone is (rightly) much higher than simply not hiring them, fixing a false positive is much more expensive than missing out on a good candidate. Also you can fix some false negatives - allow people to interview again for other positions at the company. I've seen multiple people get hired on the 2nd or 3rd time around and ultimately be successful.
    - scarface74 8y ago
      Simple - contract to perm.
      - gaius 8y ago

bradleyjg 8y ago

It may be understandable, but the prevailing attitude towards type I vs type II errors in tech industry hiring certainly makes me less sympathetic to claims that “we can’t find anyone to hire” and therefore governments should enact policies X,Y,&Z. If they are choosers they ought not to go begging.