11 ms·

It does always amaze me that literally what you learn in HR 101 in an undergraduate management curriculum is controversial. The r^2 of a ton of different method

It does always amaze me that literally what you learn in HR 101 in an undergraduate management curriculum is controversial. The r^2 of a ton of different methods for predicting job performance is something large companies are highly incentivized to get studied by academics (and they do). Spoiler alert: work-sample tests are the only practical and generally tolerated one to use (for software engineers) that is actually correlated with job performance. @tptacek: how do you feel about people who have large amounts of open source work that is easily reviewed? This obviously doesn’t add new data to your company’s test but is definitely some kind of work sample, albeit not necessarily in a similar environment to the one being hired for.

notjesse 8y ago
But it’s not generally tolerated. You end up optimizing for sub-prime candidates because those are the only ones desperate enough to take 4-6 hours out of their free time for a company that hasn’t even bothered interviewing you, yet. If they want to turn the onsite into one big work sample, by all means, that sounds very effective (and something I’ve seen work well). But in my experience, you’re going to deter qualified candidates by forcing them to do take-home assignments.
- Aeolun 8y ago
  Hah, I had a sample test that I had to do after first being met on-site, which I thought was great. Then they said my work sample was amazing, and they’d like to do an on-site Q&A about it, but when I arrived the engineer hadn’t even seen my work, and proceeded to just quiz me on obscure JS trivia. Which I promptly failed. They then rejected me even though they were happy about my work :/
  - droithomme 8y ago
    In addition to covering all interview expenses such as travel food and lodging, candidates need to be paid for their time interviewing by the companies interviewing them. Six hour take home tests is fine, but I want $1350 for that in advance as a consultation fee, and if I hit 6 hrs and it's not done yet they can keep paying until I am done or we can just end it, no refunds. The idea that I should spend six hours doing free programming for random companies that say they are desperate to find anyone qualified is absolutely absurd and insulting. No one should put up with that, particularly anyone with an established and verifiable career.
    - mancerayder 8y ago
      It's even worse when you're a consultant. At the moment I'm faced with scheduling my fourth round with a company, and of course it's 5 hours onsite. I have the option of skipping the free lunch, they said. It's nice of them, but my monetary loss for that time (including commute) is enough to pay lunch for a number of people. All that, and they can still can you a few weeks or months into the job if you're a dud. It's employment, not marriage, guys, lighten up with your strenuous and time-draining processes. We senior employees aren't as, is the word excitable, as the entry-levels about joining your workforce.
      - ams6110 8y ago
        That does sound excessive. I've never had more than 2 rounds of interview, and even at that the first round was typically over the phone and the second on-site.
    - kayoone 8y ago
      why are you even interviewing for these "random companies desperate to find anyone qualified" if you are so disgusted by them? The elitism of some engineers is mind boggling, instead of being grateful working in an industry that has so much demand that you can easily find a job at anytime, you complain about the process being insulting to your oh-so-important persona. If i really want to work for a certain company because what they are doing excites me, then yes, i'd do a 6 hour take home test where I can probably also learn a thing or two and I am willing to bet a lot of other developers would too. As an interviewer, someone charging >$1000 for a take home test would be an immediate red flag and our mindsets probably don't match up. If this works fine for you, kudos.
      - mancerayder 8y ago
        It's good to see a full gamut of opinions on here. I'll say that mentalities like yours are why I'm a contractor in the first place -- you should be grateful for the opportunity, etc and so forth. No. I exchange my time for your money. If that's entitlement to you, we come from opposite worlds. I'm not the guy who says he should be compensated for take-home tests above. I am however a senior engineer. I'm spurning any long interview processes because they cost money. I don't think it's entitlement, but if that's what you want to call it, so be it. It's called hours worked, hours paid, and in the West it's been a concept since at least the 18th century.
  - mancerayder 8y ago
    That sounds like disorganization and poor internal communication, two traits of the organization probably not just evident in their evaluation of you. You may have dodged a bullet.
- nicoburns 8y ago
  The best interview I ever had was a 2-hour onsite work sample, followed by 1/2 an hour discussing what I'd come up with. I was offered the job the next day. Surely most people would prefer this to whiteboard tasks?
  - dleslie 8y ago
    The best interview I had was a few hours of friendly conversation about the details of my resume. Then I was hired under probation, as everyone there was, and the understanding was that I could be easily dismissed if it was clear that I wasn't working out.
    - jay_kyburz 8y ago
      Yeah, I think 1-3 month probation is really the only way to do it.
    - projektir 8y ago
      This would make me damn uncomfortable and stressed out once I started the job, and I'd likely take any other offer over this.
      - dleslie 8y ago
        You've had this condition at every job thus far. The real interview is always the work you do.
        joshuamorton 8y ago
        This is only true in the most extreme case. Many companies have formal procedures around firing people for performance issues. Short of hr violations or literally refusing to do anything, I can't imagine someone being fired before 6 months.
        dleslie 8y ago
        Probationary periods are industry standard here in British Columbia.
- ryanworl 8y ago
  I don’t have an opinion on whether or not it is done synchronously or asynchronously and I hope I didn’t imply work-sample = take home test.
  - notjesse 8y ago
    Gotcha, yes that makes more sense. I find a lot of unicorns are already doing this sort of interview, anyway. While I have never observed studies, it seems to provide better signals than white-boarding.
- tptacek 8y ago
  Again: this is true of "work sample" tests in their mainstream implementation in our industry (spend 6 hours jumping through a hoop for the privilege of running a standard, nondeterministic interview gauntlet), and those hiring processes are a scourge. But there's a right way to do it: give work sample challenges and then, at least for the most part, end the technical qualification part of your process there. You spend 4-6 hours at home instead of spending 4-6 hours in front of a whiteboard doing dumb coding challenges. If I was looking for a job right now, I'd probably refuse to interview anywhere that gave me take-home problems that couldn't promise that a strong result on those problems would take me all the way through technical qualification. But a company that could offer me take-home problems and conclusively make the technical part of the decision on whether to hire me based on those problem would be a strong prospect.
  - bradleyjg 8y ago
    You ever see or suspect fraud in a take home? I can't imagine it doesn't happen giving that cheating is so prevalent in colleges.
    - tptacek 8y ago
      If I was doing this at Google, I would spend a lot of time thinking about test fraud. At a 40-50 person company? Not so much. We do simple follow-up things that raise the amount of effort you'd have to put in to fraudulently submitting work sample tests, and we know pretty exactly how we'd randomize our work sample tests to make it hard to cheat (at least, hard to cheat without doing something we'd be interested in anyways) --- but it's just not worth it right now. Work sample test fraud is one of those things that sounds like a huge deal on message boards, but when you game through what would be involved in doing it in real life, it makes very little sense.
    - wool_gather 8y ago
      To my mind, the simplest and most straightforward way to combat any fraud is also beneficial because it gives you even more information about the candidate: Talk to them about the code they wrote. Have a conversation, as if they were already your co-worker, with the exercise as the subject. Go through it, ask them -- non-adversarially -- why they did X, what they thought about requirement 2, how they could get better test coverage for Z. If you see something that seems to be a mistake, talk about it. If you see something awesome, discuss it. I can't imagine someone incompetent being able to bullshit their way through detailed discussion of code they were supposed to have written. And this provides valuable info about how this person thinks and communicates about the work you want them to do.
- LaserToy 8y ago
  Well, I actually willing to spend time on a problem, because it is interesting and I can do it in a fun way and learn something new. But I have 0 tolerance policy towards puzzle whiteboarding tests. It is total waste of time.
- seanmcdirmid 8y ago
  I’m really mixed about this. As a new immigrant, my wife had trouble finding work as a designer until she was given a take home work sample test. After that, the company that gave her the test hired her quickly and she has been doing well with them ever since. I’m currently doing the gauntlet thing myself, but none of the interviews I’ve done have asked any technical questions pertaining to the role that they actually want to hire me for. It’s all generic stuff that frankly I would have done better on 20 years ago.
zzzcpan 8y ago
> highly incentivized to get studied by academics (and they do) Why do you think they are incentivized to get studied? If anything, power dynamics in large hierarchical organizations keep away any studying done that can threaten the power. Sometimes studies do happen though, but large organization can never apply the results to anything on their scale. They are more worried where to even get such massive stream of professionals to hire.
- ryanworl 8y ago
  You’re getting too philosophical here for something that isn’t that complicated. This isn’t related to power dynamics. Asking someone if they can flip burgers is obviously not as effective as asking them to flip a burger and seeing if they can do it correctly. Getting employees who are unquestioning robots with no ambitions for power is a different interviewing issue.
  - blotter_paper 8y ago
    "Everything in the world is about sex except sex. Sex is about power." -Oscar Wilde
walshemj 8y ago
Hr isn't exactly known for rigorous scientific research and I don't recall it being hr that introduced this trend of hazing candidates. Also have you got any examples of this research that actually proves this correlation I haven't heard of any and I would have as I have spent decades semi involved in IR (industrial relations) in the UK
- zzzcpan 8y ago
  Hazing candidates is mostly negative reinforcement introduced to keep people from wanting to go to interviews and change jobs. It also selects for those willing to do a lot and put up with a lot for a corporation.
jonathankoren 8y ago
>It does always amaze me that literally what you learn in HR 101 in an undergraduate management curriculum is controversial. The r^2 of a ton of different methods for predicting job performance is something large companies are highly incentivized to get studied by academics (and they do). Large companies are not "highly incentivized" to optimize they're hiring process. Once a certain throughput is achieved, there's very little reason to revisit it, because if they revisit it, "they're lowering the bar", or not "optimizing for recall" or whatever. Mindlessly copying large companies isn't all that useful. Microsoft famously loved brain teasers. Can't get the fox, chicken, and grain to the other side of of the river? You must not be able to code either. Google, loved asking your SAT scores because, "obviously" someone that got some arbitrary score on a standardized test, a minimum of 6 years ago, certainly means something today. Large companies aren't immune from bullshit. In fact, they have the habit of metastasizing bullshit, because of "Well, X does it, so it must be good." X only does it, because they had a stupid idea, became successful because of completely unrelated means, and then fooled themselves into thinking their process was good, "Well, I've been hitting candidates with ball-pein hammer for years, and I'm successful, so screw you." Interestingly enough, eventually both Microsoft and Google abandoned these interview questions, because eventually, they realized that one had nothing to do with the other, but only after years doing it, and others copying them.
- ryanworl 8y ago
  Microsoft and Google do not collectively hire that many people and are not who I am talking about. “Brain-teaser” type questions are explicitly not work-sample tests and, yes, have no correlation with job performance.
  - jonathankoren 8y ago
    The number of people that they collectively hire is irrelevant. They have an outsized influence on interview practices industry wide.
  - Consultant32452 8y ago
    IQ does correlate with work performance, and SAT tests are IQ tests. Brain teasers are an attempt at seeing how well/quickly your brain works to solve problems, is a rudimentary IQ test. The problem is that requiring IQ tests for employment introduces liability that employers do not want.

notjesse 8y ago

But it’s not generally tolerated. You end up optimizing for sub-prime candidates because those are the only ones desperate enough to take 4-6 hours out of their free time for a company that hasn’t even bothered interviewing you, yet. If they want to turn the onsite into one big work sample, by all means, that sounds very effective (and something I’ve seen work well). But in my experience, you’re going to deter qualified candidates by forcing them to do take-home assignments.

Aeolun 8y ago
Hah, I had a sample test that I had to do after first being met on-site, which I thought was great. Then they said my work sample was amazing, and they’d like to do an on-site Q&A about it, but when I arrived the engineer hadn’t even seen my work, and proceeded to just quiz me on obscure JS trivia. Which I promptly failed. They then rejected me even though they were happy about my work :/
- droithomme 8y ago
  In addition to covering all interview expenses such as travel food and lodging, candidates need to be paid for their time interviewing by the companies interviewing them. Six hour take home tests is fine, but I want $1350 for that in advance as a consultation fee, and if I hit 6 hrs and it's not done yet they can keep paying until I am done or we can just end it, no refunds. The idea that I should spend six hours doing free programming for random companies that say they are desperate to find anyone qualified is absolutely absurd and insulting. No one should put up with that, particularly anyone with an established and verifiable career.
  - mancerayder 8y ago
    It's even worse when you're a consultant. At the moment I'm faced with scheduling my fourth round with a company, and of course it's 5 hours onsite. I have the option of skipping the free lunch, they said. It's nice of them, but my monetary loss for that time (including commute) is enough to pay lunch for a number of people. All that, and they can still can you a few weeks or months into the job if you're a dud. It's employment, not marriage, guys, lighten up with your strenuous and time-draining processes. We senior employees aren't as, is the word excitable, as the entry-levels about joining your workforce.
    - ams6110 8y ago
      That does sound excessive. I've never had more than 2 rounds of interview, and even at that the first round was typically over the phone and the second on-site.
  - kayoone 8y ago
    why are you even interviewing for these "random companies desperate to find anyone qualified" if you are so disgusted by them? The elitism of some engineers is mind boggling, instead of being grateful working in an industry that has so much demand that you can easily find a job at anytime, you complain about the process being insulting to your oh-so-important persona. If i really want to work for a certain company because what they are doing excites me, then yes, i'd do a 6 hour take home test where I can probably also learn a thing or two and I am willing to bet a lot of other developers would too. As an interviewer, someone charging >$1000 for a take home test would be an immediate red flag and our mindsets probably don't match up. If this works fine for you, kudos.
    - mancerayder 8y ago
      It's good to see a full gamut of opinions on here. I'll say that mentalities like yours are why I'm a contractor in the first place -- you should be grateful for the opportunity, etc and so forth. No. I exchange my time for your money. If that's entitlement to you, we come from opposite worlds. I'm not the guy who says he should be compensated for take-home tests above. I am however a senior engineer. I'm spurning any long interview processes because they cost money. I don't think it's entitlement, but if that's what you want to call it, so be it. It's called hours worked, hours paid, and in the West it's been a concept since at least the 18th century.
- mancerayder 8y ago
  That sounds like disorganization and poor internal communication, two traits of the organization probably not just evident in their evaluation of you. You may have dodged a bullet.
nicoburns 8y ago
The best interview I ever had was a 2-hour onsite work sample, followed by 1/2 an hour discussing what I'd come up with. I was offered the job the next day. Surely most people would prefer this to whiteboard tasks?
- dleslie 8y ago
  The best interview I had was a few hours of friendly conversation about the details of my resume. Then I was hired under probation, as everyone there was, and the understanding was that I could be easily dismissed if it was clear that I wasn't working out.
  - jay_kyburz 8y ago
    Yeah, I think 1-3 month probation is really the only way to do it.
  - projektir 8y ago
    This would make me damn uncomfortable and stressed out once I started the job, and I'd likely take any other offer over this.
    - dleslie 8y ago
      You've had this condition at every job thus far. The real interview is always the work you do.
      - joshuamorton 8y ago
        This is only true in the most extreme case. Many companies have formal procedures around firing people for performance issues. Short of hr violations or literally refusing to do anything, I can't imagine someone being fired before 6 months.
        dleslie 8y ago
        Probationary periods are industry standard here in British Columbia.
ryanworl 8y ago
I don’t have an opinion on whether or not it is done synchronously or asynchronously and I hope I didn’t imply work-sample = take home test.
- notjesse 8y ago
  Gotcha, yes that makes more sense. I find a lot of unicorns are already doing this sort of interview, anyway. While I have never observed studies, it seems to provide better signals than white-boarding.
tptacek 8y ago
Again: this is true of "work sample" tests in their mainstream implementation in our industry (spend 6 hours jumping through a hoop for the privilege of running a standard, nondeterministic interview gauntlet), and those hiring processes are a scourge. But there's a right way to do it: give work sample challenges and then, at least for the most part, end the technical qualification part of your process there. You spend 4-6 hours at home instead of spending 4-6 hours in front of a whiteboard doing dumb coding challenges. If I was looking for a job right now, I'd probably refuse to interview anywhere that gave me take-home problems that couldn't promise that a strong result on those problems would take me all the way through technical qualification. But a company that could offer me take-home problems and conclusively make the technical part of the decision on whether to hire me based on those problem would be a strong prospect.
- bradleyjg 8y ago
  You ever see or suspect fraud in a take home? I can't imagine it doesn't happen giving that cheating is so prevalent in colleges.
  - tptacek 8y ago
    If I was doing this at Google, I would spend a lot of time thinking about test fraud. At a 40-50 person company? Not so much. We do simple follow-up things that raise the amount of effort you'd have to put in to fraudulently submitting work sample tests, and we know pretty exactly how we'd randomize our work sample tests to make it hard to cheat (at least, hard to cheat without doing something we'd be interested in anyways) --- but it's just not worth it right now. Work sample test fraud is one of those things that sounds like a huge deal on message boards, but when you game through what would be involved in doing it in real life, it makes very little sense.
  - wool_gather 8y ago
    To my mind, the simplest and most straightforward way to combat any fraud is also beneficial because it gives you even more information about the candidate: Talk to them about the code they wrote. Have a conversation, as if they were already your co-worker, with the exercise as the subject. Go through it, ask them -- non-adversarially -- why they did X, what they thought about requirement 2, how they could get better test coverage for Z. If you see something that seems to be a mistake, talk about it. If you see something awesome, discuss it. I can't imagine someone incompetent being able to bullshit their way through detailed discussion of code they were supposed to have written. And this provides valuable info about how this person thinks and communicates about the work you want them to do.
LaserToy 8y ago
Well, I actually willing to spend time on a problem, because it is interesting and I can do it in a fun way and learn something new. But I have 0 tolerance policy towards puzzle whiteboarding tests. It is total waste of time.
seanmcdirmid 8y ago
I’m really mixed about this. As a new immigrant, my wife had trouble finding work as a designer until she was given a take home work sample test. After that, the company that gave her the test hired her quickly and she has been doing well with them ever since. I’m currently doing the gauntlet thing myself, but none of the interviews I’ve done have asked any technical questions pertaining to the role that they actually want to hire me for. It’s all generic stuff that frankly I would have done better on 20 years ago.

zzzcpan 8y ago

> highly incentivized to get studied by academics (and they do) Why do you think they are incentivized to get studied? If anything, power dynamics in large hierarchical organizations keep away any studying done that can threaten the power. Sometimes studies do happen though, but large organization can never apply the results to anything on their scale. They are more worried where to even get such massive stream of professionals to hire.

ryanworl 8y ago
You’re getting too philosophical here for something that isn’t that complicated. This isn’t related to power dynamics. Asking someone if they can flip burgers is obviously not as effective as asking them to flip a burger and seeing if they can do it correctly. Getting employees who are unquestioning robots with no ambitions for power is a different interviewing issue.
- blotter_paper 8y ago
  "Everything in the world is about sex except sex. Sex is about power." -Oscar Wilde

walshemj 8y ago

Hr isn't exactly known for rigorous scientific research and I don't recall it being hr that introduced this trend of hazing candidates. Also have you got any examples of this research that actually proves this correlation I haven't heard of any and I would have as I have spent decades semi involved in IR (industrial relations) in the UK

zzzcpan 8y ago
Hazing candidates is mostly negative reinforcement introduced to keep people from wanting to go to interviews and change jobs. It also selects for those willing to do a lot and put up with a lot for a corporation.

jonathankoren 8y ago

>It does always amaze me that literally what you learn in HR 101 in an undergraduate management curriculum is controversial. The r^2 of a ton of different methods for predicting job performance is something large companies are highly incentivized to get studied by academics (and they do). Large companies are not "highly incentivized" to optimize they're hiring process. Once a certain throughput is achieved, there's very little reason to revisit it, because if they revisit it, "they're lowering the bar", or not "optimizing for recall" or whatever. Mindlessly copying large companies isn't all that useful. Microsoft famously loved brain teasers. Can't get the fox, chicken, and grain to the other side of of the river? You must not be able to code either. Google, loved asking your SAT scores because, "obviously" someone that got some arbitrary score on a standardized test, a minimum of 6 years ago, certainly means something today. Large companies aren't immune from bullshit. In fact, they have the habit of metastasizing bullshit, because of "Well, X does it, so it must be good." X only does it, because they had a stupid idea, became successful because of completely unrelated means, and then fooled themselves into thinking their process was good, "Well, I've been hitting candidates with ball-pein hammer for years, and I'm successful, so screw you." Interestingly enough, eventually both Microsoft and Google abandoned these interview questions, because eventually, they realized that one had nothing to do with the other, but only after years doing it, and others copying them.

ryanworl 8y ago
Microsoft and Google do not collectively hire that many people and are not who I am talking about. “Brain-teaser” type questions are explicitly not work-sample tests and, yes, have no correlation with job performance.
- jonathankoren 8y ago
  The number of people that they collectively hire is irrelevant. They have an outsized influence on interview practices industry wide.
- Consultant32452 8y ago
  IQ does correlate with work performance, and SAT tests are IQ tests. Brain teasers are an attempt at seeing how well/quickly your brain works to solve problems, is a rudimentary IQ test. The problem is that requiring IQ tests for employment introduces liability that employers do not want.