5 ms·

The gold standard in hiring qualification is work-sample testing. It works fine. You do not need to "make hiring a profit center" or "provisionally hire" or do

by tptacek 5mo ago

The gold standard in hiring qualification is work-sample testing. It works fine. You do not need to "make hiring a profit center" or "provisionally hire" or do internships. Work samples done correctly demand less time from candidates than interviews and scale better than interviews. They are standardizable and iterable. What I feel like I'm reading here is someone who has been poisoned by FAANG hiring practices --- and they are terrible --- and has missed most of the work that's been done (outside of Google's admirable work in debunking their own processes). I appreciate the "kitchen confidential" here, but with respect to Yegge, I think he's been working at the Olive Garden this whole time. Go stage at Gramercy Tavern! They're working at a different scale, yes, but you'll at least get a different perspective on the "gold standard".

enraged_camel 5mo ago
The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold: a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc. b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes. Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.
- tptacek 5mo ago
  A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.
  - littlexsparkee 5mo ago
    I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
    - tptacek 5mo ago
      We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
    - esafak 5mo ago
      As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
      - tptacek 5mo ago
        I've been hearing these kinds of things since 2014 (when I wrote a long post about the work sample process we had used at Matasano). I've been hiring continuously since 2008, so 18 years, and in that entire time I have never come close to hiring a scammer. It might be a more salient concern now, in the era of AI agents, and we are much warier today than I was at Matasano, but generally I think this risk is more talked about than experienced.
        dilyevsky 5mo ago
        Ive had people show up at onsites that were clearly not the same person I screened over the phone earlier as far back as 2013 at google. It’s real. Just ask any recruiters from well known companies and they’ll tell you. It’s happening to more and more startups too now because it’s gotten easier
- dpark 5mo ago
  I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop. Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.
  - tptacek 5mo ago
    Adding take-home problems to a traditional 4-6 hour interview loop is odious. But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
    - dpark 5mo ago
      This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment. With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
      - tptacek 5mo ago
        We don't ask or check how long candidates take. You're a professional, we give you a challenge, you can decide (up front, 30 minutes in, whatever) whether you're likely to be able to finish in the budgeted time. Maybe you can't because you've got a lunch date and don't have the contiguous block, and want to do it in chunks. Fine by us. Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.
- chollida1 5mo ago
  We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far. If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.
- mplanchard 5mo ago
  And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs. There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo. Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?
- NoMoreNicksLeft 5mo ago
  >The candidate gets 3-6 months of stable employment To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience. If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.
kevin_nisbet 5mo ago
I'm a little bit more on the fence with the work sample interviews having designed them and also interviewed through them. I've also done my fair share of "traditional" tech interviews, all at startups, never FAANG. As an interviewer, I much prefer the signals generated through a work-sample interview. I'm much more confident in the hiring recommendation than I get from a 1 hour zoom session. However, if I look at teams that were built through the work-sample and zoom interviews, I'm not sure the outcomes were that noticeably better. As an interviewee, I think I understand the frustration being on the other side has. With an in-person interview, I often have a good sense that I bombed the interview or something to improve on or replay in my head, less surprising outcomes. On the work samples it's harder to know whether you're making mistakes, or are being out-competed by someone putting in 4 times the effort to polish the solution beyond what their regular work product would be. Although I had one really good outcome where the work-sample interview really flagged the internal dysfunction of a company. And then with both interview processes, I still think there is a really big unknown on what the false no-hire rate is, how much effort is getting wasted rejecting candidates that would actually fit the team. So having to choose a process as an interviewer, I'm with you and would always choose a work-sample interview. On whether it should be considered the "gold standard", I'm much more hesitant, I think there are some limitations that are still hard to control for. I do wish Starfighter/Stockfighter model had gained more traction, would've been interesting to see a recruiting company specialize in this and then seeding the interview results to multiple companies model work out.
- sethammons 5mo ago
  Work sample interviews don't have to be take home. We ran our technical interviews as close to work samples as possible and in person.
- jghn 5mo ago
  > I'm not sure the outcomes were that noticeably better. It's not just you. At the end of the day interviewing has been demonstrated to be close to a crapshoot in the best of circumstances, and very few interview schemes are the best of circumstances. Work samples are part of the optimal strategy [1] but even then the signal is quite low. [1] https://psycnet.apa.org/record/1998-10661-006 https://psycnet.apa.org/record/1998-10661-006
brennanpeterson 5mo ago
It would be nice to have portfolios, but systems are broken enough that that becomes hard to see. I suspect one of the reasons for the bias to hiring PHds in fields where it really isn't necessary is at least you have a work portfolio.
- tptacek 5mo ago
  Why do I care about your portfolio if I can just see how effectively you do the actual work?
dilyevsky 5mo ago
Ime take-homes was a good way to purge all the good candidates with options from your pipeline as they would simply not turn anything in and go with someone else. Maybe with AI and so many layoffs it’s better now I haven’t tried in last couple of years though i doubt it
- tptacek 5mo ago
  People have been telling me that for 18 years and (a) we've hired our fair share of candidates with gold-plated resumes (and stopped hiring candidates that only had gold-plated resumes, acquired from failing up) and (b) the nontraditional background candidates we've hired have generally been our top performers. Hiring in most of the tech industry is gatekept, and like every protectionist scheme, it isn't organized for productivity.
- dalyons 5mo ago
  Take homes are much lower signal now with AI
  - dilyevsky 5mo ago
    Depends on what you ask, i suppose. I’m sure i can come up with something that can’t simply be one shot or the result would be bad if you do. A bit more difficult on logistics tho as you’d have to arrange for an environment with some prepaid llm access
    - HenriTEL 5mo ago
      Without a way to estimate "AI power" used for the task I don't see how you can fairly rate home assignments.
      - dilyevsky 5mo ago
        Like i said either use pre-baked env or give a candidate an auth token with something like $100-200 quota from the provider your company already uses

by tptacek 5mo ago

enraged_camel 5mo ago

The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold: a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc. b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes. Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.

tptacek 5mo ago
A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.
- littlexsparkee 5mo ago
  I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
  - tptacek 5mo ago
    We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
  - esafak 5mo ago
    As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
    - tptacek 5mo ago
      I've been hearing these kinds of things since 2014 (when I wrote a long post about the work sample process we had used at Matasano). I've been hiring continuously since 2008, so 18 years, and in that entire time I have never come close to hiring a scammer. It might be a more salient concern now, in the era of AI agents, and we are much warier today than I was at Matasano, but generally I think this risk is more talked about than experienced.
      - dilyevsky 5mo ago
        Ive had people show up at onsites that were clearly not the same person I screened over the phone earlier as far back as 2013 at google. It’s real. Just ask any recruiters from well known companies and they’ll tell you. It’s happening to more and more startups too now because it’s gotten easier
dpark 5mo ago
I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop. Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.
- tptacek 5mo ago
  Adding take-home problems to a traditional 4-6 hour interview loop is odious. But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
  - dpark 5mo ago
    This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment. With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
    - tptacek 5mo ago
      We don't ask or check how long candidates take. You're a professional, we give you a challenge, you can decide (up front, 30 minutes in, whatever) whether you're likely to be able to finish in the budgeted time. Maybe you can't because you've got a lunch date and don't have the contiguous block, and want to do it in chunks. Fine by us. Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.
chollida1 5mo ago
We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far. If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.
mplanchard 5mo ago
And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs. There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo. Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?
NoMoreNicksLeft 5mo ago
>The candidate gets 3-6 months of stable employment To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience. If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.

kevin_nisbet 5mo ago

I'm a little bit more on the fence with the work sample interviews having designed them and also interviewed through them. I've also done my fair share of "traditional" tech interviews, all at startups, never FAANG. As an interviewer, I much prefer the signals generated through a work-sample interview. I'm much more confident in the hiring recommendation than I get from a 1 hour zoom session. However, if I look at teams that were built through the work-sample and zoom interviews, I'm not sure the outcomes were that noticeably better. As an interviewee, I think I understand the frustration being on the other side has. With an in-person interview, I often have a good sense that I bombed the interview or something to improve on or replay in my head, less surprising outcomes. On the work samples it's harder to know whether you're making mistakes, or are being out-competed by someone putting in 4 times the effort to polish the solution beyond what their regular work product would be. Although I had one really good outcome where the work-sample interview really flagged the internal dysfunction of a company. And then with both interview processes, I still think there is a really big unknown on what the false no-hire rate is, how much effort is getting wasted rejecting candidates that would actually fit the team. So having to choose a process as an interviewer, I'm with you and would always choose a work-sample interview. On whether it should be considered the "gold standard", I'm much more hesitant, I think there are some limitations that are still hard to control for. I do wish Starfighter/Stockfighter model had gained more traction, would've been interesting to see a recruiting company specialize in this and then seeding the interview results to multiple companies model work out.

sethammons 5mo ago
Work sample interviews don't have to be take home. We ran our technical interviews as close to work samples as possible and in person.
jghn 5mo ago
> I'm not sure the outcomes were that noticeably better. It's not just you. At the end of the day interviewing has been demonstrated to be close to a crapshoot in the best of circumstances, and very few interview schemes are the best of circumstances. Work samples are part of the optimal strategy [1] but even then the signal is quite low. [1] https://psycnet.apa.org/record/1998-10661-006 https://psycnet.apa.org/record/1998-10661-006

brennanpeterson 5mo ago

It would be nice to have portfolios, but systems are broken enough that that becomes hard to see. I suspect one of the reasons for the bias to hiring PHds in fields where it really isn't necessary is at least you have a work portfolio.

tptacek 5mo ago
Why do I care about your portfolio if I can just see how effectively you do the actual work?

dilyevsky 5mo ago

Ime take-homes was a good way to purge all the good candidates with options from your pipeline as they would simply not turn anything in and go with someone else. Maybe with AI and so many layoffs it’s better now I haven’t tried in last couple of years though i doubt it

tptacek 5mo ago
People have been telling me that for 18 years and (a) we've hired our fair share of candidates with gold-plated resumes (and stopped hiring candidates that only had gold-plated resumes, acquired from failing up) and (b) the nontraditional background candidates we've hired have generally been our top performers. Hiring in most of the tech industry is gatekept, and like every protectionist scheme, it isn't organized for productivity.
dalyons 5mo ago
Take homes are much lower signal now with AI
- dilyevsky 5mo ago
  Depends on what you ask, i suppose. I’m sure i can come up with something that can’t simply be one shot or the result would be bad if you do. A bit more difficult on logistics tho as you’d have to arrange for an environment with some prepaid llm access
  - HenriTEL 5mo ago
    Without a way to estimate "AI power" used for the task I don't see how you can fairly rate home assignments.
    - dilyevsky 5mo ago
      Like i said either use pre-baked env or give a candidate an auth token with something like $100-200 quota from the provider your company already uses