5 ms·

The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold: a) it discrimina

The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold: a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc. b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes. Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.

tptacek 5mo ago
A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.
- littlexsparkee 5mo ago
  I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
  - tptacek 5mo ago
    We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
  - esafak 5mo ago
    As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
    - tptacek 5mo ago
      I've been hearing these kinds of things since 2014 (when I wrote a long post about the work sample process we had used at Matasano). I've been hiring continuously since 2008, so 18 years, and in that entire time I have never come close to hiring a scammer. It might be a more salient concern now, in the era of AI agents, and we are much warier today than I was at Matasano, but generally I think this risk is more talked about than experienced.
      - dilyevsky 5mo ago
        Ive had people show up at onsites that were clearly not the same person I screened over the phone earlier as far back as 2013 at google. It’s real. Just ask any recruiters from well known companies and they’ll tell you. It’s happening to more and more startups too now because it’s gotten easier
        tptacek 5mo ago
        I believe you, but again: I haven't experienced anything like this, ever. One of the clearest results we got at Matasano adopting this approach was a drastic reduction in turnover, which is exactly the opposite of what you expect to see if people are scamming their way into jobs they can't do. That's held up (I don't have the before/after comparison, but I do have the industry-wide comparison) over the years afterwards. We hire these people, then we work with them, and they're amazing.
      - The_Fox 4mo ago
        I read that post back in the day and have remembered it ever since, and used it as a basis for the process at my current employment. Thank you! (For those interested: https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)
- enraged_camel 5mo ago
  >> A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. Yes, standard interview loops also discriminate, and the more time they take, the more discriminatory they are. Any on-site requirements compound the issues. Like Yegge says: provisional employment/internships solve all of these issues. You get the best of all worlds: stable employment for the candidate where they get paid a regular wage and aren't under a stressful interview setting, and lots and lots of work samples for you, the employer. It's not perfect. For example, it can be difficult to entrust the provisional employee/intern with anything impactful if you don't know whether they'll be employed at the end. But it is significantly better than the alternatives in most contexts.
  - tptacek 5mo ago
    Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. I concede that provisional hires are higher-signal than work sample testing (or rather: that they're the platonic ideal of work sample testing), but the entire problem of hiring qualification is to make decisions in the context of a candidate doing a job search.
    - enraged_camel 5mo ago
      >> Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. It works. I’ve seen it in two different places. At the second one, the fundamental realization I came to was that it is virtually no different than “regular” employment, where the new employee can get fired for not meeting expectations within an arbitrary time period after being hired. This can be months, or even weeks. From the perspective of the candidate, regular employment and provisional employment have roughly the same level of risk: in both cases they take a job where they might be let go at some point. The benefit of provisional employment is that they know how long they will be evaluated for and against whom. It turns out a lot of candidates do in fact like the all-cards-on-the-table approach and enjoy being given the opportunity to prove themselves on the job.
- throwburn202605 5mo ago
  The question is whether companies would tolerate their own process if their employees did that recruiting process at a different company. They obviously do to some extent via plausible deniability; I have a 1 hour "doctors" appointment this afternoon, or I'm taking leave on Monday. Using it as cover to attend an interview. Would this company permit an employee taking 3 months unpaid leave to provisional hire somewhere else and have free choice whether they stay or go at the end of it.
dpark 5mo ago
I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop. Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.
- tptacek 5mo ago
  Adding take-home problems to a traditional 4-6 hour interview loop is odious. But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
  - dpark 5mo ago
    This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment. With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
    - tptacek 5mo ago
      We don't ask or check how long candidates take. You're a professional, we give you a challenge, you can decide (up front, 30 minutes in, whatever) whether you're likely to be able to finish in the budgeted time. Maybe you can't because you've got a lunch date and don't have the contiguous block, and want to do it in chunks. Fine by us. Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.
chollida1 5mo ago
We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far. If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.
mplanchard 5mo ago
And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs. There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo. Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?
NoMoreNicksLeft 5mo ago
>The candidate gets 3-6 months of stable employment To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience. If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.

tptacek 5mo ago

A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.

littlexsparkee 5mo ago
I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
- tptacek 5mo ago
  We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
- esafak 5mo ago
  As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
  - tptacek 5mo ago
    I've been hearing these kinds of things since 2014 (when I wrote a long post about the work sample process we had used at Matasano). I've been hiring continuously since 2008, so 18 years, and in that entire time I have never come close to hiring a scammer. It might be a more salient concern now, in the era of AI agents, and we are much warier today than I was at Matasano, but generally I think this risk is more talked about than experienced.
    - dilyevsky 5mo ago
      Ive had people show up at onsites that were clearly not the same person I screened over the phone earlier as far back as 2013 at google. It’s real. Just ask any recruiters from well known companies and they’ll tell you. It’s happening to more and more startups too now because it’s gotten easier
      - tptacek 5mo ago
        I believe you, but again: I haven't experienced anything like this, ever. One of the clearest results we got at Matasano adopting this approach was a drastic reduction in turnover, which is exactly the opposite of what you expect to see if people are scamming their way into jobs they can't do. That's held up (I don't have the before/after comparison, but I do have the industry-wide comparison) over the years afterwards. We hire these people, then we work with them, and they're amazing.
    - The_Fox 4mo ago
      I read that post back in the day and have remembered it ever since, and used it as a basis for the process at my current employment. Thank you! (For those interested: https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)
enraged_camel 5mo ago
>> A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room. Yes, standard interview loops also discriminate, and the more time they take, the more discriminatory they are. Any on-site requirements compound the issues. Like Yegge says: provisional employment/internships solve all of these issues. You get the best of all worlds: stable employment for the candidate where they get paid a regular wage and aren't under a stressful interview setting, and lots and lots of work samples for you, the employer. It's not perfect. For example, it can be difficult to entrust the provisional employee/intern with anything impactful if you don't know whether they'll be employed at the end. But it is significantly better than the alternatives in most contexts.
- tptacek 5mo ago
  Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. I concede that provisional hires are higher-signal than work sample testing (or rather: that they're the platonic ideal of work sample testing), but the entire problem of hiring qualification is to make decisions in the context of a candidate doing a job search.
  - enraged_camel 5mo ago
    >> Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. It works. I’ve seen it in two different places. At the second one, the fundamental realization I came to was that it is virtually no different than “regular” employment, where the new employee can get fired for not meeting expectations within an arbitrary time period after being hired. This can be months, or even weeks. From the perspective of the candidate, regular employment and provisional employment have roughly the same level of risk: in both cases they take a job where they might be let go at some point. The benefit of provisional employment is that they know how long they will be evaluated for and against whom. It turns out a lot of candidates do in fact like the all-cards-on-the-table approach and enjoy being given the opportunity to prove themselves on the job.
throwburn202605 5mo ago
The question is whether companies would tolerate their own process if their employees did that recruiting process at a different company. They obviously do to some extent via plausible deniability; I have a 1 hour "doctors" appointment this afternoon, or I'm taking leave on Monday. Using it as cover to attend an interview. Would this company permit an employee taking 3 months unpaid leave to provisional hire somewhere else and have free choice whether they stay or go at the end of it.

dpark 5mo ago

I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop. Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.

tptacek 5mo ago
Adding take-home problems to a traditional 4-6 hour interview loop is odious. But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
- dpark 5mo ago
  This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment. With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
  - tptacek 5mo ago
    We don't ask or check how long candidates take. You're a professional, we give you a challenge, you can decide (up front, 30 minutes in, whatever) whether you're likely to be able to finish in the budgeted time. Maybe you can't because you've got a lunch date and don't have the contiguous block, and want to do it in chunks. Fine by us. Again, the underlying smuggled premise here is that candidates have to finish the work sample. No they don't. In fact, that's a strong sign it's not an effective work sample. A test that everybody passes isn't a real test; it's just a hazing ritual.

chollida1 5mo ago

We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far. If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.

mplanchard 5mo ago

And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs. There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo. Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?

NoMoreNicksLeft 5mo ago

>The candidate gets 3-6 months of stable employment To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience. If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.