6 ms·

The last time I was hiring I gave out a take-home test, and I thought it was the opposite of an imposition on candidates' time. I'd be curious to hear your thou

by hansvm 7mo ago

The last time I was hiring I gave out a take-home test, and I thought it was the opposite of an imposition on candidates' time. I'd be curious to hear your thoughts: - It was designed to be fast to complete (20min max -- not a huge imposition if being hired is likely, obviously very expensive if you're taking one for every job posting). - I only gave them out after a resume screen. If you had a 0% chance then I didn't waste your time. If you had enough other proof of abilities then I skipped the take-home. - Candidates were told that it was designed to be fast and that if they couldn't complete it quickly they were unlikely to be successful interviewing either. They still had the option to spend a lot of time if they thought my assessment of the situation was wrong, but part of the point was to allow candidates to gauge their own abilities and not waste their time interviewing without a chance of being hired. - I did a lot of work behind the scenes calibrating and re-writing the questions individually and as a whole so that the test score correlated very well with interview performance (most interviews administered by not-me, removing a form of bias that's easy to creep in there).

crobertsbmw 7mo ago
That seems reasonable. Some employers will hand out 3-6 hour assignments after a candidate’s resumes make it through an AI screening.
heavyset_go 7mo ago
For every "20 min max" take home assignment, there will be people who are willing to spend 4+ hours doing it to outshine candidates who have jobs, families and lives. If you want to make it more of a fair consideration of time, consider moving your take home to interviews, that way there isn't a time cost asymmetry. You can enforce your "20 min max" claim this way, you can judge a candidate's performance, thought process and filter out anyone who is LLMing or spending inordinate amounts of time on them. You will also make a better impression on candidates by investing your time in them in the same way they are with you. Maybe you're hiring kids out of college without experience, but you only have to do so many take home tests before you realize that they're a waste of time, and pass on potential employers who throw them at you, or you learn to just send them your hourly rate for the test.
- parpfish 7mo ago
  One other way to keep things true to the “20 min max” is to have a clear objective/scoring rubric. Nothing open ended (data science jobs LOVE handing out open ended data analyses). I need to know that it’s okay to stop and that anything I’m doing would just be overkill.
- stbtrax 7mo ago
  you can put a time limit on it from when they start to submit. It's really the only way to solve high volume of unqualified applicants. So much time wasted talking to people who could barely code
  - Barbing 7mo ago
    Submit 30min after interview, “you have 20min” (remainder for bio break or whatever)?
- seer 7mo ago
  Live coding during an interview is one of the most oppressive things I’ve witnessed in the industry in general. There is usually a huge disconnect between someone who knows that “this task should take 20mins” and doing it cold in a super high-pressure environment. People sweat, panic, brain freeze, and are just plain out stressed. I’ll only OK something like this if we give out a similar but not the same task before the interview so a person can train a bit beforehand. I’ve heard it all justified as “we want to see how you perform under pressure” but to me that has always sounded super flimsy - like if this is representative of how work is done at this organisation, then do I want to work there in the first place? And if it isn’t, why the hell are you putting people through this ringer in the first place, just sounds inhumane.
  - ryandrake 7mo ago
    Yea, there's really no way to do an "interview assignment" well. If you give unlimited amount of time, you're giving an advantage to people with no life who can just focus on your assignment and polish it as if it were a full time job. If you give a limited amount of time, then you're making the interview a pressure cooker with a countdown clock, giving a disadvantage to people who are just not great at working under minute-to-minute time pressure.
komali2 7mo ago
> but part of the point was to allow candidates to gauge their own abilities and not waste their time interviewing without a chance of being hired. In my experience this is the wrong game theory. Unemployed people can make job hunting their full time job, so a 20 minute take home doesn't select for "who delivers the highest quality solution in the least amount of time," it selects for "who is the richest applicant who can burn hours on a take home to deliver a higher quality result than people with less time they can afford to spend?" Also, nobody should ever self-select themselves out of an interview process. Passing a resume review and getting a callback is about 10% likely: for every job hunt, in my experience , candidates get about 10 callbacks for every 100 resume sends. From there, it's about 20% chance to get to final stage, and from there, maybe 50% to get an offer (you're either their first choice or second; if second, your hiring hinges on whether the first choice accepts). Math is right there: once you pass a resume check, in terms of the volume of applications you've sent, it's optimal to spend far more effort into this gig than into firing off ten or twenty more resumes. Therefore, even if the candidate doesn't think they're a good fit, they should do everything they can to stay in the game, including lying by omission. After all they might be engaging in imposter syndrome, right? Why assume for the interviewer that your python skills aren't good enough - maybe the interviewer understands perfectly well that you've only used it for scripts and one off tools, but doesn't care because they personally believe your startup experience is more valuable to them and they believe you can up skill! Maybe the take home was designed poorly by someone who was tasked randomly by a lead to shit out a take home, and it's not an accurate indication of what the job would be like. Maybe they sent you the wrong take home? Maybe it's a good take home but you need money so fuck it, if you manage to sneak in despite not being a good fit, you can just bust ass to upskill and make up the difference before anyone notices. Or fuck it twice, it's a shit market and who knows how much longer you'll be able to sell your labor as an engineer, even if you can only fool them for two weeks, that's two weeks of income while you still keep up your job hunt.
- fc417fc802 7mo ago
  > After all they might be engaging in imposter syndrome, right? GP specifically stated that this was the point of the takehome though. If the person handing it out specifically warns you that struggling with it means you aren't a good fit then if you struggle with it that's not imposter syndrome - you aren't a good fit! Not dropping out at that point is just refusing to acknowledge reality and insisting on wasting everyone's time.
Mario9382 7mo ago
20 minutes max seems fair to me. For context I was once given a 1 week assignment just to be discarded without any feedback. From then on if it takes more than a day I won't do it.
manfre 7mo ago
Any take home test trivial enough to complete in under 20 minutes could be completed by an AI. The only signal you get from a take home test is whether or not they can submit answers. It doesn't let you know if the candidate is capable of passing the test unassisted. Take home tests were never a worthwhile signal. Pre-AI, people would search for solutions or have another person complete it.
- Esophagus4 7mo ago
  Major interview platforms all can identify AI usage, and record the interview for humans to review later. Employers can, with a somewhat high (?) degree of certainty, weed out candidates using AI.
- hansvm 7mo ago
  Cheating is possible in the abstract, but I found a tight correlation between interview and take-home performance. For whatever reason, candidates didn't seem to cheat much. The AI point is worth diving into a little. This was a year ago, so SOTA was worse, but I didn't find it terribly hard to write questions AI couldn't solve, whose answers you couldn't search for, and which good candidates could solve. The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section. I don't think that moat will exist indefinitely, but today's AI just isn't very good at a lot of incredibly basic tasks unless the operator has enough outside knowledge to guide it in the right direction (and if a candidate did that I mostly wouldn't care because, by definition, they had the knowledge I was looking for). I use AI a lot, it's great at a lot of things, some even quite complicated, but it was weaknesses, and those are pretty easy to exploit.
  - jonpurdy 7mo ago
    Your description of the test and your replies to questions indicate you've come up with a pretty great assessment for the role(s) you hire for. Especially where you mentioned: > The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section I also like how you allow/encourage self-assessment, where if a candidate can't do the test in ~20 minutes under zero pressure, they probably won't be a good fit in the role itself.
Esophagus4 7mo ago
I generally don’t do take home tests unless it’s really a job I would love to have. If you’re some random startup or no name company, I don’t bother. I already have a good job. If you’re a top name hot company offering $600k+ total comp, I’m going to spend the hour shooting my shot. Even if it’s lower likelihood. For random company XYZ, I expect humans to sink as much of their time into the process as I do.
nsxwolf 7mo ago
How many candidates did you send the take home to? You may have done a resume screen first, but it's unlikely your time commitment was equal. I've worked at places that would send out 100. People would spend their weekends working on it and we often wouldn't even look at the submissions.
- hansvm 7mo ago
  I sent out screens to ~15-25% of resumes (a higher rate for new grads, lower for seniors, not wanting to unnecessarily rule them out just because they didn't have positive evidence of potential success and didn't know how to write a resume, only ruling them out of there was positive evidence they'd be unsuccessful). That amounted to ~100 per position filled. Around half of those completed the take-home. Some of the rest should have self-selected out and didn't, which is something I'd like to improve if I run a take-home again. You're right the time commitment wasn't equal. Early on I spent much more time than the candidates designing and analyzing the test. Afterward, their 20 minutes would usually take me <5min (often <1min for obvious failures and obvious passes, the average brought up due to time analyzing edge cases). I did read every submission though. It wasn't wasted time for candidates.

by hansvm 7mo ago

crobertsbmw 7mo ago

That seems reasonable. Some employers will hand out 3-6 hour assignments after a candidate’s resumes make it through an AI screening.

heavyset_go 7mo ago

For every "20 min max" take home assignment, there will be people who are willing to spend 4+ hours doing it to outshine candidates who have jobs, families and lives. If you want to make it more of a fair consideration of time, consider moving your take home to interviews, that way there isn't a time cost asymmetry. You can enforce your "20 min max" claim this way, you can judge a candidate's performance, thought process and filter out anyone who is LLMing or spending inordinate amounts of time on them. You will also make a better impression on candidates by investing your time in them in the same way they are with you. Maybe you're hiring kids out of college without experience, but you only have to do so many take home tests before you realize that they're a waste of time, and pass on potential employers who throw them at you, or you learn to just send them your hourly rate for the test.

parpfish 7mo ago
One other way to keep things true to the “20 min max” is to have a clear objective/scoring rubric. Nothing open ended (data science jobs LOVE handing out open ended data analyses). I need to know that it’s okay to stop and that anything I’m doing would just be overkill.
stbtrax 7mo ago
you can put a time limit on it from when they start to submit. It's really the only way to solve high volume of unqualified applicants. So much time wasted talking to people who could barely code
- Barbing 7mo ago
  Submit 30min after interview, “you have 20min” (remainder for bio break or whatever)?
seer 7mo ago
Live coding during an interview is one of the most oppressive things I’ve witnessed in the industry in general. There is usually a huge disconnect between someone who knows that “this task should take 20mins” and doing it cold in a super high-pressure environment. People sweat, panic, brain freeze, and are just plain out stressed. I’ll only OK something like this if we give out a similar but not the same task before the interview so a person can train a bit beforehand. I’ve heard it all justified as “we want to see how you perform under pressure” but to me that has always sounded super flimsy - like if this is representative of how work is done at this organisation, then do I want to work there in the first place? And if it isn’t, why the hell are you putting people through this ringer in the first place, just sounds inhumane.
- ryandrake 7mo ago
  Yea, there's really no way to do an "interview assignment" well. If you give unlimited amount of time, you're giving an advantage to people with no life who can just focus on your assignment and polish it as if it were a full time job. If you give a limited amount of time, then you're making the interview a pressure cooker with a countdown clock, giving a disadvantage to people who are just not great at working under minute-to-minute time pressure.

komali2 7mo ago

> but part of the point was to allow candidates to gauge their own abilities and not waste their time interviewing without a chance of being hired. In my experience this is the wrong game theory. Unemployed people can make job hunting their full time job, so a 20 minute take home doesn't select for "who delivers the highest quality solution in the least amount of time," it selects for "who is the richest applicant who can burn hours on a take home to deliver a higher quality result than people with less time they can afford to spend?" Also, nobody should ever self-select themselves out of an interview process. Passing a resume review and getting a callback is about 10% likely: for every job hunt, in my experience , candidates get about 10 callbacks for every 100 resume sends. From there, it's about 20% chance to get to final stage, and from there, maybe 50% to get an offer (you're either their first choice or second; if second, your hiring hinges on whether the first choice accepts). Math is right there: once you pass a resume check, in terms of the volume of applications you've sent, it's optimal to spend far more effort into this gig than into firing off ten or twenty more resumes. Therefore, even if the candidate doesn't think they're a good fit, they should do everything they can to stay in the game, including lying by omission. After all they might be engaging in imposter syndrome, right? Why assume for the interviewer that your python skills aren't good enough - maybe the interviewer understands perfectly well that you've only used it for scripts and one off tools, but doesn't care because they personally believe your startup experience is more valuable to them and they believe you can up skill! Maybe the take home was designed poorly by someone who was tasked randomly by a lead to shit out a take home, and it's not an accurate indication of what the job would be like. Maybe they sent you the wrong take home? Maybe it's a good take home but you need money so fuck it, if you manage to sneak in despite not being a good fit, you can just bust ass to upskill and make up the difference before anyone notices. Or fuck it twice, it's a shit market and who knows how much longer you'll be able to sell your labor as an engineer, even if you can only fool them for two weeks, that's two weeks of income while you still keep up your job hunt.

fc417fc802 7mo ago
> After all they might be engaging in imposter syndrome, right? GP specifically stated that this was the point of the takehome though. If the person handing it out specifically warns you that struggling with it means you aren't a good fit then if you struggle with it that's not imposter syndrome - you aren't a good fit! Not dropping out at that point is just refusing to acknowledge reality and insisting on wasting everyone's time.

Mario9382 7mo ago

20 minutes max seems fair to me. For context I was once given a 1 week assignment just to be discarded without any feedback. From then on if it takes more than a day I won't do it.

manfre 7mo ago

Any take home test trivial enough to complete in under 20 minutes could be completed by an AI. The only signal you get from a take home test is whether or not they can submit answers. It doesn't let you know if the candidate is capable of passing the test unassisted. Take home tests were never a worthwhile signal. Pre-AI, people would search for solutions or have another person complete it.

Esophagus4 7mo ago
Major interview platforms all can identify AI usage, and record the interview for humans to review later. Employers can, with a somewhat high (?) degree of certainty, weed out candidates using AI.
hansvm 7mo ago
Cheating is possible in the abstract, but I found a tight correlation between interview and take-home performance. For whatever reason, candidates didn't seem to cheat much. The AI point is worth diving into a little. This was a year ago, so SOTA was worse, but I didn't find it terribly hard to write questions AI couldn't solve, whose answers you couldn't search for, and which good candidates could solve. The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section. I don't think that moat will exist indefinitely, but today's AI just isn't very good at a lot of incredibly basic tasks unless the operator has enough outside knowledge to guide it in the right direction (and if a candidate did that I mostly wouldn't care because, by definition, they had the knowledge I was looking for). I use AI a lot, it's great at a lot of things, some even quite complicated, but it was weaknesses, and those are pretty easy to exploit.
- jonpurdy 7mo ago
  Your description of the test and your replies to questions indicate you've come up with a pretty great assessment for the role(s) you hire for. Especially where you mentioned: > The test was a few of those questions and a few which were easier to cheat, and almost nobody had good scores on just the cheatable section I also like how you allow/encourage self-assessment, where if a candidate can't do the test in ~20 minutes under zero pressure, they probably won't be a good fit in the role itself.

Esophagus4 7mo ago

I generally don’t do take home tests unless it’s really a job I would love to have. If you’re some random startup or no name company, I don’t bother. I already have a good job. If you’re a top name hot company offering $600k+ total comp, I’m going to spend the hour shooting my shot. Even if it’s lower likelihood. For random company XYZ, I expect humans to sink as much of their time into the process as I do.

nsxwolf 7mo ago

How many candidates did you send the take home to? You may have done a resume screen first, but it's unlikely your time commitment was equal. I've worked at places that would send out 100. People would spend their weekends working on it and we often wouldn't even look at the submissions.

hansvm 7mo ago
I sent out screens to ~15-25% of resumes (a higher rate for new grads, lower for seniors, not wanting to unnecessarily rule them out just because they didn't have positive evidence of potential success and didn't know how to write a resume, only ruling them out of there was positive evidence they'd be unsuccessful). That amounted to ~100 per position filled. Around half of those completed the take-home. Some of the rest should have self-selected out and didn't, which is something I'd like to improve if I run a take-home again. You're right the time commitment wasn't equal. Early on I spent much more time than the candidates designing and analyzing the test. Afterward, their 20 minutes would usually take me <5min (often <1min for obvious failures and obvious passes, the average brought up due to time analyzing edge cases). I did read every submission though. It wasn't wasted time for candidates.