6 ms·

While there are a variety of ways you can conduct a programming interview, the goals typically are the same: Will the candidate be able to write working code i

While there are a variety of ways you can conduct a programming interview, the goals typically are the same: Will the candidate be able to write working code if they join the team? Can the candidate discuss code and problems with the people they'll be working with? Can the candidate reason about arbitrary problems and constraints? Is the candidate someone we would enjoy working with? All of these can be solved directly by asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. I'm doing it right now as a candidate and it's by far the best interview experience I've had. The others were all artificial. A lot of people have a problem with this model for one reason or another, but in terms of effectiveness it's probably at the top 1%.

rckclmbr 9y ago
This is 100% the "best" way to interview/hire a person. A small company who screens someone for basic competence, then does this is great. It does has its drawbacks though 1. Requires a significant time investment on your part 2. Isn't the most efficient -- if you have someone work a week on a project, and decide they're not a good fit, you just lost a week. 3. You can't really have multiple candidates for the same position coding at the same time. They would feel like they would be competing (and really, they would be) 4. For the reasons above, this would never work for a bigger company. By the time they are doing their week-long "trial", you should be pretty confident through screens anyway. Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?
- morgante 9y ago
  > Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site? I have. I've done plenty of trial weeks where the candidate didn't work out. They got paid for their time, but we didn't extend them further offers.
wjossey 9y ago
I also agree. As someone who has recently co-founded a startup, this is something my co-founder and I have spent a lot of discussing and exploring. My personal experience with this style of interview was that it really helped me to personally demonstrate my abilities. I was interviewing at a startup in San Francisco in 2012 who wanted me to do a quick version of "battleship" as part of their hiring exercise. I proposed, mid interview, "how about I help you solve a performance problem with your current system. What's your biggest issue right now?" My interviewer actually showed me, and we ended up solving the problem together and then deploying out the change. The funny thing about that experience was that when we were done improving their platform, the VP of engineering was told what I did, and then had the interviewer go back and do the battleship exercise anyway. They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". I did far more poorly with the battleship exercise than I did helping them on their platform.
- mousa 9y ago
  Can't really blame him, it might seem kind of arrogant to refuse to do the task they give and propose one yourself, although it is pretty impressive that you could do something useful on a codebase you haven't seen during the code interview.
- sillysaurus3 9y ago
  It's unfortunate, isn't it? We spend all this time constructing artificial tests when the real thing is all that matters.
- ThrowawayR2 9y ago
  > They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". If your employer were ever sued by a candidate for discrimination and it came to light that you had received different treatment than other candidates, some very uncomfortable questions regarding the reason would be asked in the courtroom. An unfortunate bit of fallout from the litigious environment we live in today, but it is what it is.
  - 9y ago
morgante 9y ago
I used to think that, but honestly after recently going through a job search I think it's untenable for most scenarios. As a candidate, I want to be able to interview with multiple companies simultaneously—if I had to do a trial week with each company, I'd never be able to get multiple parallel offers. There's also significant onboarding complexity which I hadn't factored in. One company had me do a trial period and it took me half a day to get productive—time which is a sunk cost for them (since they had to pay for it) and made me feel bad about my lack of productivity. In the same time period it took to onboard with their codebase/build tools, I could have easily completed a full interview loop with another company.
- sillysaurus3 9y ago
  True enough, but it sounds like the above model isn't for you. It's for everyone who is talented but doesn't have the option to go interview with multiple companies simultaneously. Most devs are in that position. I'd much rather commit to a single company that I want to work at and do this process to put me ahead of the other candidates. But that's not a decision that everyone feels the same about.
  - deleted 9y ago
    [deleted]
  - readittwice 9y ago
    I get your sentiment. But this is quite problematic if you already have a job. It's not so easy to take a week or month off on short notice. What happens if you don't get your dream job? You can't just quit your current job.
- TomK32 9y ago
  pair programming should cut the startup time.
  - morgante 9y ago
    It would also substantially increase the cost of this interviewing method.
baddox 9y ago
You can still simulate that within the more standard interview process. Pick a real bug or small task in your codebase and give that to every candidate. Obviously you'll want to complete the task for real in your codebase, so just keep an interview branch in your source control.
gaius 9y ago
They're compensated for their time, of course. Taking on paid work for another company, possibly a competitor, would put nearly anyone who did so in breach of their contract with their current employer, as well as complicating their tax situation. Not to mention the vacation time they would be burning on it. You will only be able to hire freelancers or people currently unemployed by this method. Do candidates set their rate, or is it just a token amount?
- sillysaurus3 9y ago
  In my case, I offered to do it for free. (It wasn't their idea to give me a task on the actual codebase.) Almost nobody would be fine with that, but I want to work there. The other option was to hope that none of the other candidates seemed like a better choice. It's good to lock in an opportunity. Perhaps it's unworkable in the general case. That would be unfortunate; it's the most effective type of interview.
- alkonaut 9y ago
  I think it would be prudent for the interviewer to just say that "of course we will compensate for your time because this takes a couple of hours". If you happen to have a contract that forbids you from doing that, just say "Can't take any comp, I'm happy to do it for free". I don't think the key is compensation, but at least not assuming the subject will work for free. If you are the interviewer in that situation, give them the t-shirt and mug.
  - ng12 9y ago
    Unless your company was exceptionally unique in some way I'd probably just pass instead. It's awfully one-sided: you're just sending me some code and I'm supposed to invest hours on it. At least with an onsite interview our priorities are aligned: neither of us want to waste more time than we have to.
    - alkonaut 9y ago
      Yeah we all have different taste in interviews, I'd much rather spend 2h working for free than writing som tree traversal or fib function on a whiteboard for ten minutes. Interviewing is hard.
pcmaffey 9y ago
Rather than working on a real issue, I'd recommend a made-up task/feature, modeled after something they might actually do. Just put it in a repo and share to all candidates. It shouldn't take more than half a day's work. Then the next round is all about the candidates presenting their work and explaining their thought process. It's simulated, but exemplary of real work. And easily repeatable. (Also no need for NDA, etc.) Just requires a thoughtful engineer to develop the task.
vitus 9y ago
> asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. At Google, we call these internships. Well, more or less -- there are restrictions regarding enrollment in a degree program, the work is for a fixed duration rather than for a fixed scope, etc. But the goal from the company's perspective is largely the same. I'd be surprised if internships at other companies in the sector don't serve the same purpose. (obligatory disclaimer about how these are my opinion's, not the company's, etc etc)
- walshemj 9y ago
  doesn't work for those not still at university and there are strict rules in the USA about the role of an "internship" its meant to educate the intern not be a source of cheap labour
  - vitus 9y ago
    I never said anything about internships being a source of cheap labor. Please don't put words in my mouth. While yes, an intern typically makes less than a full-time engineer, you have to realize that the intern reduces the productivity of the host engineer, who has to spend time training the intern, looking over the intern's work, and so forth. The real value in the internship program is that it essentially creates a extended interview process after which we can evaluate the individual (given firsthand experience of how they operate in a work environment), and the individual can evaluate the company.
Jabbles 9y ago
This is an interesting candidate-first approach, but I have some concerns: This requires that candidates are experienced at the language or framework you use. Are you not prepared to take someone with different experience? What about graduates? What if their ability with one framework does not transfer to another that you end up assigning them to? Not all bugfixes are equal. If you haven't solved it yourself before, how do you know how hard it will be? If you give candidates questions that vary in difficulty, how do you compensate for that? Is the ability to dive into a codebase and fix (presumably) trivial bugs really what you should be optimizing for? Is it actually better than abstract algorithmic questions for determining if the candidate [can] reason about arbitrary problems and constraints? How would you overcome your unconscious biases when answering the question Is the candidate someone we would enjoy working with?
- sillysaurus3 9y ago
  After I proposed the idea, they said they liked it and were willing to give it a try. They offered 1.) slight changes to the UX, 2.) add a payments feature, or 3.) some back-end bugs that need to be resolved. They asked which of them sounds most appealing as a test. This was perfect from my perspective, and after I signed the NDA I told them all of those sounded fine. I mentioned that fixing a bug is one of the best ways to get familiar with a new codebase, so I'd probably focus on those, but that I'd like to hear about the rest too. I can probably throw in some of the feature requests or UI tweaks. That's a bit more candidate-driven than might be expected, but it's really wonderful so far. I feel like I have autonomy, while at the same time they'll get the data about whether I'm effective in the field. The bias question is important, and I'm not quite sure the best way for us to collectively deal with it. All I know is that there seems to be far more bias in a traditional interview pipeline. One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?" Obviously the point of that is to see how you think through a problem. But such problems are useless. They're abstracted from what we as engineers do in the field, and the candidate knows that their entire fate rests on how well they answer that question. Yet those are the kind of metrics real companies are currently using.
  - 9y ago
tptacek 9y ago
Almost. Instead of doing this, take an old set of bugfixes from the tree, 1-3 of them, give the same bugfixes to every candidate, and come up with a standard rubric for judging the fixes. (If it's an open source project, synthesize 1-3 bugs, using the history of bugs on the project as a guidepost). This is a "work sample test", and it's the gold standard for judging technical qualification. Firms should rely on them to the exclusion of subjective interviews. Within reason, to a first approximation, every candidate gets the same problems (you may iterate over time). You can judge candidates apples-apples. You don't need to pay candidates for their time, because they're not doing work for you; you already fixed these bugs.
- sillysaurus3 9y ago
  Yeah, I'm a big fan of your approach. Unfortunately it seems quite difficult to get anyone to do it. Most companies have interview pipelines that are well-established and immutable, so only new companies are inclined to try it. And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria. It was a neat hack as a candidate to propose to the interviewing company that they do essentially the same thing as a work-sample test and hear them say yes. That only works if you propose they let you try your hand at doing their actual work. I wonder if it can be generalized?
  - tptacek 9y ago
    I think asking candidates to do "real" work is bad for a bunch of reasons: * It's intrinsically exploitative, even when compensated, because real work done under similar terms is paid far more highly than prorated salary (it also does weird things to salary negotiation, by extracting an up-front concession on rate). * It requires different candidates to work on different problems, which means you're not getting a reliable record of how candidates do on the interview so you can tune it over time. * Because the best candidates will generally refuse to work under these terms, most firms that do paid up-front work have a fast-path for elite candidates, which is an obvious process failure.
    - deleted 9y ago
johnvega 9y ago
I wonder where companies or hiring managers can post their job openings that supports this hiring process sometimes called Trial Employment or Job Audition for Software Engineers. Looks like a great way to get the best hire and also prevent a bad hire, although it can only work for a very limited number of companies and candidates.

rckclmbr 9y ago

This is 100% the "best" way to interview/hire a person. A small company who screens someone for basic competence, then does this is great. It does has its drawbacks though 1. Requires a significant time investment on your part 2. Isn't the most efficient -- if you have someone work a week on a project, and decide they're not a good fit, you just lost a week. 3. You can't really have multiple candidates for the same position coding at the same time. They would feel like they would be competing (and really, they would be) 4. For the reasons above, this would never work for a bigger company. By the time they are doing their week-long "trial", you should be pretty confident through screens anyway. Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?

morgante 9y ago
> Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site? I have. I've done plenty of trial weeks where the candidate didn't work out. They got paid for their time, but we didn't extend them further offers.

wjossey 9y ago

I also agree. As someone who has recently co-founded a startup, this is something my co-founder and I have spent a lot of discussing and exploring. My personal experience with this style of interview was that it really helped me to personally demonstrate my abilities. I was interviewing at a startup in San Francisco in 2012 who wanted me to do a quick version of "battleship" as part of their hiring exercise. I proposed, mid interview, "how about I help you solve a performance problem with your current system. What's your biggest issue right now?" My interviewer actually showed me, and we ended up solving the problem together and then deploying out the change. The funny thing about that experience was that when we were done improving their platform, the VP of engineering was told what I did, and then had the interviewer go back and do the battleship exercise anyway. They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". I did far more poorly with the battleship exercise than I did helping them on their platform.

mousa 9y ago
Can't really blame him, it might seem kind of arrogant to refuse to do the task they give and propose one yourself, although it is pretty impressive that you could do something useful on a codebase you haven't seen during the code interview.
sillysaurus3 9y ago
It's unfortunate, isn't it? We spend all this time constructing artificial tests when the real thing is all that matters.
ThrowawayR2 9y ago
> They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". If your employer were ever sued by a candidate for discrimination and it came to light that you had received different treatment than other candidates, some very uncomfortable questions regarding the reason would be asked in the courtroom. An unfortunate bit of fallout from the litigious environment we live in today, but it is what it is.
- 9y ago

morgante 9y ago

I used to think that, but honestly after recently going through a job search I think it's untenable for most scenarios. As a candidate, I want to be able to interview with multiple companies simultaneously—if I had to do a trial week with each company, I'd never be able to get multiple parallel offers. There's also significant onboarding complexity which I hadn't factored in. One company had me do a trial period and it took me half a day to get productive—time which is a sunk cost for them (since they had to pay for it) and made me feel bad about my lack of productivity. In the same time period it took to onboard with their codebase/build tools, I could have easily completed a full interview loop with another company.

sillysaurus3 9y ago
True enough, but it sounds like the above model isn't for you. It's for everyone who is talented but doesn't have the option to go interview with multiple companies simultaneously. Most devs are in that position. I'd much rather commit to a single company that I want to work at and do this process to put me ahead of the other candidates. But that's not a decision that everyone feels the same about.
- deleted 9y ago
  [deleted]
- readittwice 9y ago
  I get your sentiment. But this is quite problematic if you already have a job. It's not so easy to take a week or month off on short notice. What happens if you don't get your dream job? You can't just quit your current job.
TomK32 9y ago
pair programming should cut the startup time.
- morgante 9y ago
  It would also substantially increase the cost of this interviewing method.

baddox 9y ago

You can still simulate that within the more standard interview process. Pick a real bug or small task in your codebase and give that to every candidate. Obviously you'll want to complete the task for real in your codebase, so just keep an interview branch in your source control.

gaius 9y ago

They're compensated for their time, of course. Taking on paid work for another company, possibly a competitor, would put nearly anyone who did so in breach of their contract with their current employer, as well as complicating their tax situation. Not to mention the vacation time they would be burning on it. You will only be able to hire freelancers or people currently unemployed by this method. Do candidates set their rate, or is it just a token amount?

sillysaurus3 9y ago
In my case, I offered to do it for free. (It wasn't their idea to give me a task on the actual codebase.) Almost nobody would be fine with that, but I want to work there. The other option was to hope that none of the other candidates seemed like a better choice. It's good to lock in an opportunity. Perhaps it's unworkable in the general case. That would be unfortunate; it's the most effective type of interview.
alkonaut 9y ago
I think it would be prudent for the interviewer to just say that "of course we will compensate for your time because this takes a couple of hours". If you happen to have a contract that forbids you from doing that, just say "Can't take any comp, I'm happy to do it for free". I don't think the key is compensation, but at least not assuming the subject will work for free. If you are the interviewer in that situation, give them the t-shirt and mug.
- ng12 9y ago
  Unless your company was exceptionally unique in some way I'd probably just pass instead. It's awfully one-sided: you're just sending me some code and I'm supposed to invest hours on it. At least with an onsite interview our priorities are aligned: neither of us want to waste more time than we have to.
  - alkonaut 9y ago
    Yeah we all have different taste in interviews, I'd much rather spend 2h working for free than writing som tree traversal or fib function on a whiteboard for ten minutes. Interviewing is hard.

pcmaffey 9y ago

Rather than working on a real issue, I'd recommend a made-up task/feature, modeled after something they might actually do. Just put it in a repo and share to all candidates. It shouldn't take more than half a day's work. Then the next round is all about the candidates presenting their work and explaining their thought process. It's simulated, but exemplary of real work. And easily repeatable. (Also no need for NDA, etc.) Just requires a thoughtful engineer to develop the task.

vitus 9y ago

> asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. At Google, we call these internships. Well, more or less -- there are restrictions regarding enrollment in a degree program, the work is for a fixed duration rather than for a fixed scope, etc. But the goal from the company's perspective is largely the same. I'd be surprised if internships at other companies in the sector don't serve the same purpose. (obligatory disclaimer about how these are my opinion's, not the company's, etc etc)

walshemj 9y ago
doesn't work for those not still at university and there are strict rules in the USA about the role of an "internship" its meant to educate the intern not be a source of cheap labour
- vitus 9y ago
  I never said anything about internships being a source of cheap labor. Please don't put words in my mouth. While yes, an intern typically makes less than a full-time engineer, you have to realize that the intern reduces the productivity of the host engineer, who has to spend time training the intern, looking over the intern's work, and so forth. The real value in the internship program is that it essentially creates a extended interview process after which we can evaluate the individual (given firsthand experience of how they operate in a work environment), and the individual can evaluate the company.

Jabbles 9y ago

This is an interesting candidate-first approach, but I have some concerns: This requires that candidates are experienced at the language or framework you use. Are you not prepared to take someone with different experience? What about graduates? What if their ability with one framework does not transfer to another that you end up assigning them to? Not all bugfixes are equal. If you haven't solved it yourself before, how do you know how hard it will be? If you give candidates questions that vary in difficulty, how do you compensate for that? Is the ability to dive into a codebase and fix (presumably) trivial bugs really what you should be optimizing for? Is it actually better than abstract algorithmic questions for determining if the candidate [can] reason about arbitrary problems and constraints? How would you overcome your unconscious biases when answering the question Is the candidate someone we would enjoy working with?

sillysaurus3 9y ago
After I proposed the idea, they said they liked it and were willing to give it a try. They offered 1.) slight changes to the UX, 2.) add a payments feature, or 3.) some back-end bugs that need to be resolved. They asked which of them sounds most appealing as a test. This was perfect from my perspective, and after I signed the NDA I told them all of those sounded fine. I mentioned that fixing a bug is one of the best ways to get familiar with a new codebase, so I'd probably focus on those, but that I'd like to hear about the rest too. I can probably throw in some of the feature requests or UI tweaks. That's a bit more candidate-driven than might be expected, but it's really wonderful so far. I feel like I have autonomy, while at the same time they'll get the data about whether I'm effective in the field. The bias question is important, and I'm not quite sure the best way for us to collectively deal with it. All I know is that there seems to be far more bias in a traditional interview pipeline. One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?" Obviously the point of that is to see how you think through a problem. But such problems are useless. They're abstracted from what we as engineers do in the field, and the candidate knows that their entire fate rests on how well they answer that question. Yet those are the kind of metrics real companies are currently using.
- 9y ago

tptacek 9y ago

Almost. Instead of doing this, take an old set of bugfixes from the tree, 1-3 of them, give the same bugfixes to every candidate, and come up with a standard rubric for judging the fixes. (If it's an open source project, synthesize 1-3 bugs, using the history of bugs on the project as a guidepost). This is a "work sample test", and it's the gold standard for judging technical qualification. Firms should rely on them to the exclusion of subjective interviews. Within reason, to a first approximation, every candidate gets the same problems (you may iterate over time). You can judge candidates apples-apples. You don't need to pay candidates for their time, because they're not doing work for you; you already fixed these bugs.

sillysaurus3 9y ago
Yeah, I'm a big fan of your approach. Unfortunately it seems quite difficult to get anyone to do it. Most companies have interview pipelines that are well-established and immutable, so only new companies are inclined to try it. And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria. It was a neat hack as a candidate to propose to the interviewing company that they do essentially the same thing as a work-sample test and hear them say yes. That only works if you propose they let you try your hand at doing their actual work. I wonder if it can be generalized?
- tptacek 9y ago
  I think asking candidates to do "real" work is bad for a bunch of reasons: * It's intrinsically exploitative, even when compensated, because real work done under similar terms is paid far more highly than prorated salary (it also does weird things to salary negotiation, by extracting an up-front concession on rate). * It requires different candidates to work on different problems, which means you're not getting a reliable record of how candidates do on the interview so you can tune it over time. * Because the best candidates will generally refuse to work under these terms, most firms that do paid up-front work have a fast-path for elite candidates, which is an obvious process failure.
  - deleted 9y ago

johnvega 9y ago

I wonder where companies or hiring managers can post their job openings that supports this hiring process sometimes called Trial Employment or Job Audition for Software Engineers. Looks like a great way to get the best hire and also prevent a bad hire, although it can only work for a very limited number of companies and candidates.