6 ms·

From being on both sides of whiteboard and take home interviews over the last few years, I’m all for whiteboard interviews, for a few reasons: - take home inte

From being on both sides of whiteboard and take home interviews over the last few years, I’m all for whiteboard interviews, for a few reasons: - take home interviews incentivize the candidate to spend more time than they’re told. You can choose a three hour problem, but candidates that really want the job will probably spend more time. Less experienced candidates will spend more time on it too. - I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result. - whiteboarding can go badly too. The caveat is that if it does, only so much of the candidate’s time is taken up; there’s a hard ceiling. - from the perspective of conducting whiteboard interviews, the company loses out if the interviewer doesn’t do a good job. This is really on the company though - it’s worth investing in a good whiteboarding process. If there’s no investment here, it’s only the company that suffers (competent candidates will get hired by a company with a better process). In terms of conducting a good whiteboard interview, my $0.02 is no trick questions, just solid problem solving with plenty of scope to show creativity and proficiency.

colordrops 7y ago
Take home interviews should be time-limited and proctored real-time. If it's more than 4-6 hours then it's too long. The problem should have multiple parts and just difficult enough to make it unlikely for most candidates to complete everything. The problem should unique enough and broad enough to allow the candidate to demonstrate good code structure, problem solving, creativity, and algorithmic ability. It should obviously not be something so straightforward and simple as to be a question on stack overflow. The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction. Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure. Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency. I and my team used this technique for years to great success, hiring some of the world's top talent.
- ProZsolt 7y ago
  > just difficult enough to make it unlikely for most candidates to complete everything. How do you judge someone's half-finished work? Do you expect me to write half-finished code on the job?
  - rsanek 7y ago
    I assume they mean there are individually relatively self-contained pieces of functionality in the problem statement. Not completing an additional feature would still leave the rest of the project in a good state. Perhaps if you use a vcs you could even leave WIP code in an unmerged branch.
    - ProZsolt 7y ago
      My question still stands. How do you evaluate someone's half-finished work? Do you prefer if I finish more features or my already finished code is better quality or better tested? It's just an unnecessary source of stress during the test.
      - rsanek 7y ago
        It would probably be best if someone was available to answer these kinds of questions for the interviewee when their time period starts. Though I think it could also be interesting to have this be part of the assessment -- how does the candidate trade off between scope / quality when they are given a fixed time period?
        ProZsolt 7y ago
        It is as interesting as evaluating them based on how they indent their code. I can do both but if you don't tell me what you prefer how can I choose?
ninjapenguin54 7y ago
Re: time spent on take home Take homes can & ought to be timed. I've had the opportunity to do this fornat twice and I think it works well. Here's how it works: They ask you to pick a 3 hour time window to work on the project. You pick the time. They send you the project prompt in an email right at the beginning of your time interval. You have 3 hours to read the prompt, design, implement, test a solution. If you miss the deadline, project counts for nothing. Both the times I did this the prompts were absolutely doable in 3 hours. They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days. Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time. It forces a real time constraint that keeps you from piling time into something you just don't know. Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.
- eru 7y ago
  Yes. Though in practice I'd give people some grace period after the three hours: the three hour time limit is to help the candidate not commit endless hours to the project, it's not meant as a punishment.
  - ddek 7y ago
    I agree here - given the pressures of an interview are much higher than on the job, enforcing a time limit should only prevent extremes rather than add yet more pressure. I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
- amatecha 7y ago
  Yup I've done a couple interviews like that. They send the email at the time of your choosing and you have to submit the finished challenge at whatever deadline. Works for me! I can do things how I want to, just as if I had been given such a task on the job.
PZ81JUXJE7uJ 7y ago
I'm not looking gor a dev that knows it all by heart. I'm looking for a dev who knows how to get information about stuff he doesn't know. A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview. Also: Give your candidate a real problem to solve and pay them for their time!
- chrisbennet 7y ago
  There's a famous quote that is attributed to Albert Einstein: “Never memorise something that you can look up.” It is said to be related to a time when a colleague asked him for his phone number, and he reached for his telephone directory to look it up.
  - ProZsolt 7y ago
    At an interview, Albert Szent-Györgyi states basically the same: "You shouldn't fill your head with "szecska" (chaff?). You should keep knowledge in books where you can look it up and use your head only for thinking." The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian: https://youtu.be/rtqhA4TtBrI https://youtu.be/rtqhA4TtBrI
koonsolo 7y ago
> are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow Strange reasoning. If you want to cut corners, you use stack overflow. Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow. When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.
- sokoloff 7y ago
  IMO, using stackoverflow is closer to the opposite of cutting corners. No sense laboring to create something that is a 45 second Google search away.
  - avgDev 7y ago
    I agree with googling stuff but this gives me anxiety, even though I can create relational databases, use ORM/stored procedures/queries, write web apps, deploy to IIS/Docker. I still worry about looking for the next gig. I did write a complete mission critical solution for a mid-size business, used by 100+ employees but then I am asked about algorithms and I could answer 3 years ago but now I would get stuck on it. Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
- chrisbennet 7y ago
  There are two good reasons to look things up on the internet: 1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.) 2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.
bryanrasmussen 7y ago
> Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result. One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers). At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it) I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere) Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.
- sokoloff 7y ago
  I would say that while only 10-20% of programmers might be literally not worth paying any amount for, it might be as much as half of the field that isn’t worth the current, major US market compensation.
  - bryanrasmussen 7y ago
    once you're willing to admit that they are worth paying something for you're just arguing price, which is whatever the market says it is. either that or 25 dollars, same as in town.
- zimpenfish 7y ago
  > a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things. (Ok, it's often Perl people...) For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
ProZsolt 7y ago
> I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result. If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.

colordrops 7y ago

Take home interviews should be time-limited and proctored real-time. If it's more than 4-6 hours then it's too long. The problem should have multiple parts and just difficult enough to make it unlikely for most candidates to complete everything. The problem should unique enough and broad enough to allow the candidate to demonstrate good code structure, problem solving, creativity, and algorithmic ability. It should obviously not be something so straightforward and simple as to be a question on stack overflow. The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction. Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure. Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency. I and my team used this technique for years to great success, hiring some of the world's top talent.

ProZsolt 7y ago
> just difficult enough to make it unlikely for most candidates to complete everything. How do you judge someone's half-finished work? Do you expect me to write half-finished code on the job?
- rsanek 7y ago
  I assume they mean there are individually relatively self-contained pieces of functionality in the problem statement. Not completing an additional feature would still leave the rest of the project in a good state. Perhaps if you use a vcs you could even leave WIP code in an unmerged branch.
  - ProZsolt 7y ago
    My question still stands. How do you evaluate someone's half-finished work? Do you prefer if I finish more features or my already finished code is better quality or better tested? It's just an unnecessary source of stress during the test.
    - rsanek 7y ago
      It would probably be best if someone was available to answer these kinds of questions for the interviewee when their time period starts. Though I think it could also be interesting to have this be part of the assessment -- how does the candidate trade off between scope / quality when they are given a fixed time period?
      - ProZsolt 7y ago
        It is as interesting as evaluating them based on how they indent their code. I can do both but if you don't tell me what you prefer how can I choose?

ninjapenguin54 7y ago

Re: time spent on take home Take homes can & ought to be timed. I've had the opportunity to do this fornat twice and I think it works well. Here's how it works: They ask you to pick a 3 hour time window to work on the project. You pick the time. They send you the project prompt in an email right at the beginning of your time interval. You have 3 hours to read the prompt, design, implement, test a solution. If you miss the deadline, project counts for nothing. Both the times I did this the prompts were absolutely doable in 3 hours. They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days. Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time. It forces a real time constraint that keeps you from piling time into something you just don't know. Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.

eru 7y ago
Yes. Though in practice I'd give people some grace period after the three hours: the three hour time limit is to help the candidate not commit endless hours to the project, it's not meant as a punishment.
- ddek 7y ago
  I agree here - given the pressures of an interview are much higher than on the job, enforcing a time limit should only prevent extremes rather than add yet more pressure. I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
amatecha 7y ago
Yup I've done a couple interviews like that. They send the email at the time of your choosing and you have to submit the finished challenge at whatever deadline. Works for me! I can do things how I want to, just as if I had been given such a task on the job.

PZ81JUXJE7uJ 7y ago

I'm not looking gor a dev that knows it all by heart. I'm looking for a dev who knows how to get information about stuff he doesn't know. A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview. Also: Give your candidate a real problem to solve and pay them for their time!

chrisbennet 7y ago
There's a famous quote that is attributed to Albert Einstein: “Never memorise something that you can look up.” It is said to be related to a time when a colleague asked him for his phone number, and he reached for his telephone directory to look it up.
- ProZsolt 7y ago
  At an interview, Albert Szent-Györgyi states basically the same: "You shouldn't fill your head with "szecska" (chaff?). You should keep knowledge in books where you can look it up and use your head only for thinking." The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian: https://youtu.be/rtqhA4TtBrI https://youtu.be/rtqhA4TtBrI

koonsolo 7y ago

> are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow Strange reasoning. If you want to cut corners, you use stack overflow. Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow. When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.

sokoloff 7y ago
IMO, using stackoverflow is closer to the opposite of cutting corners. No sense laboring to create something that is a 45 second Google search away.
- avgDev 7y ago
  I agree with googling stuff but this gives me anxiety, even though I can create relational databases, use ORM/stored procedures/queries, write web apps, deploy to IIS/Docker. I still worry about looking for the next gig. I did write a complete mission critical solution for a mid-size business, used by 100+ employees but then I am asked about algorithms and I could answer 3 years ago but now I would get stuck on it. Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
chrisbennet 7y ago
There are two good reasons to look things up on the internet: 1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.) 2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.

bryanrasmussen 7y ago

> Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result. One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers). At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it) I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere) Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.

sokoloff 7y ago
I would say that while only 10-20% of programmers might be literally not worth paying any amount for, it might be as much as half of the field that isn’t worth the current, major US market compensation.
- bryanrasmussen 7y ago
  once you're willing to admit that they are worth paying something for you're just arguing price, which is whatever the market says it is. either that or 25 dollars, same as in town.
zimpenfish 7y ago
> a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things. (Ok, it's often Perl people...) For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.

ProZsolt 7y ago

> I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result. If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.