6 ms·

I hate take-home projects more than I hate whiteboard interviews. I think the best approach would be giving the candidate a broken app and have they debug it l

I hate take-home projects more than I hate whiteboard interviews. I think the best approach would be giving the candidate a broken app and have they debug it live or in a take-home debug-and-fix or debug-and-improve exercise. Could also have some refactor assignment. Has any company tried something like this before?

Ozzie_osman 6y ago
Yes. I worked at Quora and we had a "practical interview" that was exactly this (it wasn't take-home though). https://www.quora.com/What-is-it-like-having-a-practical-interview-at-Quora?ch=10&share=25fcacf6&srid=OYIQ https://www.quora.com/What-is-it-like-having-a-practical-int...
sethhochberg 6y ago
We do this at String and Key (and I adopted the process from my prior teams at Digitally Imported. I've posted jobs from both here on HN in the past). Its a one-hour hands-on programing lab via screenshare - a pretty straightforward task of consuming some data from an API, processing it a little, and displaying it. We use Ruby for our work, and ask the same of our candidates in the lab. No handwritten syntax, exotic/tricky algorithms, or gotchas. Just a task to see how a candidate approaches a problem, deals with requirements, and implements them in our language of choice. Happy to go into more detail if anyone is trying to implement something similar in their own hiring processes.
- leonidasv 6y ago
  I'm interested in hearing more. I'm currently in the process of improving my interview approaches — currently a live coding session with a real-world problem involving an internal dataset.
  - sethhochberg 6y ago
    It sounds like you're on the right track. I think the important parts are: 1) It should be a live exercise (at least mostly live). I'm less interested in the end result than how someone got there, and most people chose to narrate their thought process as they work through the exercise. 2) The candidate should be able to work in their own environment. Online assessment tools can be different enough to introduce friction in a timed interview scenario. I'm interested in seeing how fluid a candidate is with the tools they'll use every day. 3) It should have the same expectations as your day-to-day work. Did the editor have a linter configured, and did they use it? Were tests written for the parts of the lab that introduced nontrivial logic? Is there a README? Is it all one git commit, or did they habitually make commits as they worked? 4) The design of the lab should have a decision point - I'm currently using a lab which could be implemented successfully with either a relational database or some kind of cache store. The strongest candidates recognize there are multiple paths and can justify why they chose the one they did. This works best for reasonably experienced devs who can actually be expected to create something from scratch from a set of requirements on a short time frame. For more junior people, I'd typically give a version of this lab that I completed myself and introduced a couple of bugs into. The task is to fix the bugs.
toadi 6y ago
We are a remote company. I give take home projects too. Mostly I give them couple of days but mention they shouldn't work more then 4 hours on it. Our test is actually quite simple and everyone that did any kind of development should have done these basic things. But it tells me a log of the current state of the candidate. Most code samples on internet are not production ready code. Security and error handling are mostly not included. So this is where a candidate can shine... but I'm happy if you can string it together working even if it is by googling.
- leonidasv 6y ago
  > they shouldn't work more then 4 hours on it So you're saying spending 3 and a half hours on the assignment is fine. It's not. Take-home assignments have little to no value for the candidate, they're only feasible if you are jobless or in college. The last thing I want to do in my spare time is to stay focusing on a bogus problem that may end up discarded.
  - cutemonster 6y ago
    What would you like to do instead? Talking on-site or on the phone? Is that better spent time
jb775 6y ago
Take-home projects are frustrating when they basically just say "Build an app that does $xyz", and we'll be in touch if we like it. Obviously no one is going to submit an app that doesn't accomplish the primary task. I feel like you get judged more-so on how you decided to get the initial app up and running, which likely isn't something you'll need to do working there.
- koreth1 6y ago
  > Obviously no one is going to submit an app that doesn't accomplish the primary task. I would have guessed the same thing before my company started doing homework interviews. But in fact, the ones that don't get to the finish line are sometimes the most interesting ones to review, and they aren't an automatic no-hire. (And we say so explicitly, if a bit more diplomatically, in our instructions.) If someone hands us clean, well-thought-out code and it's clear that they honored our time limit and just didn't quite get there before the clock ran out, we'll still talk to them.
underwater 6y ago
I don't understand how take home interviews are being held up as a better solution. They have two major flaws: They lower the cost for the company to zero. So they can widen their pool, and take a chance on more candidates, with no extra cost incurred for them. If you take this to the extreme – every company asking every candidate to complete a take home – then the result is far worse for candidates. There is no feedback mechanism for candidates during the process. I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable. On occasions where we had a system for capturing progress, I found that most candidates far exceeded the time recommended to them. Some were spending up to six hours. The frustrating thing was the extra time never made a difference to the outcome. I could tell from their initial hour whether they were going to pass. As a hiring manager I've used them, but work hard to be considerate to the candidate. I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.
- WalterBright 6y ago
  At Caltech, most exams were take home with a time limit of 2 hours. Some were "infinite time". The students hated the latter, because it means there was no end to the time you needed to spend on the exam, meanwhile there were other exams, etc., to take. With a 2 hour exam, there's an end to it, and you know how much time it should take to do the problems. Much better. It's even worse if you're being graded on a curve, as you would be in a job interview. You're competing against others who will spend infinite time on it.
  - azangru 6y ago
    Like you said, "there were other exams, etc., to take.". Presumably other competitors have other exams as well? You'll thus be competing against people with different time management skills.
- koreth1 6y ago
  My company does take-home interviews (or rather, we give candidates the option to do them; nearly all candidates choose take-home instead of a live coding session) and I don't think I agree with the "lowers the cost to zero" point. First of all, they are never the starting point; we only give a take-home problem to a candidate after we've had an initial discussion with them. When we get a take-home submission, we essentially run it through our code review process, and as one of the engineers who does reviews, it usually takes me nearly as long to thoroughly examine a homework solution and write up my feedback for internal discussion as it would to do an in-person interview. Being able to review a candidate's code on my own schedule rather than being yanked from the middle of some knotty problem to go do an interview at a fixed time is a big quality-of-life win for me. But the rate limit isn't much different than it would be with regular interviews. All that said, it's totally possible our process is super atypical.
Aqua_Geek 6y ago
Apple did this in one of the sessions last time I interviewed there. They handed me a project in Xcode that had a number of bugs with varying degrees of difficulty — some caused crashes, others were UI glitches, etc. The assignment was to identify the problems and fix them. If you have more than a couple years’ experience the issues were pretty straightforward. I really enjoyed this session, but I can see how it could be stressful to others.
- barrenko 6y ago
  This sounds really cool.
- twic 6y ago
  We used to do this at an old job. We were a consultancy which specialised in a particular e-commerce framework, and were hiring people with experience in it. So we set up a small project using this framework, and mildly wrecked it. Made some really basic mistakes, made some more subtle mistakes, added a bunch of dead code, wrote good tests for some of the broken stuff, wrote broken tests for some of the good stuff, etc. Candidates came in, and we asked them to add some simple feature to the app, which we warned them may be broken. We had someone sort-of-pair with them as they work, to watch what they do and keep them on the rails to an extent. We then evaluated whether what the candidate did was sensible: did they run the tests, did they pay attention to the failures, did they find and fix the mistakes, did they get distracted by the dead code, etc. Candidates didn't have to do everything perfectly, but there is plenty of scope to go really wrong. I remember one case where the very first thing the candidate did was delete all the tests. Thanks for coming in, we'll let you know! The company did quite a lot of rescue missions, where we would pick up a project that another consultancy had started and failed to deliver, and get it live. This interview wasn't just a trick, it was testing everyday skills!
collyw 6y ago
Take homes are too time consuming.
zerr 6y ago
Paid (premium rate) take home tasks is the way to go, after the initial screening.
bhntr3 6y ago
> I think the best approach would be giving the candidate a broken app and have they debug it live I had a company do this and it went really poorly for me. It was a python app and at the time I wasn't very expert in python. They told me to use my laptop and all I had for python was a vim environment. I knew enough python to do white-boarding problems. But debugging a complex app exercises a whole different set of skills I just hadn't developed in the language. I was also learning vim on the side. I looked like an incompetent amateur. So there's a language and tooling issue. The bigger problem is that debugging an unfamiliar code base under time pressure is almost as niche as doing algorithm problems under time pressure. I've been on enough company wide on calls that I actually think I'm pretty good at debugging unfamiliar code under time pressure. But if I have someone who understands the code looking over my shoulder, they're usually at least trying to help. It's still time pressure. It's still a somewhat contrived problem and code base. The interviewer is still comfortably omniscient. And it introduces some more complexity due to unfamiliarity of the language, tooling, code base or problem space. Ultimately, I'm not sure it's any better than white-boarding. A lot of these problems are solved with a take home question but then all the problems with take homes rise in their place.
- deleted 6y ago
  [deleted]

Ozzie_osman 6y ago

Yes. I worked at Quora and we had a "practical interview" that was exactly this (it wasn't take-home though). https://www.quora.com/What-is-it-like-having-a-practical-interview-at-Quora?ch=10&share=25fcacf6&srid=OYIQ https://www.quora.com/What-is-it-like-having-a-practical-int...

sethhochberg 6y ago

We do this at String and Key (and I adopted the process from my prior teams at Digitally Imported. I've posted jobs from both here on HN in the past). Its a one-hour hands-on programing lab via screenshare - a pretty straightforward task of consuming some data from an API, processing it a little, and displaying it. We use Ruby for our work, and ask the same of our candidates in the lab. No handwritten syntax, exotic/tricky algorithms, or gotchas. Just a task to see how a candidate approaches a problem, deals with requirements, and implements them in our language of choice. Happy to go into more detail if anyone is trying to implement something similar in their own hiring processes.

leonidasv 6y ago
I'm interested in hearing more. I'm currently in the process of improving my interview approaches — currently a live coding session with a real-world problem involving an internal dataset.
- sethhochberg 6y ago
  It sounds like you're on the right track. I think the important parts are: 1) It should be a live exercise (at least mostly live). I'm less interested in the end result than how someone got there, and most people chose to narrate their thought process as they work through the exercise. 2) The candidate should be able to work in their own environment. Online assessment tools can be different enough to introduce friction in a timed interview scenario. I'm interested in seeing how fluid a candidate is with the tools they'll use every day. 3) It should have the same expectations as your day-to-day work. Did the editor have a linter configured, and did they use it? Were tests written for the parts of the lab that introduced nontrivial logic? Is there a README? Is it all one git commit, or did they habitually make commits as they worked? 4) The design of the lab should have a decision point - I'm currently using a lab which could be implemented successfully with either a relational database or some kind of cache store. The strongest candidates recognize there are multiple paths and can justify why they chose the one they did. This works best for reasonably experienced devs who can actually be expected to create something from scratch from a set of requirements on a short time frame. For more junior people, I'd typically give a version of this lab that I completed myself and introduced a couple of bugs into. The task is to fix the bugs.

toadi 6y ago

We are a remote company. I give take home projects too. Mostly I give them couple of days but mention they shouldn't work more then 4 hours on it. Our test is actually quite simple and everyone that did any kind of development should have done these basic things. But it tells me a log of the current state of the candidate. Most code samples on internet are not production ready code. Security and error handling are mostly not included. So this is where a candidate can shine... but I'm happy if you can string it together working even if it is by googling.

leonidasv 6y ago
> they shouldn't work more then 4 hours on it So you're saying spending 3 and a half hours on the assignment is fine. It's not. Take-home assignments have little to no value for the candidate, they're only feasible if you are jobless or in college. The last thing I want to do in my spare time is to stay focusing on a bogus problem that may end up discarded.
- cutemonster 6y ago
  What would you like to do instead? Talking on-site or on the phone? Is that better spent time

jb775 6y ago

Take-home projects are frustrating when they basically just say "Build an app that does $xyz", and we'll be in touch if we like it. Obviously no one is going to submit an app that doesn't accomplish the primary task. I feel like you get judged more-so on how you decided to get the initial app up and running, which likely isn't something you'll need to do working there.

koreth1 6y ago
> Obviously no one is going to submit an app that doesn't accomplish the primary task. I would have guessed the same thing before my company started doing homework interviews. But in fact, the ones that don't get to the finish line are sometimes the most interesting ones to review, and they aren't an automatic no-hire. (And we say so explicitly, if a bit more diplomatically, in our instructions.) If someone hands us clean, well-thought-out code and it's clear that they honored our time limit and just didn't quite get there before the clock ran out, we'll still talk to them.

underwater 6y ago

I don't understand how take home interviews are being held up as a better solution. They have two major flaws: They lower the cost for the company to zero. So they can widen their pool, and take a chance on more candidates, with no extra cost incurred for them. If you take this to the extreme – every company asking every candidate to complete a take home – then the result is far worse for candidates. There is no feedback mechanism for candidates during the process. I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable. On occasions where we had a system for capturing progress, I found that most candidates far exceeded the time recommended to them. Some were spending up to six hours. The frustrating thing was the extra time never made a difference to the outcome. I could tell from their initial hour whether they were going to pass. As a hiring manager I've used them, but work hard to be considerate to the candidate. I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

WalterBright 6y ago
At Caltech, most exams were take home with a time limit of 2 hours. Some were "infinite time". The students hated the latter, because it means there was no end to the time you needed to spend on the exam, meanwhile there were other exams, etc., to take. With a 2 hour exam, there's an end to it, and you know how much time it should take to do the problems. Much better. It's even worse if you're being graded on a curve, as you would be in a job interview. You're competing against others who will spend infinite time on it.
- azangru 6y ago
  Like you said, "there were other exams, etc., to take.". Presumably other competitors have other exams as well? You'll thus be competing against people with different time management skills.
koreth1 6y ago
My company does take-home interviews (or rather, we give candidates the option to do them; nearly all candidates choose take-home instead of a live coding session) and I don't think I agree with the "lowers the cost to zero" point. First of all, they are never the starting point; we only give a take-home problem to a candidate after we've had an initial discussion with them. When we get a take-home submission, we essentially run it through our code review process, and as one of the engineers who does reviews, it usually takes me nearly as long to thoroughly examine a homework solution and write up my feedback for internal discussion as it would to do an in-person interview. Being able to review a candidate's code on my own schedule rather than being yanked from the middle of some knotty problem to go do an interview at a fixed time is a big quality-of-life win for me. But the rate limit isn't much different than it would be with regular interviews. All that said, it's totally possible our process is super atypical.

Aqua_Geek 6y ago

Apple did this in one of the sessions last time I interviewed there. They handed me a project in Xcode that had a number of bugs with varying degrees of difficulty — some caused crashes, others were UI glitches, etc. The assignment was to identify the problems and fix them. If you have more than a couple years’ experience the issues were pretty straightforward. I really enjoyed this session, but I can see how it could be stressful to others.

barrenko 6y ago
This sounds really cool.
twic 6y ago
We used to do this at an old job. We were a consultancy which specialised in a particular e-commerce framework, and were hiring people with experience in it. So we set up a small project using this framework, and mildly wrecked it. Made some really basic mistakes, made some more subtle mistakes, added a bunch of dead code, wrote good tests for some of the broken stuff, wrote broken tests for some of the good stuff, etc. Candidates came in, and we asked them to add some simple feature to the app, which we warned them may be broken. We had someone sort-of-pair with them as they work, to watch what they do and keep them on the rails to an extent. We then evaluated whether what the candidate did was sensible: did they run the tests, did they pay attention to the failures, did they find and fix the mistakes, did they get distracted by the dead code, etc. Candidates didn't have to do everything perfectly, but there is plenty of scope to go really wrong. I remember one case where the very first thing the candidate did was delete all the tests. Thanks for coming in, we'll let you know! The company did quite a lot of rescue missions, where we would pick up a project that another consultancy had started and failed to deliver, and get it live. This interview wasn't just a trick, it was testing everyday skills!

collyw 6y ago

Take homes are too time consuming.

zerr 6y ago

Paid (premium rate) take home tasks is the way to go, after the initial screening.

bhntr3 6y ago

> I think the best approach would be giving the candidate a broken app and have they debug it live I had a company do this and it went really poorly for me. It was a python app and at the time I wasn't very expert in python. They told me to use my laptop and all I had for python was a vim environment. I knew enough python to do white-boarding problems. But debugging a complex app exercises a whole different set of skills I just hadn't developed in the language. I was also learning vim on the side. I looked like an incompetent amateur. So there's a language and tooling issue. The bigger problem is that debugging an unfamiliar code base under time pressure is almost as niche as doing algorithm problems under time pressure. I've been on enough company wide on calls that I actually think I'm pretty good at debugging unfamiliar code under time pressure. But if I have someone who understands the code looking over my shoulder, they're usually at least trying to help. It's still time pressure. It's still a somewhat contrived problem and code base. The interviewer is still comfortably omniscient. And it introduces some more complexity due to unfamiliarity of the language, tooling, code base or problem space. Ultimately, I'm not sure it's any better than white-boarding. A lot of these problems are solved with a take home question but then all the problems with take homes rise in their place.

deleted 6y ago
[deleted]