10 ms·

We're having a bit of a debate about this internally as the company I work for is struggling to hire good candidates. For a while we've had a take-home test and

We're having a bit of a debate about this internally as the company I work for is struggling to hire good candidates. For a while we've had a take-home test and that has increased our success rate somewhat. But as you say, we really don't want to lay unreasonable expectations on people who may have other obligations, but we've had many people in the past who've interviewed very well but turned out to be completely incompetent when assigned to a real project. Maybe we're just really bad at interviews? It probably doesn't help that management here is almost entirely non-technical. There's usually a developer or two sitting in on the interview to try and balance it out, but I don't think any of us would consider ourselves particularly good at interviewing people.

notsureaboutpg 6y ago
>For a while we've had a take-home test and that has increased our success rate somewhat. But as you say, we really don't want to lay unreasonable expectations on people who may have other obligations These are mutually exclusive things. Candidates who have more free time to code are more likely to be better at it than those who don't (all else equal). Candidates who have OSS maintenance / leadership experience are more likely to work well in teams (all else equal). If you choose not to weight those things, to balance the playing field for people who have more obligations outside of work, then you'll also have lesser quality candidates (again, generally). So if you're struggling to hire good candidates, maybe it's a good idea to weight these things (and bias against people with less free time outside work). Once you have good candidates, and this is no longer a pressure on your business, then it might be a good time to try and balance the playing field for new hires. But you cannot balance the playing field and also hire the best candidates. The best candidates will have unfair advantages in general. You can either lean towards having the best or lean towards balancing the playing field.
- sokoloff 6y ago
  We should try to eliminate irrelevant biases in the hiring process but we are fundamentally trying to select people to hire who will join our company and write good code. That quality is not evenly distributed in the population. Some of the gymnastics thinking that I see seems to suggest that we’d seek to hire fluent English speakers by interviewing evenly across all populations. By all means, I’d be more than happy to hire someone fluent in English from China, but if I’m looking for a fluent English speaker, I shouldn’t spent 18.5% of my recruiting efforts/budget in China out of "fairness".
  - potatoz2 6y ago
    People in China often don't speak fluent English because they grow up in a country where it's not a spoken language. People who don't contribute to OSS typically don't do it because they don't have the ability (some OSS code is held together by duct tape) but because they don't have time, interest or think it's harder than it is. If you have better tools to determine if someone is good at writing code for your company, why use a proxy measure that's different in many ways?
    - sokoloff 6y ago
      Unless I’ve worked with you or someone that I deeply trust has and will vouch for you, I don’t have any stronger signal of your coding and teamwork abilities than a strong OSS contribution history can provide. It’s rare for an interview process to provide more than 6 hours of content, some of which is non-evaluatable. Reference checks are all but useless compared to seeing actual work over time. About 1 time in 6, you’ll interview a full standard deviation above or below your “true” ability. About 1 time in 50, you’ll turn in a +2σ (or -2σ) performance. That’s largely eliminated with personal experience, a trusted vouch, or a strong OSS portfolio.
PragmaticPulp 6y ago
> we've had many people in the past who've interviewed very well but turned out to be completely incompetent when assigned to a real project. Unpopular opinion on HN: This is actually quite common when you hire based purely on resumes or credentials. Some people are really good at interviewing and being charismatic enough to convince people to hire them. There are a lot of candidates who can talk the talk but really just want a job where they can browse Reddit and Twitter all day while writing a couple lines of code every once in a while. There are a lot of companies that are big enough for these people to blend in for years. Take home tests in the range of 1-4 hours shouldn’t be an undue burden on any applicants, if you give sufficient time to return the test. Many candidates wouldn’t bat an eye at taking a day to interview on site, so asking them to spend a couple hours of their free time on an interview isn’t really a disproportionate ask. Giving someone a 20-40 hour take home project would be ridiculous, of course, but reasonably sized problems are perfectly reasonable.
- ofrzeta 6y ago
  We spent some time on making up a quite simple project with a readme about the tasks to be implemented. It's something that can be done in half an hour when you are fast and we always thought we need to make it a bit harder. Turns out it's good enough to weed out quite many people who can't keep a deadline or don't get their shit together in other ways. From reviewing the code you can judge how people think, if they keep their files and code in order, use git etc. We also prompt people do document their process in finding a solution.
  - portoal 6y ago
    Interesting, could u share a bit more pls?
    - ofrzeta 6y ago
      We made up a problem with technologies that we use at work targeted at frontend engineers. So we set up a project with a Symfony backend and made the candidates create Twig templates and SASS styling based on Bootstrap. Then we put that up on Github with a README. So the candidates could focus on the frontend stuff without caring for the backend. Still they had to figure out how to set up SASS compilation and modularization of Twig templates. As I said it's not very hard but you can get bonus points if you come up with considerations for responsive design or stuff like that. Overall the aim is to create readable idiomatic code and not come up with something clever that no one but you can understand. I guess that's a very specific task that's of no great use to others but the point is to have a specific problem with some blanks that can be filled out with a reasonable amount of work by the candidates. Plus the general workflow using git and such.
HeyLaughingBoy 6y ago
I don't know how receptive your management would be to this, or what it costs, but at my last job, all interviewers were required to take interview training. It made a real improvement in my ability to suss out a candidate and understand what they could actually do vs what they claimed they could.
908B64B197 6y ago
> We're having a bit of a debate about this internally as the company I work for is struggling to hire good candidates. What's the compensation like? Stock? > For a while we've had a take-home test and that has increased our success rate somewhat. But as you say, we really don't want to lay unreasonable expectations on people who may have other obligations You'll also find out that the people you really want to hire are not on the market for a long time. So if they are interviewing at several places and you give them a 4 hours take-home, they'll put it on their to-do list but by the time they get to it they might be in the final rounds at 2-3 other places that brought them on-site immediately. > It probably doesn't help that management here is almost entirely non-technical That's a much bigger issue than most assume. > There's usually a developer or two sitting in on the interview to try and balance it out That's a huge no from me. Final approval of a technical candidate should only be in the hands of the technical staff. I've heard horror story of a "senior" engineer from "his country's top school" being interviewed for a technical position by several non-technical managers and HR reps. They only included an engineer in the final round, which was basically supposed to be rubberstamped anyways. He was then asked to implement something trivial like fizzbuzz or wordcount on the whiteboard. The candidate then became extremely defensive and tried to argue that such task was "beneath him", arguing for a good 15 minutes why he shouldn't have to do it. Then the dev just left the room and said that he used this question as a warmup with new hires and it typically takes them less than 10 minutes.
- elanning 6y ago
  A lot of senior engineers would refuse to do a fizzbuzz. I'm really not seeing the problem here.
  - fbleibel 6y ago
    On the contrary, I think this is a huge red flag. Just go along with the interviewer, maybe highlight that this is typically and entry-level problem, but solve it. You really don't want to hire someone who not only can't solve fizzbuzz, but also refuses to hear about it and complain that it's 'beneath them' (what a annoying attitude!).
watwut 6y ago
> we've had many people in the past who've interviewed very well but turned out to be completely incompetent when assigned to a real project. There is absolutely nothing wrong with coding test. Timed with internet. The one that does not tests for algorithms, but for basic competency. For example, ask them to parse some data out of xml file and give them tons of time. That will check competency without being burden. Yes, non tech managera sux at recognizing who is good programmer in discusion. But, so do programmers and technical people. It is easy to pretend competence if you read enough blogs and can project right attitudes.
anyfoo 6y ago
> Maybe we're just really bad at interviews? It probably doesn't help that management here is almost entirely non-technical. There's usually a developer or two sitting in on the interview to try and balance it out, but I don't think any of us would consider ourselves particularly good at interviewing people. This seems to be the issue, in fact it's quite baffling to me. If you are interviewing for a technical role, why would you not perform a technical interview led by technical staff? I'm not saying the other stuff is not important as well, but this reads to me like you are essentially leaving the core of the role out of the interview (I'm not sure what "usually a developer or two are sitting in as well" means).
indigochill 6y ago
Our take-home obviously isn't a trade secret, so we have three trivial problems in ours: 1. Write code that takes in a rectangle, coordinate, and distance, and tells if that coordinate is within the distance of the rectangle. 2. Determine whether a string has permutations which are palindromes. (I'm of the opinion this one is too simple, but it's been fascinating seeing what mental gymnastics some developers will go through to solve this problem) 3. Write a CLI dice game (we provide the rules) in which the computer player wins 70% of the time. Make the method by which the computer cheats undetectable (of course this is impossible, but candidates just give it a best shot and we talk about their approach in the interview). Assuming the test passes muster, we have three interviews right now, I think. One with managers to talk about the role, one with senior devs outside the team to perform the technical interview, and one with the devs in the team to talk shop. In the technical interview, we primarily go through their test solutions (we do no whiteboard coding aside from reviewing the test and even that's very loose - not actual code). We have hired candidates before who have failed some of these tasks (mainly our candidates have trouble with #1) but who showed a capacity for being able to think on their feet and correct their mistakes during the interview. Of course a candidate who aces the test is going to have an advantage, but absent having the right answers, quick thinking/adaptability is probably the #2 trait we look for that seems to be a good predictor for developer success. Every candidate we've hired through our process has been a success (which could be dumb luck, since none of us are super experienced/knowledgeable interviewers either, but we wing it best we can) and the take-home (and, equally importantly, the technical interview) has absolutely helped us weed out candidates with impressive CVs whose test results did not measure up.
- orangecat 6y ago
  2. Determine whether a string has permutations which are palindromes. If I understand this correctly, it's the sort of problem that's simple if you see the trick and hard if you don't. I'd expect many candidates to think that you actually want them to generate all the permutations and then get bogged down in recursion. #1 is good, and #3 sounds interesting.
  - concreteblock 6y ago
ChefboyOG 6y ago
Oddly, the last technical interview process I went through was the best and in some ways, the most old fashioned. There were three "rounds": 1. Phone screen with actual lead developer. There were some quiz-y questions here, which I'd previously thought of as a silly outdated approach, but it honestly was a low stress filter for basic technical knowledge. 2. 90 minute pairing exercise with the same lead developer. We built a small example app together. Resources were sent over ahead of time so my environment was good to go, and the expectation was set from the start that the goal was to assess how I thought and approached code, not to see how far I could get in 90 minutes. 3. 4 hour "on site" where I talked to each developer on the team I'd be joining. No technical exercises. Each person came with their own questions, and expected me to ask mine. What I took away from the experience was that companies are seriously overthinking and over-engineering the process. There isn't a magic heuristic you're going to discover that will identify "10x engineers." You can vet if someone is technically competent enough for the role, approaches software in a way that gels with your org, and if they have any red flags in fairly straightforward fashion that doesn't require enormous amounts of prep for them or your team.
autarch 6y ago
I've written a fair bit about hiring processes on my blog. My most recent post about is at https://blog.urth.org/2019/07/11/a-technical-hiring-process-revisited/ https://blog.urth.org/2019/07/11/a-technical-hiring-process-... We use this process at my current employer, and we've gotten mostly good feedback about it (including from people we didn't hire), though some candidates don't like it. The people we've hired using this process have been quite good, though I can't say if that's the process or something else that leads to that outcome.

notsureaboutpg 6y ago

>For a while we've had a take-home test and that has increased our success rate somewhat. But as you say, we really don't want to lay unreasonable expectations on people who may have other obligations These are mutually exclusive things. Candidates who have more free time to code are more likely to be better at it than those who don't (all else equal). Candidates who have OSS maintenance / leadership experience are more likely to work well in teams (all else equal). If you choose not to weight those things, to balance the playing field for people who have more obligations outside of work, then you'll also have lesser quality candidates (again, generally). So if you're struggling to hire good candidates, maybe it's a good idea to weight these things (and bias against people with less free time outside work). Once you have good candidates, and this is no longer a pressure on your business, then it might be a good time to try and balance the playing field for new hires. But you cannot balance the playing field and also hire the best candidates. The best candidates will have unfair advantages in general. You can either lean towards having the best or lean towards balancing the playing field.

sokoloff 6y ago
We should try to eliminate irrelevant biases in the hiring process but we are fundamentally trying to select people to hire who will join our company and write good code. That quality is not evenly distributed in the population. Some of the gymnastics thinking that I see seems to suggest that we’d seek to hire fluent English speakers by interviewing evenly across all populations. By all means, I’d be more than happy to hire someone fluent in English from China, but if I’m looking for a fluent English speaker, I shouldn’t spent 18.5% of my recruiting efforts/budget in China out of "fairness".
- potatoz2 6y ago
  People in China often don't speak fluent English because they grow up in a country where it's not a spoken language. People who don't contribute to OSS typically don't do it because they don't have the ability (some OSS code is held together by duct tape) but because they don't have time, interest or think it's harder than it is. If you have better tools to determine if someone is good at writing code for your company, why use a proxy measure that's different in many ways?
  - sokoloff 6y ago
    Unless I’ve worked with you or someone that I deeply trust has and will vouch for you, I don’t have any stronger signal of your coding and teamwork abilities than a strong OSS contribution history can provide. It’s rare for an interview process to provide more than 6 hours of content, some of which is non-evaluatable. Reference checks are all but useless compared to seeing actual work over time. About 1 time in 6, you’ll interview a full standard deviation above or below your “true” ability. About 1 time in 50, you’ll turn in a +2σ (or -2σ) performance. That’s largely eliminated with personal experience, a trusted vouch, or a strong OSS portfolio.

PragmaticPulp 6y ago

> we've had many people in the past who've interviewed very well but turned out to be completely incompetent when assigned to a real project. Unpopular opinion on HN: This is actually quite common when you hire based purely on resumes or credentials. Some people are really good at interviewing and being charismatic enough to convince people to hire them. There are a lot of candidates who can talk the talk but really just want a job where they can browse Reddit and Twitter all day while writing a couple lines of code every once in a while. There are a lot of companies that are big enough for these people to blend in for years. Take home tests in the range of 1-4 hours shouldn’t be an undue burden on any applicants, if you give sufficient time to return the test. Many candidates wouldn’t bat an eye at taking a day to interview on site, so asking them to spend a couple hours of their free time on an interview isn’t really a disproportionate ask. Giving someone a 20-40 hour take home project would be ridiculous, of course, but reasonably sized problems are perfectly reasonable.

ofrzeta 6y ago
We spent some time on making up a quite simple project with a readme about the tasks to be implemented. It's something that can be done in half an hour when you are fast and we always thought we need to make it a bit harder. Turns out it's good enough to weed out quite many people who can't keep a deadline or don't get their shit together in other ways. From reviewing the code you can judge how people think, if they keep their files and code in order, use git etc. We also prompt people do document their process in finding a solution.
- portoal 6y ago
  Interesting, could u share a bit more pls?
  - ofrzeta 6y ago
    We made up a problem with technologies that we use at work targeted at frontend engineers. So we set up a project with a Symfony backend and made the candidates create Twig templates and SASS styling based on Bootstrap. Then we put that up on Github with a README. So the candidates could focus on the frontend stuff without caring for the backend. Still they had to figure out how to set up SASS compilation and modularization of Twig templates. As I said it's not very hard but you can get bonus points if you come up with considerations for responsive design or stuff like that. Overall the aim is to create readable idiomatic code and not come up with something clever that no one but you can understand. I guess that's a very specific task that's of no great use to others but the point is to have a specific problem with some blanks that can be filled out with a reasonable amount of work by the candidates. Plus the general workflow using git and such.

HeyLaughingBoy 6y ago

I don't know how receptive your management would be to this, or what it costs, but at my last job, all interviewers were required to take interview training. It made a real improvement in my ability to suss out a candidate and understand what they could actually do vs what they claimed they could.

908B64B197 6y ago

> We're having a bit of a debate about this internally as the company I work for is struggling to hire good candidates. What's the compensation like? Stock? > For a while we've had a take-home test and that has increased our success rate somewhat. But as you say, we really don't want to lay unreasonable expectations on people who may have other obligations You'll also find out that the people you really want to hire are not on the market for a long time. So if they are interviewing at several places and you give them a 4 hours take-home, they'll put it on their to-do list but by the time they get to it they might be in the final rounds at 2-3 other places that brought them on-site immediately. > It probably doesn't help that management here is almost entirely non-technical That's a much bigger issue than most assume. > There's usually a developer or two sitting in on the interview to try and balance it out That's a huge no from me. Final approval of a technical candidate should only be in the hands of the technical staff. I've heard horror story of a "senior" engineer from "his country's top school" being interviewed for a technical position by several non-technical managers and HR reps. They only included an engineer in the final round, which was basically supposed to be rubberstamped anyways. He was then asked to implement something trivial like fizzbuzz or wordcount on the whiteboard. The candidate then became extremely defensive and tried to argue that such task was "beneath him", arguing for a good 15 minutes why he shouldn't have to do it. Then the dev just left the room and said that he used this question as a warmup with new hires and it typically takes them less than 10 minutes.

elanning 6y ago
A lot of senior engineers would refuse to do a fizzbuzz. I'm really not seeing the problem here.
- fbleibel 6y ago
  On the contrary, I think this is a huge red flag. Just go along with the interviewer, maybe highlight that this is typically and entry-level problem, but solve it. You really don't want to hire someone who not only can't solve fizzbuzz, but also refuses to hear about it and complain that it's 'beneath them' (what a annoying attitude!).

watwut 6y ago

> we've had many people in the past who've interviewed very well but turned out to be completely incompetent when assigned to a real project. There is absolutely nothing wrong with coding test. Timed with internet. The one that does not tests for algorithms, but for basic competency. For example, ask them to parse some data out of xml file and give them tons of time. That will check competency without being burden. Yes, non tech managera sux at recognizing who is good programmer in discusion. But, so do programmers and technical people. It is easy to pretend competence if you read enough blogs and can project right attitudes.

anyfoo 6y ago

> Maybe we're just really bad at interviews? It probably doesn't help that management here is almost entirely non-technical. There's usually a developer or two sitting in on the interview to try and balance it out, but I don't think any of us would consider ourselves particularly good at interviewing people. This seems to be the issue, in fact it's quite baffling to me. If you are interviewing for a technical role, why would you not perform a technical interview led by technical staff? I'm not saying the other stuff is not important as well, but this reads to me like you are essentially leaving the core of the role out of the interview (I'm not sure what "usually a developer or two are sitting in as well" means).

indigochill 6y ago

Our take-home obviously isn't a trade secret, so we have three trivial problems in ours: 1. Write code that takes in a rectangle, coordinate, and distance, and tells if that coordinate is within the distance of the rectangle. 2. Determine whether a string has permutations which are palindromes. (I'm of the opinion this one is too simple, but it's been fascinating seeing what mental gymnastics some developers will go through to solve this problem) 3. Write a CLI dice game (we provide the rules) in which the computer player wins 70% of the time. Make the method by which the computer cheats undetectable (of course this is impossible, but candidates just give it a best shot and we talk about their approach in the interview). Assuming the test passes muster, we have three interviews right now, I think. One with managers to talk about the role, one with senior devs outside the team to perform the technical interview, and one with the devs in the team to talk shop. In the technical interview, we primarily go through their test solutions (we do no whiteboard coding aside from reviewing the test and even that's very loose - not actual code). We have hired candidates before who have failed some of these tasks (mainly our candidates have trouble with #1) but who showed a capacity for being able to think on their feet and correct their mistakes during the interview. Of course a candidate who aces the test is going to have an advantage, but absent having the right answers, quick thinking/adaptability is probably the #2 trait we look for that seems to be a good predictor for developer success. Every candidate we've hired through our process has been a success (which could be dumb luck, since none of us are super experienced/knowledgeable interviewers either, but we wing it best we can) and the take-home (and, equally importantly, the technical interview) has absolutely helped us weed out candidates with impressive CVs whose test results did not measure up.

orangecat 6y ago
2. Determine whether a string has permutations which are palindromes. If I understand this correctly, it's the sort of problem that's simple if you see the trick and hard if you don't. I'd expect many candidates to think that you actually want them to generate all the permutations and then get bogged down in recursion. #1 is good, and #3 sounds interesting.
- concreteblock 6y ago

ChefboyOG 6y ago

Oddly, the last technical interview process I went through was the best and in some ways, the most old fashioned. There were three "rounds": 1. Phone screen with actual lead developer. There were some quiz-y questions here, which I'd previously thought of as a silly outdated approach, but it honestly was a low stress filter for basic technical knowledge. 2. 90 minute pairing exercise with the same lead developer. We built a small example app together. Resources were sent over ahead of time so my environment was good to go, and the expectation was set from the start that the goal was to assess how I thought and approached code, not to see how far I could get in 90 minutes. 3. 4 hour "on site" where I talked to each developer on the team I'd be joining. No technical exercises. Each person came with their own questions, and expected me to ask mine. What I took away from the experience was that companies are seriously overthinking and over-engineering the process. There isn't a magic heuristic you're going to discover that will identify "10x engineers." You can vet if someone is technically competent enough for the role, approaches software in a way that gels with your org, and if they have any red flags in fairly straightforward fashion that doesn't require enormous amounts of prep for them or your team.

autarch 6y ago

I've written a fair bit about hiring processes on my blog. My most recent post about is at https://blog.urth.org/2019/07/11/a-technical-hiring-process-revisited/ https://blog.urth.org/2019/07/11/a-technical-hiring-process-... We use this process at my current employer, and we've gotten mostly good feedback about it (including from people we didn't hire), though some candidates don't like it. The people we've hired using this process have been quite good, though I can't say if that's the process or something else that leads to that outcome.