8 ms·

Hiring Great Developers: The Pre-Screening

51 points by pufuwozu 16y ago

bugsy 16y ago
A fine article for hiring adequate or average developers. You can't really hire great developers using any process involving resumes and ads. Hiring great developers involves recruitment. You already know you would hire the developer before you even contact them. The entire issue is can you offer them enough to lure them from where they are now.
- variety 16y ago
  Or, it should be seen more as a tool for filtering people out rather than for selecting people in. And if they are used, these tests should be (1) done at the very earliest stages in the process and (2) be conducted in a manner that's as time-efficient as humanely possible.
sp4rki 16y ago
And just to be sure, we disregarded the results from the screening anyway, and based our hiring decisions only on the three onsite-interviews. Am I missing something here?
- chrisaycock 16y ago
  The screening process he's describing is just to throw applications away, not to actually accept anyone.
- bigiain 16y ago
  I suspect it means passing the screening gets you through to the onsite-interviews, but the subsequent decision about hiring is purely based on the interviews, not on any results or rankings from the screenings.
huhtenberg 16y ago
We tried it and while it does help to filter out completely clueless people, however cheating is a real problem. This clearly depends on the industry and specifics of the position, but out of three people that hired, two clearly cheated on the remote coding test. This was for a C position in embedded context, and one would think that no one in their sane mind would cheat, and still many people did. So, yeah, good idea, but use with caution as YMMV.
- chrisaycock 16y ago
  > out of three people that hired, two clearly cheated on the remote coding test What screening did you do onsite?
  - huhtenberg 16y ago
    No, all from other cities. Sent the task over email and wanted to get a solution back in 8 hours or less.
    - chrisaycock 16y ago
      Right, I mean after the remote coding test, did you bring them onsite for another battery of screening?
      - huhtenberg 16y ago
        We had several phone interviews - one technical before the test, one - after and another one with HR, and then CTO insisted on hiring and flying them over right away. With a three month probation period... that did come handy. So basically the first two weeks were the on-site screening if you will, and which they spectacularly failed.
        chrisaycock 16y ago
        Ok, that makes a lot more sense then.
tav 16y ago
Here's a much better approach: use IRC. I've successfully recruited dozens of exceptional individuals through IRC and am surprised that it's not often used for recruitment. Networks like freenode are filled with amazing individuals who are quite amenable to interesting startups. Even with my first company back in '99, when I didn't even know how to code, I found that the developers hired through IRC performed magnitudes better than the ones who came through more traditional hiring processes. Just hang out for a few months in the IRC channels of open source projects that you find relevant; get to know the various individuals; once you find someone that you believe you can work well with, engage them about joining your venture. Easy, effective and the social bond even lasts for years afterwards.
trotsky 16y ago
But I am still surprised at how few companies do this Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity. I'd be willing to bet that the 30% who didn't bother to respond to the challenges included some of your best applicants. You'd never hire an accountant by insisting they prepare a quarterly for you before you ever met them and you wouldn't make prospective truck drivers arbitrarily drive hundreds of miles to an interview. If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for. But just like your little tests, you'll be pre-screening out the best as well as the worst.
- praptak 16y ago
  "Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity." My impression is exactly the opposite. It flags them as a company that cares about technical competence. "If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for." Their test screen for more than that. And of course those professional certifications are useless if you go beyond the very basic level, just like "CV-years of experience" are useless for determining competence.
  - smikhanov 16y ago
    It flags them as a company that cares about technical competence I bet the tasks being sent out are most probably something in the vein of parsing a CSV file or (in case of Atlassian) writing a simple JSP or trivial XSLT or what have you. Not the best check for competence.
    - perfragemann 16y ago
      You should give it a try, see what they send you! The stuff we did when I was still there was definitely more advanced. But again, pre-screening is never a test for competence, it is a test for lack of competence. The real test for great skill comes in the form of a 90m+ coding test and a 2h technical interview.
jchonphoenix 16y ago
There's a major issue with your method. The best developers know they can get jobs wherever they want. What incentive do they have to bother with the extra work?
- cageface 16y ago
  When I see application processes like this I'm generally more interested because it signals to me that the company cares about hiring and that I'm more likely to have colleagues there that have been selected for ability rather than having exactly the right checkboxes on their CVs.
- ajg1977 16y ago
  Because, a) the 'best developers tend to love beating challenges so see it as a fun exercise to do one evening, and b) good developers want to work with good developers and seeing the screening process is a benefit. Good developers of a certain experience level generally realize that if they apply for a job somewhere and waltz through the interview, or worse, the interview is a basically a pleasant chat with the tech director, then the chances are the hiring standards really aren't that great . Oh and c) - if a developer does feel they're too good for your screening test, or such a thing is enough to put them off your company, then weeding them out at such an early stage may actually be a positive thing. I do think it's crass to send out a problem sheet without at least verbally talking to the candidate briefly though. We currently do - 5-15 minute chat with a recruiter or PM who is savvy enough to filter out the people who really aren't experienced enough. 3 hour timed problem (debugging, coding, math, domain knowledge) Technical phone screen In-person interview (6-7 hours, all expenses paid).
  - bugsy 16y ago
    What you say is true for mediocre to average developers. Before I had my own company, every place I worked whenever I would talk to clients or go to conferences I would get on the spot job offers. Principals would ask "Would you ever consider leaving XYZ?" I would say I'm not looking at the moment. They would always then hand me their cards and say "If you're ever looking to move, give me a call first." In addition, I would get calls at home to my unlisted number because of articles I had written or patents I held. When it was time to move on I would have dozens of open offers already to chose from. Why the hell would I or any other great developer bother dredging through monster hoping to find the 0.001% of ads that are both legitimate and for a place than isn't some web shop with slave hours and low wages? Answer is I wouldn't. Any company that only relies on job ads and resumes is not a place that great developers work for.
sciboy 16y ago
I simply give them a call and ask them what the most interesting papers they have read over the last year are. Without any forewarning of course. People who are passionate about things know their stuff; and it's not from blog posts or forums that they obtain this information. And I can tell you, if I learn something while on the phone with them then it's a guaranteed good hire.
- perfragemann 16y ago
  I believe that with your approach you are guaranteed to only invite great applicants, so that's cool. But I worry that you also dismiss many great applicants who are not that snappy, have a bad day (minute, I guess!) I couldn't come up with a great paper. I read blogs, sure, but papers? If I needed a developer who's great at answering tricky questions in front of an audience, your approach is perfect. But I'd also love a developer who solves hard questions with a bit more time to ponder the problem space. I worry I couldn't spot them in a such a tough phone interview. Oh well, next post! :-)
  - sciboy 16y ago
    No offence but discussing the most recent/interesting paper you have read shouldn't be a tricky question.
BweezySearch 16y ago
Very cool. I'm thinking it would be great if elance-type of sites, or even experts exchange, would have an area where a prospective employer can submit small coding questions to potential employees.
tomjen3 16y ago
> I am asking applicants to write some unit tests, to summarise the key point of a Google I/O video presentation that's actually fun to watch. I am asking them to learn enough about a web framework to be able to write a very basic program. I am asking them to review some really dumb code. And since I don't need code-monkeys but people with opinions, I am asking them for their opinion about my protoype: What would you improve? Shit you are kidding right? Or do people just really value their time that low?
VladRussian 16y ago
it isn't about process, it is about people. As Steve Jobs put it : "A players hire A players, B players hire C players." http://www.folklore.org/StoryView.py?project=Macintosh&story=How_to_Hire_Insanely_Great_Employees.txt&sortOrder=Sort%20by%20Date http://www.folklore.org/StoryView.py?project=Macintosh&s...
marcin 16y ago
Wrote it in the post comment, but will write again. The company I helped get traction (Codility) would address many issues people have with remote testing: manual work, some of the cheating and qualitative vs quantitative dilema. I may be biased, due to my former involvement, but it costs nothing to try it, so you can see for yourself: http://codility.com/demo/take-sample-test/ http://codility.com/demo/take-sample-test/
- kranner 16y ago
  Bunging in my own http://codeboff.in http://codeboff.in in the same spirit.
  - brlewis 16y ago
    Upvoted you both for on-topic self-promotion. HN needs more of that.
bengl3rt 16y ago
An interesting side note to gather from this is that it confirms something I've long suspected: LinkedIn recommendations are useless. FWIW I have a couple good ones, but I don't think they've ever even helped me get an interview much less a job. I don't think anybody who actually makes hiring decisions has time to look at them; seems the only purpose they serve is to let people who casually cruise your profile know that you are competent.
xiaoma 16y ago
Interspersing all kinds of rebuttals to complaints others have made about the post inside the post itself looks really defensive. It disrupted my reading.
benaston 16y ago
Posts like these reenforce my dismay at the total and utter devaluation of the education system when dealing with recruiters for full-time positions (15 years minimum of full-time education, but you still need that one final test). That said, I don't have a problem with these tests if the setter understands that they are not there to filter in or out the competent, but instead serve as a filter of convenience because dealing with 60 resumes is easier than 600. This seems an acceptable pragmatic approach. When rubber hits the road a business needs to make money by being competitive. If having read a book on JUnit is your metric for hiring competitive employees, so be it. I can't help but wonder if there is a sizeable amount of latent competitive advantage in the complement of this set.

51 points by pufuwozu 16y ago

bugsy 16y ago

A fine article for hiring adequate or average developers. You can't really hire great developers using any process involving resumes and ads. Hiring great developers involves recruitment. You already know you would hire the developer before you even contact them. The entire issue is can you offer them enough to lure them from where they are now.

variety 16y ago
Or, it should be seen more as a tool for filtering people out rather than for selecting people in. And if they are used, these tests should be (1) done at the very earliest stages in the process and (2) be conducted in a manner that's as time-efficient as humanely possible.

sp4rki 16y ago

And just to be sure, we disregarded the results from the screening anyway, and based our hiring decisions only on the three onsite-interviews. Am I missing something here?

chrisaycock 16y ago
The screening process he's describing is just to throw applications away, not to actually accept anyone.
bigiain 16y ago
I suspect it means passing the screening gets you through to the onsite-interviews, but the subsequent decision about hiring is purely based on the interviews, not on any results or rankings from the screenings.

huhtenberg 16y ago

We tried it and while it does help to filter out completely clueless people, however cheating is a real problem. This clearly depends on the industry and specifics of the position, but out of three people that hired, two clearly cheated on the remote coding test. This was for a C position in embedded context, and one would think that no one in their sane mind would cheat, and still many people did. So, yeah, good idea, but use with caution as YMMV.

chrisaycock 16y ago
> out of three people that hired, two clearly cheated on the remote coding test What screening did you do onsite?
- huhtenberg 16y ago
  No, all from other cities. Sent the task over email and wanted to get a solution back in 8 hours or less.
  - chrisaycock 16y ago
    Right, I mean after the remote coding test, did you bring them onsite for another battery of screening?
    - huhtenberg 16y ago
      We had several phone interviews - one technical before the test, one - after and another one with HR, and then CTO insisted on hiring and flying them over right away. With a three month probation period... that did come handy. So basically the first two weeks were the on-site screening if you will, and which they spectacularly failed.
      - chrisaycock 16y ago
        Ok, that makes a lot more sense then.

tav 16y ago

Here's a much better approach: use IRC. I've successfully recruited dozens of exceptional individuals through IRC and am surprised that it's not often used for recruitment. Networks like freenode are filled with amazing individuals who are quite amenable to interesting startups. Even with my first company back in '99, when I didn't even know how to code, I found that the developers hired through IRC performed magnitudes better than the ones who came through more traditional hiring processes. Just hang out for a few months in the IRC channels of open source projects that you find relevant; get to know the various individuals; once you find someone that you believe you can work well with, engage them about joining your venture. Easy, effective and the social bond even lasts for years afterwards.

trotsky 16y ago

But I am still surprised at how few companies do this Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity. I'd be willing to bet that the 30% who didn't bother to respond to the challenges included some of your best applicants. You'd never hire an accountant by insisting they prepare a quarterly for you before you ever met them and you wouldn't make prospective truck drivers arbitrarily drive hundreds of miles to an interview. If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for. But just like your little tests, you'll be pre-screening out the best as well as the worst.

praptak 16y ago
"Because it flags you as a mill, uninterested in the applicant and likely a firm that treats development as a commodity." My impression is exactly the opposite. It flags them as a company that cares about technical competence. "If you want to screen for the absolute basics of rote capability that's what those various certified professional type credentials are for." Their test screen for more than that. And of course those professional certifications are useless if you go beyond the very basic level, just like "CV-years of experience" are useless for determining competence.
- smikhanov 16y ago
  It flags them as a company that cares about technical competence I bet the tasks being sent out are most probably something in the vein of parsing a CSV file or (in case of Atlassian) writing a simple JSP or trivial XSLT or what have you. Not the best check for competence.
  - perfragemann 16y ago
    You should give it a try, see what they send you! The stuff we did when I was still there was definitely more advanced. But again, pre-screening is never a test for competence, it is a test for lack of competence. The real test for great skill comes in the form of a 90m+ coding test and a 2h technical interview.

jchonphoenix 16y ago

There's a major issue with your method. The best developers know they can get jobs wherever they want. What incentive do they have to bother with the extra work?

cageface 16y ago
When I see application processes like this I'm generally more interested because it signals to me that the company cares about hiring and that I'm more likely to have colleagues there that have been selected for ability rather than having exactly the right checkboxes on their CVs.
ajg1977 16y ago
Because, a) the 'best developers tend to love beating challenges so see it as a fun exercise to do one evening, and b) good developers want to work with good developers and seeing the screening process is a benefit. Good developers of a certain experience level generally realize that if they apply for a job somewhere and waltz through the interview, or worse, the interview is a basically a pleasant chat with the tech director, then the chances are the hiring standards really aren't that great . Oh and c) - if a developer does feel they're too good for your screening test, or such a thing is enough to put them off your company, then weeding them out at such an early stage may actually be a positive thing. I do think it's crass to send out a problem sheet without at least verbally talking to the candidate briefly though. We currently do - 5-15 minute chat with a recruiter or PM who is savvy enough to filter out the people who really aren't experienced enough. 3 hour timed problem (debugging, coding, math, domain knowledge) Technical phone screen In-person interview (6-7 hours, all expenses paid).
- bugsy 16y ago
  What you say is true for mediocre to average developers. Before I had my own company, every place I worked whenever I would talk to clients or go to conferences I would get on the spot job offers. Principals would ask "Would you ever consider leaving XYZ?" I would say I'm not looking at the moment. They would always then hand me their cards and say "If you're ever looking to move, give me a call first." In addition, I would get calls at home to my unlisted number because of articles I had written or patents I held. When it was time to move on I would have dozens of open offers already to chose from. Why the hell would I or any other great developer bother dredging through monster hoping to find the 0.001% of ads that are both legitimate and for a place than isn't some web shop with slave hours and low wages? Answer is I wouldn't. Any company that only relies on job ads and resumes is not a place that great developers work for.

sciboy 16y ago

I simply give them a call and ask them what the most interesting papers they have read over the last year are. Without any forewarning of course. People who are passionate about things know their stuff; and it's not from blog posts or forums that they obtain this information. And I can tell you, if I learn something while on the phone with them then it's a guaranteed good hire.

perfragemann 16y ago
I believe that with your approach you are guaranteed to only invite great applicants, so that's cool. But I worry that you also dismiss many great applicants who are not that snappy, have a bad day (minute, I guess!) I couldn't come up with a great paper. I read blogs, sure, but papers? If I needed a developer who's great at answering tricky questions in front of an audience, your approach is perfect. But I'd also love a developer who solves hard questions with a bit more time to ponder the problem space. I worry I couldn't spot them in a such a tough phone interview. Oh well, next post! :-)
- sciboy 16y ago
  No offence but discussing the most recent/interesting paper you have read shouldn't be a tricky question.

BweezySearch 16y ago

Very cool. I'm thinking it would be great if elance-type of sites, or even experts exchange, would have an area where a prospective employer can submit small coding questions to potential employees.

tomjen3 16y ago

> I am asking applicants to write some unit tests, to summarise the key point of a Google I/O video presentation that's actually fun to watch. I am asking them to learn enough about a web framework to be able to write a very basic program. I am asking them to review some really dumb code. And since I don't need code-monkeys but people with opinions, I am asking them for their opinion about my protoype: What would you improve? Shit you are kidding right? Or do people just really value their time that low?

VladRussian 16y ago

it isn't about process, it is about people. As Steve Jobs put it : "A players hire A players, B players hire C players." http://www.folklore.org/StoryView.py?project=Macintosh&story=How_to_Hire_Insanely_Great_Employees.txt&sortOrder=Sort%20by%20Date http://www.folklore.org/StoryView.py?project=Macintosh&s...

marcin 16y ago

Wrote it in the post comment, but will write again. The company I helped get traction (Codility) would address many issues people have with remote testing: manual work, some of the cheating and qualitative vs quantitative dilema. I may be biased, due to my former involvement, but it costs nothing to try it, so you can see for yourself: http://codility.com/demo/take-sample-test/ http://codility.com/demo/take-sample-test/

kranner 16y ago
Bunging in my own http://codeboff.in http://codeboff.in in the same spirit.
- brlewis 16y ago
  Upvoted you both for on-topic self-promotion. HN needs more of that.

bengl3rt 16y ago

An interesting side note to gather from this is that it confirms something I've long suspected: LinkedIn recommendations are useless. FWIW I have a couple good ones, but I don't think they've ever even helped me get an interview much less a job. I don't think anybody who actually makes hiring decisions has time to look at them; seems the only purpose they serve is to let people who casually cruise your profile know that you are competent.

xiaoma 16y ago

Interspersing all kinds of rebuttals to complaints others have made about the post inside the post itself looks really defensive. It disrupted my reading.

benaston 16y ago

Posts like these reenforce my dismay at the total and utter devaluation of the education system when dealing with recruiters for full-time positions (15 years minimum of full-time education, but you still need that one final test). That said, I don't have a problem with these tests if the setter understands that they are not there to filter in or out the competent, but instead serve as a filter of convenience because dealing with 60 resumes is easier than 600. This seems an acceptable pragmatic approach. When rubber hits the road a business needs to make money by being competitive. If having read a book on JUnit is your metric for hiring competitive employees, so be it. I can't help but wonder if there is a sizeable amount of latent competitive advantage in the complement of this set.