2 ms·
Thanks. I'm glad to hear your description of our process, it's nice to get some positive feedback. ... candidates won't perform work-sample challenges. I've in
by mike_hearn 7y ago
Thanks. I'm glad to hear your description of our process, it's nice to get some positive feedback.
... candidates won't perform work-sample challenges. I've interviewed almost as many candidates as this person has (and screened out far more based on challenges) and that simply hasn't been my experience
I'm curious what you mean exactly by work-sample challenge here - I guess you mean homework assignments, as a coding test is in effect a work-sample challenge of a sort.
The belief that senior candidates won't do them is based on observation of behaviour when asking them to take 'robot' automated interviews. The people with the most experience and best known companies on their CV (the sort of places that do hire carefully) were the most likely to drop out of the process at that point. We didn't do a big study because, well, we didn't want to lose more senior candidates, but there were enough samples to see a pattern. We haven't seen that problem with video-call interviews.
I guess you're asking more about either showing open source code, or homework assignments.
High quality open source code can be a good factor in favour of a candidate and it is checked if someone puts some on their CV. Unfortunately in the past I got burned by hiring a candidate based purely on code on their github repository and a Q&A style interview, without doing a live coding test, and the person turned out to be largely incapable of writing code to the necessary standard. We never figured out why his code on github seemed so much better. There's just so much more information you get from watching someone work, like whether it takes 10x longer than it should to produce that code, or how much help they need.
The issue with homework assignments can be summed up as - I wouldn't be willing to do them if I were looking for a job right now, unless I was completely unemployed and desperate. There are very limited hours in a day and most programmers looking for a new job already have one. Also, you can't prove it was actually them that wrote the code, and you can't tell how long it took them to do so.
I make no claim our process is the best possible, or even very good. It was just descriptive and seems to work well enough - other than the one bad hire where we didn't follow the standard process, we have a pretty uniformly competent team.
- tptacek 7y agoA work-sample challenge is a practical problem that mimics the actual work done on a job (the term comes from W. Edwards Deming back in the 1950s). Doing a Github-PR-style code review is a perfect example of a work-sample challenge; you can come up with any number of work-sample challenges that involve actually coding; a friend used them to recruit for his sales team by having them write outbound mails or handle simulated inbound requests. Our work sample challenges at Latacora involve as assessment of a Django web application, an assessment of an AWS deployment environment hosting that application, and the construction of a simple scanning tool to spot the vulnerabilities discovered. At my previous firm, the challenges were similar – there was no programming challenge, but there was an assessment of a custom binary trading protocol. I'd balk at robot automated interviews too. I'd be concerned that my responses wouldn't fit whatever weird template they were using; also, I'd be almost certain that any robo-tronic assessment thingy they were doing almost certainly wouldn't be determinative of the outcome of the qualification process, but rather just an arbitrary hurdle I'd have to jump over before the "real" interview happened. Obviously, if my options include equally attractive companies that don't present that same extra hurdle, I'm not going to bother applying! I also think there is certainly a class of senior developer that won't do practical at-home challenges at all, no matter how carefully designed, even if they're entirely determinative of outcome and even if they're calibrated carefully to make the interview process less logistically arduous (for instance, by reducing the time commitment for actual interviews from 4-6 hours down to a single hour, and assuring before the interview happens that you're going in already tech-qualified). Certainly, work-sample qualification isn't the norm in our industry, and senior developers also tend to move from job to job through networking and very warm introductions. What I'll contend, though, is that there's a pretty significant adverse selection problem in that pool of senior developers. I don't doubt that the bulk of them have earned their experience and reputation, but there're also a lot of developers swimming in that pool that have failed upward into it, and move from job to job largely through interviewing skill. What I've learned (or think I have) is that there's virtually no correlation between someone's ability to talk about programming and their ability to effectively deliver code on a commercial project. What makes the work-sample process so attractive to me is that the opposite phenomenon also seems to exist: in the pool of "not labeled senior" developers, there seems to be a usefully large number of exceptionally talented developers who have, owing to circumstance, simply not acquired the resume labels to navigate themselves into good jobs. Hiring from that pool is very risky if you rely on interviews! But what we learned over the last 10 years is that you can build practical challenges and get to a level of confidence where you can rely on them over interviews and resumes, and when you do, you can hire a lot of talented people that other firms can't, because their interview processes won't let them.