6 ms·

How long does a work-sample test take? I'm typically hiring people to work on on-going projects over a period of months or years; the fact that candidate A does

by jleader 13y ago

How long does a work-sample test take? I'm typically hiring people to work on on-going projects over a period of months or years; the fact that candidate A does better on a 2-hour or 2-day task than candidate B may not give me that much information about things like how effective they will be at sharing their own knowledge with the rest of the team (or taking guidance from more senior members of the team), developing a good working rapport with their QA teammates so they can get to the root of any problems as efficiently as possible, convincing the sysadmins that the candidate's proposed deployment tool is the right way to do it (or gracefully agreeing to use the sysadmins' tool of choice, either because it's actually better, or just because the benefit of the sysadmins' familiarity with it outweighs any technical benefits of the other tool), etc. Unless I'm hiring someone to sit off in a corner developing their own solo project, I'm not convinced that "work sample" tests are that informative. I don't agree that this question is a silver bullet, but I think it's valuable to see what sort of achievment the candidate considers significant, to ask follow-up questions about how they handled problems along the way to that achievment, etc. The question helps reduce the power assymetry of the interview, by letting the candidate choose something they're interested in talking about and presumably familiar with. I find that people seem more relaxed when they get into the flow of describing something they're interested in, which lets me explore various aspects of their working style with as little stress to them as possible. Job interviews do suck, but the alternatives I've seen proposed suck at least as bad, though in different ways.

tptacek 13y ago
In our case? Call it ~6 hours, over the course of several (potentially discontiguous) days, offsite and at the candidate's leisure. Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE). We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too. Some other things to know about that guy: * No prior crypto experience * No prior software security experience * Less than 2.5 years dev experience across both jobs * From flyover country * Physically uncomfortable during the interview (something noted by interviewers on his feedback) How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess. Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf. Yes, I think someone is going to make a lot of money correcting that market inefficiency. † If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.
- saturdayplace 13y ago
  If this comment shows up hereafter on every "developers need passion" and hiring thread, I'll up vote it. every. single. time. Alternately, a blog post expanding on the thesis, if we could draft you to write it...
- ScottBurson 13y ago
  Wait -- there are two questions here. One is whether developers need passion, and the other is whether an interview process is a good way to tell whether they have it (and the other qualities required -- I haven't seen anyone say that passion is sufficient). I think most of us are speaking to the former -- I certainly have been. I agree that work sample tests can be quite valuable.

by jleader 13y ago

tptacek 13y ago

In our case? Call it ~6 hours, over the course of several (potentially discontiguous) days, offsite and at the candidate's leisure. Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE). We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too. Some other things to know about that guy: * No prior crypto experience * No prior software security experience * Less than 2.5 years dev experience across both jobs * From flyover country * Physically uncomfortable during the interview (something noted by interviewers on his feedback) How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess. Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf. Yes, I think someone is going to make a lot of money correcting that market inefficiency. † If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.

saturdayplace 13y ago
If this comment shows up hereafter on every "developers need passion" and hiring thread, I'll up vote it. every. single. time. Alternately, a blog post expanding on the thesis, if we could draft you to write it...
ScottBurson 13y ago
Wait -- there are two questions here. One is whether developers need passion, and the other is whether an interview process is a good way to tell whether they have it (and the other qualities required -- I haven't seen anyone say that passion is sufficient). I think most of us are speaking to the former -- I certainly have been. I agree that work sample tests can be quite valuable.