3 ms·

So, there's a step missing here. What you need going into this interview is a solid understanding of your corporate culture -- what behaviors you value. Maybe

So, there's a step missing here. What you need going into this interview is a solid understanding of your corporate culture -- what behaviors you value. Maybe you work for a bank, and the most important things for you are integrity and conservatism. Or, maybe you work for an early stage start-up, and calculated risk-taking and team spirit are most important. In either case, as you walk through these questions, you ask about the how's and why's of the person's behavior. Doing so is the best way to figure out if the person is going to act in ways that you would want and expect them to. Again, the significance of the biggest thing itself is small, it's how and why they achieved it, gleaned through the 20-odd follow-up questions, that will get you to the answer. You mention college hiring -- yes, that's a different beast. A lot of companies have success with paid internships the summer before graduation. You can also ask some of these behavioral questions, but unfortunately our educational system doesn't include enough collaborative, project-based work to get a good baseline. Nonetheless, a college student who has done significant summer project work, or who has worked on an extracurricular project, should have some data to extract.

tptacek 13y ago
No, asking questions in a face-to-face interview is not the best way to figure out if a person is going to do what you want them to. It is not a very effective way of doing that at all. To be meaningful at all, it requires an interviewer with many of the same skills as a psychologist, and a candidate who can represent themselves effectively in an extraordinarily hostile situation. Even if you have both, you're still not even in the vicinity of the power of a work-sample test. Job interviews suck, and I feel like people who have been hiring in this industry for a long time recognize at some level that they're essentially random functions, made workable only by dev teams willingness to put up with employee churn and extreme inefficiency.
- jleader 13y ago
  How long does a work-sample test take? I'm typically hiring people to work on on-going projects over a period of months or years; the fact that candidate A does better on a 2-hour or 2-day task than candidate B may not give me that much information about things like how effective they will be at sharing their own knowledge with the rest of the team (or taking guidance from more senior members of the team), developing a good working rapport with their QA teammates so they can get to the root of any problems as efficiently as possible, convincing the sysadmins that the candidate's proposed deployment tool is the right way to do it (or gracefully agreeing to use the sysadmins' tool of choice, either because it's actually better, or just because the benefit of the sysadmins' familiarity with it outweighs any technical benefits of the other tool), etc. Unless I'm hiring someone to sit off in a corner developing their own solo project, I'm not convinced that "work sample" tests are that informative. I don't agree that this question is a silver bullet, but I think it's valuable to see what sort of achievment the candidate considers significant, to ask follow-up questions about how they handled problems along the way to that achievment, etc. The question helps reduce the power assymetry of the interview, by letting the candidate choose something they're interested in talking about and presumably familiar with. I find that people seem more relaxed when they get into the flow of describing something they're interested in, which lets me explore various aspects of their working style with as little stress to them as possible. Job interviews do suck, but the alternatives I've seen proposed suck at least as bad, though in different ways.
  - tptacek 13y ago
    In our case? Call it ~6 hours, over the course of several (potentially discontiguous) days, offsite and at the candidate's leisure. Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE). We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too. Some other things to know about that guy: * No prior crypto experience * No prior software security experience * Less than 2.5 years dev experience across both jobs * From flyover country * Physically uncomfortable during the interview (something noted by interviewers on his feedback) How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess. Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf. Yes, I think someone is going to make a lot of money correcting that market inefficiency. † If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.
    - 13y ago

tptacek 13y ago

No, asking questions in a face-to-face interview is not the best way to figure out if a person is going to do what you want them to. It is not a very effective way of doing that at all. To be meaningful at all, it requires an interviewer with many of the same skills as a psychologist, and a candidate who can represent themselves effectively in an extraordinarily hostile situation. Even if you have both, you're still not even in the vicinity of the power of a work-sample test. Job interviews suck, and I feel like people who have been hiring in this industry for a long time recognize at some level that they're essentially random functions, made workable only by dev teams willingness to put up with employee churn and extreme inefficiency.

jleader 13y ago
How long does a work-sample test take? I'm typically hiring people to work on on-going projects over a period of months or years; the fact that candidate A does better on a 2-hour or 2-day task than candidate B may not give me that much information about things like how effective they will be at sharing their own knowledge with the rest of the team (or taking guidance from more senior members of the team), developing a good working rapport with their QA teammates so they can get to the root of any problems as efficiently as possible, convincing the sysadmins that the candidate's proposed deployment tool is the right way to do it (or gracefully agreeing to use the sysadmins' tool of choice, either because it's actually better, or just because the benefit of the sysadmins' familiarity with it outweighs any technical benefits of the other tool), etc. Unless I'm hiring someone to sit off in a corner developing their own solo project, I'm not convinced that "work sample" tests are that informative. I don't agree that this question is a silver bullet, but I think it's valuable to see what sort of achievment the candidate considers significant, to ask follow-up questions about how they handled problems along the way to that achievment, etc. The question helps reduce the power assymetry of the interview, by letting the candidate choose something they're interested in talking about and presumably familiar with. I find that people seem more relaxed when they get into the flow of describing something they're interested in, which lets me explore various aspects of their working style with as little stress to them as possible. Job interviews do suck, but the alternatives I've seen proposed suck at least as bad, though in different ways.
- tptacek 13y ago
  In our case? Call it ~6 hours, over the course of several (potentially discontiguous) days, offsite and at the candidate's leisure. Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE). We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too. Some other things to know about that guy: * No prior crypto experience * No prior software security experience * Less than 2.5 years dev experience across both jobs * From flyover country * Physically uncomfortable during the interview (something noted by interviewers on his feedback) How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess. Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf. Yes, I think someone is going to make a lot of money correcting that market inefficiency. † If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.
  - 13y ago