10 ms·

> All our technical hires, whether in engineering or product management, go through a work sample test of sorts, where they are asked to solve engineering probl

> All our technical hires, whether in engineering or product management, go through a work sample test of sorts, where they are asked to solve engineering problems during the interview. I disagree that a whiteboard session can be considered a work sample test. Whats wrong with 'complete steps 1,2 & 3 and i'll be back in 30 minutes'? For engineering we actually found giving a 2-3 hr 1 problem test has been a better predictor, and 3-4 30 minute face to face (usually 2 on skype prior to onsite Test, then 2 follow ups). Alone time gives the candidate time to digest and solve the problem on their own which is more 'real world' like and shows what can be done on their own. Some questions are intentionally vague to compensate for 'able to make decisions with little information'. We have no right answer but will ask why you made these decisions. This frees up our staff time tremendously and weeds out those that may need a lot of hand holding early on.

tptacek 12y ago
This is exactly correct. A work-sample test needs to capture the normal parameters of the work. If team members aren't routinely called on to solve programming problems in a high-stakes stick-the-landing-or-you're-fired exercise on a whiteboard in front of an audience, then the prediction the test is trying to make is confounded by all those factors. * Eliminate live audiences * Let people work in the environment they'll be able to choose on the job * Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job * If you're worried about cheating, build that assessment into your in-person followup interview
- Estragon 12y ago
  How often did people cheat in remote work-sample tests?
  - tptacek 12y ago
    I never once saw it happen.
    - JonMR 12y ago
      I've seen it several dozen times. A solution to one of our sample tests made it out to Github and that was all she wrote. For a while we kept the same sample test as a honeypot, but disqualifying half of the candidates for cheating was tiring. I still agree it's the best approach.
      - endtime 12y ago
        > disqualifying half of the candidates for cheating was tiring. That actually sounds awesome. You could reduce your interview evaluation overhead by nearly 50%!
    - buro9 12y ago
      I helped a friend cheat. Originally he'd simply asked if I could be there whilst he did the test so that he could talk aloud and bounce ideas off of me as it was timed and he gets nervous in tests and was afraid he'd not think clearly. He froze. Totally. I took over and did the test for him, and he aced it and was offered the job at the top salary band. He is a perfectly good engineer and the company were very satisfied with their hire, but he never did that test. The test in it's entirety was completed by myself. I cannot imagine this is such a rare thing with remote technical tests.
      - tptacek 12y ago
        I believe both of you, but our process did/does nothing to overtly catch cheaters, we hired directly off its conclusions, we hired at a rate faster than most VC-funded cash-flow-positive YC companies, including in SFBA, and we never let anyone go (nor did anyone ever quit while I was there) once we made an offer. (Matasano is also not a company where it's easy to duck attention and coast; the tempo is 2-3 week engagements that wrap up with metrics that everyone cares deeply about). The conclusion I draw is that cheating just isn't as big an issue as people think it is.
pXMzR2A 12y ago
> weeds out those that may need a lot of hand holding early on. i.e. You are using interviews in order to avoid paid in-house training.
- ryen 12y ago
  True, but theres a difference between paid in-house training and being able to investigate problems on your own without constantly bothering everyone on the team to do your thinking. The latter is a candidate I try to avoid like the plague.
r00fus 12y ago
I'd agree with everything but "alone". While having an interviewer (or more than one) staring at you while you solve a problem is unnerving, I'd also expect to be able to elicit requirements during the exercise, either before, during or near completion - because that's how it has always happened on my past assignments. Do you let your interviewees reach out (possibly via IM) to your interviewers with specific questions?
- bengali3 12y ago
  Yes, we all have skype on our machines and I show them how to dial my phone extension if they need anything. Edit: also for clarification, we discuss the problem ahead of time to make sure they understand it and ask any questions. For my group it's always a basic full stack problem. Here's a mockup and a basic install of visual studio and MSSQL, show me you can CRUD some data and present a decent UI. then we'll talk about how you did it.
Consultant32452 12y ago
Meh. IT unemployment has been about 2% for YEARS. I can get a good job on a team I'll enjoy working with in less than a week. If you're a top tier IT name then it might be worth jumping through hoops, but I will wind up at companies with the least hassle in the hiring department. 15 years in the industry has told me that there's no strong correlation between hiring practices and team quality.
- IndianAstronaut 12y ago
  Precisely why I hit the back button if my employment application is through something like Taleo. My time is valuable, too valuable to waste on a resume rabbit hole.
  - bengali3 12y ago
    actually, that brings up a really great point. This could be an interesting article. Considering how much the web focuses on 'conversion of web leads' and making the entire process seamless, you'd think you could quantify the candidates lost (and put a dollar figure on it) simply because the interface wasn't friendly and turned away candidates. However, all too often i feel that the goal for everyone involved but the hiring manager, is a seat filled. Anecdotally, I was prompted 'Zip code required' on a multinational company talent portal. Tried to fix and submit for at least 60 seconds before finally had to modify the CSS to show the input so I could put in a value, as Zip input was set to display:none. I figured at least there would be less competition for the role.
    - grrowl 12y ago
      That's a killer idea for a front-end developer test.
      - dennisgorelik 12y ago
        Front-end developer may not want to work in the place that makes such silly mistakes.

tptacek 12y ago

This is exactly correct. A work-sample test needs to capture the normal parameters of the work. If team members aren't routinely called on to solve programming problems in a high-stakes stick-the-landing-or-you're-fired exercise on a whiteboard in front of an audience, then the prediction the test is trying to make is confounded by all those factors. * Eliminate live audiences * Let people work in the environment they'll be able to choose on the job * Ideally, let people work in their own comfortable environs, even if they won't be able to do that on the job * If you're worried about cheating, build that assessment into your in-person followup interview

Estragon 12y ago
How often did people cheat in remote work-sample tests?
- tptacek 12y ago
  I never once saw it happen.
  - JonMR 12y ago
    I've seen it several dozen times. A solution to one of our sample tests made it out to Github and that was all she wrote. For a while we kept the same sample test as a honeypot, but disqualifying half of the candidates for cheating was tiring. I still agree it's the best approach.
    - endtime 12y ago
      > disqualifying half of the candidates for cheating was tiring. That actually sounds awesome. You could reduce your interview evaluation overhead by nearly 50%!
  - buro9 12y ago
    I helped a friend cheat. Originally he'd simply asked if I could be there whilst he did the test so that he could talk aloud and bounce ideas off of me as it was timed and he gets nervous in tests and was afraid he'd not think clearly. He froze. Totally. I took over and did the test for him, and he aced it and was offered the job at the top salary band. He is a perfectly good engineer and the company were very satisfied with their hire, but he never did that test. The test in it's entirety was completed by myself. I cannot imagine this is such a rare thing with remote technical tests.
    - tptacek 12y ago
      I believe both of you, but our process did/does nothing to overtly catch cheaters, we hired directly off its conclusions, we hired at a rate faster than most VC-funded cash-flow-positive YC companies, including in SFBA, and we never let anyone go (nor did anyone ever quit while I was there) once we made an offer. (Matasano is also not a company where it's easy to duck attention and coast; the tempo is 2-3 week engagements that wrap up with metrics that everyone cares deeply about). The conclusion I draw is that cheating just isn't as big an issue as people think it is.

pXMzR2A 12y ago

> weeds out those that may need a lot of hand holding early on. i.e. You are using interviews in order to avoid paid in-house training.

ryen 12y ago
True, but theres a difference between paid in-house training and being able to investigate problems on your own without constantly bothering everyone on the team to do your thinking. The latter is a candidate I try to avoid like the plague.

r00fus 12y ago

I'd agree with everything but "alone". While having an interviewer (or more than one) staring at you while you solve a problem is unnerving, I'd also expect to be able to elicit requirements during the exercise, either before, during or near completion - because that's how it has always happened on my past assignments. Do you let your interviewees reach out (possibly via IM) to your interviewers with specific questions?

bengali3 12y ago
Yes, we all have skype on our machines and I show them how to dial my phone extension if they need anything. Edit: also for clarification, we discuss the problem ahead of time to make sure they understand it and ask any questions. For my group it's always a basic full stack problem. Here's a mockup and a basic install of visual studio and MSSQL, show me you can CRUD some data and present a decent UI. then we'll talk about how you did it.

Consultant32452 12y ago

Meh. IT unemployment has been about 2% for YEARS. I can get a good job on a team I'll enjoy working with in less than a week. If you're a top tier IT name then it might be worth jumping through hoops, but I will wind up at companies with the least hassle in the hiring department. 15 years in the industry has told me that there's no strong correlation between hiring practices and team quality.

IndianAstronaut 12y ago
Precisely why I hit the back button if my employment application is through something like Taleo. My time is valuable, too valuable to waste on a resume rabbit hole.
- bengali3 12y ago
  actually, that brings up a really great point. This could be an interesting article. Considering how much the web focuses on 'conversion of web leads' and making the entire process seamless, you'd think you could quantify the candidates lost (and put a dollar figure on it) simply because the interface wasn't friendly and turned away candidates. However, all too often i feel that the goal for everyone involved but the hiring manager, is a seat filled. Anecdotally, I was prompted 'Zip code required' on a multinational company talent portal. Tried to fix and submit for at least 60 seconds before finally had to modify the CSS to show the input so I could put in a value, as Zip input was set to display:none. I figured at least there would be less competition for the role.
  - grrowl 12y ago
    That's a killer idea for a front-end developer test.
    - dennisgorelik 12y ago
      Front-end developer may not want to work in the place that makes such silly mistakes.