3 ms·

But this takes far too long.

tptacek 11y ago
It saved us enormous amounts of time at Matasano. When we designed our work-sample rubric, our #1 concern was filtering out people who interviewed well and/or had great resumes but weren't worth a billable hour on a real project, but our #2 concern was interviewing faster and getting results back to candidates quickly. (We didn't discover the best reason to do work-samples --- discovering hidden talent --- until after we'd started doing them). With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.
- rorykoehler 11y ago
  Did you ever consider the candidate in all of this?
  - tptacek 11y ago
    Yes: like I said, the goal was to get results back to candidates faster. The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us). After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week. Also, see: https://news.ycombinator.com/item?id=11328090 https://news.ycombinator.com/item?id=11328090 The process I'm advocating for is far better for candidates.
    - marktd 11y ago
      As a candidate, I would be a little wary of this process. You're asking me to put in multiple hours of work, but you only put in 7 minutes. So the incentives are misaligned: you are incented to give these challenges to many people, even if they have a low chance of passing through. But as a candidate, I don't want to sped multiple hours if I have a low chance of passing through. This is one reason why full-day on-site interviews aren't so bad - if I've gotten to that stage, I'm probably pretty likely to get an offer. What do you do to ameliorate this concern?
      - tptacek 11y ago
        First, I don't concede the idea that this is a concern I need to ameliorate, because the work-sample process consumes fewer hours of candidate time than the conventional interview does, and, better still, consumes those times as, when, and where the candidate chooses to make them available: an hour a night during the week, say. Do the work from your favorite quiet bar. Do it during your coffee break. I am spending a lot of time these days talking to people interviewing in the valley, and what I'm seeing is that the norm candidates are subjected to is 7+ hours of 6-7 on-site interviews. Candidates have to go through all the interviews, even if the first interview has effectively ejected them from the process. Compared to that horrid process, I don't believe I have to justify anything about my process. But, if you read downthread, you'll see that we in fact did a lot to ameliorate the (bogus, I think) concern that we were incentivized to soak up hopeless effort from lots of people.
      - chrisbennet 11y ago
        I was thinking something similar. When one party can waste the other party's time at little cost to their own, the situation can be abused. If a job ad wants you to submit to a test before you even talk to a hiring manager, this is a signal that the employer doesn't care about wasting your time. I can see the advantages to automating the hiring process but as a candidate I am less inclined to engage with a party that has no "skin in the game". Think of the design world: Potential client "A" asks multiple desperate artists to work on spec in hopes that they will get the commission. Potential client "B" call you up and talks to you, sends you some napkin sketches and generally engages with you for an hour before asking you to do a design. Both clients want you to do a design (test) but one clearly doesn't have any skin in the game. Assuming you're a decent designer, which client do you respond to?
- ArkyBeagle 11y ago
  Aha! I sort of like that, then. Assuming of course that it actually does work. Back when things were simpler, we had a multiple guess questionnaire that did the similar. For onsites, we then looked for evidence of rigor and thoroughness, the ability to estimate how much communication was required, had they deployed real systems? You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion.
  - e15ctr0n 11y ago
    > You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion. Announcing Starfighter https://news.ycombinator.com/item?id=9173939 https://news.ycombinator.com/item?id=9173939
    - tptacek 11y ago
      I am talking my book in the sense that my business works a lot better if companies get saner about how they hire people, but Starfighter is not the commercialization of the idea I'm talking about here.

It saved us enormous amounts of time at Matasano. When we designed our work-sample rubric, our #1 concern was filtering out people who interviewed well and/or had great resumes but weren't worth a billable hour on a real project, but our #2 concern was interviewing faster and getting results back to candidates quickly. (We didn't discover the best reason to do work-samples --- discovering hidden talent --- until after we'd started doing them). With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.

rorykoehler 11y ago
Did you ever consider the candidate in all of this?
- tptacek 11y ago
  Yes: like I said, the goal was to get results back to candidates faster. The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us). After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week. Also, see: https://news.ycombinator.com/item?id=11328090 https://news.ycombinator.com/item?id=11328090 The process I'm advocating for is far better for candidates.
  - marktd 11y ago
    As a candidate, I would be a little wary of this process. You're asking me to put in multiple hours of work, but you only put in 7 minutes. So the incentives are misaligned: you are incented to give these challenges to many people, even if they have a low chance of passing through. But as a candidate, I don't want to sped multiple hours if I have a low chance of passing through. This is one reason why full-day on-site interviews aren't so bad - if I've gotten to that stage, I'm probably pretty likely to get an offer. What do you do to ameliorate this concern?
    - tptacek 11y ago
      First, I don't concede the idea that this is a concern I need to ameliorate, because the work-sample process consumes fewer hours of candidate time than the conventional interview does, and, better still, consumes those times as, when, and where the candidate chooses to make them available: an hour a night during the week, say. Do the work from your favorite quiet bar. Do it during your coffee break. I am spending a lot of time these days talking to people interviewing in the valley, and what I'm seeing is that the norm candidates are subjected to is 7+ hours of 6-7 on-site interviews. Candidates have to go through all the interviews, even if the first interview has effectively ejected them from the process. Compared to that horrid process, I don't believe I have to justify anything about my process. But, if you read downthread, you'll see that we in fact did a lot to ameliorate the (bogus, I think) concern that we were incentivized to soak up hopeless effort from lots of people.
    - chrisbennet 11y ago
      I was thinking something similar. When one party can waste the other party's time at little cost to their own, the situation can be abused. If a job ad wants you to submit to a test before you even talk to a hiring manager, this is a signal that the employer doesn't care about wasting your time. I can see the advantages to automating the hiring process but as a candidate I am less inclined to engage with a party that has no "skin in the game". Think of the design world: Potential client "A" asks multiple desperate artists to work on spec in hopes that they will get the commission. Potential client "B" call you up and talks to you, sends you some napkin sketches and generally engages with you for an hour before asking you to do a design. Both clients want you to do a design (test) but one clearly doesn't have any skin in the game. Assuming you're a decent designer, which client do you respond to?
ArkyBeagle 11y ago
Aha! I sort of like that, then. Assuming of course that it actually does work. Back when things were simpler, we had a multiple guess questionnaire that did the similar. For onsites, we then looked for evidence of rigor and thoroughness, the ability to estimate how much communication was required, had they deployed real systems? You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion.
- e15ctr0n 11y ago
  > You understand that it sounds like you "fixed" hiring, and so we're all sorta skeptical, right? If you could bottle that, there's your billion. Announcing Starfighter https://news.ycombinator.com/item?id=9173939 https://news.ycombinator.com/item?id=9173939
  - tptacek 11y ago
    I am talking my book in the sense that my business works a lot better if companies get saner about how they hire people, but Starfighter is not the commercialization of the idea I'm talking about here.