3 ms·

Well, I “can” perform these so called “work sample tests” that a lot of companies want to give out (sometimes even before getting to speak with anyone at the co

Well, I “can” perform these so called “work sample tests” that a lot of companies want to give out (sometimes even before getting to speak with anyone at the company). Here’s my issue with them: * Either they’re strictly timed, introducing an artificial sort of pressure that’s rarely present on the job, or one ends up competing against people who spend 3 or 4 times the suggested amount of time. I don’t know about you, but I can do a way better job on most of these sorts of tasks if I spend a large multiple of the expected time on it, too. * The tasks are often poorly specified, sometimes with the justification that it tests the ability to clarify requirements. I don’t think that argument holds water, because all “clarifying requirements” does when one doesn’t have ready access to folks who can answer those kinds of questions is introduce insane amounts of latency and frustration into the process. * Even when the tasks are reasonably well specified, the grading rubric is often kept from the candidate. At work, if I don’t know if I or my team is building the right thing or doing it in the right way, I can go ask people (PM, tech lead, EM, another team member, etc.) * Finally, most of these “take home assignments” I’ve seen that are supposed to be in the 2-4 hour range to finish don’t get me as a candidate out of 2-4 hours of giving impromptu algorithms lectures at the whiteboard. If the prize for putting in 2-6x as much time as I would to get to an onsite interview with another company is just to come onsite with your company and spend all day doing the exact same thing, then, no thank you. I realize that few or none of these things apply to the process at Matasano, but there is another one that does: I can already get a job that pays reasonably well, doing things I already mostly know how to do, without spending hours studying a field I know little about. That is arguably a feature and not a bug, so, I don’t mention it as a criticism, but rather as a reason why I probably wouldn’t do Matasano’s work sample test. For the type of people they’re looking for, that’s probably fine. It’s just that I’m not one of those people.

tptacek 7y ago
This is all well said. I wonder if you have thoughts on what employers might do about the "competition with completists taking 2 days on a 2 hour challenge" versus "pressure from timing people". We've taken the tack of telling people what our timing expectations are, but also not timing or tracking time (you could ostensibly have picked up the problem on a Monday, noodled for 15 minutes, decided you'd do a better job on Wednesday, picked it back up for 45 minutes, had a sudden business call, and then had to wait until Saturday to finish it; right now, we'd like that to just seamlessly work, but it obviously creates the effect you're talking about).
- pmiller2 7y ago
  If you’re not going to track time in any way, then there’s no way to prevent anyone spending as much time on the problem as they have available. My best suggestion is to just ask people how much time they spent on it. The biggest problem with that is that by telling people the time expectations, you’ll probably get answers within or near that range. I had one of these tests say they wanted the project delivered as a git repo and would use the timestamps on the commit history to figure out how much time was spent. I just laughed at that and figured the people who knew how to forge the timestamps would do that to make themselves look good. The only alternative I can think of to explicitly tracking time is to just not give a time expectation and ask how much time they spent. That way, answers aren’t biased by you anchoring a range in their mind. This has other obvious disadvantages, but it would take care of the “candidate spent 2 days on a 2 hour project” problem.

tptacek 7y ago

This is all well said. I wonder if you have thoughts on what employers might do about the "competition with completists taking 2 days on a 2 hour challenge" versus "pressure from timing people". We've taken the tack of telling people what our timing expectations are, but also not timing or tracking time (you could ostensibly have picked up the problem on a Monday, noodled for 15 minutes, decided you'd do a better job on Wednesday, picked it back up for 45 minutes, had a sudden business call, and then had to wait until Saturday to finish it; right now, we'd like that to just seamlessly work, but it obviously creates the effect you're talking about).

pmiller2 7y ago
If you’re not going to track time in any way, then there’s no way to prevent anyone spending as much time on the problem as they have available. My best suggestion is to just ask people how much time they spent on it. The biggest problem with that is that by telling people the time expectations, you’ll probably get answers within or near that range. I had one of these tests say they wanted the project delivered as a git repo and would use the timestamps on the commit history to figure out how much time was spent. I just laughed at that and figured the people who knew how to forge the timestamps would do that to make themselves look good. The only alternative I can think of to explicitly tracking time is to just not give a time expectation and ask how much time they spent. That way, answers aren’t biased by you anchoring a range in their mind. This has other obvious disadvantages, but it would take care of the “candidate spent 2 days on a 2 hour project” problem.