5 ms·

We went through the phase when we gave candidates a problem and let them work on it remotely. It was in an embedded C shop that did a lot of kernel work. Basica

by eps 12y ago

We went through the phase when we gave candidates a problem and let them work on it remotely. It was in an embedded C shop that did a lot of kernel work. Basically, we'd give a short programming task (say, to write an intrusive AVL tree container in C) and 24 hours. Guess what? HALF of candidates cheated. Meaning that when the got called for an in-person interview, they stumbled to explain how "their" code worked. To say that I was shocked is an understatement. It's bloody obvious that such blatant cheating would surface within the first week after hiring and yet they cheated :-| In the end we had to switch to the in-person interviews only and implement few other things to discourage cheating. In other words - YMMV, what works for others may misfire miserably for you.

deleted 12y ago
[deleted]
meowface 12y ago
Yeah, that's the only real problem with remote work sample tests. Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).
- tptacek 12y ago
  Actually, I think the opposite is true; in appsec, the final work product is a list of vulns, so if you want to cheat, it only takes 4 minutes talking to anyone else who has ever succeeded in the process before. For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on. For my part: we ran this process for over two years and never discovered any plagiarism. Meanwhile, we drastically increased the size of our team and had total retention; from the time I took over recruiting to the time I left the firm, we lost a total of zero of those hires. All of them worked out.
  - meowface 12y ago
    You're right that it's definitely pretty easy to cheat if you have insider knowledge of some sort, but the odds of a candidate leaking the answers are probably pretty low unless you're a huge company and don't change up your tests.
  - lostcolony 12y ago
    The problem I'd see with using your own app and taking out some functionality, is that the first few people you throw at it, you have no data points to verify against. I.e., they do it 'wrong' (it's incomplete, it's not what you meant, whatever). What does that mean? They don't have the domain knowledge your team has, and that may be important. I'd think better would be to have a few of your devs come up with a small app, and set of instructions, that you pass to others of your devs. They can then create a reasonable baseline you can start to measure against. You may need to be a little forgiving initially, if it's not a completely made up example, but it probably is more valid than something that was built for production, with months of domain knowledge behind it.
    - rhizome 12y ago
      I've long thought that to come up with a good work-product test a company should continually save off their own bugs to put into a broken test app, after which candidates can fix the bugs and the company can see how candidates solve the problems compared to the people they've already hired. The actual production code is the baseline.
deleted 12y ago
[deleted]
Jemaclus 12y ago
I'm not sure the conclusion here is that remote tests are bad. It's also possible that your screening methods didn't do a good enough job. (As a disclaimer, I'm a terrible interviewer, but...) I believe that by the time a candidate gets into a room with me, that my team and I should have a good idea about whether this person will work out or not. If I make them take time out of their day, and I take time out of my day, and then 10 minutes into it I realize they don't know what they're talking about, I see that as my screw-up. I shouldn't have let them get this far. You know what I mean? Right now at my job, we don't do a good enough job of this. I get candidates that walk through the door that can't code more than "Hello World!" without screwing up. I'm not happy about that. This post, I think, enumerates a large number of things that you can do to determine worthiness before they even walk through the door -- the remote test is only a small part of it. If all you're doing is a remote test, then maybe you need to do more, yes?
- bduerst 12y ago
  I honestly believe that the recruiting/hiring market is ripe to be flipped on it's head for these reasons. I just can't think of any way to do it that can scale though. There are the hired.com models where independent experts thoroughly vet the candidate, but that involves a lot of manual interaction.
  - Jemaclus 12y ago
    My big problem with Hired.com and similar sites is that there's no incentive for anyone to be honest. I have no incentive to say "not interested" to anyone, and they have no incentive to back up the salary they post. It's the equivalent of the "always swipe right" strategy on Tinder -- it's in your best interest to say "maybe" or "interested" to everything, because otherwise you don't know if you're missing out on opportunities. What I really want as a hiring manager is for someone to show up and say "Here's Candidate A. We think they're strong because X, Y, Z, and here's the proof." Someone whose mission is to honestly find me the best candidate, who isn't just trying to place someone as fast as possible so they get their commission. Most recruiters I know are interested in placing a good candidate as quickly as possible, but there are a lot of problems with this. "Good" isn't "best", and often times the recruiters don't know the difference between Java and Javascript, and I wind up with junk resumes in my inbox. So how do we fix this? I have no idea. But there's gotta be a way. I don't think that way is going to be scalable with a simple web application. I think you have to have a secret weapon that nobody else has, and maybe that's just a group of in-industry professionals that vet candidates on your behalf, or maybe it's some incredible machine learning algorithm, or maybe it's something else entirely. As a job seeker, I want to get the best job I can, and as a boss, I want the best employees I can get. And I don't think those things are mutually exclusive at all.
dilap 12y ago
Huh. That's crazy to me. I worked at a company that had a remote, 3 day test. It was pretty fun, and I still sometimes try out things on it to see how they work (it was a two-parter, with part 1 being to write an evaluation function, and part 2 solve a NP-hard search problem). I strongly feel that the information I gained from looking at the code candidates wrote for that problem told me lots of really good info. As far as I know, nobody cheated. (All submitted code as unique, and everybody could talk cogently about what they did. Some people did great things, some people terrible, and lots in the middle.) In conclusion, I would strongly recommend a "take home" test. Maybe give a bit longer than 24 hours. And make sure it's the same problem all the time, so you have a point of comparison.
anonymousab 12y ago
Sounds like a rousing success in terms of qualifying/disqualifying candidates.
__mp 12y ago
Embedded C and AVL trees? Really? Could you explain how an AVL tree would be used in an embedded context?
- eps 12y ago
  It's a C test.
lsc 12y ago
eh, I think it's actually pretty easy to tell if a person wrote a particular piece of code last night. So cheating is pretty easy to ferret out. Because of this, I think the take-home test conveys a lot of useful information.

by eps 12y ago

deleted 12y ago

[deleted]

meowface 12y ago

Yeah, that's the only real problem with remote work sample tests. Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).

tptacek 12y ago
Actually, I think the opposite is true; in appsec, the final work product is a list of vulns, so if you want to cheat, it only takes 4 minutes talking to anyone else who has ever succeeded in the process before. For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on. For my part: we ran this process for over two years and never discovered any plagiarism. Meanwhile, we drastically increased the size of our team and had total retention; from the time I took over recruiting to the time I left the firm, we lost a total of zero of those hires. All of them worked out.
- meowface 12y ago
  You're right that it's definitely pretty easy to cheat if you have insider knowledge of some sort, but the odds of a candidate leaking the answers are probably pretty low unless you're a huge company and don't change up your tests.
- lostcolony 12y ago
  The problem I'd see with using your own app and taking out some functionality, is that the first few people you throw at it, you have no data points to verify against. I.e., they do it 'wrong' (it's incomplete, it's not what you meant, whatever). What does that mean? They don't have the domain knowledge your team has, and that may be important. I'd think better would be to have a few of your devs come up with a small app, and set of instructions, that you pass to others of your devs. They can then create a reasonable baseline you can start to measure against. You may need to be a little forgiving initially, if it's not a completely made up example, but it probably is more valid than something that was built for production, with months of domain knowledge behind it.
  - rhizome 12y ago
    I've long thought that to come up with a good work-product test a company should continually save off their own bugs to put into a broken test app, after which candidates can fix the bugs and the company can see how candidates solve the problems compared to the people they've already hired. The actual production code is the baseline.

deleted 12y ago

[deleted]

Jemaclus 12y ago

I'm not sure the conclusion here is that remote tests are bad. It's also possible that your screening methods didn't do a good enough job. (As a disclaimer, I'm a terrible interviewer, but...) I believe that by the time a candidate gets into a room with me, that my team and I should have a good idea about whether this person will work out or not. If I make them take time out of their day, and I take time out of my day, and then 10 minutes into it I realize they don't know what they're talking about, I see that as my screw-up. I shouldn't have let them get this far. You know what I mean? Right now at my job, we don't do a good enough job of this. I get candidates that walk through the door that can't code more than "Hello World!" without screwing up. I'm not happy about that. This post, I think, enumerates a large number of things that you can do to determine worthiness before they even walk through the door -- the remote test is only a small part of it. If all you're doing is a remote test, then maybe you need to do more, yes?

bduerst 12y ago
I honestly believe that the recruiting/hiring market is ripe to be flipped on it's head for these reasons. I just can't think of any way to do it that can scale though. There are the hired.com models where independent experts thoroughly vet the candidate, but that involves a lot of manual interaction.
- Jemaclus 12y ago
  My big problem with Hired.com and similar sites is that there's no incentive for anyone to be honest. I have no incentive to say "not interested" to anyone, and they have no incentive to back up the salary they post. It's the equivalent of the "always swipe right" strategy on Tinder -- it's in your best interest to say "maybe" or "interested" to everything, because otherwise you don't know if you're missing out on opportunities. What I really want as a hiring manager is for someone to show up and say "Here's Candidate A. We think they're strong because X, Y, Z, and here's the proof." Someone whose mission is to honestly find me the best candidate, who isn't just trying to place someone as fast as possible so they get their commission. Most recruiters I know are interested in placing a good candidate as quickly as possible, but there are a lot of problems with this. "Good" isn't "best", and often times the recruiters don't know the difference between Java and Javascript, and I wind up with junk resumes in my inbox. So how do we fix this? I have no idea. But there's gotta be a way. I don't think that way is going to be scalable with a simple web application. I think you have to have a secret weapon that nobody else has, and maybe that's just a group of in-industry professionals that vet candidates on your behalf, or maybe it's some incredible machine learning algorithm, or maybe it's something else entirely. As a job seeker, I want to get the best job I can, and as a boss, I want the best employees I can get. And I don't think those things are mutually exclusive at all.

dilap 12y ago

Huh. That's crazy to me. I worked at a company that had a remote, 3 day test. It was pretty fun, and I still sometimes try out things on it to see how they work (it was a two-parter, with part 1 being to write an evaluation function, and part 2 solve a NP-hard search problem). I strongly feel that the information I gained from looking at the code candidates wrote for that problem told me lots of really good info. As far as I know, nobody cheated. (All submitted code as unique, and everybody could talk cogently about what they did. Some people did great things, some people terrible, and lots in the middle.) In conclusion, I would strongly recommend a "take home" test. Maybe give a bit longer than 24 hours. And make sure it's the same problem all the time, so you have a point of comparison.

anonymousab 12y ago

Sounds like a rousing success in terms of qualifying/disqualifying candidates.

__mp 12y ago

Embedded C and AVL trees? Really? Could you explain how an AVL tree would be used in an embedded context?

eps 12y ago
It's a C test.

lsc 12y ago

eh, I think it's actually pretty easy to tell if a person wrote a particular piece of code last night. So cheating is pretty easy to ferret out. Because of this, I think the take-home test conveys a lot of useful information.