6 ms·

We subscribe to the work sample test as your best option for technical validation, but we also limit ourselves to a small window of time, on site. We pick somet

We subscribe to the work sample test as your best option for technical validation, but we also limit ourselves to a small window of time, on site. We pick something we recently worked on, distill it down to something we can knock out in 15 minutes, and give the candidate about 45 minutes to work through the distilled problem and then we explore for about 15 minutes their solution and how they would make it production ready (as a conversation). We then have another 45 minutes of designing a solution to something more complex we recently had to work on, again distilled down. It involves a white board and boxes and arrows. The last 15 minutes is time for them to ask whatever they want of us. We have two interviewers on this technical portion to level out false reads by the other interviewer (I thought this part was a poor answer, but the other interviewer has a different view of it). I really like it so far. The part I'm battling on it is if the current coding part selects against Java developers. Part of the code we want right now requires a unit test with dependency injection to match an interface. So many of our Java candidates simply can't set up a running unit test. They are used to layers of framework already set up in the IDE and just clicking on stuff. They have full access to Google. Maybe it is good we are filtering out these candidates, but I'm not sure. Still thinking on it.

projektir 8y ago
> The part I'm battling on it is if the current coding part selects against Java developers. Yes, it does. And let me put it this way: I've used C# and Java at most of my jobs, those theoretically would be my comfort languages, yes? I do not use those languages on interviews. I often just use C++ (!), or Rust (free unit tests!) if the company tools allow for it, or worst case I'd learn some Python basics. C#/Java are very awkward and boilerplatey in such a small time frame as 45 minutes.
- deleted 8y ago
  [deleted]
- sethammons 8y ago
  Interesting. To clarify, you are saying that these boiler-platey languages, it is not fair to write a unit test in a 45 minute time block with access to Google? I'm not a Java guy. All the languages I've used, this would not be an issue. Again, I'm wrestling with it because it I feel it should be easy but like 80% of our candidates who chose Java struggle with it.
  - cookiecaper 8y ago
    It sounds like this is a pick-your-own-language type test. I'd suggest scoping down to one, or at most, two, simple allowable options that your team is already pretty comfortable handling. I've been on the reviewer side of a handful of "choose your own language"-style take-homes recently and found that it's really not good for the candidate if they actually end up using something that wouldn't have been the interview committee's first or second choice. There have been cases where choosing a less-trendy-but-still-totally-viable toolkit has effectively disqualified a candidate, with several committee members not even considering it necessary to look at the code. This is very unfair and lame but an unfortunate reality. I asked that the test be changed to constrain the options at least to a list that wouldn't be immediately disqualifying. You could advise the candidate ahead of the interview something along the lines of "you'll be asked to write a code sample in either Ruby or Python -- you'll have full access to Google, but you may want to brush up on the basics of these languages if you haven't used them recently". This does two things: first, it prevents the issue you have, where you're essentially not sure if you're correctly scoring the samples produced. Secondly, it really requires you to constrain the problem to things that someone who has barely used language X or Y can do within 45 minutes.
    - sethammons 8y ago
      The theory with pick your own language is they should be able to feel fully comfortable (interviews are already stressful enough). If they picked Rust, Scala, or a lisp dialect, (or anything the interviewers are unfamiliar with), it can even be a better interview because we get more insight on how the candidate communicates and their ability to walk someone else through their solution. A potential other bonus is less biases leak through from an interviewer on "that is a strange way to do that in language X."
      - cookiecaper 8y ago
        > The theory with pick your own language is they should be able to feel fully comfortable (interviews are already stressful enough). Ah, but there's the rub. Candidates are trying to please the interview panel. If you don't provide guidance, the odds that they'll just use whatever they think the interviewers most prefer are just as good, if not better, than the odds that they'll actually use whatever makes them most comfortable. You said yourself that since some candidates pick a language that you don't know well, you can't really tell if the failure of a large number of those candidates is reflective of a bad test or just a mismatched candidate pool. IMO, if you're going to stick to the "pick any language" thing, you should at least find out and ensure that any language the candidate picks will have a fair shot. > it can even be a better interview because we get more insight on how the candidate communicates and their ability to walk someone else through their solution. You can still get the candidate to communicate and explain his choices if you give an option: "either Ruby or Python" or "either JavaScript or Visual Basic", etc. The problem with having this happen in a language that the interviewers don't know reasonably well is that they are much more vulnerable to the smooth talker who can present incorrect information confidently, and they won't have enough background/anchoring in the subject matter to know the difference. > A potential other bonus is less biases leak through from an interviewer on "that is a strange way to do that in language X." I would say that if you're worried that interviewers will load in biases toward their preferred shortcuts etc in a specific language, that you should be equally worried that some good candidates are being excluded for choosing the "wrong" language in an any-language-goes test. Above, you mentioned that there'd be a positive response if a candidate used "Rust, Scala, or a lisp dialect" -- these are all relatively trendy. What if the candidate used nim, Pony, or some other language that hasn't pulled in to the hype superstation yet? What if the candidate used a language that's not-so-trendy anymore, like Visual Basic, Cobol, or bash? What if the candidate used a programming language of their own design, and brought a copy of the compiler with them on a flash drive? I'm asking because I've seen this in practice. Candidates for a devops position who chose to use bash to implement the very simple take-home task they were given were laughed off by several other members of the interview committee, despite being potentially high-value senior people -- they were at least senior enough that they're more comfortable performing sysadmin-style tasks in a shell, rather than using a massive CM framework or an awkward amalgamation of Python scripts running os.spawn. It feels like this type of thing happens a lot, in the same sense that very often, "unlimited PTO" just means "guess whatever amount of PTO is acceptable around here and hope you get it right".
tpxl 8y ago
>Part of the code we want right now requires a unit test with dependency injection to match an interface. What exactly do you mean by this?
- sethammons 8y ago
  We ask that the code they write have a unit test. The nature of that unit test is that it has a dependency. They can mock however they like, but passing in an object that represents the dependency (dependency injection) is the easiest and most straight forward way to do that. That object should have a method on it (a know method, with a known signature, also known in some circles as matching an interface).

projektir 8y ago

> The part I'm battling on it is if the current coding part selects against Java developers. Yes, it does. And let me put it this way: I've used C# and Java at most of my jobs, those theoretically would be my comfort languages, yes? I do not use those languages on interviews. I often just use C++ (!), or Rust (free unit tests!) if the company tools allow for it, or worst case I'd learn some Python basics. C#/Java are very awkward and boilerplatey in such a small time frame as 45 minutes.

deleted 8y ago
[deleted]
sethammons 8y ago
Interesting. To clarify, you are saying that these boiler-platey languages, it is not fair to write a unit test in a 45 minute time block with access to Google? I'm not a Java guy. All the languages I've used, this would not be an issue. Again, I'm wrestling with it because it I feel it should be easy but like 80% of our candidates who chose Java struggle with it.
- cookiecaper 8y ago
  It sounds like this is a pick-your-own-language type test. I'd suggest scoping down to one, or at most, two, simple allowable options that your team is already pretty comfortable handling. I've been on the reviewer side of a handful of "choose your own language"-style take-homes recently and found that it's really not good for the candidate if they actually end up using something that wouldn't have been the interview committee's first or second choice. There have been cases where choosing a less-trendy-but-still-totally-viable toolkit has effectively disqualified a candidate, with several committee members not even considering it necessary to look at the code. This is very unfair and lame but an unfortunate reality. I asked that the test be changed to constrain the options at least to a list that wouldn't be immediately disqualifying. You could advise the candidate ahead of the interview something along the lines of "you'll be asked to write a code sample in either Ruby or Python -- you'll have full access to Google, but you may want to brush up on the basics of these languages if you haven't used them recently". This does two things: first, it prevents the issue you have, where you're essentially not sure if you're correctly scoring the samples produced. Secondly, it really requires you to constrain the problem to things that someone who has barely used language X or Y can do within 45 minutes.
  - sethammons 8y ago
    The theory with pick your own language is they should be able to feel fully comfortable (interviews are already stressful enough). If they picked Rust, Scala, or a lisp dialect, (or anything the interviewers are unfamiliar with), it can even be a better interview because we get more insight on how the candidate communicates and their ability to walk someone else through their solution. A potential other bonus is less biases leak through from an interviewer on "that is a strange way to do that in language X."
    - cookiecaper 8y ago
      > The theory with pick your own language is they should be able to feel fully comfortable (interviews are already stressful enough). Ah, but there's the rub. Candidates are trying to please the interview panel. If you don't provide guidance, the odds that they'll just use whatever they think the interviewers most prefer are just as good, if not better, than the odds that they'll actually use whatever makes them most comfortable. You said yourself that since some candidates pick a language that you don't know well, you can't really tell if the failure of a large number of those candidates is reflective of a bad test or just a mismatched candidate pool. IMO, if you're going to stick to the "pick any language" thing, you should at least find out and ensure that any language the candidate picks will have a fair shot. > it can even be a better interview because we get more insight on how the candidate communicates and their ability to walk someone else through their solution. You can still get the candidate to communicate and explain his choices if you give an option: "either Ruby or Python" or "either JavaScript or Visual Basic", etc. The problem with having this happen in a language that the interviewers don't know reasonably well is that they are much more vulnerable to the smooth talker who can present incorrect information confidently, and they won't have enough background/anchoring in the subject matter to know the difference. > A potential other bonus is less biases leak through from an interviewer on "that is a strange way to do that in language X." I would say that if you're worried that interviewers will load in biases toward their preferred shortcuts etc in a specific language, that you should be equally worried that some good candidates are being excluded for choosing the "wrong" language in an any-language-goes test. Above, you mentioned that there'd be a positive response if a candidate used "Rust, Scala, or a lisp dialect" -- these are all relatively trendy. What if the candidate used nim, Pony, or some other language that hasn't pulled in to the hype superstation yet? What if the candidate used a language that's not-so-trendy anymore, like Visual Basic, Cobol, or bash? What if the candidate used a programming language of their own design, and brought a copy of the compiler with them on a flash drive? I'm asking because I've seen this in practice. Candidates for a devops position who chose to use bash to implement the very simple take-home task they were given were laughed off by several other members of the interview committee, despite being potentially high-value senior people -- they were at least senior enough that they're more comfortable performing sysadmin-style tasks in a shell, rather than using a massive CM framework or an awkward amalgamation of Python scripts running os.spawn. It feels like this type of thing happens a lot, in the same sense that very often, "unlimited PTO" just means "guess whatever amount of PTO is acceptable around here and hope you get it right".

tpxl 8y ago

>Part of the code we want right now requires a unit test with dependency injection to match an interface. What exactly do you mean by this?

sethammons 8y ago
We ask that the code they write have a unit test. The nature of that unit test is that it has a dependency. They can mock however they like, but passing in an object that represents the dependency (dependency injection) is the easiest and most straight forward way to do that. That object should have a method on it (a know method, with a known signature, also known in some circles as matching an interface).