7 ms·

This is absolutely completely true: "Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to

This is absolutely completely true: "Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to talk about computers convincingly, but can’t actually write a program when asked. Any program. At all." I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test. We get a lot of English-as-second-language and shy developers (who can code) and I can still figure out by asking pretty general questions about development. The idea that you need a code test to figure out who can code and who can't is false. I'm not arguing against it, I'm simply arguing that it's not strictly necessary.

gregmac 7y ago
What has worked well for me (at least for non-junior positions) is to just ask about their prior work. "You were on a team that developed x - great. Assume I'm an expert in that and am happy to geek out about the tiny details. What was a part that you specifically were responsible for? What was most fun/interesting about it? What was one of the challenges and how did you overcome it? What was something new you learned? If you started over today, what was a part you'd do differently, and why and how?" Really just pick something they seem excited about and dive really deep into it. "Why didn't you use y for that? How did you test it?" The key is really simple: find something they're passionate about and get them to talk about it. The annoying thing about refining interview techniques is you only get a small view of effectiveness: you can't tell how many times when you didn't hire it was the correct choice vs passing on someone great, so I can't judge from that perspective. That said, I haven't regretted hiring anyone that did well on this style of interviewing, without another coding interview.
JamesBarney 7y ago
What are the questions about development?
llamataboot 7y ago
I agree 100% and I would argue that unless you are pairing together just to see how you work together and for some back and forth problem solving, no other code test provides a better signal than a good conversation. Not set questions, even, just a good conversation. What issue have they run into with Framework X, how do they refactor, what's the weirdest bug they've ever had to track down, etc etc etc. SV seems to enjoy grueling 7-10 round interviews these days. I had one a couple months ago that was over 40 hours (of my time) and over 20 hours (of their time, that I was aware of). I just can't believe that much time provides 20-40x the value of a simple relaxed conversation.
danpalmer 7y ago
> I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test. I also think I can do this, but I believe most people (myself included) introduce a lot of bias when doing this. I've spoken to developers who in 15 minutes I might have written off as not very qualified, but after hours/days have realised are great at what they do, and very experienced in types of software engineering that I have had less exposure to. I think we have to be careful when judging candidates with what amounts to "gut feeling" like this, as it's typically biased, and not aligned within the hiring team.
- sgift 7y ago
  The thing is: The method described in the post has the same problem. It just doesn't look like it. Most of these types of interview templates are designed by one or two persons and then blindly applied. Sometimes you have a view modifications, but for the most part it's fixed. Since every interviewee gets the same set people then start to think this is more scientific than "gut feeling" - it is not, but instead of your own biases you have the biases of the template designer. There are probably a few companies which extensively study if their interview process for the people hired (the only subset they have access to, which already is a problem) actually matches their job performance and then try to readjust the process, but it doesn't seem to happen very often. This also gets us back to the old problem that we have no idea what a good programmer is or how to evaluate job performance.
- codeduck 7y ago
  I find that asking "What is the worst bug you've ever encountered" is a great sounding board.
  - foldr 7y ago
    I think that question can be great, but it also puts a lot of pressure on the candidate to execute a rather complicated query on the spot. I've definitely encountered lots of interesting bugs in my time, but my initial reaction to this question is a total mind blank. Especially since "worst" could mean a lot of things, and what I really want is "worst bug I've encountered that I can say something interesting and impressive about". I'd say it's not a bad idea to forward questions like that to the candidate in advance of the interview. That way, you avoid false negatives from people who just couldn't think of a good answer under interview pressure.
jldugger 7y ago
> I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. How do you know? Is this repeatable -- e.g. can you write down the questions and the criteria such that someone who is not you can follow them and come to the conclusions you would? Do you ask all candidates the same questions, or do you go with your gut and hope there's no systemic bias?
anbop 7y ago
It’s certainly possible but it’s obviously much harder to do. Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play? Rather than trying to determine this just by asking questions, why not take the much easier approach of putting them on the court for 2 minutes to see how they play?
- irq11 7y ago
  “Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play?” No, that would be phenomenally stupid. You would do what teams actually do, and look at their recent playing history. Note that this is also entirely possible for engineers.
  - hvidgaard 7y ago
    > Note that this is also entirely possible for engineers. How exactly? Most people cannot take their code with them and show off. It belongs to previous employers, and please do not begin with the "show me your GitHub" - it's unrealistic to expect people to devote their life 100% to coding. Just as I wouldn't expect a carpenter to build houses for fun in his spare time, I do not expect programmers to spend their spare time in front of computers. In fact I believe that socially it's a benefit if they have other hobbies.
    - zimpenfish 7y ago
      > Most people cannot take their code with them and show off. And if someone turned up with code from their previous employer, that'd surely be a huge red flag anyway.
    - irq11 7y ago
      This is the problem with programmers: everyone gets obsessive about finding perfect unicorn universal solutions, and gives up on obvious stuff that works 95% of the time. Get out of your own way! First, most programmers have at least some code they can show you. Stop fretting about the 5% who don’t. If you give up here simply because you have a theory about programmers “devoting their life to coding”, you lose. Second, you call their references. Yes, really. I know it’s not “objective”. Do it anyway. You will learn a lot. Really. Third, look around for other people they’ve worked with, who weren’t listed as references. Call some of them. See if you get different opinions. If your candidate truly, honestly has no code available, no enthusiastic references, and you can’t find anyone who worked with them who will vouch for their skill, then do you really want to hire them? Finally, yes: do a trivial fizzbuzz coding test to filter out the total fakes. Listening to coders complain, you’d think that nobody else in the world hired anyone, ever. Let go of your need to make hiring a failure-proof system, and you’ll discover a bigger world.
raxxorrax 7y ago
I could do that for my specific niche for programmers (c, embedded systems [for the systems I know], a bit of web stuff), but not in general for a data analyst or ML for example. I imagine I would ask for knowledge only practical experience could provide (interest in a topic is not the greatest teacher, but a decent one) Better than fizz-buzz I think... But you can never be sure either way. I would think that actually giving people a chance while defining quantifiable expectations would help here. Costly and time intensive perhaps, but I don't believe there to be a reliable shortcut. Especially not if the labor market is in need of more engineers.
lmm 7y ago
> I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test. How do you know? Do you check your interview assessments against actual performance among those hired (measured how)? What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?
- wvenable 7y ago
  > What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected? I'm pretty forgiving -- heck it usually takes at least half the interview before the candidate's nerves settle down. Being a developer involves being able to explain yourself clearly. You can't just sit in the dark and code, talking to no one. If I came away from an interview with the impression that you can't code then I don't want you. The candidate has to do some work too -- they are there to convince me to hire them, not just pass a test and get a grade.
  - lmm 7y ago
    It's much easier to "explain yourself clearly" to someone from the same cultural/social/... background. Doesn't that approach create a huge bias towards hiring people like you?
    - wvenable 7y ago
      Literally every single candidate we get is from a different cultural/social background. I'm not sure why giving them a test first and then interviewing them would make any difference.
      - lmm 7y ago
        Obviously if you ignore the test results then the test doesn't make any difference. But if you assign some weight to whether their solution to a test problem worked (which is culture-independent, or at least less culture-dependent) rather than how well they explained themselves to a person from a particular culture (which is easier for people from that same culture), then you end up with a process that is much fairer across cultures. And similarly for non-culture aspects of a person's background.
pjc50 7y ago
How do you determine the false-positive rate of this: people who actually can code quite well but fail the interview?
speby 7y ago
It's interesting. I've encountered a very different animal, even without having the candidate do a "homework test" or a "coding test" during the interview. I have found that this different animal is competent, and certainly can code, but is of the very unproductive, slow type of developer. And so my biggest pain point is rarely around a developer's competence during the interview process (i.e. Does this person know the material and are they able to do the job?) but rather instead around their overall productivity, their "capacity" to build software using their competence. There are plenty of people in the world who know the material but are extremely "slow" in actually getting things done and problems solved in a day job. Unfortunately, the in-interview coding tests don't do a very good job of measuring the person's productive capacity to get problems solved, only whether or not they can solve some coding problem in a 30-minute timespan.

gregmac 7y ago

What has worked well for me (at least for non-junior positions) is to just ask about their prior work. "You were on a team that developed x - great. Assume I'm an expert in that and am happy to geek out about the tiny details. What was a part that you specifically were responsible for? What was most fun/interesting about it? What was one of the challenges and how did you overcome it? What was something new you learned? If you started over today, what was a part you'd do differently, and why and how?" Really just pick something they seem excited about and dive really deep into it. "Why didn't you use y for that? How did you test it?" The key is really simple: find something they're passionate about and get them to talk about it. The annoying thing about refining interview techniques is you only get a small view of effectiveness: you can't tell how many times when you didn't hire it was the correct choice vs passing on someone great, so I can't judge from that perspective. That said, I haven't regretted hiring anyone that did well on this style of interviewing, without another coding interview.

JamesBarney 7y ago

What are the questions about development?

llamataboot 7y ago

I agree 100% and I would argue that unless you are pairing together just to see how you work together and for some back and forth problem solving, no other code test provides a better signal than a good conversation. Not set questions, even, just a good conversation. What issue have they run into with Framework X, how do they refactor, what's the weirdest bug they've ever had to track down, etc etc etc. SV seems to enjoy grueling 7-10 round interviews these days. I had one a couple months ago that was over 40 hours (of my time) and over 20 hours (of their time, that I was aware of). I just can't believe that much time provides 20-40x the value of a simple relaxed conversation.

danpalmer 7y ago

> I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test. I also think I can do this, but I believe most people (myself included) introduce a lot of bias when doing this. I've spoken to developers who in 15 minutes I might have written off as not very qualified, but after hours/days have realised are great at what they do, and very experienced in types of software engineering that I have had less exposure to. I think we have to be careful when judging candidates with what amounts to "gut feeling" like this, as it's typically biased, and not aligned within the hiring team.

sgift 7y ago
The thing is: The method described in the post has the same problem. It just doesn't look like it. Most of these types of interview templates are designed by one or two persons and then blindly applied. Sometimes you have a view modifications, but for the most part it's fixed. Since every interviewee gets the same set people then start to think this is more scientific than "gut feeling" - it is not, but instead of your own biases you have the biases of the template designer. There are probably a few companies which extensively study if their interview process for the people hired (the only subset they have access to, which already is a problem) actually matches their job performance and then try to readjust the process, but it doesn't seem to happen very often. This also gets us back to the old problem that we have no idea what a good programmer is or how to evaluate job performance.
codeduck 7y ago
I find that asking "What is the worst bug you've ever encountered" is a great sounding board.
- foldr 7y ago
  I think that question can be great, but it also puts a lot of pressure on the candidate to execute a rather complicated query on the spot. I've definitely encountered lots of interesting bugs in my time, but my initial reaction to this question is a total mind blank. Especially since "worst" could mean a lot of things, and what I really want is "worst bug I've encountered that I can say something interesting and impressive about". I'd say it's not a bad idea to forward questions like that to the candidate in advance of the interview. That way, you avoid false negatives from people who just couldn't think of a good answer under interview pressure.

jldugger 7y ago

> I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. How do you know? Is this repeatable -- e.g. can you write down the questions and the criteria such that someone who is not you can follow them and come to the conclusions you would? Do you ask all candidates the same questions, or do you go with your gut and hope there's no systemic bias?

anbop 7y ago

It’s certainly possible but it’s obviously much harder to do. Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play? Rather than trying to determine this just by asking questions, why not take the much easier approach of putting them on the court for 2 minutes to see how they play?

irq11 7y ago
“Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play?” No, that would be phenomenally stupid. You would do what teams actually do, and look at their recent playing history. Note that this is also entirely possible for engineers.
- hvidgaard 7y ago
  > Note that this is also entirely possible for engineers. How exactly? Most people cannot take their code with them and show off. It belongs to previous employers, and please do not begin with the "show me your GitHub" - it's unrealistic to expect people to devote their life 100% to coding. Just as I wouldn't expect a carpenter to build houses for fun in his spare time, I do not expect programmers to spend their spare time in front of computers. In fact I believe that socially it's a benefit if they have other hobbies.
  - zimpenfish 7y ago
    > Most people cannot take their code with them and show off. And if someone turned up with code from their previous employer, that'd surely be a huge red flag anyway.
  - irq11 7y ago
    This is the problem with programmers: everyone gets obsessive about finding perfect unicorn universal solutions, and gives up on obvious stuff that works 95% of the time. Get out of your own way! First, most programmers have at least some code they can show you. Stop fretting about the 5% who don’t. If you give up here simply because you have a theory about programmers “devoting their life to coding”, you lose. Second, you call their references. Yes, really. I know it’s not “objective”. Do it anyway. You will learn a lot. Really. Third, look around for other people they’ve worked with, who weren’t listed as references. Call some of them. See if you get different opinions. If your candidate truly, honestly has no code available, no enthusiastic references, and you can’t find anyone who worked with them who will vouch for their skill, then do you really want to hire them? Finally, yes: do a trivial fizzbuzz coding test to filter out the total fakes. Listening to coders complain, you’d think that nobody else in the world hired anyone, ever. Let go of your need to make hiring a failure-proof system, and you’ll discover a bigger world.

raxxorrax 7y ago

I could do that for my specific niche for programmers (c, embedded systems [for the systems I know], a bit of web stuff), but not in general for a data analyst or ML for example. I imagine I would ask for knowledge only practical experience could provide (interest in a topic is not the greatest teacher, but a decent one) Better than fizz-buzz I think... But you can never be sure either way. I would think that actually giving people a chance while defining quantifiable expectations would help here. Costly and time intensive perhaps, but I don't believe there to be a reliable shortcut. Especially not if the labor market is in need of more engineers.

lmm 7y ago

> I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test. How do you know? Do you check your interview assessments against actual performance among those hired (measured how)? What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?

wvenable 7y ago
> What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected? I'm pretty forgiving -- heck it usually takes at least half the interview before the candidate's nerves settle down. Being a developer involves being able to explain yourself clearly. You can't just sit in the dark and code, talking to no one. If I came away from an interview with the impression that you can't code then I don't want you. The candidate has to do some work too -- they are there to convince me to hire them, not just pass a test and get a grade.
- lmm 7y ago
  It's much easier to "explain yourself clearly" to someone from the same cultural/social/... background. Doesn't that approach create a huge bias towards hiring people like you?
  - wvenable 7y ago
    Literally every single candidate we get is from a different cultural/social background. I'm not sure why giving them a test first and then interviewing them would make any difference.
    - lmm 7y ago
      Obviously if you ignore the test results then the test doesn't make any difference. But if you assign some weight to whether their solution to a test problem worked (which is culture-independent, or at least less culture-dependent) rather than how well they explained themselves to a person from a particular culture (which is easier for people from that same culture), then you end up with a process that is much fairer across cultures. And similarly for non-culture aspects of a person's background.

pjc50 7y ago

How do you determine the false-positive rate of this: people who actually can code quite well but fail the interview?

speby 7y ago

It's interesting. I've encountered a very different animal, even without having the candidate do a "homework test" or a "coding test" during the interview. I have found that this different animal is competent, and certainly can code, but is of the very unproductive, slow type of developer. And so my biggest pain point is rarely around a developer's competence during the interview process (i.e. Does this person know the material and are they able to do the job?) but rather instead around their overall productivity, their "capacity" to build software using their competence. There are plenty of people in the world who know the material but are extremely "slow" in actually getting things done and problems solved in a day job. Unfortunately, the in-interview coding tests don't do a very good job of measuring the person's productive capacity to get problems solved, only whether or not they can solve some coding problem in a 30-minute timespan.