34 ms·

> Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for

> Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for candidates to perform well technically but not effectively communicate what they’re doing (or vice versa), largely (and fortunately) debunking the myth of the incoherent, fast-talking, awkward engineer. My interpretation of this is that interviewees who can communicate clearly about code (whether they wrote it or not) correlate with high technical ability. Does this suggest that rather than having the interviewee write code on the spot, one could give them some new code they've never seen before and ask them to reason about it aloud for 30 minutes, then gauge their technical ability based on their ability to communicate clearly about the code? In other words, could you replace live-coding with "here's some code, tell me about it"?

soulnothing 9y ago
This was the style of interview I ran. I usually picked snippets from an open source project. I usually grabbed pull requests that fixed bugs. Have them delve through it. Reason aloud, ask questions about it. See how they gain context, and how they troubleshoot. The biggest down side was they sometimes need to reference external projects, or documentation. I would often have to search on my work laptop and then let them should surf the docs.
qntmfred 9y ago
this is also a prominent technique I've used in the hundreds of interviews I've conducted.
vonmoltke 9y ago
My previous company did this. I have seen three variants over my time: explain what this code does; explain why this code is wrong and how to fix it; explain why this code is inefficient and how to optimize it.
- DoofusOfDeath 9y ago
  > explain why this code is inefficient and how to optimize it. That sounds like a trick question. I've done a lot of code optimization. Here are a few generalizations I've drawn: (a) You rarely know what part of a program is the bottleneck until you profile it on an appropriate workload. (b) You often don't know why it's slow until you dig into performance metrics like cache-miss ratios, branch-mispredicts, etc. Often at the disassembly level.
  - igorbark 9y ago
    But for a short, isolated piece of code I assume you would be able to suggest a handful of plausible inefficiencies and explain how they could be tested for and optimized for.
  - mmcnl 9y ago
    Well that would be a perfect answer if you ask me.
    - bluGill 9y ago
      No, it would not be a perfect answer, and could work against you: it shows you are not willing to work with me and assume some intelligence on my part (until proven otherwise) and thus you are potentially not going to work well with the rest of the team. I would respond that we have already profiled this code and discovered it is a bottleneck, now answer the question. It won't be hard to overcome the negative impression, but it left a negative that you don't want. Now if you responded in the form "Of course in the real world we don't look at code until after a profiler shows the need. I see X which is suspicious, let me look deeper". In this case the comment is perfect: it shows you trust, but know to verify that trust. It also has the side effect of giving you time to think.
      - mmcnl 9y ago
        I fail to see why analyzing a problem before solving it somehow translates into "not going to work well with the team". Problem analysis skills are far more valuable than problem solving skills. Anyone can search on Google for a solution to a given problem, but no one on Google can tell you what the actual problem is. If you really thing the answer I was responding to is not acceptable, then the question is a lousy question imho. "It doesn't make sense to hire smart people and tell them what to do; we hire smart people so they can tell us what to do." - S. Jobs
- b0rsuk 9y ago
  I think "find the bug in this code piece" questions are dumb! When you program, you don't get a helpful fairy (or Clippy for you Windows users) telling you the code is wrong and needs fixing. The bug manifests itself in production or when you try to execute the code. Bad output, exceptions or segfaults, slow operation, infinite loop. Then you have a vague idea what to look for, or at least what kind of input causes errors. Furthermore, if you're the author you should know what the code is trying to do, and how. I believe programmers in teams are usually told to fix their bugs, not fix someone else's bug ? I'd rather see a question like "Please modify this piece of code to add X functionality."
lj3 9y ago
Those are my preferred types of interviews when I'm an interviewee and a technique I've used a few times on the other side of the table. It's so effective and efficient. I don't know why it isn't used more. Maybe it finally will be, now that there's some solid data supporting it.
lettergram 9y ago
That's actually how I conduct interviews. It's been immensely successful at weeding out candidates within 5 minutes; although most interviews are 45 - 60 minutes. Basically, give them less than ten lines of code, ask them what it does, where are a couple bugs, ask what would you name the function, etc. Then we talk about how to improve it. I'd say, less than 20% of people I interview pass. It's actually amazingly low how many people can find a bug and communicate it. Half the people don't even tell me what they are thinking. And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down. They think in their head, don't work through the problem at all.
- marcosdumay 9y ago
  > They think in their head I'm left wondering why this would be a problem.
  - swimfar 9y ago
    for one thing, you can't get partial credit for ideas/work that only exists in your head. (Similar to why it's a good idea to show work on homework assignments and tests) If they sat there for a bit thinking about the problem and then provided a solution, I'm sure that would be fine. But that's not the situation he was describing.
  - elgenie 9y ago
    Not being able to successfully and clearly communicate their thought process about a simple problem one-on-one is a bad signal for being able to communicate clearly about a difficult problem in a group setting or being a useful sounding board for technical ideas.
    - krzyk 9y ago
      Yeah, but if you shoot for elaborate ones you will get politicians/lawyers not coders. I don't have any source but from my experience mathematical/computer minded people are more likely to be introverted/talk less then arts people.
jghn 9y ago
I used to take "relatively rare" a step further and viewed it as a myth. I'd get into debates with coworkers that all one needs to do is have a conversation with a candidate to fully judge their abilities. No need for coding tests or the typical questions. Unfortunately after we brought in a ton of candidates there were a few people who fell into this bucket. I have no idea how it happened but they couldn't code themselves out of a wet paper bag when it came time to do so even they kicked ass in technical conversations. Sadly it became a bit of an "I told you so" moment where the outliers are made up to be more frequent than they actually are. I like your idea though.
- MichaelGG 9y ago
  Had a similar experience with a remote sysadmin. Interviewing, he seemed to understand everything. He explained how he'd setup networks, details on using SSH for bastion hosts, etc. Seemed to really know stuff. After hiring him, he was unable to even SSH to a box. As in he didn't know how to give us his private key, or how to configure his SSH client. Maybe one guy takes interviews under other people's names then lets them mess up the work.
  - troels 9y ago
    OTOH if he had given you his private key, surely he would have failed the sysadm test right there.
- DoofusOfDeath 9y ago
  That happened to me once, during a phone interview with Google. My brain simply froze during the live coding test. I looked like an idiot, even though the coding challenge was one I could have normally handled in my sleep. My guess is that it was due to anxiety about doing well on the interview. But to this day I'm gun-shy regarding live coding tests.
  - meggar 9y ago
    I did that at a Google interview too, on a variation of a question I'd practiced a few days before no less.
Jemmeh 9y ago
Something I've learned from making Youtube tutorials is that you need to have a deeper knowledge to actually teach something. The point at which I study a topic and have it "click in my head" comes way before I can actually manage to "explain it back clearly". As I'm writing the script for the tutorial I often realize there's a gap in my understanding that I have to go back and work on. The old saying "If you can't explain it simply, you don't understand it well enough" applies here I think. :)
- benjaminjackman 9y ago
  That has been my experience as well, one level abstracted. Back when I was a position when I last had to organize teaching employees technical domain-specific skills, the best way we found for them to learn was for them to teach others, this was accomplished by having one generation of employees train the next and then reviewing the newly trained employees. Then for the next round of hiring, advance the employees up the education pyramid. The trainees become trainers, and the previous trainers mentor the new trainers by reviewing the newly trained employees.[1] It was one of the things I think we did we really well and that I am really proud of (even though the business went belly-up). When training is done is this pattern, hopefully a virtuous cycle of education is established throughout your organization. 1: Note that this was at a quickly growing company, with recruits that had little to no relevant actual know-how to do what we did.
  - enobrev 9y ago
    I occasionally wonder if something like this would significantly improve our education systems, where part of our schooling includes tutoring / teaching someone from the next year.
cr0sh 9y ago
I've never experienced an interview like this, but it sounds like it would be both challenging and fun - especially if it was done in a language (or pseudo-language) that you'd never seen before (then again, they could lob BF at you or something eso like that!). Based on other responses to your post, it's heartening to see that others do this kind of interview, so maybe in the future I might encounter it.
jondubois 9y ago
Some companies just love fast-talking engineers because they sound like they know what they're doing. Some engineers spend a lot of effort trying to project a specific image of themselves at the expense of bringing up the morale of others in their team. On a few occasions, I've seen an engineer tear apart the work of another engineer at a team meeting in order to make themselves look good and more senior in front of their boss - Often, the critique is delivered with such vigour that it sounds rehearsed. I've seen this behaviour in startups mostly.
desdiv 9y ago
When I'm nervous, I don't communicate well. When I'm nervous, I don't code well. I don't know about other people, but for me personally, when you measure me under a high-stress situation, all you are measuring is just my level of nervousness.
- poikniok 9y ago
  I must ask, did you get a university degree and if so how? Personally I have found interviews to be trivial compared to exams in school, so while I do get quite nervous for some of my math examinations I have never had this experience in interviews.
  - desdiv 9y ago
    I have a CS degree from a first-tier school. My experience with exams and interviews is almost the exact opposite of yours. In exams I don't get as nervous because I know that if I know the material then I will pass, if I don't then I won't. You can't bullshit your way past the professor/TA grading your exam. But in interviews I know that people who are far better than me have failed[0], and that people less competent than me could get the job. So it's not my coding ability or my experience that's being measured here, it's something else. And I don't know whether I have that something else or not. Maybe I do, maybe I just don't. [0] https://twitter.com/mxcl/status/608682016205344768?lang=en https://twitter.com/mxcl/status/608682016205344768?lang=en
    - deleted 9y ago
      [deleted]
    - javabean22 9y ago
      I feel exactly the same way. The good thing is you keep doing the interviews for some time at some point you run into normal people and then it's easy.
  - b0rsuk 9y ago
    Last but not least, after you fail an exam you can usually see it with errors underlined and often have the teacher explain them for you. If you answer interview questions in a suboptimal way, you get no feedback. You don't know what was the error you made, if any. Maybe another candidate simply read about the logic puzzle that was used. The canned response I got was "We're sorry, but we can't currently recruit you. Please try again in half a year or so, maybe we will have some new opportunities." If you press on, you just get evasive answers.
maxxxxx 9y ago
That's what I do. If I can have a meaningful conversation with someone about stuff he did or stuff I am describing most likely they will do well. If the conversation gets very vague or confusing most likely things won't work out.
11thEarlOfMar 9y ago
We do it in reverse. In the first 2 hour interview, candidates learn our tech from 2 or 3 engineers. We then send them home with a set of coding tasks that are all relevant to what we do. As in functions of utilities that we actually would need to write, as opposed to puzzles. They choose one of 6 tasks, write ~100 lines of code, then come back for the next 2 hour interview and lead a code review of the code they wrote. This approach allows them to take the same time and care they would if working for us, rather than putting them in an unnatural state of coding at a white board in front of one to n scowling veterans. Engineers who are capable can clearly explain their coding choices and their style is evident. If they've borrowed too liberally or had someone else write it for them, it becomes apparent pretty quickly because they can't clearly explain the code they supposedly wrote. This is a fairly new approach for us, but so far, the candidates have appreciated it and the team reports that this approach gives them a good sense of the candidates ability.
- lj3 9y ago
  So, based on the new information presented in the article that people who can talk clearly about code can also write code, are you likely to change your interview practices to omit the take home task and ask the candidate to talk about code that somebody else wrote? It would be faster, easier and according to the data in TFA just as accurate, yes?
  - 11thEarlOfMar 9y ago
    Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data. I do give candidates the option of the take home vs. white boarding. So far (it's admittedly still a pretty small sample) they've all opted for the take home. However, we'll take a look at this approach as well.
    - lj3 9y ago
      > Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data. I'm not aware of any solid data around take home tests. All I can give is yet more anecdotal data, which is that they take a lot of time on both ends. So much time, in fact, that I've had several interviews where the interviewer never looked at the test code I was required to send in. Whether they work or not is irrelevant if nobody ever looks at the code.
graphememes 9y ago
I do this frequently and I utilize their resume to power the code. Someone says "I am a 9/10 in JS" I give them something that uses the animation frame api, the dom api, and a few more javascript APIs that I personally am not fully comfortable saying that I am a 9/10 on (been working with JS for over ten years) and see how they decipher what it is doing, what jumps out, and how they would review it / give feedback. 9/10 times they don't actually understand Javascript.
kasey_junk 9y ago
Its fairly easy to see how this correlation breaks down. It is almost a truism that someone who is good at communicating with a wide range of people, is likely to make that wide range of people think they are more technically competent than people who aren't good at communicating with a wide range of people. Especially given the virtually random nature who you will communicate with during any given interview cycle. But, its quite possible for people who are not naturally able to communicate with a wide range of people to be able to communicate with a smaller subset (say people they are more comfortable with, or people working on similar problems to them). The more interesting question is, could those people who communicate less broadly, in a situation where they do communicate well, perform technically at the same level (or better) as people who are more natural communicators? Another way to say this is, evaluating interviews is fairly prone to bias towards people who interview well. As someone who is hiring I'd be much more enthused about their results if it correlated to job performance, not hiring results. Its almost gospel at this point that there are people in software who interview well who do not make good hires, but the methodology of this article doesn't talk about this at all. All told, nothing about this research is persuasive to me on my opinion of "interviews are largely worthless in determining who will be a good hire".
amalcon 9y ago
The version of this where that code is clearly unsuitable for its stated purpose is my favorite coding interview question. The unsuitability should be extremely obvious, no "spot the missing semicolon". Maybe it has a severe performance problem, maybe an obvious security flaw, maybe a logic error. The point is to prompt a discussion of better alternative ways to do what the code is trying to do.

soulnothing 9y ago

This was the style of interview I ran. I usually picked snippets from an open source project. I usually grabbed pull requests that fixed bugs. Have them delve through it. Reason aloud, ask questions about it. See how they gain context, and how they troubleshoot. The biggest down side was they sometimes need to reference external projects, or documentation. I would often have to search on my work laptop and then let them should surf the docs.

qntmfred 9y ago

this is also a prominent technique I've used in the hundreds of interviews I've conducted.

vonmoltke 9y ago

My previous company did this. I have seen three variants over my time: explain what this code does; explain why this code is wrong and how to fix it; explain why this code is inefficient and how to optimize it.

DoofusOfDeath 9y ago
> explain why this code is inefficient and how to optimize it. That sounds like a trick question. I've done a lot of code optimization. Here are a few generalizations I've drawn: (a) You rarely know what part of a program is the bottleneck until you profile it on an appropriate workload. (b) You often don't know why it's slow until you dig into performance metrics like cache-miss ratios, branch-mispredicts, etc. Often at the disassembly level.
- igorbark 9y ago
  But for a short, isolated piece of code I assume you would be able to suggest a handful of plausible inefficiencies and explain how they could be tested for and optimized for.
- mmcnl 9y ago
  Well that would be a perfect answer if you ask me.
  - bluGill 9y ago
    No, it would not be a perfect answer, and could work against you: it shows you are not willing to work with me and assume some intelligence on my part (until proven otherwise) and thus you are potentially not going to work well with the rest of the team. I would respond that we have already profiled this code and discovered it is a bottleneck, now answer the question. It won't be hard to overcome the negative impression, but it left a negative that you don't want. Now if you responded in the form "Of course in the real world we don't look at code until after a profiler shows the need. I see X which is suspicious, let me look deeper". In this case the comment is perfect: it shows you trust, but know to verify that trust. It also has the side effect of giving you time to think.
    - mmcnl 9y ago
      I fail to see why analyzing a problem before solving it somehow translates into "not going to work well with the team". Problem analysis skills are far more valuable than problem solving skills. Anyone can search on Google for a solution to a given problem, but no one on Google can tell you what the actual problem is. If you really thing the answer I was responding to is not acceptable, then the question is a lousy question imho. "It doesn't make sense to hire smart people and tell them what to do; we hire smart people so they can tell us what to do." - S. Jobs
b0rsuk 9y ago
I think "find the bug in this code piece" questions are dumb! When you program, you don't get a helpful fairy (or Clippy for you Windows users) telling you the code is wrong and needs fixing. The bug manifests itself in production or when you try to execute the code. Bad output, exceptions or segfaults, slow operation, infinite loop. Then you have a vague idea what to look for, or at least what kind of input causes errors. Furthermore, if you're the author you should know what the code is trying to do, and how. I believe programmers in teams are usually told to fix their bugs, not fix someone else's bug ? I'd rather see a question like "Please modify this piece of code to add X functionality."

lj3 9y ago

Those are my preferred types of interviews when I'm an interviewee and a technique I've used a few times on the other side of the table. It's so effective and efficient. I don't know why it isn't used more. Maybe it finally will be, now that there's some solid data supporting it.

lettergram 9y ago

That's actually how I conduct interviews. It's been immensely successful at weeding out candidates within 5 minutes; although most interviews are 45 - 60 minutes. Basically, give them less than ten lines of code, ask them what it does, where are a couple bugs, ask what would you name the function, etc. Then we talk about how to improve it. I'd say, less than 20% of people I interview pass. It's actually amazingly low how many people can find a bug and communicate it. Half the people don't even tell me what they are thinking. And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down. They think in their head, don't work through the problem at all.

marcosdumay 9y ago
> They think in their head I'm left wondering why this would be a problem.
- swimfar 9y ago
  for one thing, you can't get partial credit for ideas/work that only exists in your head. (Similar to why it's a good idea to show work on homework assignments and tests) If they sat there for a bit thinking about the problem and then provided a solution, I'm sure that would be fine. But that's not the situation he was describing.
- elgenie 9y ago
  Not being able to successfully and clearly communicate their thought process about a simple problem one-on-one is a bad signal for being able to communicate clearly about a difficult problem in a group setting or being a useful sounding board for technical ideas.
  - krzyk 9y ago
    Yeah, but if you shoot for elaborate ones you will get politicians/lawyers not coders. I don't have any source but from my experience mathematical/computer minded people are more likely to be introverted/talk less then arts people.

jghn 9y ago

I used to take "relatively rare" a step further and viewed it as a myth. I'd get into debates with coworkers that all one needs to do is have a conversation with a candidate to fully judge their abilities. No need for coding tests or the typical questions. Unfortunately after we brought in a ton of candidates there were a few people who fell into this bucket. I have no idea how it happened but they couldn't code themselves out of a wet paper bag when it came time to do so even they kicked ass in technical conversations. Sadly it became a bit of an "I told you so" moment where the outliers are made up to be more frequent than they actually are. I like your idea though.

MichaelGG 9y ago
Had a similar experience with a remote sysadmin. Interviewing, he seemed to understand everything. He explained how he'd setup networks, details on using SSH for bastion hosts, etc. Seemed to really know stuff. After hiring him, he was unable to even SSH to a box. As in he didn't know how to give us his private key, or how to configure his SSH client. Maybe one guy takes interviews under other people's names then lets them mess up the work.
- troels 9y ago
  OTOH if he had given you his private key, surely he would have failed the sysadm test right there.
DoofusOfDeath 9y ago
That happened to me once, during a phone interview with Google. My brain simply froze during the live coding test. I looked like an idiot, even though the coding challenge was one I could have normally handled in my sleep. My guess is that it was due to anxiety about doing well on the interview. But to this day I'm gun-shy regarding live coding tests.
- meggar 9y ago
  I did that at a Google interview too, on a variation of a question I'd practiced a few days before no less.

Jemmeh 9y ago

Something I've learned from making Youtube tutorials is that you need to have a deeper knowledge to actually teach something. The point at which I study a topic and have it "click in my head" comes way before I can actually manage to "explain it back clearly". As I'm writing the script for the tutorial I often realize there's a gap in my understanding that I have to go back and work on. The old saying "If you can't explain it simply, you don't understand it well enough" applies here I think. :)

benjaminjackman 9y ago
That has been my experience as well, one level abstracted. Back when I was a position when I last had to organize teaching employees technical domain-specific skills, the best way we found for them to learn was for them to teach others, this was accomplished by having one generation of employees train the next and then reviewing the newly trained employees. Then for the next round of hiring, advance the employees up the education pyramid. The trainees become trainers, and the previous trainers mentor the new trainers by reviewing the newly trained employees.[1] It was one of the things I think we did we really well and that I am really proud of (even though the business went belly-up). When training is done is this pattern, hopefully a virtuous cycle of education is established throughout your organization. 1: Note that this was at a quickly growing company, with recruits that had little to no relevant actual know-how to do what we did.
- enobrev 9y ago
  I occasionally wonder if something like this would significantly improve our education systems, where part of our schooling includes tutoring / teaching someone from the next year.

cr0sh 9y ago

I've never experienced an interview like this, but it sounds like it would be both challenging and fun - especially if it was done in a language (or pseudo-language) that you'd never seen before (then again, they could lob BF at you or something eso like that!). Based on other responses to your post, it's heartening to see that others do this kind of interview, so maybe in the future I might encounter it.

jondubois 9y ago

Some companies just love fast-talking engineers because they sound like they know what they're doing. Some engineers spend a lot of effort trying to project a specific image of themselves at the expense of bringing up the morale of others in their team. On a few occasions, I've seen an engineer tear apart the work of another engineer at a team meeting in order to make themselves look good and more senior in front of their boss - Often, the critique is delivered with such vigour that it sounds rehearsed. I've seen this behaviour in startups mostly.

desdiv 9y ago

When I'm nervous, I don't communicate well. When I'm nervous, I don't code well. I don't know about other people, but for me personally, when you measure me under a high-stress situation, all you are measuring is just my level of nervousness.

poikniok 9y ago
I must ask, did you get a university degree and if so how? Personally I have found interviews to be trivial compared to exams in school, so while I do get quite nervous for some of my math examinations I have never had this experience in interviews.
- desdiv 9y ago
  I have a CS degree from a first-tier school. My experience with exams and interviews is almost the exact opposite of yours. In exams I don't get as nervous because I know that if I know the material then I will pass, if I don't then I won't. You can't bullshit your way past the professor/TA grading your exam. But in interviews I know that people who are far better than me have failed[0], and that people less competent than me could get the job. So it's not my coding ability or my experience that's being measured here, it's something else. And I don't know whether I have that something else or not. Maybe I do, maybe I just don't. [0] https://twitter.com/mxcl/status/608682016205344768?lang=en https://twitter.com/mxcl/status/608682016205344768?lang=en
  - deleted 9y ago
    [deleted]
  - javabean22 9y ago
    I feel exactly the same way. The good thing is you keep doing the interviews for some time at some point you run into normal people and then it's easy.
- b0rsuk 9y ago
  Last but not least, after you fail an exam you can usually see it with errors underlined and often have the teacher explain them for you. If you answer interview questions in a suboptimal way, you get no feedback. You don't know what was the error you made, if any. Maybe another candidate simply read about the logic puzzle that was used. The canned response I got was "We're sorry, but we can't currently recruit you. Please try again in half a year or so, maybe we will have some new opportunities." If you press on, you just get evasive answers.

maxxxxx 9y ago

That's what I do. If I can have a meaningful conversation with someone about stuff he did or stuff I am describing most likely they will do well. If the conversation gets very vague or confusing most likely things won't work out.

11thEarlOfMar 9y ago

We do it in reverse. In the first 2 hour interview, candidates learn our tech from 2 or 3 engineers. We then send them home with a set of coding tasks that are all relevant to what we do. As in functions of utilities that we actually would need to write, as opposed to puzzles. They choose one of 6 tasks, write ~100 lines of code, then come back for the next 2 hour interview and lead a code review of the code they wrote. This approach allows them to take the same time and care they would if working for us, rather than putting them in an unnatural state of coding at a white board in front of one to n scowling veterans. Engineers who are capable can clearly explain their coding choices and their style is evident. If they've borrowed too liberally or had someone else write it for them, it becomes apparent pretty quickly because they can't clearly explain the code they supposedly wrote. This is a fairly new approach for us, but so far, the candidates have appreciated it and the team reports that this approach gives them a good sense of the candidates ability.

lj3 9y ago
So, based on the new information presented in the article that people who can talk clearly about code can also write code, are you likely to change your interview practices to omit the take home task and ask the candidate to talk about code that somebody else wrote? It would be faster, easier and according to the data in TFA just as accurate, yes?
- 11thEarlOfMar 9y ago
  Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data. I do give candidates the option of the take home vs. white boarding. So far (it's admittedly still a pretty small sample) they've all opted for the take home. However, we'll take a look at this approach as well.
  - lj3 9y ago
    > Do you have data that shows how fast and easy the take home test approach is so I can compare? I only have anecdotal data. I'm not aware of any solid data around take home tests. All I can give is yet more anecdotal data, which is that they take a lot of time on both ends. So much time, in fact, that I've had several interviews where the interviewer never looked at the test code I was required to send in. Whether they work or not is irrelevant if nobody ever looks at the code.

graphememes 9y ago

I do this frequently and I utilize their resume to power the code. Someone says "I am a 9/10 in JS" I give them something that uses the animation frame api, the dom api, and a few more javascript APIs that I personally am not fully comfortable saying that I am a 9/10 on (been working with JS for over ten years) and see how they decipher what it is doing, what jumps out, and how they would review it / give feedback. 9/10 times they don't actually understand Javascript.

kasey_junk 9y ago

Its fairly easy to see how this correlation breaks down. It is almost a truism that someone who is good at communicating with a wide range of people, is likely to make that wide range of people think they are more technically competent than people who aren't good at communicating with a wide range of people. Especially given the virtually random nature who you will communicate with during any given interview cycle. But, its quite possible for people who are not naturally able to communicate with a wide range of people to be able to communicate with a smaller subset (say people they are more comfortable with, or people working on similar problems to them). The more interesting question is, could those people who communicate less broadly, in a situation where they do communicate well, perform technically at the same level (or better) as people who are more natural communicators? Another way to say this is, evaluating interviews is fairly prone to bias towards people who interview well. As someone who is hiring I'd be much more enthused about their results if it correlated to job performance, not hiring results. Its almost gospel at this point that there are people in software who interview well who do not make good hires, but the methodology of this article doesn't talk about this at all. All told, nothing about this research is persuasive to me on my opinion of "interviews are largely worthless in determining who will be a good hire".

amalcon 9y ago

The version of this where that code is clearly unsuitable for its stated purpose is my favorite coding interview question. The unsuitability should be extremely obvious, no "spot the missing semicolon". Maybe it has a severe performance problem, maybe an obvious security flaw, maybe a logic error. The point is to prompt a discussion of better alternative ways to do what the code is trying to do.