4 ms·

Probing someone with very technical conversation about past technical projects is a much stronger filter to prevent unqualified candidates than passing CoderPad

by mlthoughts2018 8y ago

Probing someone with very technical conversation about past technical projects is a much stronger filter to prevent unqualified candidates than passing CoderPad tests, whiteboard algorithms, etc. The conditional probability you are hopelessly lacking software skills to do a job given that you nonetheless passed a TripleByte exam or something is quite high. Overfitting & memorization for the sake of the test is extremely common. But it’s much, much harder to fake competence when needing to dynamically and verbally explain technical details in a conversational interview about past work experience.

joshuamorton 8y ago
I can, with quite a high level of competence, speak about projects I had no part in implementing or that don't yet exist. It's far, far easier to fake expertise when you have more context than the person asking questions. Technical interview questions make sure that the question giver has more context than the recipient, ignoring pathological cases.
- mlthoughts2018 8y ago
  I’m sorry but you cannot do what you are claiming. It has nothing to do with whether the candidate has more context or not. In fact, conversational interviewing like this has very little to do with any of the domain specifics of the project. The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome. It is precisely the situation when someone did not have to dig into the technical weeds of a project for themselves that they will not be able to fake or fast talk their way through this type of interview. That is the number one, defining characteristic of this way of interviewing.
  - joshuamorton 8y ago
    >I’m sorry but you cannot do what you are claiming. But I have. Not in the specific context of a job interview, but I have absolutely convinced technical experts that my level of expertise in a field is above my actual level of expertise. Ironically, if this were a technical interview, I'd fail it, not because I lack the skills to convince technical experts of my non-existent abilities, but because you don't find me trustworthy. >The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome. But without context, you can't effectively do that. I, the candidate, am in control. I can steer the conversation to avoid areas where I don't have expertise by answering all kinds of things: "investigating that was someone else's responsibility", "well we never tried anything else and our current implementation works well enough that we never needed to", etc. You can't know if those are lies or not. There are all kinds of completely valid non-technical reasons for decisions that may be completely outside of a candidate's control. You're either forced to completely trust the candidate, or attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.
    - mlthoughts2018 8y ago
      > “But without context, you can't effectively do that.” This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context. > “investigating that was someone else's responsibility" This just confirms to me that you are not correct in asserting people can just skate by these discussions. If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting. > “attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.” This is just wrong. You don’t ever “just trust” the candidate, that’s the opposite of this interview style. Further, you never “attempt” to verify authenticity.. you just do verify it, since it does not require context or domain specialty to analyze reductionist decomposition of any engineering work down into primitive constituent tasks or decisions that are universal. This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
      - joshuamorton 8y ago
        >This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context. I mean, yes you do. Context tells you which decisions are important. You're asking for the context you need to assess someone's technical competency from the person whose technical competency you're attempting to assess. They have every incentive to lie, mislead, or stretch the truth to make the context they give you highlight their skills more than the real context did. And no, you can't verify that. >If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting. But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way. This is just a bias against people whose development practices aren't the same as yours. >This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure. Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic. This comment thread exactly demonstrates just how you can fail to identify a good candidate with this method because if you dislike what they're saying, you'll unconsciously convert that to them being less authentic. So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level. You've just demonstrated this perfectly.
kragen 8y ago
Are you talking about TripleByte exams from experience, or are you speculating that they are similar to other software interviewing processes? The questions they asked me covered what I think of as a staggeringly wide range of knowledge, including sysadmin stuff, detailed knowledge of four or five programming languages, POSIX semantics, high-level scalable systems architecture, and so on. It seemed to me that it would be very difficult to "memorize for the sake of the test".
- mlthoughts2018 8y ago
  Maybe we just see it differently, but I’d consider everything you’ve listed as exactly the sort of stuff that just gets memorized. I don’t even think the breadth of what you listed is that bad actually. Especially for recent grad brogrammer types with no serious life obligation time commitments, memorizing an entire program of study like that is probably only an investment of ~1 month of time, and has practically no correlation to on the job effectiveness once hired. I’m speaking from ~10 years of experience running my team’s recruiting in a quant finance firm, where many interview requirements / tests / etc., came down from executive managers, so I got to see a wide range of performance on tests of all sorts, riddles, hardcore algo trivia, etc. The sum total of all that leads me to believe quite strongly that the best signal to noise comes from super careful and tedious resume selection followed by conversational and behavioral interviews that recursively probe into more specific technical details.
  - kragen 8y ago
    I agree that any one of the questions could have been memorized, but it seems to me like it would take a year or two of constant memorization, like more than 8 hours a day, to get all of it, even assuming there was a place that had the relevant information conveniently formatted for memorization — which, as far as I know, there isn't. But maybe I'm just bad at memorizing things? I note that you didn't answer my question. Have you taken the TripleByte exam or not?

joshuamorton 8y ago

I can, with quite a high level of competence, speak about projects I had no part in implementing or that don't yet exist. It's far, far easier to fake expertise when you have more context than the person asking questions. Technical interview questions make sure that the question giver has more context than the recipient, ignoring pathological cases.

mlthoughts2018 8y ago
I’m sorry but you cannot do what you are claiming. It has nothing to do with whether the candidate has more context or not. In fact, conversational interviewing like this has very little to do with any of the domain specifics of the project. The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome. It is precisely the situation when someone did not have to dig into the technical weeds of a project for themselves that they will not be able to fake or fast talk their way through this type of interview. That is the number one, defining characteristic of this way of interviewing.
- joshuamorton 8y ago
  >I’m sorry but you cannot do what you are claiming. But I have. Not in the specific context of a job interview, but I have absolutely convinced technical experts that my level of expertise in a field is above my actual level of expertise. Ironically, if this were a technical interview, I'd fail it, not because I lack the skills to convince technical experts of my non-existent abilities, but because you don't find me trustworthy. >The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome. But without context, you can't effectively do that. I, the candidate, am in control. I can steer the conversation to avoid areas where I don't have expertise by answering all kinds of things: "investigating that was someone else's responsibility", "well we never tried anything else and our current implementation works well enough that we never needed to", etc. You can't know if those are lies or not. There are all kinds of completely valid non-technical reasons for decisions that may be completely outside of a candidate's control. You're either forced to completely trust the candidate, or attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.
  - mlthoughts2018 8y ago
    > “But without context, you can't effectively do that.” This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context. > “investigating that was someone else's responsibility" This just confirms to me that you are not correct in asserting people can just skate by these discussions. If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting. > “attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.” This is just wrong. You don’t ever “just trust” the candidate, that’s the opposite of this interview style. Further, you never “attempt” to verify authenticity.. you just do verify it, since it does not require context or domain specialty to analyze reductionist decomposition of any engineering work down into primitive constituent tasks or decisions that are universal. This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
    - joshuamorton 8y ago
      >This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context. I mean, yes you do. Context tells you which decisions are important. You're asking for the context you need to assess someone's technical competency from the person whose technical competency you're attempting to assess. They have every incentive to lie, mislead, or stretch the truth to make the context they give you highlight their skills more than the real context did. And no, you can't verify that. >If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting. But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way. This is just a bias against people whose development practices aren't the same as yours. >This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure. Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic. This comment thread exactly demonstrates just how you can fail to identify a good candidate with this method because if you dislike what they're saying, you'll unconsciously convert that to them being less authentic. So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level. You've just demonstrated this perfectly.

kragen 8y ago

Are you talking about TripleByte exams from experience, or are you speculating that they are similar to other software interviewing processes? The questions they asked me covered what I think of as a staggeringly wide range of knowledge, including sysadmin stuff, detailed knowledge of four or five programming languages, POSIX semantics, high-level scalable systems architecture, and so on. It seemed to me that it would be very difficult to "memorize for the sake of the test".

mlthoughts2018 8y ago
Maybe we just see it differently, but I’d consider everything you’ve listed as exactly the sort of stuff that just gets memorized. I don’t even think the breadth of what you listed is that bad actually. Especially for recent grad brogrammer types with no serious life obligation time commitments, memorizing an entire program of study like that is probably only an investment of ~1 month of time, and has practically no correlation to on the job effectiveness once hired. I’m speaking from ~10 years of experience running my team’s recruiting in a quant finance firm, where many interview requirements / tests / etc., came down from executive managers, so I got to see a wide range of performance on tests of all sorts, riddles, hardcore algo trivia, etc. The sum total of all that leads me to believe quite strongly that the best signal to noise comes from super careful and tedious resume selection followed by conversational and behavioral interviews that recursively probe into more specific technical details.
- kragen 8y ago
  I agree that any one of the questions could have been memorized, but it seems to me like it would take a year or two of constant memorization, like more than 8 hours a day, to get all of it, even assuming there was a place that had the relevant information conveniently formatted for memorization — which, as far as I know, there isn't. But maybe I'm just bad at memorizing things? I note that you didn't answer my question. Have you taken the TripleByte exam or not?