4 ms·

"When a measure becomes a target, it ceases to be a good measure."[1] These coding challenges used to be a very good measure of ability, but now applicants are

"When a measure becomes a target, it ceases to be a good measure."[1] These coding challenges used to be a very good measure of ability, but now applicants are just memorizing the most common problems for each company. We're not at the point where these tests are useless but we will get there in the near future. The problem is that I haven't seen any good alternatives. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

cole-k 6y ago
The same can be said for the SAT. Unfortunately there seems to be significant enough overlap between "good test taker" (or in this case "good interviewee") and "good worker".
- centimeter 6y ago
  Studying for the SAT won’t actually improve your score much on expectation, but taking it multiple times of course will increase your expected score (since they take the max). SAT is basically an IQ test, which is of course a reasonably strong predictor of white collar job performance (ceterus paribus).
  - JoeAltmaier 6y ago
    I don't think its 'of course'. A study at the UofIowa showed only one correlation between tested psychological features and job performance (as measured by 1-year evaluation). It was 'conscientiousness'. That is, if you kept at a task until it was actually and fully done, your boss/coworkers valued your contribution. That's it. Not about being smart or educated.
    - centimeter 6y ago
      The conclusion of this IQ vs job perf paper is “it’s complicated”, but essentially every meta study they cite is pretty unequivocal that IQ predicts job performance. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4557354/#!po=0.480769 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4557354/#!po=0.... The only room for doubt is in confounding effects like the fact that people with higher IQs are likely to enter more mentally difficult professions.
indigochill 6y ago
> These coding challenges used to be a very good measure of ability I'm skeptical. The stereotypical interview question involves reciting algorithms from memory, where the thing you actually want to know of a software engineer is do they appreciate how things fit together to meet the constraints of the project. > The problem is that I haven't seen any good alternatives. Something we've done before (and had good success with in hiring good candidates) has been to pose a real problem we're dealing with now or have recently solved (so the solution and problems are fresh in the reviewers' mind) and ask the candidate for their solution at a high level. Not whiteboard coding, though. Something like if I tell you we're trying to filter out strings that contain creatively obfuscated obscenities, what should we do in order to maximize true positives while minimizing false negatives (and as the candidate offers solutions, provide sample strings that would fail their solution and see how they adapt)?
- will_pseudonym 6y ago
  Do you mind if I ask what company you're with? I'd love to put it on my list of companies with good interview practices that I would want to apply to. If you're not comfortable with getting that specific, just for my personal curiosity, is it a FAAMG/big tech company, or other? haha :)
  - indigochill 6y ago
    It's definitely not silicon valley. Small European tech company, very much just making things up as we go.
centimeter 6y ago
I agree that the predictive power of jeopardy-style interview questions has collapsed. However, I think it’s still quite predictive to ask more design-oriented and less algorithm-oriented questions, especially if there is some leeway for creative choice. It’s also relatively easy to detect people who have simply heard and memorized the question, since when you ask them for extemporaneous expansion on some aspect of the question, they usually fall apart.
MivLives 6y ago
The best coding challenge I had to do was when the company basically gave me a set of requirements, said build this. Took about an hour. Then during the interview, I was asked about my decisions, and blockers. Afterwards I was given another feature to add. This took another half an hour. I wish more companies would do this. The hardest part for me at least is staring at a blank document.
- catwind7 6y ago
  different companies need different skills and it changes even as a company grows. Part of the frustration about this entire process is that it feels very cookie cutter and sort of like a lowest-common-denominator type effort on behalf of companies. Huge companies like google are for the most part hiring people they can slot into almost any team in the org so it makes sense for them to place less emphasis on tooling expertise. They have plenty of time and money to upskill employees. The same can't be said for the vast majority of consumer start ups that really just need people who can rapidly prototype. Of course those groups are not mutually exclusive, I just think it's sad that everyone has adopted the approach that works for a select few companies
- deleted 6y ago
  [deleted]

cole-k 6y ago

The same can be said for the SAT. Unfortunately there seems to be significant enough overlap between "good test taker" (or in this case "good interviewee") and "good worker".

centimeter 6y ago
Studying for the SAT won’t actually improve your score much on expectation, but taking it multiple times of course will increase your expected score (since they take the max). SAT is basically an IQ test, which is of course a reasonably strong predictor of white collar job performance (ceterus paribus).
- JoeAltmaier 6y ago
  I don't think its 'of course'. A study at the UofIowa showed only one correlation between tested psychological features and job performance (as measured by 1-year evaluation). It was 'conscientiousness'. That is, if you kept at a task until it was actually and fully done, your boss/coworkers valued your contribution. That's it. Not about being smart or educated.
  - centimeter 6y ago
    The conclusion of this IQ vs job perf paper is “it’s complicated”, but essentially every meta study they cite is pretty unequivocal that IQ predicts job performance. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4557354/#!po=0.480769 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4557354/#!po=0.... The only room for doubt is in confounding effects like the fact that people with higher IQs are likely to enter more mentally difficult professions.

indigochill 6y ago

> These coding challenges used to be a very good measure of ability I'm skeptical. The stereotypical interview question involves reciting algorithms from memory, where the thing you actually want to know of a software engineer is do they appreciate how things fit together to meet the constraints of the project. > The problem is that I haven't seen any good alternatives. Something we've done before (and had good success with in hiring good candidates) has been to pose a real problem we're dealing with now or have recently solved (so the solution and problems are fresh in the reviewers' mind) and ask the candidate for their solution at a high level. Not whiteboard coding, though. Something like if I tell you we're trying to filter out strings that contain creatively obfuscated obscenities, what should we do in order to maximize true positives while minimizing false negatives (and as the candidate offers solutions, provide sample strings that would fail their solution and see how they adapt)?

will_pseudonym 6y ago
Do you mind if I ask what company you're with? I'd love to put it on my list of companies with good interview practices that I would want to apply to. If you're not comfortable with getting that specific, just for my personal curiosity, is it a FAAMG/big tech company, or other? haha :)
- indigochill 6y ago
  It's definitely not silicon valley. Small European tech company, very much just making things up as we go.

centimeter 6y ago

I agree that the predictive power of jeopardy-style interview questions has collapsed. However, I think it’s still quite predictive to ask more design-oriented and less algorithm-oriented questions, especially if there is some leeway for creative choice. It’s also relatively easy to detect people who have simply heard and memorized the question, since when you ask them for extemporaneous expansion on some aspect of the question, they usually fall apart.

MivLives 6y ago

The best coding challenge I had to do was when the company basically gave me a set of requirements, said build this. Took about an hour. Then during the interview, I was asked about my decisions, and blockers. Afterwards I was given another feature to add. This took another half an hour. I wish more companies would do this. The hardest part for me at least is staring at a blank document.

catwind7 6y ago
different companies need different skills and it changes even as a company grows. Part of the frustration about this entire process is that it feels very cookie cutter and sort of like a lowest-common-denominator type effort on behalf of companies. Huge companies like google are for the most part hiring people they can slot into almost any team in the org so it makes sense for them to place less emphasis on tooling expertise. They have plenty of time and money to upskill employees. The same can't be said for the vast majority of consumer start ups that really just need people who can rapidly prototype. Of course those groups are not mutually exclusive, I just think it's sad that everyone has adopted the approach that works for a select few companies
deleted 6y ago
[deleted]