5 ms·

Sometimes I wonder about the how much better leetcode interviews are at picking the best candidate than a random number generator would be (obviously after doin

Sometimes I wonder about the how much better leetcode interviews are at picking the best candidate than a random number generator would be (obviously after doing all the other interview stuff other than the leetcode)

disgruntledphd2 5y ago
This is a great question, more generally what is the expected improvement in the quality of hire conditional on some selection method. The famous Hunter & Schmidt meta-analysis suggests that only work-sample tests and IQ tests work well enough to use, but it's an old set of studies so hard to know if it still applies.
- emodendroket 5y ago
  But isn't this a work sample test?
  - disgruntledphd2 5y ago
    Sortof. From my vague remembrance of this study, the closer the work sample was to the actual work the better it predicted job performance. So debugging or implementing some small stuff against a real application would perform better than leetcode. TBH, I think the original point of algorithms was to act as disguised intelligence test, which are tricky to use in US hiring.
    - emodendroket 5y ago
      Well, the problem is that you can't actually have a "realistic" work problem in the space of an interview. Given that constraint I think it's a reasonable approximation.
      - disgruntledphd2 5y ago
        If programmers spent their time implementing sorting problems, I would agree ;)
        emodendroket 5y ago
        Have you ever actually been asked to implement a sorting algorithm or balance a tree from memory? I've done a lot of whiteboard interviews, including at some of the Big Ns, and I can't say I ever experienced this, despite those exercises being used as a kind of metonym for whiteboarding.
        disgruntledphd2 5y ago
        It was mostly a joke, I'm a DS so I get arbitrary take homes rather than leetcode. The more general point is that the algorithmic approaches from leetcode problems have not a lot of relation to what most programmers do all day, and as such, are less useful as a work-sample test. Doing a take-home where you fix some bugs would probably work better (i.e. more correlated with outcomes) than leetcode interviews.
        emodendroket 5y ago
        Well I think this is an important point, because the tests, IME, are asking you to apply the concepts to solve a toy problem, not to actually implement stuff like sorting algorithms from memory. The latter is indeed unrealistic, but the former is something my job actually does entail.
      - noleetcode 5y ago
        Sure you can. At least, if our baseline for "realism" is, "an approximation of what someone will do on the job". I'd say asking someone to regurgitate some solution for a random leetcode problem is far worse an approximation than asking someone to write, say, a little toy API that does nothing more than retrieve a value out of a set. See how well they're able to develop in a language of their choosing. Can they get started immediately or do they stumble putting together the first little building blocks? Treat it like a "real-world example" and make that clear up front. Do they think about logging and metrics? (for the purposes of a toy interview problem, just writing to stdout for both would be sufficient). Do they think about dependency injection? What about unit tests? Then follow it up by asking them to modify a bit of their logic. ("okay, we've got it returning a matching value from the set if it exists - what if we wanted to add in wildcard support at the end of the incoming string?"). Tons of very real things to consider, even in the constraints of a simple toy problem. As someone who works in Big Tech, I would much rather have people on my team who think about maintainability, debuggability, monitoring, what can go wrong, etc. etc. (and have shown during an interview they're capable of writing some trivial business logic around that) than someone who absolutely nailed mirroring a binary tree and solving the longest common sub-sequence problem.
- mavelikara 5y ago
  I can't find a link to the original 1998 paper, but the authors have an updated paper here: https://home.ubalt.edu/tmitch/645/session%204/Schmidt%20&%20Oh%20validity%20and%20util%20100%20yrs%20of%20research%20Wk%20PPR%202016.pdf https://home.ubalt.edu/tmitch/645/session%204/Schmidt%20&%20... In the updated paper, work sample tests do not come up as important as they were in the original paper: It appears that almost all of the validity of work sample tests is captured by GMA measures, because the incremental validity is essentially zero. But work sample tests are a great mechanism when GMA cannot be used for legal or other reasons. The paper concludes with: The research evidence summarized in this article shows that different methods and combinations of methods have very different validities for predicting future job performance. Some, such as person-job fit, person- organization fit, and amount of education, have low validity. Others, such as graphology, have essentially no validity; they are equivalent to hiring randomly. Still others, such as GMA tests and integrity tests, have high validity. Of the combinations of predictors examined, two stand out as being both practical to use for most hiring and as having high composite validity: the combination of a GMA test and an integrity test (composite validity of .78); and the combination of a GMA test and a structured interview (composite validity of .76).

disgruntledphd2 5y ago

This is a great question, more generally what is the expected improvement in the quality of hire conditional on some selection method. The famous Hunter & Schmidt meta-analysis suggests that only work-sample tests and IQ tests work well enough to use, but it's an old set of studies so hard to know if it still applies.

emodendroket 5y ago
But isn't this a work sample test?
- disgruntledphd2 5y ago
  Sortof. From my vague remembrance of this study, the closer the work sample was to the actual work the better it predicted job performance. So debugging or implementing some small stuff against a real application would perform better than leetcode. TBH, I think the original point of algorithms was to act as disguised intelligence test, which are tricky to use in US hiring.
  - emodendroket 5y ago
    Well, the problem is that you can't actually have a "realistic" work problem in the space of an interview. Given that constraint I think it's a reasonable approximation.
    - disgruntledphd2 5y ago
      If programmers spent their time implementing sorting problems, I would agree ;)
      - emodendroket 5y ago
        Have you ever actually been asked to implement a sorting algorithm or balance a tree from memory? I've done a lot of whiteboard interviews, including at some of the Big Ns, and I can't say I ever experienced this, despite those exercises being used as a kind of metonym for whiteboarding.
        disgruntledphd2 5y ago
        It was mostly a joke, I'm a DS so I get arbitrary take homes rather than leetcode. The more general point is that the algorithmic approaches from leetcode problems have not a lot of relation to what most programmers do all day, and as such, are less useful as a work-sample test. Doing a take-home where you fix some bugs would probably work better (i.e. more correlated with outcomes) than leetcode interviews.
        emodendroket 5y ago
        Well I think this is an important point, because the tests, IME, are asking you to apply the concepts to solve a toy problem, not to actually implement stuff like sorting algorithms from memory. The latter is indeed unrealistic, but the former is something my job actually does entail.
    - noleetcode 5y ago
      Sure you can. At least, if our baseline for "realism" is, "an approximation of what someone will do on the job". I'd say asking someone to regurgitate some solution for a random leetcode problem is far worse an approximation than asking someone to write, say, a little toy API that does nothing more than retrieve a value out of a set. See how well they're able to develop in a language of their choosing. Can they get started immediately or do they stumble putting together the first little building blocks? Treat it like a "real-world example" and make that clear up front. Do they think about logging and metrics? (for the purposes of a toy interview problem, just writing to stdout for both would be sufficient). Do they think about dependency injection? What about unit tests? Then follow it up by asking them to modify a bit of their logic. ("okay, we've got it returning a matching value from the set if it exists - what if we wanted to add in wildcard support at the end of the incoming string?"). Tons of very real things to consider, even in the constraints of a simple toy problem. As someone who works in Big Tech, I would much rather have people on my team who think about maintainability, debuggability, monitoring, what can go wrong, etc. etc. (and have shown during an interview they're capable of writing some trivial business logic around that) than someone who absolutely nailed mirroring a binary tree and solving the longest common sub-sequence problem.
mavelikara 5y ago
I can't find a link to the original 1998 paper, but the authors have an updated paper here: https://home.ubalt.edu/tmitch/645/session%204/Schmidt%20&%20Oh%20validity%20and%20util%20100%20yrs%20of%20research%20Wk%20PPR%202016.pdf https://home.ubalt.edu/tmitch/645/session%204/Schmidt%20&%20... In the updated paper, work sample tests do not come up as important as they were in the original paper: It appears that almost all of the validity of work sample tests is captured by GMA measures, because the incremental validity is essentially zero. But work sample tests are a great mechanism when GMA cannot be used for legal or other reasons. The paper concludes with: The research evidence summarized in this article shows that different methods and combinations of methods have very different validities for predicting future job performance. Some, such as person-job fit, person- organization fit, and amount of education, have low validity. Others, such as graphology, have essentially no validity; they are equivalent to hiring randomly. Still others, such as GMA tests and integrity tests, have high validity. Of the combinations of predictors examined, two stand out as being both practical to use for most hiring and as having high composite validity: the combination of a GMA test and an integrity test (composite validity of .78); and the combination of a GMA test and a structured interview (composite validity of .76).