3 ms·

Leetcode is simple: performance on it is a proxy for a combination of two metrics, raw IQ and willingness to consistently jump through a series of arbitrary hoo

Leetcode is simple: performance on it is a proxy for a combination of two metrics, raw IQ and willingness to consistently jump through a series of arbitrary hoops to achieve some end, which is a predictor of success at large companies. If you're hiring at scale, there's nothing else that matches it in terms of difficulty to game and cheapness of cost to implement. And for every engineer who says that they refuse out of principle to do Leetcode, there are ten who refuse to do time consuming take home projects and the nth implementation of a bespoke open source static site generator just to build a portfolio on Github. Most smaller companies can and must figure out other hiring niches for overlooked/available candidates since they can't effectively compete with large companies for good conventional candidates using the same hiring process (e.g. by offering candidates more flexibility, taking more hiring risks), but pretending it's not a tradeoff isn't taking the problem seriously.

mjburgess 4y ago
The problem is that we have no theory of intelligence that tells us what, if anything, IQ is actually measuring. If you renamed IQ to "linear-correlated ability to take non-domain specific tests", or "GTTQ" (general test-taking quotient). Then you'd explain every observed correlation between IQ and non-IQ performance. So, do businesses want intelligent software engineers, or those with a high GTTQ score? As research at google, iirc, has shown: high performers on coding challenges tend to do worse in practice. Any plausible theory of "competent software engineering" would, I think, place an extremely minor emphasis on a GTTQ. Intelligence is a much richer phenomenon (need for cognition, curiosity, resilliance to failure, domain-specific depth-wise thinking, cross-domain lateral, etc.)....
- scarmig 4y ago
  That research was of candidates who had experience in competitive programming competitions, and IIRC it didn't show that they did worse but that they didn't do better. There's a restriction of range effect at play here: if you look at the top "GTTQ" quartile, you're not going to see it have a huge effect in performance within that range (and the factors you mention would be far more predictive of on the job performance). But the relevant comparison isn't between how well actual candidates who barely got in perform versus how well candidates who steamrolled the GTTQ questions perform; the comparison is between the hypothetical performance of the candidates who didn't get in because they failed the GTTQ questions and the actual performance of candidates who did get in because they did well enough on GTTQ questions. For that you'd get a much stronger signal. Though it would be interesting for Google to arbitrarily give offers to 1% of people who flunk the existing interview process just to track how well they perform on the job; that would give the world some really neat metrics to look at. (At the cost of giving everyone they hire even worse imposter syndrome.)

mjburgess 4y ago

The problem is that we have no theory of intelligence that tells us what, if anything, IQ is actually measuring. If you renamed IQ to "linear-correlated ability to take non-domain specific tests", or "GTTQ" (general test-taking quotient). Then you'd explain every observed correlation between IQ and non-IQ performance. So, do businesses want intelligent software engineers, or those with a high GTTQ score? As research at google, iirc, has shown: high performers on coding challenges tend to do worse in practice. Any plausible theory of "competent software engineering" would, I think, place an extremely minor emphasis on a GTTQ. Intelligence is a much richer phenomenon (need for cognition, curiosity, resilliance to failure, domain-specific depth-wise thinking, cross-domain lateral, etc.)....

scarmig 4y ago
That research was of candidates who had experience in competitive programming competitions, and IIRC it didn't show that they did worse but that they didn't do better. There's a restriction of range effect at play here: if you look at the top "GTTQ" quartile, you're not going to see it have a huge effect in performance within that range (and the factors you mention would be far more predictive of on the job performance). But the relevant comparison isn't between how well actual candidates who barely got in perform versus how well candidates who steamrolled the GTTQ questions perform; the comparison is between the hypothetical performance of the candidates who didn't get in because they failed the GTTQ questions and the actual performance of candidates who did get in because they did well enough on GTTQ questions. For that you'd get a much stronger signal. Though it would be interesting for Google to arbitrarily give offers to 1% of people who flunk the existing interview process just to track how well they perform on the job; that would give the world some really neat metrics to look at. (At the cost of giving everyone they hire even worse imposter syndrome.)