3 ms·

I think what you describe is a real problem and unfortunately one that getting data around is really tough, for the reasons you describe. For what it's worth,

by leeny 11y ago

I think what you describe is a real problem and unfortunately one that getting data around is really tough, for the reasons you describe. For what it's worth, when I observed a disconnect between how well people spoke about their projects and how well they coded, it was generally a situation where someone had perfected a pretty polished self-pitch rather than a situation where I drilled down deeply into what they had done, asked them what they'd have done differently if we varied up certain constraints, etc. And when they fucked up on coding, it was on warmup problems that was something you'd reasonably expect anyone with some experience to be able to do (e.g. explain why you might want to use a hash table over a linked list for certain scenarios, reverse a string in place). That said, one of the reasons I'm really psyched about interviewing.io (the thing I'm working on now) is that we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't.

msandford 11y ago
> we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't. So I think that only works if you hire everyone, whether they interview well or not. Or else the process is biasing the results and it's not representative anymore. If you really wanted to get better information you'd have to go interview people who are already employees at a particular company and have outsiders (people who don't already know them) conduct the interviews. Then when you're done you can compare the simulated hire/no hire results and the interviewers recorded confidence numbers in their evaluation against the performance evaluations of the interviewed employees. So long as the outsiders conduct many different types of interviews (especially besides what the company normally does) you might get a clearer view into what kind of interviewing works well and what doesn't. I know some people that applied to and got hired by Google. Google seems painfully aware of how uncorrelated their interviewing process is with their hiring results. The hoops that these guys jumped through I never would. So even if I was talented enough to work at Google (I won't speculate here) they'll never actually be able to hire me unless they actively recruit me and don't make me run the gauntlet. The whole problem is a really tough nut to crack. I suspect that all the pipelines are going to be biased one way or another. If I were in charge of hiring, I'd want to try and use several of them so as to not miss out on good candidates who are undervalued for whatever reason. There's a lot of talent out there, despite everyone thinking that there's a talent shortage. The error actually lies in trying to have a one-size-fits-all solution to a problem that's definitely not uniform. Companies are failing to adapt to the human-ness of their "human resources" and it's costing them.
jrapdx3 11y ago
Actually, the experiments that need to be conducted wouldn't be that hard to do. In fact I'd be surprised if research of this kind has not already been done. The problem of poor performance under scrutiny, where there is pressure for superior performance (as in a job interview) is well-described as a form of social anxiety disorder (or social phobia), a common condition affecting ~10% of adults in the US (lifetime prevalence). Of course, there's a range of mild to severe symptoms, nonetheless it affects a significant population. The implication is that in a not-so-pressured setting candidates might perform very differently. Furthermore, writing code, solving software problems are generally incremental processes more akin to watching paint dry than putting out fires for all the externally visible action there is to see. A "whiteboard" exam likely isn't a good model of the real requirements of the job. There's an enormous amount of research on testing methodology, testing is a huge industry. Ironically enough, one that is extremely reliant on software for analyzing test data in order to determine what is a good test of sets of knowledge or actual abilities. Seems like there's a clue in there somewhere about how software enterprises could find out who is really good at creating software.

by leeny 11y ago

msandford 11y ago

> we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't. So I think that only works if you hire everyone, whether they interview well or not. Or else the process is biasing the results and it's not representative anymore. If you really wanted to get better information you'd have to go interview people who are already employees at a particular company and have outsiders (people who don't already know them) conduct the interviews. Then when you're done you can compare the simulated hire/no hire results and the interviewers recorded confidence numbers in their evaluation against the performance evaluations of the interviewed employees. So long as the outsiders conduct many different types of interviews (especially besides what the company normally does) you might get a clearer view into what kind of interviewing works well and what doesn't. I know some people that applied to and got hired by Google. Google seems painfully aware of how uncorrelated their interviewing process is with their hiring results. The hoops that these guys jumped through I never would. So even if I was talented enough to work at Google (I won't speculate here) they'll never actually be able to hire me unless they actively recruit me and don't make me run the gauntlet. The whole problem is a really tough nut to crack. I suspect that all the pipelines are going to be biased one way or another. If I were in charge of hiring, I'd want to try and use several of them so as to not miss out on good candidates who are undervalued for whatever reason. There's a lot of talent out there, despite everyone thinking that there's a talent shortage. The error actually lies in trying to have a one-size-fits-all solution to a problem that's definitely not uniform. Companies are failing to adapt to the human-ness of their "human resources" and it's costing them.

jrapdx3 11y ago

Actually, the experiments that need to be conducted wouldn't be that hard to do. In fact I'd be surprised if research of this kind has not already been done. The problem of poor performance under scrutiny, where there is pressure for superior performance (as in a job interview) is well-described as a form of social anxiety disorder (or social phobia), a common condition affecting ~10% of adults in the US (lifetime prevalence). Of course, there's a range of mild to severe symptoms, nonetheless it affects a significant population. The implication is that in a not-so-pressured setting candidates might perform very differently. Furthermore, writing code, solving software problems are generally incremental processes more akin to watching paint dry than putting out fires for all the externally visible action there is to see. A "whiteboard" exam likely isn't a good model of the real requirements of the job. There's an enormous amount of research on testing methodology, testing is a huge industry. Ironically enough, one that is extremely reliant on software for analyzing test data in order to determine what is a good test of sets of knowledge or actual abilities. Seems like there's a clue in there somewhere about how software enterprises could find out who is really good at creating software.