5 ms·

I think this is very interesting counterpoint to TripleByte's data which implied that talking about passion projects was lower signal than their coding quizzes.

I think this is very interesting counterpoint to TripleByte's data which implied that talking about passion projects was lower signal than their coding quizzes. The benchmark was performance in a long form coding interview for TripleByte, whereas Aline's is the final offer, so not exactly apples to apples.

leeny 11y ago
Aline here. Fwiw, I don't disagree with TripleByte's findings... anecdotally, after having interviewed somewhere between 500 and 1000 people in my career, I think they're right -- I, too, have observed a disconnect between how polished people sound when they talk about their work and what happens when they actually have to write code. What we're actually comparing here, though, isn't coding vs. describing projects. It's describing projects vs. resumes. And I expect that, there, resumes provide a lower signal.
- Kluny 11y ago
  Hey Aline, thanks for the article. I've subscribed to your RSS because your posts are so consistently high quality and well researched, but I actually saw this one on HN before I saw it in my inbox. Cheers.
- msandford 11y ago
  > I think they're right -- I, too, have observed a disconnect between how polished people sound when they talk about their work and what happens when they actually have to write code. I've done a lot of work that people would probably find very interesting and useful. But I tend to choke on whiteboard code interviews because they're so high stakes. Any time spent thinking about the problem looks bad, so you have to talk a lot. But I can't really think and talk at the same time. So I end up talking rather than thinking and I do poorly. Now obviously I'm going to push to move the status quo towards something that doesn't put me at a competitive disadvantage. So we both know that I'm biased. But the idea that people can talk about what they've done and answer any questions that you have (about what they did and programming in general) but still screw up on the actual "coding" part might mean that the part where you make them write code is more noise than signal. The problem is that you never find out because if someone bombs the coding part you simply chuckle and say "well that person is clearly a liar, or something!" and they don't go any further in the hiring process. So they never get hired, and because they're never hired, you can't evaluate their work performance. Which might be excellent when they're not being actively scrutinized by multiple people all at the same time in a high stakes situation. Unfortunately to try and get some objective data on this you'd have to hire several people who talk about their projects well but don't do well on the coding part. An understandably impossible task unless your client is a Google or Microsoft and they know it's just a big experiment regarding hiring. But until someone does that and reports back (and they won't because it'll be a competitive advantage) it's tough for me to swallow the "talks good but can't code so NOPE" that I tend to see bandied about. Putting someone in a pressure cooker and then measuring their performance will only tell you how they perform in a pressure cooker. Which is usually quite distinct from what they're going to do day-to-day.
  - leeny 11y ago
    I think what you describe is a real problem and unfortunately one that getting data around is really tough, for the reasons you describe. For what it's worth, when I observed a disconnect between how well people spoke about their projects and how well they coded, it was generally a situation where someone had perfected a pretty polished self-pitch rather than a situation where I drilled down deeply into what they had done, asked them what they'd have done differently if we varied up certain constraints, etc. And when they fucked up on coding, it was on warmup problems that was something you'd reasonably expect anyone with some experience to be able to do (e.g. explain why you might want to use a hash table over a linked list for certain scenarios, reverse a string in place). That said, one of the reasons I'm really psyched about interviewing.io (the thing I'm working on now) is that we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't.
    - msandford 11y ago
      > we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't. So I think that only works if you hire everyone, whether they interview well or not. Or else the process is biasing the results and it's not representative anymore. If you really wanted to get better information you'd have to go interview people who are already employees at a particular company and have outsiders (people who don't already know them) conduct the interviews. Then when you're done you can compare the simulated hire/no hire results and the interviewers recorded confidence numbers in their evaluation against the performance evaluations of the interviewed employees. So long as the outsiders conduct many different types of interviews (especially besides what the company normally does) you might get a clearer view into what kind of interviewing works well and what doesn't. I know some people that applied to and got hired by Google. Google seems painfully aware of how uncorrelated their interviewing process is with their hiring results. The hoops that these guys jumped through I never would. So even if I was talented enough to work at Google (I won't speculate here) they'll never actually be able to hire me unless they actively recruit me and don't make me run the gauntlet. The whole problem is a really tough nut to crack. I suspect that all the pipelines are going to be biased one way or another. If I were in charge of hiring, I'd want to try and use several of them so as to not miss out on good candidates who are undervalued for whatever reason. There's a lot of talent out there, despite everyone thinking that there's a talent shortage. The error actually lies in trying to have a one-size-fits-all solution to a problem that's definitely not uniform. Companies are failing to adapt to the human-ness of their "human resources" and it's costing them.

leeny 11y ago

Aline here. Fwiw, I don't disagree with TripleByte's findings... anecdotally, after having interviewed somewhere between 500 and 1000 people in my career, I think they're right -- I, too, have observed a disconnect between how polished people sound when they talk about their work and what happens when they actually have to write code. What we're actually comparing here, though, isn't coding vs. describing projects. It's describing projects vs. resumes. And I expect that, there, resumes provide a lower signal.

Kluny 11y ago
Hey Aline, thanks for the article. I've subscribed to your RSS because your posts are so consistently high quality and well researched, but I actually saw this one on HN before I saw it in my inbox. Cheers.
msandford 11y ago
> I think they're right -- I, too, have observed a disconnect between how polished people sound when they talk about their work and what happens when they actually have to write code. I've done a lot of work that people would probably find very interesting and useful. But I tend to choke on whiteboard code interviews because they're so high stakes. Any time spent thinking about the problem looks bad, so you have to talk a lot. But I can't really think and talk at the same time. So I end up talking rather than thinking and I do poorly. Now obviously I'm going to push to move the status quo towards something that doesn't put me at a competitive disadvantage. So we both know that I'm biased. But the idea that people can talk about what they've done and answer any questions that you have (about what they did and programming in general) but still screw up on the actual "coding" part might mean that the part where you make them write code is more noise than signal. The problem is that you never find out because if someone bombs the coding part you simply chuckle and say "well that person is clearly a liar, or something!" and they don't go any further in the hiring process. So they never get hired, and because they're never hired, you can't evaluate their work performance. Which might be excellent when they're not being actively scrutinized by multiple people all at the same time in a high stakes situation. Unfortunately to try and get some objective data on this you'd have to hire several people who talk about their projects well but don't do well on the coding part. An understandably impossible task unless your client is a Google or Microsoft and they know it's just a big experiment regarding hiring. But until someone does that and reports back (and they won't because it'll be a competitive advantage) it's tough for me to swallow the "talks good but can't code so NOPE" that I tend to see bandied about. Putting someone in a pressure cooker and then measuring their performance will only tell you how they perform in a pressure cooker. Which is usually quite distinct from what they're going to do day-to-day.
- leeny 11y ago
  I think what you describe is a real problem and unfortunately one that getting data around is really tough, for the reasons you describe. For what it's worth, when I observed a disconnect between how well people spoke about their projects and how well they coded, it was generally a situation where someone had perfected a pretty polished self-pitch rather than a situation where I drilled down deeply into what they had done, asked them what they'd have done differently if we varied up certain constraints, etc. And when they fucked up on coding, it was on warmup problems that was something you'd reasonably expect anyone with some experience to be able to do (e.g. explain why you might want to use a hash table over a linked list for certain scenarios, reverse a string in place). That said, one of the reasons I'm really psyched about interviewing.io (the thing I'm working on now) is that we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't.
  - msandford 11y ago
    > we're getting a lot of comparative interview data, i.e. where the same person gets interviewed a bunch of different ways. Excited to see if we can draw some good conclusions about what works and what doesn't. So I think that only works if you hire everyone, whether they interview well or not. Or else the process is biasing the results and it's not representative anymore. If you really wanted to get better information you'd have to go interview people who are already employees at a particular company and have outsiders (people who don't already know them) conduct the interviews. Then when you're done you can compare the simulated hire/no hire results and the interviewers recorded confidence numbers in their evaluation against the performance evaluations of the interviewed employees. So long as the outsiders conduct many different types of interviews (especially besides what the company normally does) you might get a clearer view into what kind of interviewing works well and what doesn't. I know some people that applied to and got hired by Google. Google seems painfully aware of how uncorrelated their interviewing process is with their hiring results. The hoops that these guys jumped through I never would. So even if I was talented enough to work at Google (I won't speculate here) they'll never actually be able to hire me unless they actively recruit me and don't make me run the gauntlet. The whole problem is a really tough nut to crack. I suspect that all the pipelines are going to be biased one way or another. If I were in charge of hiring, I'd want to try and use several of them so as to not miss out on good candidates who are undervalued for whatever reason. There's a lot of talent out there, despite everyone thinking that there's a talent shortage. The error actually lies in trying to have a one-size-fits-all solution to a problem that's definitely not uniform. Companies are failing to adapt to the human-ness of their "human resources" and it's costing them.