3 ms·

"it's incredibly hard to identify an engineer's real ability levels in an interview process". It is more or less true (I would not use "incredibly hard", just "

"it's incredibly hard to identify an engineer's real ability levels in an interview process". It is more or less true (I would not use "incredibly hard", just "hard") with the qualifier "using a standardized, working-at-scale interview process". If you had to choose one engineer among 100 candidates using whatever approach you want but within a time limit of 10 hours per candidate, I am pretty confident you would accurately gauge the "quality" of candidates.

brundolf 4y ago
I'm not. I think I could filter out the ones that are totally unqualified. Probably reliably end up with one in the top 50%; maybe even the top 25%. But one that's going to be sustainably effective, alongside the rest of the team members, through a variety of real-world situations? Nah. It's basically random. The same interviewee could turn out to be anywhere from "the best hire we've ever had" to "can scrape by but isn't very effective overall", and in many cases there would be no way to tell beforehand. This is why some companies just give up and use Leetcode. If it's going to be mostly random anyway, they figure they might as well save time and effort. I wouldn't choose to go that route, but I get it.
- borroka 4y ago
  I am sure we agree that it is not random for many other professions. Why should the filter almost random for SWEs? What's so special about them? Let's say you are asked to rate a boxer. You have their resume (who and when they fought) and you can "interview" (no sparring partner, even) them for 10 hours. If I am going to rate them on a scale 1-10, I could certainly rate them with an error of +-10% (on average), that is I could make an accurate prediction of their "worth" in the next year. It is not just the "interview" without sparring partners, it is the interview + resume. Only the interview and no resume, well, there have been a lot of promising boxers who went nowhere, and in fact it is much more challenging to evaluate juniors than seniors. In my experience, I was able to rate SWE and ML types pretty accurately, conditioned on the fact that they were hired (and they hired some people I rejected and some I approved). I had them for one hour and I read the resume. I think the secret was that I asked good questions.
  - brundolf 4y ago
    Because software engineering isn't boxing? It's hard to separate a team's performance from the rest of the org, and it's hard to separate an individual's performance from the rest of their team. And even if you could- how do you measure success? Unlike boxing, there's no binary win/loss condition. KPIs? But what about an engineer who does a brilliant job quietly making sure a piece of software will be maintainable over the next decade? How do you measure that, even from inside the organization, much less outside of it? And if you could, how do you know that success in one org (with a particular codebase, stack, team culture, velocity, etc) translates to success in a different one? And then because of all the above: even a resume doesn't tell you very much. It tells you they convinced other employers they were worth hiring/keeping, but those other employers were subject to the same imprecisions/measurement problems that you are. It's very possible they were all wrong about the quality of this person's contributions. Software development is a deeply human activity that's impossible to reduce to simplified signifiers. It's my opinion that the only way to evaluate an engineer is to be another engineer working alongside them for an extended time. And even then- that doesn't necessarily translate to other scenarios they might work in. Another commenter here said they do a 30-day trial period with new engineers, to see if they're a good fit. From a hiring perspective, this is the only process I can think of that would be decently accurate. Though obviously it would be pretty rough on the engineer to not know if they have job security for 30 days.

brundolf 4y ago

I'm not. I think I could filter out the ones that are totally unqualified. Probably reliably end up with one in the top 50%; maybe even the top 25%. But one that's going to be sustainably effective, alongside the rest of the team members, through a variety of real-world situations? Nah. It's basically random. The same interviewee could turn out to be anywhere from "the best hire we've ever had" to "can scrape by but isn't very effective overall", and in many cases there would be no way to tell beforehand. This is why some companies just give up and use Leetcode. If it's going to be mostly random anyway, they figure they might as well save time and effort. I wouldn't choose to go that route, but I get it.

borroka 4y ago
I am sure we agree that it is not random for many other professions. Why should the filter almost random for SWEs? What's so special about them? Let's say you are asked to rate a boxer. You have their resume (who and when they fought) and you can "interview" (no sparring partner, even) them for 10 hours. If I am going to rate them on a scale 1-10, I could certainly rate them with an error of +-10% (on average), that is I could make an accurate prediction of their "worth" in the next year. It is not just the "interview" without sparring partners, it is the interview + resume. Only the interview and no resume, well, there have been a lot of promising boxers who went nowhere, and in fact it is much more challenging to evaluate juniors than seniors. In my experience, I was able to rate SWE and ML types pretty accurately, conditioned on the fact that they were hired (and they hired some people I rejected and some I approved). I had them for one hour and I read the resume. I think the secret was that I asked good questions.
- brundolf 4y ago
  Because software engineering isn't boxing? It's hard to separate a team's performance from the rest of the org, and it's hard to separate an individual's performance from the rest of their team. And even if you could- how do you measure success? Unlike boxing, there's no binary win/loss condition. KPIs? But what about an engineer who does a brilliant job quietly making sure a piece of software will be maintainable over the next decade? How do you measure that, even from inside the organization, much less outside of it? And if you could, how do you know that success in one org (with a particular codebase, stack, team culture, velocity, etc) translates to success in a different one? And then because of all the above: even a resume doesn't tell you very much. It tells you they convinced other employers they were worth hiring/keeping, but those other employers were subject to the same imprecisions/measurement problems that you are. It's very possible they were all wrong about the quality of this person's contributions. Software development is a deeply human activity that's impossible to reduce to simplified signifiers. It's my opinion that the only way to evaluate an engineer is to be another engineer working alongside them for an extended time. And even then- that doesn't necessarily translate to other scenarios they might work in. Another commenter here said they do a 30-day trial period with new engineers, to see if they're a good fit. From a hiring perspective, this is the only process I can think of that would be decently accurate. Though obviously it would be pretty rough on the engineer to not know if they have job security for 30 days.