7 ms·

The interviewer skills ladder for high growth companies

182 points by leeny 7y ago

the_arun 7y ago
I found this is a great summary of non-technical side of the interviews. Thanks for sharing!
draw_down 7y ago
I noticed that the people who interview a lot don’t really get anything in return; in our quarterly engineering org meetings or whatever there’s a slide thanking the people who interviewed most. I’ve never gotten a thanks when I interviewed more than other times, nor any other indication that this is something my manager values. (And what your manager values is kind of everything, in my experience.) So I almost never volunteer when they need someone to cover, don’t try to put more effort into it than necessary, and have blocks in my calendar so they don’t try to schedule when I don’t want them. It may be crucial for the company (and ethically, you owe it to candidates to treat them fairly) but it’s not treated like something they value. For what it’s worth, I somehow slipped through the cracks and didn’t have to interview for quite a long time after I started. No one complained or made a note of it at all.
- csydas 7y ago
  Ideally, the trade-off from Interviewing should be that a person has some agency in the direction of the company, that their values can be reflected in the influx of new recruits, and that persons with similar ideals or interesting skillsets can be added to the floor to make it a more interesting place to work. I've had the role of global interviewing for our technical team thrust on me, and most of the time, it's just a chore. I'm constantly battling with HR/Managers who simply have a "fill seats" attitude towards hiring, and it can be a political minefield. (Professional tip; if someone asks you to take-over hiring from someone (or a group) that has not fully ceded this responsibility for any reason, don't accept) Despite the above though, our global team now looks, works, and acts like I want. Even our technically weaker Engineers have the interest to improve and the drive to get better, and universally our team agrees that the flood of questions from these weaker engineers are not wasted. This is the one bit of satisfaction I get from doing this type of hiring -- I get to work with people that do the type of work I want to see.
- kinkrtyavimoodh 7y ago
  You touch on a very important thing. It feels like no one cares about interviewing UNTIL they can use it like a cudgel to hit you in the head when you ask for a promotion. In the normal course of affairs, interviewing gets you nothing. But if you don't interview, they get a convenient excuse to not promote you.
  - draw_down 7y ago
    I’m sure you’re right, but honestly they will always find a reason if they’re looking to find one.
- deleted 7y ago
  [deleted]
emtel 7y ago
I don't think "assessing talent" during interviewing is a skill you can learn. The skill you're learning is assessing how well the candidate performed during the interviewing process. The interviewing process itself is what is assessing the talent. Put another way, I would rather have a novice interviewer use a good process than have an experienced interviewer use a bad or mediocre process. In my experience, FAANG-style whiteboard interviews are a bad process. They don't work for assessing developers, and the skill of the interviewer can't compensate for that. The only thing I have found that works consistently is some form of work-sample interview, where the candidate produces working code in their preferred development environment.
- endymi0n 7y ago
  After more than 500 interviews and having hired about 50 engineers, I have to wholeheartedly disagree. I've A/B tested my interview script and technique over all those years and there's a massive inverse correlation between how hard the answer to a question is to grade and how good of a predictor they are. Deep and open questions are the best ones, period, and learning to interpret the answers to them takes years. Interviewing is really a skill to be learned and it's a hyperdimensional optimization. One of my currently best engineers started out with one of the worst coding challenges I've ever received (think JSON marshalling by string concatenation), but he was so smart, empathetic and driven I took the time and energy to build him up. One of the best challenges was by an asshole that was so intelligent he constantly managed to sabotage himself as well as the team. TL;DR: These days I don't believe anybody anymore who proposes there's just a single skill to be checked in interviewing. Tech Interviewing is just a hard job, just like engineering itself, but it's equally fulfilling if you take the time to learn it and get good.
  - braythwayt 7y ago
    I see your answer as boiling down to "That's a false dichotomy. We need experienced interviewers working with a good interview process in the form of deep and open questions that require experience to grade correctly."
  - emtel 7y ago
pkteison 7y ago
Can anyone recommend a good book on becoming a better technical interviewer? I’ve found plenty of books on being a better interview candidate, and a few for hiring ceos or sales people, but I haven’t been able to find anything with useful suggestions for the hiring side of the table in a tech interview.
- jsjohnst 7y ago
  While I can’t recommend a specific book off hand, I’d suggest looking into books in the realm of profiling. Technics I’ve read in that area made the biggest improvement in my overall ability to interview engineering candidates.
  - Ozzie_osman 7y ago
    Me and over a dozen other contributors are working on a Guide [1] to Technical Hiring and Recruiting that's due out this fall (published by Holloway). Alex Allain (author of this blog post) is actually a contributing author and wrote our section on interviewing. If you're curious enough, and willing to spend time reviewing an early draft and giving feedback, please use the contact us link on Holloway.com. [1] https://www.holloway.com/g/technical-recruiting-hiring/about https://www.holloway.com/g/technical-recruiting-hiring/about
  - wool_gather 7y ago
    Could you be more specific about the kind of profiling you mean? My assumption is software performance profiling, but I'm not sure if that makes sense. The other kind that occurs to me is law enforcement profiling, but that makes even less sense to me. I think there's a use of the word that I'm missing to understand your comment.
- alexallain 7y ago
  (original author here) I got inspired to write this post after helping contribute a few chapters to The Holloway Guide to Technical Interviewing and Recruiting, which is in early access and has a more material that might be interesting+relevant to you: https://www.holloway.com/g/technical-recruiting-hiring/about https://www.holloway.com/g/technical-recruiting-hiring/about (It goes beyond just interviewing, as well, to cover the whole hiring process end-to-end)
deleted 7y ago
[deleted]
choppaface 7y ago
This article has some interesting tips and useful encouragement but the system it outlines has a glaring hole: there is no objective feedback loop based upon outcomes back to the interviewer. The author appears to encourage that interviewers seek feedback, but the message is not to look for data but sentiment: “One way you can tell you’re proficient is that recruiters and hiring managers try to find ways to include you in their interview loops because they know you’ll both get the candidate excited and be able to assess their work fairly, even if they take an unusual approach.” If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned. Is it possible to be aligned and “wrong” ? Of course! Invariably, gluttonous hiring practices result in the need for “bar raisers” to restore consistency, entire teams with no real focus, and an explosion of communication channels. Hypergrowth is cancerous. And if everybody is in it together, nobody will be hold another accountable. This article outlines very little about proficiency and a lot about how to brainwash new grads into believing alignment itself with a recruiting process is a very special skill. Seek evidence about real outcomes instead. Do your own critical thinking.
- bsder 7y ago
  > If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned. Even recruiters will cut you a lot of slack if your feedback is occasionally something like: "Well, we passed on that candidate. However, his knowledge of X, Y and Z were quite outstanding. Try and funnel him to a place that needs that--maybe company H?" This shows that you, as an interviewer, really did dedicate the time and effort to interview the candidate properly. Good recruiters consider this to be gold. And other hiring managers have never begrudged my negative judgments. Especially because I generally try to find positive points in a candidate and give some details--"Well, I liked X, Y, and Q, however, I am deeply concerned about lack of Z. If anybody else has counter-evidence that shows he does know Z, I'm open for it." Generally, the result is a couple other managers also saying "No, that lack of Z is real, and there are some other holes we discovered." Occasionally, you get a surprise. Someone will pipe up: "Huh? We talked about Z for almost 40 minutes and he was phenomenal. You must have had a communication misfire." It happens--then I will defer on that. I have had managers probe my consistency by sending me "ringers" that they were obviously going to hire just to see if I was being overly harsh. One manager decided to quit doing that after I threatened to hire the candidate for my own group.
tignaj 7y ago
Engineer / engineering manager here. Nicely articulated article. There is one observation that I respectfully disagree with - the one that says interviews are monotonous and boring. I have done hundreds of interviews in my career and never found them to be boring. Perhaps one the reasons is I don't do scripted interviews. It is always an exploration for me. I will start with simple questions and if I get good answers I will go deeper and deeper into the subject matter or into the adjacent area. If I don't get good answers I will retract and probe another direction. This is sort of a depth-first traversal of candidate's skills with a heuristic to stop traversing a particular branch if I see it is not a promising direction. My goal is to discover the area where the candidate is the strongest (obviously within the areas I am interested in for the particular role I am hiring for). It is very fun and almost never boring. The experience is always different because people are never the same. Yes, some candidates are just not good, but even bad ones are bad in different ways. I know common wisdom for using scripts is to have repeatable and comparable evaluations so that you can choose the best candidate. I understand the desire but I prefer to find the best in each candidate and if it means I have to use a lot of gut feeling and intuition to make the final decision then so be it, I believe it still results in better overall results. The idea to choose the person I want to hire based on how many answers to my standard questionnaire they got right was never appealing to me. I can totally see how if you do scripted interviews it can become boring, so don't. Interviews can be fun and it can be a creative process. Another way interviews can be fun is if you find someone who has stronger skills than you do in a particular area. I love it when a candidate teaches me something new. Find where their strength is and let them shine.
- Ozzie_osman 7y ago
  I agree that scripted interviews can be boring... but it's not just common wisdom that using structured interviews leads to better assessments, there's plenty of research* that shows that too. Of course, you should always take research in this field with a grain of salt, since things are different from industry to industry and company to company. But I think it's safe to assume that unstructured interviews are more prone to unconscious bias. Research also shows that most people think they are better at assessing candidates in an unstructured environment than they actually are. If you're really experienced, you might be calibrated enough to be immune to that to a large extent, but it's still hard. But yeah, I agree that it can be boring and a poor experience for both candidate and interviewer if interviews are too scripted. It also has other failures, e.g. it can fail to capture different talents that candidates might have. I personally think the best way to handle this is to mix both structured and unstructured interviews. There are plenty of other ways to make interviewing rewarding. Think of it not as a burden, but as a way to help both the company and the candidate make a possibly life-changing decision as accurately as possible. Empathy can go a long way towards humanizing the process. Honestly, one of the most rewarding things I felt as an interviewer (and then a manager) was seeing someone I interviewed (or hired) later thriving in a role. Being a part of making that happen is really fulfilling. * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172...
neilv 7y ago
The article names "making the candidate want to work with you" as the first of the two critical skillsets, and goes on to reinforce that this is important. I've thought for a while that this might be one reason why some large dotcoms (which reportedly study their interview efficacy) put even very experienced candidates through one-way leetcode whiteboard batteries. It seems similar to "negging"[1]. If there's an attraction intent behind it, that would explain a lot. If so, then perhaps we have countless other companies mimicing the big dotcoms, or at least being influenced by the practice, without being in on the dark pattern side to it. [1] https://xkcd.com/1027/ https://xkcd.com/1027/
sixtypoundhound 7y ago
My reaction to this article - we've utterly failed when it comes to training effective mid-level software engineering leaders. It feel like we are substituting rule by committee for what should ultimately be a leadership decision. If you cannot select engineers, you should not be leading them. Employee Input is certainly welcome. We don't want to hire jerks or idiots. But the manager is responsible and accountable for the quality of hires and employee performance. The other half of my life is in manufacturing / distribution. We have engineers too, managing great big piles of machinery. We RUN on engineers and mechanics. Generally led by engineers. Hell, my last CEO was a former engineer and my current COO (who runs the plants) is an engineer himself. It's engineers, all the way up. They don't need to shop candidates all over the building to figure out if they can handle the job. They know. And can make an appropriate decision. It starts with setting the expectation that engineering centered functions are to be led by engineers. And the price of leading those functions is knowing your stuff... No Excuses.

182 points by leeny 7y ago

I found this is a great summary of non-technical side of the interviews. Thanks for sharing!

I noticed that the people who interview a lot don’t really get anything in return; in our quarterly engineering org meetings or whatever there’s a slide thanking the people who interviewed most. I’ve never gotten a thanks when I interviewed more than other times, nor any other indication that this is something my manager values. (And what your manager values is kind of everything, in my experience.) So I almost never volunteer when they need someone to cover, don’t try to put more effort into it than necessary, and have blocks in my calendar so they don’t try to schedule when I don’t want them. It may be crucial for the company (and ethically, you owe it to candidates to treat them fairly) but it’s not treated like something they value. For what it’s worth, I somehow slipped through the cracks and didn’t have to interview for quite a long time after I started. No one complained or made a note of it at all.

csydas 7y ago
Ideally, the trade-off from Interviewing should be that a person has some agency in the direction of the company, that their values can be reflected in the influx of new recruits, and that persons with similar ideals or interesting skillsets can be added to the floor to make it a more interesting place to work. I've had the role of global interviewing for our technical team thrust on me, and most of the time, it's just a chore. I'm constantly battling with HR/Managers who simply have a "fill seats" attitude towards hiring, and it can be a political minefield. (Professional tip; if someone asks you to take-over hiring from someone (or a group) that has not fully ceded this responsibility for any reason, don't accept) Despite the above though, our global team now looks, works, and acts like I want. Even our technically weaker Engineers have the interest to improve and the drive to get better, and universally our team agrees that the flood of questions from these weaker engineers are not wasted. This is the one bit of satisfaction I get from doing this type of hiring -- I get to work with people that do the type of work I want to see.
kinkrtyavimoodh 7y ago
You touch on a very important thing. It feels like no one cares about interviewing UNTIL they can use it like a cudgel to hit you in the head when you ask for a promotion. In the normal course of affairs, interviewing gets you nothing. But if you don't interview, they get a convenient excuse to not promote you.
- draw_down 7y ago
  I’m sure you’re right, but honestly they will always find a reason if they’re looking to find one.
deleted 7y ago
[deleted]

emtel 7y ago

I don't think "assessing talent" during interviewing is a skill you can learn. The skill you're learning is assessing how well the candidate performed during the interviewing process. The interviewing process itself is what is assessing the talent. Put another way, I would rather have a novice interviewer use a good process than have an experienced interviewer use a bad or mediocre process. In my experience, FAANG-style whiteboard interviews are a bad process. They don't work for assessing developers, and the skill of the interviewer can't compensate for that. The only thing I have found that works consistently is some form of work-sample interview, where the candidate produces working code in their preferred development environment.

endymi0n 7y ago
After more than 500 interviews and having hired about 50 engineers, I have to wholeheartedly disagree. I've A/B tested my interview script and technique over all those years and there's a massive inverse correlation between how hard the answer to a question is to grade and how good of a predictor they are. Deep and open questions are the best ones, period, and learning to interpret the answers to them takes years. Interviewing is really a skill to be learned and it's a hyperdimensional optimization. One of my currently best engineers started out with one of the worst coding challenges I've ever received (think JSON marshalling by string concatenation), but he was so smart, empathetic and driven I took the time and energy to build him up. One of the best challenges was by an asshole that was so intelligent he constantly managed to sabotage himself as well as the team. TL;DR: These days I don't believe anybody anymore who proposes there's just a single skill to be checked in interviewing. Tech Interviewing is just a hard job, just like engineering itself, but it's equally fulfilling if you take the time to learn it and get good.
- braythwayt 7y ago
  I see your answer as boiling down to "That's a false dichotomy. We need experienced interviewers working with a good interview process in the form of deep and open questions that require experience to grade correctly."
- emtel 7y ago

pkteison 7y ago

Can anyone recommend a good book on becoming a better technical interviewer? I’ve found plenty of books on being a better interview candidate, and a few for hiring ceos or sales people, but I haven’t been able to find anything with useful suggestions for the hiring side of the table in a tech interview.

jsjohnst 7y ago
While I can’t recommend a specific book off hand, I’d suggest looking into books in the realm of profiling. Technics I’ve read in that area made the biggest improvement in my overall ability to interview engineering candidates.
- Ozzie_osman 7y ago
  Me and over a dozen other contributors are working on a Guide [1] to Technical Hiring and Recruiting that's due out this fall (published by Holloway). Alex Allain (author of this blog post) is actually a contributing author and wrote our section on interviewing. If you're curious enough, and willing to spend time reviewing an early draft and giving feedback, please use the contact us link on Holloway.com. [1] https://www.holloway.com/g/technical-recruiting-hiring/about https://www.holloway.com/g/technical-recruiting-hiring/about
- wool_gather 7y ago
  Could you be more specific about the kind of profiling you mean? My assumption is software performance profiling, but I'm not sure if that makes sense. The other kind that occurs to me is law enforcement profiling, but that makes even less sense to me. I think there's a use of the word that I'm missing to understand your comment.
alexallain 7y ago
(original author here) I got inspired to write this post after helping contribute a few chapters to The Holloway Guide to Technical Interviewing and Recruiting, which is in early access and has a more material that might be interesting+relevant to you: https://www.holloway.com/g/technical-recruiting-hiring/about https://www.holloway.com/g/technical-recruiting-hiring/about (It goes beyond just interviewing, as well, to cover the whole hiring process end-to-end)

deleted 7y ago

[deleted]

choppaface 7y ago

This article has some interesting tips and useful encouragement but the system it outlines has a glaring hole: there is no objective feedback loop based upon outcomes back to the interviewer. The author appears to encourage that interviewers seek feedback, but the message is not to look for data but sentiment: “One way you can tell you’re proficient is that recruiters and hiring managers try to find ways to include you in their interview loops because they know you’ll both get the candidate excited and be able to assess their work fairly, even if they take an unusual approach.” If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned. Is it possible to be aligned and “wrong” ? Of course! Invariably, gluttonous hiring practices result in the need for “bar raisers” to restore consistency, entire teams with no real focus, and an explosion of communication channels. Hypergrowth is cancerous. And if everybody is in it together, nobody will be hold another accountable. This article outlines very little about proficiency and a lot about how to brainwash new grads into believing alignment itself with a recruiting process is a very special skill. Seek evidence about real outcomes instead. Do your own critical thinking.

bsder 7y ago
> If recruiters and hiring managers want you on a panel, it signals that you’re unlikely to disrupt their pipelines by rejecting candidates. This is especially true for high-growth companies (e.g. Twitter where ICs would do 8 onsites a week; I did 5 per week for years). They like you not because you’re proficient but because you’re aligned. Even recruiters will cut you a lot of slack if your feedback is occasionally something like: "Well, we passed on that candidate. However, his knowledge of X, Y and Z were quite outstanding. Try and funnel him to a place that needs that--maybe company H?" This shows that you, as an interviewer, really did dedicate the time and effort to interview the candidate properly. Good recruiters consider this to be gold. And other hiring managers have never begrudged my negative judgments. Especially because I generally try to find positive points in a candidate and give some details--"Well, I liked X, Y, and Q, however, I am deeply concerned about lack of Z. If anybody else has counter-evidence that shows he does know Z, I'm open for it." Generally, the result is a couple other managers also saying "No, that lack of Z is real, and there are some other holes we discovered." Occasionally, you get a surprise. Someone will pipe up: "Huh? We talked about Z for almost 40 minutes and he was phenomenal. You must have had a communication misfire." It happens--then I will defer on that. I have had managers probe my consistency by sending me "ringers" that they were obviously going to hire just to see if I was being overly harsh. One manager decided to quit doing that after I threatened to hire the candidate for my own group.

tignaj 7y ago

Engineer / engineering manager here. Nicely articulated article. There is one observation that I respectfully disagree with - the one that says interviews are monotonous and boring. I have done hundreds of interviews in my career and never found them to be boring. Perhaps one the reasons is I don't do scripted interviews. It is always an exploration for me. I will start with simple questions and if I get good answers I will go deeper and deeper into the subject matter or into the adjacent area. If I don't get good answers I will retract and probe another direction. This is sort of a depth-first traversal of candidate's skills with a heuristic to stop traversing a particular branch if I see it is not a promising direction. My goal is to discover the area where the candidate is the strongest (obviously within the areas I am interested in for the particular role I am hiring for). It is very fun and almost never boring. The experience is always different because people are never the same. Yes, some candidates are just not good, but even bad ones are bad in different ways. I know common wisdom for using scripts is to have repeatable and comparable evaluations so that you can choose the best candidate. I understand the desire but I prefer to find the best in each candidate and if it means I have to use a lot of gut feeling and intuition to make the final decision then so be it, I believe it still results in better overall results. The idea to choose the person I want to hire based on how many answers to my standard questionnaire they got right was never appealing to me. I can totally see how if you do scripted interviews it can become boring, so don't. Interviews can be fun and it can be a creative process. Another way interviews can be fun is if you find someone who has stronger skills than you do in a particular area. I love it when a candidate teaches me something new. Find where their strength is and let them shine.

Ozzie_osman 7y ago
I agree that scripted interviews can be boring... but it's not just common wisdom that using structured interviews leads to better assessments, there's plenty of research* that shows that too. Of course, you should always take research in this field with a grain of salt, since things are different from industry to industry and company to company. But I think it's safe to assume that unstructured interviews are more prone to unconscious bias. Research also shows that most people think they are better at assessing candidates in an unstructured environment than they actually are. If you're really experienced, you might be calibrated enough to be immune to that to a large extent, but it's still hard. But yeah, I agree that it can be boring and a poor experience for both candidate and interviewer if interviews are too scripted. It also has other failures, e.g. it can fail to capture different talents that candidates might have. I personally think the best way to handle this is to mix both structured and unstructured interviews. There are plenty of other ways to make interviewing rewarding. Think of it not as a burden, but as a way to help both the company and the candidate make a possibly life-changing decision as accurately as possible. Empathy can go a long way towards humanizing the process. Honestly, one of the most rewarding things I felt as an interviewer (and then a manager) was seeing someone I interviewed (or hired) later thriving in a role. Being a part of making that happen is really fulfilling. * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172.1733&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.172...

neilv 7y ago

The article names "making the candidate want to work with you" as the first of the two critical skillsets, and goes on to reinforce that this is important. I've thought for a while that this might be one reason why some large dotcoms (which reportedly study their interview efficacy) put even very experienced candidates through one-way leetcode whiteboard batteries. It seems similar to "negging"[1]. If there's an attraction intent behind it, that would explain a lot. If so, then perhaps we have countless other companies mimicing the big dotcoms, or at least being influenced by the practice, without being in on the dark pattern side to it. [1] https://xkcd.com/1027/ https://xkcd.com/1027/

sixtypoundhound 7y ago

My reaction to this article - we've utterly failed when it comes to training effective mid-level software engineering leaders. It feel like we are substituting rule by committee for what should ultimately be a leadership decision. If you cannot select engineers, you should not be leading them. Employee Input is certainly welcome. We don't want to hire jerks or idiots. But the manager is responsible and accountable for the quality of hires and employee performance. The other half of my life is in manufacturing / distribution. We have engineers too, managing great big piles of machinery. We RUN on engineers and mechanics. Generally led by engineers. Hell, my last CEO was a former engineer and my current COO (who runs the plants) is an engineer himself. It's engineers, all the way up. They don't need to shop candidates all over the building to figure out if they can handle the job. They know. And can make an appropriate decision. It starts with setting the expectation that engineering centered functions are to be led by engineers. And the price of leading those functions is knowing your stuff... No Excuses.