6 ms·

The main goal of hiring someone should be to assess how well they can do the job you are hiring for, and for anyone with job experience (i.e. not fresh out of c

The main goal of hiring someone should be to assess how well they can do the job you are hiring for, and for anyone with job experience (i.e. not fresh out of college) the best indicator of that is what have they previously achieved (especially in more recent years). How well someone can solve a whiteboard challenge or brainteaser is irrelevant unless you are hiring someone to solve 10min whiteboard challenges. Of course the difficulty is how do you assess what the candidate has honestly achieved in prior jobs - what was there personal contribution, and do they seem to have approached things in a way that suggests transferable/general skills that can be applied to the position you are hiring for. The graphic at the beginning of the article is certainly one problem - the interviewer needs to themselves have the experience to assess the candidate. There is no point having someone with 5 years of experience interview someone with 20 years, assuming you are hiring them for that level of experience. I think the best interviewing technique is just to go over the candidates experience, from most recent to as far back as you think is relevant, and get them to talk about each project. What was their role, what were their contributions, why did they make the decisions they did, what might they have done differently, etc. etc. Ask them to summarize and deep dive into the architecture, draw it on a whiteboard perhaps.

tptacek 5mo ago
Experience-based interviews are a fantastic way to select for candidates who have "failed up" through a long series of jobs. The underlying dynamic is that it usually takes more than a year to sever a technical employee; you can faceplant in a role and still wind up with a resume improvement.
- skydhash 5mo ago
  Experience as in numbers of years or project participation is flawed. But experience as in contributions and knowledge of some specific domains is good IMO.
- sanderjd 5mo ago
  I hear this a lot, but man, I just really don't think so. First of all, 1-year stints come off very poorly in this kind of discussion. (I would say that a bias against people with a bunch of short stints would be a failure mode of "experience-based interviews" rather than the failure mode you're describing.) But also, I have had these discussions with (in my judgement) both "failed up" candidates and "tons of valuable experience" candidates, and I just don't have this experience that it's difficult to differentiate. I'm fully aware that according to the internet, there is an epidemic of bullshit artists who can go deep on the architecture and tradeoffs and their contribution to the things they worked on, without having actually contributed to those things, but I dunno, the narrative just doesn't jibe with my anecdotal experience. My only uncertainty here is that I do think I have been very fortunate in the people I have worked with in my career, so I might just be getting lucky. But I have truly never worked with someone who is hired largely on the basis of a strong resume, but genuinely can't grok fizzbuzz, or whatever the contemporary equivalent of that is. I also recognize that you ran a real company doing real work on this, so I'm generally inclined to defer to your wisdom... You can tell that I'm very torn on this, because the conventional wisdom is so strongly against my perspective on it, and I generally put a good deal of weight on conventional wisdom. But man, I dunno, it all really feels like an inertia thing that I increasingly question the original foundations of. Sometimes the emperor actually doesn't have any clothes on...
  - tptacek 5mo ago
    First, they're not "1 year stints". It takes about a year to manage someone out of a role once you've decided you need to do that. Nobody makes that decision on day 1 of a report's tenure. Second, how well the "stint" comes across in the interview is a question of how skilled the candidate is at talking. You can make anything sound like anything in an interview. The interviewer has no reliable way of checking.
    - sanderjd 5mo ago
      If someone has a multi-year stint where the last year was spent managing them out, I am not convinced that this demonstrates that they are an incompetent bullshit artist. Not every employee is valuable to every company forever. I've written a bunch of words in this thread about how I'm skeptical of your second paragraph. I recognize that you and many people I know and have discussed this with are adamant about this being true, and yet I remain unconvinced. I have interviewed many people, some of whom attempted to talk up their previous experience unconvincingly, and some of whom were impressive but nevertheless unsuccessful after being hired, for various reasons. I still haven't come across this person who can go into depth about their previous projects, while being a total incompetent bullshitter. I'm certain such people exist. I'm unconvinced that they are as pervasive as all the discourse on this would have me believe.
- HarHarVeryFunny 5mo ago
  Sure, that's a risk, but that's part of the purpose of an interview - both to discover what they claim to have done, and to assess how accurate that is. This is the purpose of asking about decision making processes, alternatives, architectural summaries and deep dives, etc - if they really did everything they claimed then all relevant details should come rapidly and fluently. The problems with the alternative - interviewing based on problem solving and coding challenges are: 1) For the most part it doesn't work. Companies like Google, that have reflected on and assessed their own interviewing success, have come to the conclusion that there is little correlation between how well people interviewed in this style do on the interview vs how well they perform on the job. This is a pretty low bar to beat! 2) If you are interviewing for 10-20-30 year senior with a track record of success and adding value, how can you possibly hope to assess that via problem based challenges, especially if conducted by a less senior developer who doesn't themself have the skill set and track record you are hiring for? How do you know if the person just talks a good game and can throw around high level concepts, or can actually deliver? I do think there is some value to problem solving as a small part of the interview process, but certainly not if this comes at the expense of ignoring the candidate's actual experience and accomplishments which is the best indication of what they are capable of.
  - tptacek 5mo ago
    I have zero confidence in the ability of interviewers to reliably and repeatable parse out subtle characterological details in candidates. Like I said, the interviews you're talking about are essentially random functions. Google has notoriously one of the worst hiring processes in the entire tech industry.
    - HarHarVeryFunny 5mo ago
      > Like I said, the interviews you're talking about are essentially random functions. Only if you are REALLY bad at interviewing ! And what's the alternative - you want to hire someone to head up your infrastructure modernization based on acing LeetCode challenges and drawing a few diagrams on the whiteboard ?
sanderjd 5mo ago
Yeah this is my "natural" interviewing style. Like, I have a resume, I'm talking to a person, my natural curiosity about the person and their work leads me to exactly the kind of conversation you're describing. Then my hire / no-hire intuition is basically "am I impressed with them after that conversation?". But then I've also read a huge amount about interviewing "correctly" over the years, probably starting with the fizzbuzz article, and eventually participating in "Big Tech" interview panel training, etc. And all of this directly contradicts this natural intuition that I have, and which your comment is espousing. So I'm honestly left with pretty strong cognitive dissonance about it at this point. Are we wrong? Or is everyone else wrong? How can this consensus on the "right way" have become so ingrained for so long while being so wrong? (I also haven't been involved in a lot of hiring or at a big tech company since 2022, so I also have no idea how things have evolved to adapt to the advent of AI tools. Surely nobody is doing the same kinds of whiteboard problems as they used to do! Right?!)
- jghn 5mo ago
  This is still my preferred method. The one counterargument that I accept as valid is that this method struggles with equity and bias. It's impossible to have a conversational interview style and ensure everyone is getting equal treatment. And it's impossible to rule out subconscious biases as a factor while you're having that conversation. While giving people cookie cutter panels doesn't completely remove those issues, it helps a lot.
  - sanderjd 5mo ago
    Yes. Agreed.
- Aurornis 5mo ago
  Hiring based on gut feeling about how impressive the candidate feels can be misleading when you only do a little hiring. It can work for small samples sizes if you have a strong front end filter or you are primarily getting candidates through trusted referrals. Then one day you encounter a candidate who is great at impressing people. They leave you feeling excited with the possibility of working with them. You feel delighted after each encounter. Then you hire them and they’re not good. At all. They don’t know basic things that you assumed they would based on how they spoke in conversation. They used all the right words and maybe even recited the precisely correct things to say for a system design question about past work, but when they have to do the same work they’re lost. It’s a weird feeling to discover your intuition about someone was completely wrong, because we all think we’re better than average at separating the wheat from the chaff. I think it happens to everyone who does hiring at scale.
Aurornis 5mo ago
> Of course the difficulty is how to you assess what the candidate has honestly achieved in prior jobs - what was there personal contribution, > What was their role, what were their contributions I got a hard reality check on this when my company was getting close to hiring someone I worked with at a previous company. Their resume claimed a lot of achievements that other teams had handled. Their work was always a problem and had to be quarantined from production systems because they were very careless and would even submit broken code without testing it. They would spend a lot of time in Slack discussing other team’s work so they got good at talking about it. I think they also used a trick where they claim to have competing offers from other companies and not a lot of time for interviews, pressuring the interviewers to do discussion and resume reviews instead of getting into technical questions. This wasn’t the only time, just the first time. After that I started doing more checking for claims on resumes, including calling old friends who worked with a person. It’s crazy how often someone will claim some responsibility or experience on their resume assuming that nobody will ever check it.
- sanderjd 5mo ago
  I definitely think checking references should be more core to the process. I don't really understand why it isn't. Legal concerns? Or just because it takes a lot of time? I would personally feel way better about interviewing for jobs if it was like peer reviews during a performance review cycle.
siliconc0w 5mo ago
Just do one project. Ask them for an interesting or challenging problem they had, go over it at a high level from a business and technical perspective, and then dig in as deep as you can on specific parts. Ask about business impact, how was that measured, what were the tradeoffs, what made it especially difficult, what were the alternatives, what they'd do differently, etc. If they can go deeper than you can, then that is a good sign.

tptacek 5mo ago

Experience-based interviews are a fantastic way to select for candidates who have "failed up" through a long series of jobs. The underlying dynamic is that it usually takes more than a year to sever a technical employee; you can faceplant in a role and still wind up with a resume improvement.

skydhash 5mo ago
Experience as in numbers of years or project participation is flawed. But experience as in contributions and knowledge of some specific domains is good IMO.
sanderjd 5mo ago
I hear this a lot, but man, I just really don't think so. First of all, 1-year stints come off very poorly in this kind of discussion. (I would say that a bias against people with a bunch of short stints would be a failure mode of "experience-based interviews" rather than the failure mode you're describing.) But also, I have had these discussions with (in my judgement) both "failed up" candidates and "tons of valuable experience" candidates, and I just don't have this experience that it's difficult to differentiate. I'm fully aware that according to the internet, there is an epidemic of bullshit artists who can go deep on the architecture and tradeoffs and their contribution to the things they worked on, without having actually contributed to those things, but I dunno, the narrative just doesn't jibe with my anecdotal experience. My only uncertainty here is that I do think I have been very fortunate in the people I have worked with in my career, so I might just be getting lucky. But I have truly never worked with someone who is hired largely on the basis of a strong resume, but genuinely can't grok fizzbuzz, or whatever the contemporary equivalent of that is. I also recognize that you ran a real company doing real work on this, so I'm generally inclined to defer to your wisdom... You can tell that I'm very torn on this, because the conventional wisdom is so strongly against my perspective on it, and I generally put a good deal of weight on conventional wisdom. But man, I dunno, it all really feels like an inertia thing that I increasingly question the original foundations of. Sometimes the emperor actually doesn't have any clothes on...
- tptacek 5mo ago
  First, they're not "1 year stints". It takes about a year to manage someone out of a role once you've decided you need to do that. Nobody makes that decision on day 1 of a report's tenure. Second, how well the "stint" comes across in the interview is a question of how skilled the candidate is at talking. You can make anything sound like anything in an interview. The interviewer has no reliable way of checking.
  - sanderjd 5mo ago
    If someone has a multi-year stint where the last year was spent managing them out, I am not convinced that this demonstrates that they are an incompetent bullshit artist. Not every employee is valuable to every company forever. I've written a bunch of words in this thread about how I'm skeptical of your second paragraph. I recognize that you and many people I know and have discussed this with are adamant about this being true, and yet I remain unconvinced. I have interviewed many people, some of whom attempted to talk up their previous experience unconvincingly, and some of whom were impressive but nevertheless unsuccessful after being hired, for various reasons. I still haven't come across this person who can go into depth about their previous projects, while being a total incompetent bullshitter. I'm certain such people exist. I'm unconvinced that they are as pervasive as all the discourse on this would have me believe.
HarHarVeryFunny 5mo ago
Sure, that's a risk, but that's part of the purpose of an interview - both to discover what they claim to have done, and to assess how accurate that is. This is the purpose of asking about decision making processes, alternatives, architectural summaries and deep dives, etc - if they really did everything they claimed then all relevant details should come rapidly and fluently. The problems with the alternative - interviewing based on problem solving and coding challenges are: 1) For the most part it doesn't work. Companies like Google, that have reflected on and assessed their own interviewing success, have come to the conclusion that there is little correlation between how well people interviewed in this style do on the interview vs how well they perform on the job. This is a pretty low bar to beat! 2) If you are interviewing for 10-20-30 year senior with a track record of success and adding value, how can you possibly hope to assess that via problem based challenges, especially if conducted by a less senior developer who doesn't themself have the skill set and track record you are hiring for? How do you know if the person just talks a good game and can throw around high level concepts, or can actually deliver? I do think there is some value to problem solving as a small part of the interview process, but certainly not if this comes at the expense of ignoring the candidate's actual experience and accomplishments which is the best indication of what they are capable of.
- tptacek 5mo ago
  I have zero confidence in the ability of interviewers to reliably and repeatable parse out subtle characterological details in candidates. Like I said, the interviews you're talking about are essentially random functions. Google has notoriously one of the worst hiring processes in the entire tech industry.
  - HarHarVeryFunny 5mo ago
    > Like I said, the interviews you're talking about are essentially random functions. Only if you are REALLY bad at interviewing ! And what's the alternative - you want to hire someone to head up your infrastructure modernization based on acing LeetCode challenges and drawing a few diagrams on the whiteboard ?

sanderjd 5mo ago

Yeah this is my "natural" interviewing style. Like, I have a resume, I'm talking to a person, my natural curiosity about the person and their work leads me to exactly the kind of conversation you're describing. Then my hire / no-hire intuition is basically "am I impressed with them after that conversation?". But then I've also read a huge amount about interviewing "correctly" over the years, probably starting with the fizzbuzz article, and eventually participating in "Big Tech" interview panel training, etc. And all of this directly contradicts this natural intuition that I have, and which your comment is espousing. So I'm honestly left with pretty strong cognitive dissonance about it at this point. Are we wrong? Or is everyone else wrong? How can this consensus on the "right way" have become so ingrained for so long while being so wrong? (I also haven't been involved in a lot of hiring or at a big tech company since 2022, so I also have no idea how things have evolved to adapt to the advent of AI tools. Surely nobody is doing the same kinds of whiteboard problems as they used to do! Right?!)

jghn 5mo ago
This is still my preferred method. The one counterargument that I accept as valid is that this method struggles with equity and bias. It's impossible to have a conversational interview style and ensure everyone is getting equal treatment. And it's impossible to rule out subconscious biases as a factor while you're having that conversation. While giving people cookie cutter panels doesn't completely remove those issues, it helps a lot.
- sanderjd 5mo ago
  Yes. Agreed.
Aurornis 5mo ago
Hiring based on gut feeling about how impressive the candidate feels can be misleading when you only do a little hiring. It can work for small samples sizes if you have a strong front end filter or you are primarily getting candidates through trusted referrals. Then one day you encounter a candidate who is great at impressing people. They leave you feeling excited with the possibility of working with them. You feel delighted after each encounter. Then you hire them and they’re not good. At all. They don’t know basic things that you assumed they would based on how they spoke in conversation. They used all the right words and maybe even recited the precisely correct things to say for a system design question about past work, but when they have to do the same work they’re lost. It’s a weird feeling to discover your intuition about someone was completely wrong, because we all think we’re better than average at separating the wheat from the chaff. I think it happens to everyone who does hiring at scale.

Aurornis 5mo ago

> Of course the difficulty is how to you assess what the candidate has honestly achieved in prior jobs - what was there personal contribution, > What was their role, what were their contributions I got a hard reality check on this when my company was getting close to hiring someone I worked with at a previous company. Their resume claimed a lot of achievements that other teams had handled. Their work was always a problem and had to be quarantined from production systems because they were very careless and would even submit broken code without testing it. They would spend a lot of time in Slack discussing other team’s work so they got good at talking about it. I think they also used a trick where they claim to have competing offers from other companies and not a lot of time for interviews, pressuring the interviewers to do discussion and resume reviews instead of getting into technical questions. This wasn’t the only time, just the first time. After that I started doing more checking for claims on resumes, including calling old friends who worked with a person. It’s crazy how often someone will claim some responsibility or experience on their resume assuming that nobody will ever check it.

sanderjd 5mo ago
I definitely think checking references should be more core to the process. I don't really understand why it isn't. Legal concerns? Or just because it takes a lot of time? I would personally feel way better about interviewing for jobs if it was like peer reviews during a performance review cycle.

siliconc0w 5mo ago

Just do one project. Ask them for an interesting or challenging problem they had, go over it at a high level from a business and technical perspective, and then dig in as deep as you can on specific parts. Ask about business impact, how was that measured, what were the tradeoffs, what made it especially difficult, what were the alternatives, what they'd do differently, etc. If they can go deeper than you can, then that is a good sign.