14 ms·

> There are better ways to test candidates. Genuine question here: what's the better way? I'm not a fan of whiteboard coding challenges, even though they got

> There are better ways to test candidates. Genuine question here: what's the better way? I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job. On the other hand, take home challenges eat up a lot of time and credentialist filters like going to a name brand school (or even having the right major) filter me out. My go-to for when I'm in a hiring position again would be recommendations from my personal network, but that has its own flaws and doesn't scale very well. I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires.

chapium 6y ago
How do you interview for any other job?
- mattkrause 6y ago
  It's often less...adversarial. I work somewhere in the middle of biology/neuroscience and ML. Interviews for essentially the same job (e.g., building brain-inspired robots) are hilariously different depending on how the org views itself. In a "science" organization, the interviews mostly involved talking about what I've done in the past and how I'd approach new issues (technical and behavioral). There were certainly a few probing questions about things that seemed designed to check up on claims, but my experience seemed mostly taken at face value. A coding test, for example, was skipped because an interviewer had already seen some of my code and thought it was fine. "Tech" interviewers seemed to want everything "proved" on-site during an all-day slog. I reversed strings and traversed graphs on a whiteboard. Someone quizzed me on facts and figures about the human eye (which they were reading out of the same book I learned them from). To me, this seems to test immediate recall and stamina/stress tolerance more than anything actually related to my R&D skills.
  - UncleOxidant 6y ago
    I think adversarial is the right word for the current style of tech interviewing. It wasn't always this way. Back in the 80s when I first got into tech it was a lot more conversational and in the spirit of "we really need help, what can you do for us?". I think something more akin to a thesis defense would be better than what we're doing now. Prior to the interview have the candidate write up a description of a project they worked on and particularly liked along with a set of slides. Have the candidate send this presentation to the main interviewer who will distribute it to all of the interviewers. The interview then becomes a presentation where interviewers can ask questions about the presentation. I've seen this done in industry, but it was only when we interviewed folks who were PhDs coming from academia - I don't see why we couldn't do this for all kinds of candidates, not just those coming from academia.
- mdorazio 6y ago
  I interview, on average, two people a month for non- or semi-technical roles. Everything from financial analysts to contracts specialists and project managers. Interviews are generally structured to give interviewers a good idea about four items: 1) What did you, personally, work on previously that's relevant to this role? This usually means pretty in-depth discussions about previous roles and projects, accomplishments, tasks, skills gained, etc. 2) How do you approach tasks and problems at work? This is usually a combination of asking about why the candidate did things a certain way previously and how they would approach theoretical situations in the future. The goal is to uncover the mindset the candidate uses to actually get stuff done in a changing business environment. 3) How do you interact with teammates and managers? This is the concrete version of "culture fit" and includes asking about various interpersonal situations likely to have been encountered in the past as well as discussions about how the candidate would handle quirks of the office environment at the new company (ex. some companies love people to make decisions quickly on their own while others hate it and mandate team consensus). 4) Do you have the base-level skills needed for the role? This is often far more than the technical aspect. After some basic questions to establish that the candidate knows the basic processes, software tools, artifacts, and terminology, it's much more important to establish if they have the chops to actually be successful rather than a "task completed" robot. Can the candidate communicate ideas and opinions clearly? Did they previously work in a collaborative way rather than an adversarial way with teammates? Can they present findings and issues to management effectively? Have they previously worked with supporting teams and vendors outside their main role to solve problems? Etc. In most jobs the technical skill part is a small piece of what the person will actually be doing so the interview is a means to collaboratively establish if the role is a fit for the candidate as a whole and vice versa. If I only looked at technical skill and immediately disqualified people for failing concocted and unrealistic tests, I'd end up with mostly shitty candidates who aren't actually very good at the wider job.
  - chapium 6y ago
    Thank you for sharing your experience.
s17n 6y ago
Jeff Dean (circa 1998) would pass the resume screen anywhere and I'm pretty sure he'd do well on a coding challenge.
- wolco2 6y ago
  I would highly doubt he would get rehired today at google first try. Leetcode is a different beast and things might not be all that fresh. Someone on the hiring committee might have said yes too many times this month.
  - mtlynch 6y ago
    Former Google engineer and technical interviewer here. Google doesn't use Leetcode for interviews (at least as of 2018). I'd be extremely surprised if Jeff Dean (at any point in his employment) would fail a SWE interview loop at Google. The Google interview is heavily based on data structures and algorithms, and Jeff Dean likely knows them cold. Sure there's always a degree of luck, and Google errs on the side of false negatives, but there are certain people that are so high above the hiring bar that luck becomes too tiny a factor to be relevant.
    - humanlion87 6y ago
      Yes, Google doesn't use Leetcode per se for interviews. What the parent commenter was referring to was that Google interview questions are of the type for which Leetcode has become famous for.
    - bosswipe 6y ago
      > The Google interview is heavily based on data structures and algorithms That's pretty much what Leetcode is. When I interviewed at Google I was asked tree traversal and hashmap design questions that were very similar to leetcode questions.
  - joshuamorton 6y ago
    Given the things Jeff actively works on, I have little doubt he could pass a technical interview loop.
burnthrow 6y ago
^^^ This is why you need tests of hard skills. This is the narration of a C player hiring D player friends. And it's "risk averse."
wolco2 6y ago
What about looking at employment history: types of companies, position, length, languages used... the basics..
- loa_in_ 6y ago
  That looks good in a conversation but under scrutiny, unless you know someone in a company mentioned, all you get is an opinion or you need to do a deep economic check of the company in question, which is nothing more than a statistic at best.
  - wolco2 6y ago
    Why would you need to put so much judgement into a single company. If the person has only worked for one company than their employment history won't be much of a factor. If someone has worked at 6-8 places over 15/20 years covering a range of skills required I don't see the risk.
thewebcount 6y ago
One of the things I've done is to hand a candidate some existing code and ask, "What do you think of this code? Do you see any obvious problems? Are there things that work but could be done better? Is there anything exceptional in there? Tell me what you see." This allows you to test that they know how to read code, which is what they'll be spending the majority of their time doing. You can ask them how they'd improve it, which should give you an idea of how they write code, too. And of course, you can see their experience by checking out their resume and if possible running some of the apps they worked on. And if they have any code on GitHub, etc., you can check that out, too.
- whymauri 6y ago
  This is pretty similar to a ML Engineering interview question I got once. Basically, you get some mildly obfuscated code implementing a statistical concept and you have to tease out what the code is doing. Then you have to fix a few bugs, the variables names, and make it easier to read.
- redshirtrob 6y ago
  Yes! I've been on the interviewee side of this a couple times and absolutely love it. It has so many nice properties: 1. It fosters meaningful conversations about code that is not dependent on the candidate solving a toy problem. 2. As you said, it tests a candidate's ability to read and understand code. 3. It's actually representative of the type of work you _will_ do at any given company, i.e. code review. The major downside is if you give the candidate code in a language they're totally unfamiliar with then that candidate will likely bomb. I think this can be mitigated by sticking with mainstream languages like C/Java/Python, or domain specific languages where not-knowing would be a deal breaker anyway, i.e. must know Swift for iOS dev.
- colmvp 6y ago
  One of the companies I interviewed at had me do this exercise at the end of the interview and to be frank, I bombed it quite badly. One issue was the syntax was so weird compared to the type of code I'm used to writing and reading from others in library code that it was hard for me to comprehend the logic without the benefit of looking it up. I was filled with anxiety working under the time pressure to understand a document of code which I had very little context of, and no previous relationship with the reviewer so I kept thinking to myself whether or not the question that came into my mind was stupid obvious. In contrast, when I had take-home tests, I scored exceptionally high on all of them, leading to offers. I recognize that a lot of people here hate them because of the time involved and it's akin to spec work, but IMO it felt much more like my usual day-to-day life as a developer. One of the take home tests had existing code to work from but again, working at my own pace and having access to online resources to read docs was much less anxiety provoking.
mtlynch 6y ago
>Genuine question here: what's the better way? One strategy that's been successful for me is skipping the technical screen and jumping to a narrowly-scoped paid assignment. If they do well, I give them a larger paid assignment, and we work our way up to regular work together. If they do poorly, I pay them for their time but tell them it's not a match. I'll add the caveat that I've only used this to hire junior to mid-level developers for part-time or short-term work. I've never tested it on a full-time hire, and I imagine that would be more difficult. I've never written in detail about hiring developers, but I've shared similar strategies for hiring writers,[0] illustrators,[1] and editors.[2] [0] https://mtlynch.io/hiring-content-writers/ https://mtlynch.io/hiring-content-writers/ [1] https://mtlynch.io/how-to-hire-a-cartoonist/ https://mtlynch.io/how-to-hire-a-cartoonist/ [2] https://mtlynch.io/editor/ https://mtlynch.io/editor/
- edoceo 6y ago
  I do gig-to-hire too. Works a treat! I've found loads of talent this way, up and down the skill chain. But, for a CTO like role the gig has to be 3mo or more.
  - Balgair 6y ago
    I'm confused. You are saying that you have had CTO interviewers that did a gig role for 3 months as a CTO? Did they also have another job at the time or was this their only job?
    - edoceo 6y ago
      This one is a bit harder (naturally). A recent/typical case was a CTO type candidate, they participated part-time in some meetings, had some exposure to the team, talking through issues with other CxO. During this time candidate is also shopping around too. It's basically buying consulting while you decide to hire. It does not generally take 3mo - but you'll add a probationary period that gives you maybe 12mo room.
      - Balgair 6y ago
pluc 6y ago
How about trusting candidates' experience? If they say they have 20 years of experience in X and you hire them, you'll know very quickly if that was a lie. Those tests are so insulting to the relationship you're pretending to want to build. Why would I bother putting false information on my resume that has the potential to get me hired and get tested daily? Isn't that why there's a probation period?! I have 20 years of experience in X but prove it? Why's the burden on me to prove what I've already told you? Are you saying I'm a liar? If my date of birth is on my cv do you expect a birth certificate to validate that information too? Anyway, that's always been my philosophy - hire on facts and fit, fire quickly if anything was wrong - and it's been pretty smooth. The great thing that it does is it prevents testing for futile thing that you will never use in your position and that are just elitist criteria or vanity wishes.
- wetpaws 6y ago
  You would be surprised how many people with 20 years of experience can barely perform at junior level
  - base698 6y ago
    You can have 20 years of experience or 20 one year experiences.
    - nicoburns 6y ago
      You can also be straight up lying about having 20 years (or any) experience. Most people won't be doing this, but it seems like a basic feature of a hiring process to have some kind of check against this.
      - pluc 6y ago
        The check is what the candidate says, isn't it? If the candidate lies, you will know the day you hire them
        6y ago
lr4444lr 6y ago
I concur with you. This debate rages on endlessly. In-person whiteboarding is the worst way to assess candidates' tech abilities... except for all of the other alternatives.
xixixao 6y ago
This is a fact. A false positive is much more costly to a company than a false negative (this is also why we allow candidates to reapply).
- justin_oaks 6y ago
  How do you measure the cost of a false negative?
  - theptip 6y ago
    The highest-order term is the opportunity cost of not having a developer now, and having to wait until the next one to pass the interview process.
  - PragmaticPulp 6y ago
    It's not explicitly quantifiable. Imagine you're a team of 2 top performers. Your workload is increasing, so your manager gets approval to hire a 3rd person so you can take on more work. Unfortunately, your manager is fooled by a charismatic person who greatly exaggerated their experience on their resume, to the point that they work about 1/4 as fast as the rest of the team and tend to code by trial-and-error. Suddenly, the expectations for your team's output have increased by 50%, but the 3rd person can only really contribute another 10-20% at most. By extension, your job just got harder. Worse yet, the average talent of the team has now gone down significantly. You spend more time fixing bugs and chasing bad work. You're also faced with the fact that you're probably getting paid the same as the over-leveled person next to you. This breeds resentment, wastes time, and ultimately leads to good people quitting for other jobs. Alternatively, imagine if the company hired quickly and fired quickly. Every time a new person showed up, you had no idea if they were actually good or if they were just rushed through the interview process. You also don't know if they're going to be around after the 3 month probationary period, so why bother building a relationship? High turnover like this tends to destroy morale.
- pseudalopex 6y ago
  Depends on the company. Google can afford to reject great candidates. A small company might have to settle for a mediocre candidate 6 months after rejecting a good candidate.
almost_usual 6y ago
> I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job. You can have code challenges that aren’t HackerRank or Leetcode. Using problems from those sites is a signal the employer or team is lazy.
balls187 6y ago
> I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job. This is the fundamental part that is broken in my experience. They are often challenges/coding puzzles, when rather white board exercises should focus on collaboration, and solving problems with code. Having a "gotcha" that shows rudimentary CS understanding is fine (such as understanding why the obvious O(N^2) is a bad solution). But having a gotcha that requires some crazy leap of faith guess (or having seen the problem before) is not. I admit, my fondness for a candidate increases with the level of complexity of coding problem they can solve; however needing hints on the harder problems doesn't mean they are not qualified.
blackrock 6y ago
Are you really an interviewer? And are you actually this clueless?
deepsun 6y ago
In eastern europe interviews are mostly experience-related. Questions are mostly about technologies that hiring team needs, the more the better, and the deeper the better. I never had a problem getting the extra experience not found on previous workplace from my pet projects and blogs before moving to US. Funnily, I've seen people who moved back from SF Bay Area, and struggled to find a job there, despite being very good at whiteboard and CS problems. Turned out eastern european companies don't care about CS problems really.
mandeepj 6y ago
> Genuine question here: what's the better way? I think the candidate should be given a choice - whether they want to solve one of these tricky puzzles or work on a domain problem.
pantulis 6y ago
I've done my good share of recruiting interviews and you can measure if the CV has been inflated just by having a conversation with the candidate. Just asking "What did you do in your last job?" and following from there works wonders. No coding exercises, no whiteboards, it's all about working a conversation with them. I can confidently say that the recruiting process was faster --less interviews--, less stressful for the candidate, and finally the number of hire employees that flopped was not that much worse. I did whiteboards for Architect positions, but once again, I just asked the candidates to draw architectures to have a friendly discussion.
kylewatson 6y ago
Honest question: How do hospitals hire doctors (industry hire)? Do they quiz them like they did to the residents on Scrubs? Do they pull up a list of symptoms and ask what they would test/try next? Do they show them x-rays and ask them to spot the coronavirus? Seriously? Or do they look at your degree and experience and conclude that if you worked at General Hospital in Atlanta for 5 years you can work at Metro Hospital in Philadelphia? I honestly don't know how other professional industries hire industry-hires (not college grads/interns) but I only hear about quizzes and riddles and online tests and shit like that in programming. Hell, do they do the same shit for electrical engineers? Chemical engineers? I really want to know if they do and if not, how is it Boeing can hire a EE without a whiteboard puzzle but can't hire a CE without that bullshit?
- klintcho 6y ago
  I posed the exact same question below. I have the exact same thoughts, and genuinely curious on the process they have. I'm not that opposed in general to whiteboarding if you feel like you can't assess the candidate. However I do feel that it sometimes feels like credential and experience similar to what you describe in the medical field: if a doctor had 5 years at an inner city hospital, I don't think they would doubt his credentials to the point the want to make him put cell structures on a whiteboard?
  - theamk 6y ago
    Doctors do have professional education, harsh exams, and they can get their licenses revoked for mistakes. Do you really want the same for programmers? "only people with Ms or PhD from accredited universities can apply, and diploma is invalidated on some mistakes"
- zizee 6y ago
  Medical doctors have a professional organization enforcing strict standards. People cannot practice medicine without passing a series of standardized exams that only recognized universities can administrator, they have and a lengthy internship process, review board etc. If they break the rules they can have their licence suspended. I think it would be a shame to bring this sort of thing to software development, but it would allow us to get rid of some of the hoops people have to jump through to ensure they can actually do the job being hired for.
klintcho 6y ago
This is not a rhetorical question: How does other industries hire high knowledge well educated candidates ? How does doctors, mechanical engineers and space flight engineers get interviewed and assessed ? As have already been reiterated in each and everyone of these threads (about leetcode/hackerrank/whiteboard coding) this is what the companies seem to think is the most proven, and produces most consistent results. But I would love to understand how other industries does this as well; does a propulsion engineer start to draw up a de Laval nozzle and calculate the exhaust gas velocity on a whiteboard? (again genuine question)
- twblalock 6y ago
  The problem is that the software industry does not have the same standards as other industries. People graduate CS programs with little to no coding ability in many cases. There is no certification program like there is for, say, accountants or CPAs. Companies set the hiring bar all over the place. At a lot of companies, bad hires stick around for years. Other industries have higher and more consistent standards, and certification processes. The software industry is the wild west, and that is partly why it has succeeded and why so many self-taught programmers have found success in it. The other side of the coin is that the quality of candidates varies more widely than in other industries.
  - pseudalopex 6y ago
    I've never seen someone with a CS degree and no coding ability. Lots of them weren't taught anything else about software engineering. But most interviews don't test anything else anyway.
- pseudalopex 6y ago
  Engineers discuss work they did mostly. Interviewers ask probing questions to catch bullshitters.
suifbwish 6y ago
I will bite. What is your recommendation for a network?
- AlchemistCamp 6y ago
  It's Reid Hoffman's idea, not mine—a referral from a trusted source is a better indicator than an interview. He's previously said that if he could only decide based on one or the other, he'd hire the candidate referred by a trusted source and skip the interview. In practice, this choice never comes up, but you can weight referrals much more strongly than interviews.
geocar 6y ago
> Genuine question here: what's the better way? You tell them what the job is like, what is expected of them, and what the best way to work with you is, then you you ask if that's what they want, and how much it's going to take to get them in the door. That's it. I wouldn't ask an accountant what the Corrodium 3 Deduction is; I'd tell them I need them to file my taxes, that I need them to own the process and chase whoever they need (even if that's me). > I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires. I think you're probably right, but if someone lies to you and says they can do a job that they clearly can't, you should fire them. This should be unusual because getting a new job can be stressful, and nobody wants to put themselves into a job they can't do, so it's rare that you even have to consider this -- but it does happen, and unless you're a sociopath, it's going to make you shy about ever having to go through it again, because you're not going to want to fire anybody. It sucks for you, and it sucks for them, and it sucks for the team who see the revolving door, but nobody benefits from the wrong fit in the long run, so you just need to suck it up and do it. So be firm, and if it turns out you're firing a lot of people straight in the door, look inwards: you're probably not being honest with them (or perhaps yourself) what the job actually is. Figuring that out instead of some new programming puzzle to torture grads with is going to pay dividends.
twblalock 6y ago
> I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires. This is absolutely why whiteboarding exists, and persists. I have interviewed a lot of people at this point in my career, and it's just not possible to tell if candidates are good by looking at their credentials or job histories. I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. I'm not talking about Leetcode-style brain teasers; I'm talking about being unable to construct a working for loop and do basic array operations in a language of their choice. I'm going to continue to do whiteboarding for candidates for this reason.
- bluefirebrand 6y ago
  > I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. That honestly sounds to me like you found some people who lied like crazy about their credentials and job histories.
  - twblalock 6y ago
    Actually one of the things the bad candidates had in common was a job history at defense contractors.

chapium 6y ago

How do you interview for any other job?

mattkrause 6y ago
It's often less...adversarial. I work somewhere in the middle of biology/neuroscience and ML. Interviews for essentially the same job (e.g., building brain-inspired robots) are hilariously different depending on how the org views itself. In a "science" organization, the interviews mostly involved talking about what I've done in the past and how I'd approach new issues (technical and behavioral). There were certainly a few probing questions about things that seemed designed to check up on claims, but my experience seemed mostly taken at face value. A coding test, for example, was skipped because an interviewer had already seen some of my code and thought it was fine. "Tech" interviewers seemed to want everything "proved" on-site during an all-day slog. I reversed strings and traversed graphs on a whiteboard. Someone quizzed me on facts and figures about the human eye (which they were reading out of the same book I learned them from). To me, this seems to test immediate recall and stamina/stress tolerance more than anything actually related to my R&D skills.
- UncleOxidant 6y ago
  I think adversarial is the right word for the current style of tech interviewing. It wasn't always this way. Back in the 80s when I first got into tech it was a lot more conversational and in the spirit of "we really need help, what can you do for us?". I think something more akin to a thesis defense would be better than what we're doing now. Prior to the interview have the candidate write up a description of a project they worked on and particularly liked along with a set of slides. Have the candidate send this presentation to the main interviewer who will distribute it to all of the interviewers. The interview then becomes a presentation where interviewers can ask questions about the presentation. I've seen this done in industry, but it was only when we interviewed folks who were PhDs coming from academia - I don't see why we couldn't do this for all kinds of candidates, not just those coming from academia.
mdorazio 6y ago
I interview, on average, two people a month for non- or semi-technical roles. Everything from financial analysts to contracts specialists and project managers. Interviews are generally structured to give interviewers a good idea about four items: 1) What did you, personally, work on previously that's relevant to this role? This usually means pretty in-depth discussions about previous roles and projects, accomplishments, tasks, skills gained, etc. 2) How do you approach tasks and problems at work? This is usually a combination of asking about why the candidate did things a certain way previously and how they would approach theoretical situations in the future. The goal is to uncover the mindset the candidate uses to actually get stuff done in a changing business environment. 3) How do you interact with teammates and managers? This is the concrete version of "culture fit" and includes asking about various interpersonal situations likely to have been encountered in the past as well as discussions about how the candidate would handle quirks of the office environment at the new company (ex. some companies love people to make decisions quickly on their own while others hate it and mandate team consensus). 4) Do you have the base-level skills needed for the role? This is often far more than the technical aspect. After some basic questions to establish that the candidate knows the basic processes, software tools, artifacts, and terminology, it's much more important to establish if they have the chops to actually be successful rather than a "task completed" robot. Can the candidate communicate ideas and opinions clearly? Did they previously work in a collaborative way rather than an adversarial way with teammates? Can they present findings and issues to management effectively? Have they previously worked with supporting teams and vendors outside their main role to solve problems? Etc. In most jobs the technical skill part is a small piece of what the person will actually be doing so the interview is a means to collaboratively establish if the role is a fit for the candidate as a whole and vice versa. If I only looked at technical skill and immediately disqualified people for failing concocted and unrealistic tests, I'd end up with mostly shitty candidates who aren't actually very good at the wider job.
- chapium 6y ago
  Thank you for sharing your experience.

s17n 6y ago

Jeff Dean (circa 1998) would pass the resume screen anywhere and I'm pretty sure he'd do well on a coding challenge.

wolco2 6y ago
I would highly doubt he would get rehired today at google first try. Leetcode is a different beast and things might not be all that fresh. Someone on the hiring committee might have said yes too many times this month.
- mtlynch 6y ago
  Former Google engineer and technical interviewer here. Google doesn't use Leetcode for interviews (at least as of 2018). I'd be extremely surprised if Jeff Dean (at any point in his employment) would fail a SWE interview loop at Google. The Google interview is heavily based on data structures and algorithms, and Jeff Dean likely knows them cold. Sure there's always a degree of luck, and Google errs on the side of false negatives, but there are certain people that are so high above the hiring bar that luck becomes too tiny a factor to be relevant.
  - humanlion87 6y ago
    Yes, Google doesn't use Leetcode per se for interviews. What the parent commenter was referring to was that Google interview questions are of the type for which Leetcode has become famous for.
  - bosswipe 6y ago
    > The Google interview is heavily based on data structures and algorithms That's pretty much what Leetcode is. When I interviewed at Google I was asked tree traversal and hashmap design questions that were very similar to leetcode questions.
- joshuamorton 6y ago
  Given the things Jeff actively works on, I have little doubt he could pass a technical interview loop.

burnthrow 6y ago

^^^ This is why you need tests of hard skills. This is the narration of a C player hiring D player friends. And it's "risk averse."

wolco2 6y ago

What about looking at employment history: types of companies, position, length, languages used... the basics..

loa_in_ 6y ago
That looks good in a conversation but under scrutiny, unless you know someone in a company mentioned, all you get is an opinion or you need to do a deep economic check of the company in question, which is nothing more than a statistic at best.
- wolco2 6y ago
  Why would you need to put so much judgement into a single company. If the person has only worked for one company than their employment history won't be much of a factor. If someone has worked at 6-8 places over 15/20 years covering a range of skills required I don't see the risk.

thewebcount 6y ago

One of the things I've done is to hand a candidate some existing code and ask, "What do you think of this code? Do you see any obvious problems? Are there things that work but could be done better? Is there anything exceptional in there? Tell me what you see." This allows you to test that they know how to read code, which is what they'll be spending the majority of their time doing. You can ask them how they'd improve it, which should give you an idea of how they write code, too. And of course, you can see their experience by checking out their resume and if possible running some of the apps they worked on. And if they have any code on GitHub, etc., you can check that out, too.

whymauri 6y ago
This is pretty similar to a ML Engineering interview question I got once. Basically, you get some mildly obfuscated code implementing a statistical concept and you have to tease out what the code is doing. Then you have to fix a few bugs, the variables names, and make it easier to read.
redshirtrob 6y ago
Yes! I've been on the interviewee side of this a couple times and absolutely love it. It has so many nice properties: 1. It fosters meaningful conversations about code that is not dependent on the candidate solving a toy problem. 2. As you said, it tests a candidate's ability to read and understand code. 3. It's actually representative of the type of work you _will_ do at any given company, i.e. code review. The major downside is if you give the candidate code in a language they're totally unfamiliar with then that candidate will likely bomb. I think this can be mitigated by sticking with mainstream languages like C/Java/Python, or domain specific languages where not-knowing would be a deal breaker anyway, i.e. must know Swift for iOS dev.
colmvp 6y ago
One of the companies I interviewed at had me do this exercise at the end of the interview and to be frank, I bombed it quite badly. One issue was the syntax was so weird compared to the type of code I'm used to writing and reading from others in library code that it was hard for me to comprehend the logic without the benefit of looking it up. I was filled with anxiety working under the time pressure to understand a document of code which I had very little context of, and no previous relationship with the reviewer so I kept thinking to myself whether or not the question that came into my mind was stupid obvious. In contrast, when I had take-home tests, I scored exceptionally high on all of them, leading to offers. I recognize that a lot of people here hate them because of the time involved and it's akin to spec work, but IMO it felt much more like my usual day-to-day life as a developer. One of the take home tests had existing code to work from but again, working at my own pace and having access to online resources to read docs was much less anxiety provoking.

mtlynch 6y ago

>Genuine question here: what's the better way? One strategy that's been successful for me is skipping the technical screen and jumping to a narrowly-scoped paid assignment. If they do well, I give them a larger paid assignment, and we work our way up to regular work together. If they do poorly, I pay them for their time but tell them it's not a match. I'll add the caveat that I've only used this to hire junior to mid-level developers for part-time or short-term work. I've never tested it on a full-time hire, and I imagine that would be more difficult. I've never written in detail about hiring developers, but I've shared similar strategies for hiring writers,[0] illustrators,[1] and editors.[2] [0] https://mtlynch.io/hiring-content-writers/ https://mtlynch.io/hiring-content-writers/ [1] https://mtlynch.io/how-to-hire-a-cartoonist/ https://mtlynch.io/how-to-hire-a-cartoonist/ [2] https://mtlynch.io/editor/ https://mtlynch.io/editor/

edoceo 6y ago
I do gig-to-hire too. Works a treat! I've found loads of talent this way, up and down the skill chain. But, for a CTO like role the gig has to be 3mo or more.
- Balgair 6y ago
  I'm confused. You are saying that you have had CTO interviewers that did a gig role for 3 months as a CTO? Did they also have another job at the time or was this their only job?
  - edoceo 6y ago
    This one is a bit harder (naturally). A recent/typical case was a CTO type candidate, they participated part-time in some meetings, had some exposure to the team, talking through issues with other CxO. During this time candidate is also shopping around too. It's basically buying consulting while you decide to hire. It does not generally take 3mo - but you'll add a probationary period that gives you maybe 12mo room.
    - Balgair 6y ago

pluc 6y ago

How about trusting candidates' experience? If they say they have 20 years of experience in X and you hire them, you'll know very quickly if that was a lie. Those tests are so insulting to the relationship you're pretending to want to build. Why would I bother putting false information on my resume that has the potential to get me hired and get tested daily? Isn't that why there's a probation period?! I have 20 years of experience in X but prove it? Why's the burden on me to prove what I've already told you? Are you saying I'm a liar? If my date of birth is on my cv do you expect a birth certificate to validate that information too? Anyway, that's always been my philosophy - hire on facts and fit, fire quickly if anything was wrong - and it's been pretty smooth. The great thing that it does is it prevents testing for futile thing that you will never use in your position and that are just elitist criteria or vanity wishes.

wetpaws 6y ago
You would be surprised how many people with 20 years of experience can barely perform at junior level
- base698 6y ago
  You can have 20 years of experience or 20 one year experiences.
  - nicoburns 6y ago
    You can also be straight up lying about having 20 years (or any) experience. Most people won't be doing this, but it seems like a basic feature of a hiring process to have some kind of check against this.
    - pluc 6y ago
      The check is what the candidate says, isn't it? If the candidate lies, you will know the day you hire them
      - 6y ago

lr4444lr 6y ago

I concur with you. This debate rages on endlessly. In-person whiteboarding is the worst way to assess candidates' tech abilities... except for all of the other alternatives.

xixixao 6y ago

This is a fact. A false positive is much more costly to a company than a false negative (this is also why we allow candidates to reapply).

justin_oaks 6y ago
How do you measure the cost of a false negative?
- theptip 6y ago
  The highest-order term is the opportunity cost of not having a developer now, and having to wait until the next one to pass the interview process.
- PragmaticPulp 6y ago
  It's not explicitly quantifiable. Imagine you're a team of 2 top performers. Your workload is increasing, so your manager gets approval to hire a 3rd person so you can take on more work. Unfortunately, your manager is fooled by a charismatic person who greatly exaggerated their experience on their resume, to the point that they work about 1/4 as fast as the rest of the team and tend to code by trial-and-error. Suddenly, the expectations for your team's output have increased by 50%, but the 3rd person can only really contribute another 10-20% at most. By extension, your job just got harder. Worse yet, the average talent of the team has now gone down significantly. You spend more time fixing bugs and chasing bad work. You're also faced with the fact that you're probably getting paid the same as the over-leveled person next to you. This breeds resentment, wastes time, and ultimately leads to good people quitting for other jobs. Alternatively, imagine if the company hired quickly and fired quickly. Every time a new person showed up, you had no idea if they were actually good or if they were just rushed through the interview process. You also don't know if they're going to be around after the 3 month probationary period, so why bother building a relationship? High turnover like this tends to destroy morale.
pseudalopex 6y ago
Depends on the company. Google can afford to reject great candidates. A small company might have to settle for a mediocre candidate 6 months after rejecting a good candidate.

almost_usual 6y ago

> I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job. You can have code challenges that aren’t HackerRank or Leetcode. Using problems from those sites is a signal the employer or team is lazy.

balls187 6y ago

> I'm not a fan of whiteboard coding challenges, even though they got me my first "real" Bay Area tech job. This is the fundamental part that is broken in my experience. They are often challenges/coding puzzles, when rather white board exercises should focus on collaboration, and solving problems with code. Having a "gotcha" that shows rudimentary CS understanding is fine (such as understanding why the obvious O(N^2) is a bad solution). But having a gotcha that requires some crazy leap of faith guess (or having seen the problem before) is not. I admit, my fondness for a candidate increases with the level of complexity of coding problem they can solve; however needing hints on the harder problems doesn't mean they are not qualified.

blackrock 6y ago

Are you really an interviewer? And are you actually this clueless?

deepsun 6y ago

In eastern europe interviews are mostly experience-related. Questions are mostly about technologies that hiring team needs, the more the better, and the deeper the better. I never had a problem getting the extra experience not found on previous workplace from my pet projects and blogs before moving to US. Funnily, I've seen people who moved back from SF Bay Area, and struggled to find a job there, despite being very good at whiteboard and CS problems. Turned out eastern european companies don't care about CS problems really.

mandeepj 6y ago

> Genuine question here: what's the better way? I think the candidate should be given a choice - whether they want to solve one of these tricky puzzles or work on a domain problem.

pantulis 6y ago

I've done my good share of recruiting interviews and you can measure if the CV has been inflated just by having a conversation with the candidate. Just asking "What did you do in your last job?" and following from there works wonders. No coding exercises, no whiteboards, it's all about working a conversation with them. I can confidently say that the recruiting process was faster --less interviews--, less stressful for the candidate, and finally the number of hire employees that flopped was not that much worse. I did whiteboards for Architect positions, but once again, I just asked the candidates to draw architectures to have a friendly discussion.

kylewatson 6y ago

Honest question: How do hospitals hire doctors (industry hire)? Do they quiz them like they did to the residents on Scrubs? Do they pull up a list of symptoms and ask what they would test/try next? Do they show them x-rays and ask them to spot the coronavirus? Seriously? Or do they look at your degree and experience and conclude that if you worked at General Hospital in Atlanta for 5 years you can work at Metro Hospital in Philadelphia? I honestly don't know how other professional industries hire industry-hires (not college grads/interns) but I only hear about quizzes and riddles and online tests and shit like that in programming. Hell, do they do the same shit for electrical engineers? Chemical engineers? I really want to know if they do and if not, how is it Boeing can hire a EE without a whiteboard puzzle but can't hire a CE without that bullshit?

klintcho 6y ago
I posed the exact same question below. I have the exact same thoughts, and genuinely curious on the process they have. I'm not that opposed in general to whiteboarding if you feel like you can't assess the candidate. However I do feel that it sometimes feels like credential and experience similar to what you describe in the medical field: if a doctor had 5 years at an inner city hospital, I don't think they would doubt his credentials to the point the want to make him put cell structures on a whiteboard?
- theamk 6y ago
  Doctors do have professional education, harsh exams, and they can get their licenses revoked for mistakes. Do you really want the same for programmers? "only people with Ms or PhD from accredited universities can apply, and diploma is invalidated on some mistakes"
zizee 6y ago
Medical doctors have a professional organization enforcing strict standards. People cannot practice medicine without passing a series of standardized exams that only recognized universities can administrator, they have and a lengthy internship process, review board etc. If they break the rules they can have their licence suspended. I think it would be a shame to bring this sort of thing to software development, but it would allow us to get rid of some of the hoops people have to jump through to ensure they can actually do the job being hired for.

klintcho 6y ago

This is not a rhetorical question: How does other industries hire high knowledge well educated candidates ? How does doctors, mechanical engineers and space flight engineers get interviewed and assessed ? As have already been reiterated in each and everyone of these threads (about leetcode/hackerrank/whiteboard coding) this is what the companies seem to think is the most proven, and produces most consistent results. But I would love to understand how other industries does this as well; does a propulsion engineer start to draw up a de Laval nozzle and calculate the exhaust gas velocity on a whiteboard? (again genuine question)

twblalock 6y ago
The problem is that the software industry does not have the same standards as other industries. People graduate CS programs with little to no coding ability in many cases. There is no certification program like there is for, say, accountants or CPAs. Companies set the hiring bar all over the place. At a lot of companies, bad hires stick around for years. Other industries have higher and more consistent standards, and certification processes. The software industry is the wild west, and that is partly why it has succeeded and why so many self-taught programmers have found success in it. The other side of the coin is that the quality of candidates varies more widely than in other industries.
- pseudalopex 6y ago
  I've never seen someone with a CS degree and no coding ability. Lots of them weren't taught anything else about software engineering. But most interviews don't test anything else anyway.
pseudalopex 6y ago
Engineers discuss work they did mostly. Interviewers ask probing questions to catch bullshitters.

suifbwish 6y ago

I will bite. What is your recommendation for a network?

AlchemistCamp 6y ago
It's Reid Hoffman's idea, not mine—a referral from a trusted source is a better indicator than an interview. He's previously said that if he could only decide based on one or the other, he'd hire the candidate referred by a trusted source and skip the interview. In practice, this choice never comes up, but you can weight referrals much more strongly than interviews.

geocar 6y ago

> Genuine question here: what's the better way? You tell them what the job is like, what is expected of them, and what the best way to work with you is, then you you ask if that's what they want, and how much it's going to take to get them in the door. That's it. I wouldn't ask an accountant what the Corrodium 3 Deduction is; I'd tell them I need them to file my taxes, that I need them to own the process and chase whoever they need (even if that's me). > I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires. I think you're probably right, but if someone lies to you and says they can do a job that they clearly can't, you should fire them. This should be unusual because getting a new job can be stressful, and nobody wants to put themselves into a job they can't do, so it's rare that you even have to consider this -- but it does happen, and unless you're a sociopath, it's going to make you shy about ever having to go through it again, because you're not going to want to fire anybody. It sucks for you, and it sucks for them, and it sucks for the team who see the revolving door, but nobody benefits from the wrong fit in the long run, so you just need to suck it up and do it. So be firm, and if it turns out you're firing a lot of people straight in the door, look inwards: you're probably not being honest with them (or perhaps yourself) what the job actually is. Figuring that out instead of some new programming puzzle to torture grads with is going to pay dividends.

twblalock 6y ago

> I think at the core of the problem is that most companies are extremely risk adverse in their hiring and would gladly miss a Jeff Dean type of super-performer in order to reduce their number of bad hires. This is absolutely why whiteboarding exists, and persists. I have interviewed a lot of people at this point in my career, and it's just not possible to tell if candidates are good by looking at their credentials or job histories. I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. I'm not talking about Leetcode-style brain teasers; I'm talking about being unable to construct a working for loop and do basic array operations in a language of their choice. I'm going to continue to do whiteboarding for candidates for this reason.

bluefirebrand 6y ago
> I've seen people with impressive credentials, or impressive job histories, end up being totally unable to produce working solutions to FizzBuzz-type questions. That honestly sounds to me like you found some people who lied like crazy about their credentials and job histories.
- twblalock 6y ago
  Actually one of the things the bad candidates had in common was a job history at defense contractors.