11 ms·

Laszlo Bock (former SVP of People at Google) did a great job summarizing decades of research around structured interviewing in his book 'Work Rules!' For a qui

• andrewmutz 10y ago
  How does this work in practice? If there are consistently used questions and specific criteria for assessing responses, can a candidate just learn the likely questions and what constitutes the "right" answer?
  • tptacek 10y ago
    This is a good and important question. Also: while I have a lot of respect for people at Google trying to innovate on hiring, make no mistake: Google's heart is in the right place, but they aren't at the forefront of structured hiring, and their hiring processes are notoriously capricious. The reality is that generating good questions for a structured interview is difficult. You can't just pose a programming problem. As you've noted, most programming problems have multiple good answers. Differentiating between multiple good answers from different candidates re-introduces subjectivity. Your brain would rather convince you that one valid answer is less good than another, even when it's not, that to admit to you that it can't differentiate or generate a narrative for you. The part of your brain that generates narratives is incredibly powerful and does not care about how accurate your hiring process ends up being. What we tried to do was create questions that generated lists of facts. "Spot all the errors in this code" would be an example of this approach (but none of the three we used). We went into the process wanting to embrace epistemological uncertainty, generating a historical trail of data that we could retrofit to candidate performance. In the end, work sample testing was so much more powerful a predictor for ourselves that we never fully got around to analyzing the data. Sometimes we'd get candidates that clearly generated inferior "lists of facts"; I think there may have been 1-2 instances where that outcome actually overruled work-sample testing delivered prior (out of a few tens of engineering hires and probably ~100 interviews).
    • peripitea 10y ago
      Making sure I understand you: When you refer to work sample tests here, that refers to the crypto challenges and things like that that you published at Matasano? And you're saying that was much more predictive than the list of facts methods, right?
      • tptacek 10y ago
        That's what people ordinarily assume I mean, but while our work sample challenges were similar to the cryptopals and Microcorruption stuff, they were not the same, or even derived from them. They were designed specifically to qualify candidates, and in fact predated our public challenges.
        • peripitea 10y ago
          Ahh, interesting. Would you mind sharing an example of how they were different, or how the fact that they were specifically for qualification changed the thought process? I'm working on making tech interviewing better and am fascinated by this stuff.
          • leksak 10y ago
            NDAs? Everything I've done yet is proprietary and judiciously covered by various binding legal agreements. I wouldn't breach such a contract nor should you hire someone that would be willing to do so. The sole exception being in the cases where it would be the best for the public interest, i.e. whistle-blowing.
        • sgustard 10y ago
          Just to further clarify, by work sample you do NOT mean an example of previously produced work? This still seems fickle: my dozen years of work output (for example, building and running a site with perfect uptime for millions of users) is not as valid an indicator of my future performance as how I happen to score on some arbitrary timed test?
  • jplasmeier 10y ago
    Of course. For example, many of the questions for technical interviews involve knowing which data structure to use in a given setting, and applying it to find a solution. The interviewee is more than welcome to study data structures and interview questions about data structures. A poor student will just memorize questions and answers by rote without really understanding the data structure in question. The good student will actually learn and understand data structures in the context of the solution. One would hope that interviewers are able to ask follow-up questions about the solution which distinguish between the two. EDIT: During the best tech interview I have done, I had no idea how to solve the question asked (that is, I had not studied its solution despite this being a relatively common question). I was able to ask articulate questions and invoked understanding of data structures as I went along. Memorizing solutions is a game of luck, and not recommendable.
    • nissimk 10y ago
      Bullsh*t. Seriously, these questions are all tricks. If you memorize the tricks you can win the game. Even the "really easy" questions like reverse a string involve a trick -- swapping from the ends and stopping in the middle. The interview game involves memorizing tricks and identifying which memorized trick applies to the question.
      • sirbearington 10y ago
        I feel like 90% of writing software is identifying which "trick" applies to the problem you're trying to solve and applying it correctly.
        • nissimk 10y ago
          Yeah, but you can look them up.
          • xamuel 10y ago
            The thing about looking up tricks is you can only look them up if you remember they even exist to look up.
  • jotux 10y ago
    The same way it works for teachers that give tests. Come up with a bunch of questions and only ask a random subset of those questions to each person. Occasionally evaluate the questions and add/remove some.
  • gatsby 10y ago
    This is a great point. There are absolutely candidates who are "good interviewers." I think one of the most helpful methods of determining a good interviewer vs. a good future employee is taking a structured, situational interview question, i.e. "How would you go about selling a new product to a customer?" and turning it into an actionable work sample that is tailored to your company. "We make this piece of software that does this. Spend the next 10 minutes writing a cold email to a prospect that outlines our offering." As the interviewer, grade the work sample on structured criteria that is important.
    • tptacek 10y ago
      This is the "calibrated interviewer" system. It strikes me as funnily similar to machine learning models, in that it's a black box that somehow demonstrates statistically it's effectiveness at a business problem, and you just have to hope there isn't some glaring bias or fatal flaw in the model that only occurs sporadically.
  • StreamBright 10y ago
    There few questions that you can learn to answer but it is probably very hard to reach the 5 star answer if you do not understand what you learned in detail. An example question would be: what happens when you type www.domain.com into your browser and hit enter. You can answer this many different ways and levels.
    • autotune 10y ago
      That question has also been asked so many times most candidates who have done a few minutes of googling can give you the "right" answer. See here: https://github.com/alex/what-happens-when/blob/master/README.rst https://github.com/alex/what-happens-when/blob/master/README... I'd probably start here to lighten up the mood due to how cliche the question has become: https://github.com/alex/what-happens-when/issues/231 https://github.com/alex/what-happens-when/issues/231
      • StreamBright 10y ago
        Candidates doing a few minutes googling to find this, but will they put in the few months to get to the bottom of every aspects of it?
        • autotune 10y ago
          Probably not if they have actual work to do rather than memorize easily google-able trivia questions.
  • nostrademons 10y ago
    Google tries to keep interview questions confidential - that's why candidates sign an NDA - and periodically rotates out questions that have appeared in public. Many engineers are also continually trying to think up new questions as well, usually based on their work. For most questions, there's no "right" answer, but there are a set of points that the interviewer wants to see you touch on. For example, they might first want to see that you can code up a naive brute-force variant of the algorithm, checking whether you know the programming language claimed and can think through the problem, and ask you the algorithmic complexity. Then they'll want to see if you can get a divide-and-conquer or dynamic programming variant with lower time complexity. Then they might ask "What if it has to be an online algorithm, where new input arrives before the computation finishes?" Then they'll ask "How would you distribute this over 1000 machines, and what are the failure modes?" At each stage, they're watching how you answer, and where you get stuck. If you ask clarifying questions or spend time to think before diving into coding, that's a plus. If you have never heard of the problem before (this is frequent - many questions are not in textbooks), they want to see how you would reason through it, and break it down into subproblems that are similar to textbook problems. If you miss language trivia, most people don't care; when I did interviews I'd usually volunteer the answer if they missed some API call, and when I interviewed my interviewers did the same. If you don't know how to solve the problem and can't make any effort to move forward through a solution, that's a big negative. Similarly if you don't know what the concept of big-O is or why it's important.
    • inetknght 10y ago
      What about when you know what the concept of big O is, know why it's often touted as important, and disagree that it's as important as it's touted? My opinion is that big-O often ends up being used as a premature optimization effort hindering "just get the right answer first". Maybe that brute force method will work in a reasonable clock-time cost, despite having egregious algorithmic cost. You won't know if you're busy overengineering and optimizing before you even have a working solution.
      • nostrademons 10y ago
        That's a big red flag, at least in a Google setting. The reason is because when your dataset is in the petabytes, any algorithm bigger than O(N log N) is not going to terminate. You're not going to get any answer at all; your MapReduce is going to sit there burning CPU time for a day, and then you'll kill it, and you won't have any idea what went wrong. (This is learned from experience, if you couldn't tell.) In a startup context, I'd certainly agree with you - that's exactly the approach I'm taking with my startup, where I'm doing a lot of the debugging and code-wrangling on my local laptop after verifying that the data I want exists and calculating its size with a couple cloud-computing passes. It can sometimes be useful to prototype & iterate with a small subset of the data until you've got the algorithms down. But many of the algorithms Google cares about don't give any useful results on small data sets. I remember running a few collaborative-filtering algorithms and finding that they gave zero answers because within my small sample of 0.01% of the total data, nobody liked anything else that anybody else did.
  • wpietri 10y ago
    Only if you use a small, static body of questions. But there's nothing wrong with, say, each interviewer coming up with a couple of questions and rotating it them in a while. You do want to try out the question and criteria before depending on it too much, but it's easy enough to try it out on a colleague.
    • tptacek 10y ago
      Among the many things wrong with this approach is the fact that you aren't generating a body of questions and responses that you can evaluate against your choices and the resulting performance of candidates you hire, making it impossible to effectively iterate.
      • wpietri 10y ago
        Depends on how you think of the interview process. The theory at work here is that if you get a number of domain experts to give an independent evaluation of a candidate and then compare notes, you'll get a decent outcome. Requiring them to use a structured, repeated question with a formal scoring rubric is to subtract certain sorts of bias from each individual evaluation. When hiring somebody doesn't work out, you go back and work on the interviewers and their interviewing skills, not the questions themselves. I take it you're working from a different theory, where you try to put the expertise in a machinery of questions and answers, relying less (or not at all) on the interviewers themselves. I think that's also a valid approach, one with different limitations. Personally, as I mentioned elsewhere, I'm not a big fan of interview questions anymore period. If I want to know if somebody can do the work, now I try to create situations where they just do the work. But if one is trapped in the dominant tech interview paradigm, I personally favor investing in interviewers more than the questions themselves.
• jotux 10y ago
  The research likely behind the book: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2853669 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2853669
  • trishume 10y ago
    Easy link to the relevant table from the paper: http://imgur.com/a/YRFTh http://imgur.com/a/YRFTh
• mtkd 10y ago
  this just gives them resource liquidity for operations - the people that take things forward are mostly acquired if you have to start using structured interview questions to expand a team - you've already lost
• throwaway71958 10y ago
  Don't take his word as a word of god from heaven. Google's interview process is just as shitty as anyone else's, and shittier than some I've been through. It selects for people who do well on the whiteboard under pressure, which are very often not the best workers overall. It also wastes a ton of time both on the employer and on the candidate side. Source: interviewed ~100 people in my 6+ years at Google.
  • johan_larson 10y ago
    Yes, I don't think their process is magic. Google gets a good workforce because they are generous and prestigious, which means a lot of good people apply there. And Google is willing to say no to a lot of people in their search for good people. They reject a lot of candidates who would probably have worked out just fine.
    • mooreds 10y ago
      Or, put another way, they choose to accept a high rate of false negatives to avoid false positives.
      • lilbobbytables 10y ago
        bingo
      • thecardcheat 10y ago
        > Or, put another way, they choose to accept a high rate of false negatives to avoid false positives. Which is how it is typically presented because it sounds much better than "reject a lot of candidates who would probably have worked out just fine". It is useful to perceive both the potential value in an approach like this and the shortcomings. Google can absorb the massive expense in man hours, lost opportunity, etc. that comes with trying to craft genuinely predictive interview processes, but a lot of the companies trying to emulate them can't. Too often, interviewees don't realize a process of this sort is stacked against them, and interviewers don't appreciate the negatives of adopting a still-nascent approach that sounds more reliable simply because it is quantitative - and assuming since Google does it it must work.
• alphonsegaston 10y ago
  A book conveniently published in the wake of the PR disaster that was their wage fixing scandal. Nothing about Google's hiring process can be deemed reputable when they'd go so far as to illegally conspire with other tech giants to prevent potential employees from achieving the best possible outcomes for themselves. Maybe things have changed over there, but given the slap on the wrist they got, I'm skeptical.
• orless 10y ago
  Google is definitely not a benchmark for interviews. From what I experienced, they must have a huge rate on false negatives. You can only afford this if you're Google (or that level).
  • johan_larson 10y ago
    Yup, it was pretty common knowledge at Google that if you took all the engineers and ran them through the interview process again, it would have rejected most of them.
  • ctdonath 10y ago
    The view is: better to lose a good candidate than to hire a bad one.
• emodendroket 10y ago
  It ensures consistency, but how do you know that whatever you're measuring for is actually correlated with job performance?
  • throwaway71958 10y ago
    They don't. In fact they even did an experiment and admitted some people at random, irrespective of how well they did in the interviews. Those people were found to perform about as well as the legitimate interview "lottery winners" hired during the same time period. A lot of people (anecdotally, the majority) at Google have the "impostor syndrome", and the news of the experiment did nothing whatsoever to quell the symptoms. Now they don't know if they are, in fact, not impostors, but they do know that on average they perform about as well. :-)
    • creatine_lizard 10y ago
      Could the performance of the lottery winners have been "environmental"? That is, they benefited from being surrounded by competent people (which was, in turn, guaranteed by those people having gone through the interview process) and "leveled up" due to that? In other words, maybe as long as you let in a small number (but only a small number) of non-performers, you're fine (which is bound to happen anyway - I'm sure there is some noise in the interviews).
      • throwaway71958 10y ago
        Yes. Getting hired by Google is only part of the deal. Actually _succeeding_ when you're already there is much more difficult. It's a high pressure environment with a lot of very smart overachievers. Because of this it's sort of a self-fulfilling prophecy, and people who don't measure up also don't feel welcome, as it were. Since performance reviews are largely derived from peer feedback, hiring mistakes tend to be self-correcting. Most of the time, though, I've seen great people leave just because they didn't like the pressure. The amount of pressure depends on the team. The higher the profile -- the more pressure (but also more rewards, greater career potential, etc). But the general bar for what's considered "good work" is pretty high, and more uniform than in any other large company I have ever worked at. Then there's the issue that by the time you even get an on-site, you're already very much not a random candidate. Recruiters actually do look at your track record, etc. You can bullshit there, but I don't recommend it, since references will be spot checked, and they better line up. Google interviews are largely a roll of the dice above certain level of basic engineering competence. I.e. if you don't know the basics, you will almost certainly not pass them. But if you're a more senior candidate, Google doesn't really know how to interview you, and their interview process turns into a random number generator biased heavily towards "no hire".
• orang_utang 10y ago
  From my experience, I'd say Google's interviews need to ask harder questions and last longer. Today they ask fairly easy questions (given a list of integers, find the subsequence that has the following property etc. - basically leetcode medium level) that you can solve fairly easily using basic CS 101 knowledge. However, it ends up (again, IME) being a race against time where you have to whiteboard code a simple solution in 20 minutes approximately. I'd much rather have them as questions that take real insight to solve, but have the interviews last 90 minutes or so.
  • bitL 10y ago
    I recently suggested to them not to dumb down their interviews as they simply make the equation about talking to them not worth the time. I remember they used to have much tougher interviews in the past, which were super motivating; now it's like standard hackerrank stuff which is pretty uninteresting. OTOH, each company has its blind spots so that they won't be on top forever, so maybe it's better that way as those that were very capable but rejected could dethrone them at some later time. To me their approach lately is like forcing a top tennis player to play with kids (well, you would be surprised how many pros have issues slowing down their game) or a double diamond slope skier to ski on blue slopes only (and increasing the risk of getting an injury as their timing/moves are optimized to ultimate performance instead of basics). I've heard of people that solved stuff on interviews nobody solved before them but were rejected as they messed up some basic stuff and recruiter was communicating back to them that it looked bad in the hiring committee.
    • skybrian 10y ago
      But if you were interviewing for a tennis instructor, wouldn't that be a valid result? You probably shouldn't hire pros who aren't good at teaching when the job involves teaching. Similarly, as a senior engineer at a large company, you need to be adaptable enough to work with people with less/different experience than you. Adjusting your message to the audience is an important skill.
• draw_down 10y ago
  Yes, but even when this is done well, there is still the question of what's measured vs what results in hiring good candidates and not hiring the others. Turning away a huge number of false negatives is also not optimal, especially in a situation where talent is scarce.
• gist 10y ago
  > Laszlo Bock (former SVP of People at Google) did a great job summarizing decades of research around structured interviewing Most importantly most companies interviewing don't get to pick from the pool of applicants that google gets to. And even more importantly most companies don't even get to pick from the pool of applicants that a typical funded (or soon to be funded) startup will get. While I am sure there are things that can be learned from 'Work Rules' the question is how much of that applies to the vast majority of companies in America.
  • vonnik 10y ago
    Google's hiring process is famously painful, and they lose a lot of great candidates that way. I wouldn't trust their insights to build a hiring pipeline.
• abritinthebay 10y ago
  Google's interview process is the single shining example of the worst interview process I've ever been through. So much so I actually took the interviewer to task for it. It may be good at screening University graduates but it's pretty awful at anything else. It's also really obvious that the interviewers don't actually know how to interview people and are pretty much cargo-culting the same process that hired them. Awful. It totally destroyed my perception of the company. Now granted this was... 7 years ago, no (or more) but from colleagues who have interviewed there (or been hired!) it's not changed fundamentally that much.
  • AlexCoventry 10y ago
    Getting candidates to jump through crazy hoops is a sensible strategy for a big organization. It selects for compliance, which is important when your enterprise is too complex, uncertain or fast-moving to operate by consensus
    • abritinthebay 10y ago
      A fair point, though not really a defense as such ;)