9 ms·

How does this work in practice? If there are consistently used questions and specific criteria for assessing responses, can a candidate just learn the likely q

• tptacek 10y ago
  This is a good and important question. Also: while I have a lot of respect for people at Google trying to innovate on hiring, make no mistake: Google's heart is in the right place, but they aren't at the forefront of structured hiring, and their hiring processes are notoriously capricious. The reality is that generating good questions for a structured interview is difficult. You can't just pose a programming problem. As you've noted, most programming problems have multiple good answers. Differentiating between multiple good answers from different candidates re-introduces subjectivity. Your brain would rather convince you that one valid answer is less good than another, even when it's not, that to admit to you that it can't differentiate or generate a narrative for you. The part of your brain that generates narratives is incredibly powerful and does not care about how accurate your hiring process ends up being. What we tried to do was create questions that generated lists of facts. "Spot all the errors in this code" would be an example of this approach (but none of the three we used). We went into the process wanting to embrace epistemological uncertainty, generating a historical trail of data that we could retrofit to candidate performance. In the end, work sample testing was so much more powerful a predictor for ourselves that we never fully got around to analyzing the data. Sometimes we'd get candidates that clearly generated inferior "lists of facts"; I think there may have been 1-2 instances where that outcome actually overruled work-sample testing delivered prior (out of a few tens of engineering hires and probably ~100 interviews).
  • peripitea 10y ago
    Making sure I understand you: When you refer to work sample tests here, that refers to the crypto challenges and things like that that you published at Matasano? And you're saying that was much more predictive than the list of facts methods, right?
    • tptacek 10y ago
      That's what people ordinarily assume I mean, but while our work sample challenges were similar to the cryptopals and Microcorruption stuff, they were not the same, or even derived from them. They were designed specifically to qualify candidates, and in fact predated our public challenges.
      • peripitea 10y ago
        Ahh, interesting. Would you mind sharing an example of how they were different, or how the fact that they were specifically for qualification changed the thought process? I'm working on making tech interviewing better and am fascinated by this stuff.
        • leksak 10y ago
          NDAs? Everything I've done yet is proprietary and judiciously covered by various binding legal agreements. I wouldn't breach such a contract nor should you hire someone that would be willing to do so. The sole exception being in the cases where it would be the best for the public interest, i.e. whistle-blowing.
      • sgustard 10y ago
        Just to further clarify, by work sample you do NOT mean an example of previously produced work? This still seems fickle: my dozen years of work output (for example, building and running a site with perfect uptime for millions of users) is not as valid an indicator of my future performance as how I happen to score on some arbitrary timed test?
        • tptacek 10y ago
          Absolutely not. Samples of previous work are deceptive at the best of times.
  • mbesto 10y ago
    > but they aren't at the forefront of structured hiring, Is anyone though?
• jplasmeier 10y ago
  Of course. For example, many of the questions for technical interviews involve knowing which data structure to use in a given setting, and applying it to find a solution. The interviewee is more than welcome to study data structures and interview questions about data structures. A poor student will just memorize questions and answers by rote without really understanding the data structure in question. The good student will actually learn and understand data structures in the context of the solution. One would hope that interviewers are able to ask follow-up questions about the solution which distinguish between the two. EDIT: During the best tech interview I have done, I had no idea how to solve the question asked (that is, I had not studied its solution despite this being a relatively common question). I was able to ask articulate questions and invoked understanding of data structures as I went along. Memorizing solutions is a game of luck, and not recommendable.
  • nissimk 10y ago
    Bullsh*t. Seriously, these questions are all tricks. If you memorize the tricks you can win the game. Even the "really easy" questions like reverse a string involve a trick -- swapping from the ends and stopping in the middle. The interview game involves memorizing tricks and identifying which memorized trick applies to the question.
    • sirbearington 10y ago
      I feel like 90% of writing software is identifying which "trick" applies to the problem you're trying to solve and applying it correctly.
      • nissimk 10y ago
        Yeah, but you can look them up.
        • xamuel 10y ago
          The thing about looking up tricks is you can only look them up if you remember they even exist to look up.
• jotux 10y ago
  The same way it works for teachers that give tests. Come up with a bunch of questions and only ask a random subset of those questions to each person. Occasionally evaluate the questions and add/remove some.
• gatsby 10y ago
  This is a great point. There are absolutely candidates who are "good interviewers." I think one of the most helpful methods of determining a good interviewer vs. a good future employee is taking a structured, situational interview question, i.e. "How would you go about selling a new product to a customer?" and turning it into an actionable work sample that is tailored to your company. "We make this piece of software that does this. Spend the next 10 minutes writing a cold email to a prospect that outlines our offering." As the interviewer, grade the work sample on structured criteria that is important.
  • tptacek 10y ago
    This is the "calibrated interviewer" system. It strikes me as funnily similar to machine learning models, in that it's a black box that somehow demonstrates statistically it's effectiveness at a business problem, and you just have to hope there isn't some glaring bias or fatal flaw in the model that only occurs sporadically.
• StreamBright 10y ago
  There few questions that you can learn to answer but it is probably very hard to reach the 5 star answer if you do not understand what you learned in detail. An example question would be: what happens when you type www.domain.com into your browser and hit enter. You can answer this many different ways and levels.
  • autotune 10y ago
    That question has also been asked so many times most candidates who have done a few minutes of googling can give you the "right" answer. See here: https://github.com/alex/what-happens-when/blob/master/README.rst https://github.com/alex/what-happens-when/blob/master/README... I'd probably start here to lighten up the mood due to how cliche the question has become: https://github.com/alex/what-happens-when/issues/231 https://github.com/alex/what-happens-when/issues/231
    • StreamBright 10y ago
      Candidates doing a few minutes googling to find this, but will they put in the few months to get to the bottom of every aspects of it?
      • autotune 10y ago
        Probably not if they have actual work to do rather than memorize easily google-able trivia questions.
• nostrademons 10y ago
  Google tries to keep interview questions confidential - that's why candidates sign an NDA - and periodically rotates out questions that have appeared in public. Many engineers are also continually trying to think up new questions as well, usually based on their work. For most questions, there's no "right" answer, but there are a set of points that the interviewer wants to see you touch on. For example, they might first want to see that you can code up a naive brute-force variant of the algorithm, checking whether you know the programming language claimed and can think through the problem, and ask you the algorithmic complexity. Then they'll want to see if you can get a divide-and-conquer or dynamic programming variant with lower time complexity. Then they might ask "What if it has to be an online algorithm, where new input arrives before the computation finishes?" Then they'll ask "How would you distribute this over 1000 machines, and what are the failure modes?" At each stage, they're watching how you answer, and where you get stuck. If you ask clarifying questions or spend time to think before diving into coding, that's a plus. If you have never heard of the problem before (this is frequent - many questions are not in textbooks), they want to see how you would reason through it, and break it down into subproblems that are similar to textbook problems. If you miss language trivia, most people don't care; when I did interviews I'd usually volunteer the answer if they missed some API call, and when I interviewed my interviewers did the same. If you don't know how to solve the problem and can't make any effort to move forward through a solution, that's a big negative. Similarly if you don't know what the concept of big-O is or why it's important.
  • inetknght 10y ago
    What about when you know what the concept of big O is, know why it's often touted as important, and disagree that it's as important as it's touted? My opinion is that big-O often ends up being used as a premature optimization effort hindering "just get the right answer first". Maybe that brute force method will work in a reasonable clock-time cost, despite having egregious algorithmic cost. You won't know if you're busy overengineering and optimizing before you even have a working solution.
    • nostrademons 10y ago
      That's a big red flag, at least in a Google setting. The reason is because when your dataset is in the petabytes, any algorithm bigger than O(N log N) is not going to terminate. You're not going to get any answer at all; your MapReduce is going to sit there burning CPU time for a day, and then you'll kill it, and you won't have any idea what went wrong. (This is learned from experience, if you couldn't tell.) In a startup context, I'd certainly agree with you - that's exactly the approach I'm taking with my startup, where I'm doing a lot of the debugging and code-wrangling on my local laptop after verifying that the data I want exists and calculating its size with a couple cloud-computing passes. It can sometimes be useful to prototype & iterate with a small subset of the data until you've got the algorithms down. But many of the algorithms Google cares about don't give any useful results on small data sets. I remember running a few collaborative-filtering algorithms and finding that they gave zero answers because within my small sample of 0.01% of the total data, nobody liked anything else that anybody else did.
• wpietri 10y ago
  Only if you use a small, static body of questions. But there's nothing wrong with, say, each interviewer coming up with a couple of questions and rotating it them in a while. You do want to try out the question and criteria before depending on it too much, but it's easy enough to try it out on a colleague.
  • tptacek 10y ago
    Among the many things wrong with this approach is the fact that you aren't generating a body of questions and responses that you can evaluate against your choices and the resulting performance of candidates you hire, making it impossible to effectively iterate.
    • wpietri 10y ago
      Depends on how you think of the interview process. The theory at work here is that if you get a number of domain experts to give an independent evaluation of a candidate and then compare notes, you'll get a decent outcome. Requiring them to use a structured, repeated question with a formal scoring rubric is to subtract certain sorts of bias from each individual evaluation. When hiring somebody doesn't work out, you go back and work on the interviewers and their interviewing skills, not the questions themselves. I take it you're working from a different theory, where you try to put the expertise in a machinery of questions and answers, relying less (or not at all) on the interviewers themselves. I think that's also a valid approach, one with different limitations. Personally, as I mentioned elsewhere, I'm not a big fan of interview questions anymore period. If I want to know if somebody can do the work, now I try to create situations where they just do the work. But if one is trapped in the dominant tech interview paradigm, I personally favor investing in interviewers more than the questions themselves.