11 ms·

The anatomy of the perfect technical interview from a former Amazon VP

220 points by jnazario 13y ago

c0deporn 13y ago
I disagree with "In most cases, the “best and the brightest” already have jobs, so you’re really just on the lookout for the best available.". Just because one is employed does not mean one is taken. If you want the best you have to work at it. Appeal to their desires and they'll come to you. Looking for the "best available" is no better than settling. In my opinion you'll end up doing a disservice to both your company and the new hire.
- walshemj 13y ago
  Lets be honest Amazon is an online shop the best and brightest are either going to be working at tier 1 RnD or on wall street or the city.
  - afandian 13y ago
    Let's be honest, Amazon has a massive infrastructure supporting a very large number of very diverse customers, tonnes of products and projects, and lots of active development on them.
  - kenster07 13y ago
    Re Wall Street: It's unfortunate that some of our brightest are tempted into playing multi-million dollar poker, rather than focusing their energy on producing any significant long-term value. I believe software engineering wages will continue to rise over time, and will eventually reflect the true value the industry produces.
    - walshemj 13y ago
      Experience from other branches of engineering compared to other equivalent professions over the last 100 years suggests otherwise.
    - dylangs1030 13y ago
      I have no shame in admitting that I would rather use high-level mathematics and programming to play "multi-million dollar poker" on Wall Street than join half of the startups and big tech companies and have a boring job churning out insignificant apps at a desk. In fact, the way you phrased it, being a quant sounds really fun :) you don't want to play multi-million dollar poker? I mean, sure, meaning and existentialism and all that, but I bet it's at least somewhat exciting.
      - kenster07 13y ago
        That's the fundamental problem. The function of the finance system in a healthy economy is to ensure efficient allocation of capital. Playing poker has a neutral to negative effect on the system.
        dylangs1030 13y ago
        Well you're the one who used the word poker, so I guess it depends on your definition here...but the financial system somewhat revolves around poker. Valuations of companies are only important to investors because of the inherent gambling. If you purchase capital at a single valuation that never rises or falls, what's the point? Even an index fund pursues gradual increases over time. ...Or am I building a strawman here? Is this not where you were going?
      - chris_mahan 13y ago
        Which is why I work for a large bank, in toxic assets resolution, where the stakes are still in the billions of dollars.
    - ryandrake 13y ago
      I think I'd rather use my tech skills to play multi-million dollar poker on Wall Street then create yet another beautiful, responsive, flexible, tweet-able, location-based, modular MVC library API component framework backbone architecture.js.
      - Zimahl 13y ago
        Also, who wouldn't want to play high-stakes poker with someone else's money!
        deleted 13y ago
        [deleted]
      - dylangs1030 13y ago
        "It's beautiful! It's disruptive! It's agile and lean!" "What does it do?" "It revolutionizes the way you share pictures of your cats!" ...Yeah. Playing multi-million dollar poker on Wall Street sounds a lot sexier than the shiny startup veneer put on another photo sharing app.
  - walshemj 13y ago
    down voted eh the truth hurts doesn't it there is a world of difference between working at JPL MIT etal and some me to start up using the latest shiny new thing copying some existing SM type solution of which there are dozens already.
    - jlgreco 13y ago
      cough http://aws.amazon.com/solutions/case-studies/nasa-jpl/ http://aws.amazon.com/solutions/case-studies/nasa-jpl/ cough Sorry, caught a nasty flu..
    - jmduke 13y ago
      I didn't downvote you, but calling Amazon a startup is pretty ridiculous -- and your spelling/grammar isn't lending your argument any favors. (Though, ironically, it does fit into PG's description of "a company optimized for growth.")
      - walshemj 13y ago
        I think it should have been obvious from the context that I was talking about a substantial number of the HN commentators who are working at me to start ups (nothing wrong with that but its not world changing rocket science) Soory if you have problems with reading as a literate dyslexic I had no problem with reading writing not so much :)
        deleted 13y ago
        [deleted]
        pekk 13y ago
        > core tenant You mean "core tenet." Please don't be a dick to people about flaws that you share with them!
  - lotsofcows 13y ago
    Have you heard of a little thing called Amazon Web Services?
- vonmoltke 13y ago
  If you have a situation, which I believe we do now, where the number of engineers who fit the "best and brightest" category as defined by the companies looking is less than the number required by these companies, somebody is going to have to either "settle" or work with a hole. It will also make some of these engineers overpaid. I don't see how this benefits your company. Yes, in this system some companies will be "winners" in the sense that they get an awesome team for a good price, but most will lose. I would not be surprised if the high rate of failure in the startup community is related to this. For the record, I do not see hiring the "best available" as settling unless they do not meet some minimum requirements of the business. The purpose of hiring people is to help execute a product, service, or whatever else your business is trying to make money off of. Anyone who can increase your chances of succeeding or increase your level of success is a good hire. Yes, that includes looking at intangibles like team and culture fit, work ethic, etc. But this person does not need to be the "best" to provide this boost to your business. In fact, it is effectively impossible to hire the best, because there will always be someone better.
  - walshemj 13y ago
    "over paid" what is this concept that you speak off - there is no big guy with a big white beard sitting on a cloud who's going to reach down and say "you yes you with a third in media studies from Luton university should be paid X and those cheeky blue collar train drivers or miners who have scarce skills should be paid less than you". The PHB might think they are over paid but like it or not this is Laissez-faire capitalism pay is determined by scarcity and bargaining power. ps for those outside the UK Luton is the unfortunate University that props up the bottom of the university league table - though to be fair they do rank well for Nursing degrees.
    - saraid216 13y ago
      Paid more than one standard deviation higher than average for that level of competency, scarcity, and bargaining power.
      - humanrebar 13y ago
afandian 13y ago
"No matter how many times they say it, most still make decisions based on gut feel, basic credentials, GPAs, ivy league educations, flashy company names - even SAT scores. Roseman objects." What's a Roseman object? Is it some psychological classification of the means by which we just things? Of course, I mis-read it. Shame, that could have been an interesting topic.
- lsb 13y ago
  If it's confusing to parse, let me expand it: Neil Roseman strenuously objects to that assertion.
  - afandian 13y ago
    I understood it perfectly about half a second after I misunderstood it. I just thought I'd share my experience, as I found it amusing.
- saraid216 13y ago
  I didn't know "just" was a verb. Is that when we do something but only barely competently?
  - afandian 13y ago
    It's not a verb, it's a typo.
- VikingCoder 13y ago
  I think there should be something in science called the 'reindeer effect.' I don't know what it would be, but I think it'd be good to hear someone say, 'Gentlemen, what we have here is a terrifying example of the reindeer effect.' - Jack Handey
- learc83 13y ago
  >SAT scores Wait...what? Is this a thing? Are people actually putting this on their resumes? I had the highest SAT in my county (still just a 1360/1600), but I don't go around putting 2002 "STAR student"[1] on my resume. [1]Local award for highest SAT score. I think each school in Georgia awards this.
awakeasleep 13y ago
The one element no one ever accounts for is the interviewers. You're never going to have a perfect technical interview because you're an imperfect human. You're incapable of judging whether someone is a perfect cultural fit, and someone who doesn't fit the culture might help the company more than someone who does. That isn't even getting into the the conflict between whats good for the company and whats good for you.
- Matt_Mickiewicz 13y ago
  When Google pushed out their interview-results data this year, one of the big findings was that there is NO SUCH THING as a good interviewer. To quote the NY Times interview with Laszlo Black of Google - "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert." That being said, there are a lot of very disorganized interviewers out there, especially among seed stage companies. I've been amazed about how many people grumble about the "talent shortage", yet regularly miss interviews with "hot" candidates because of meetings, forget them altogether, or just lack a sense of urgency & speed when it comes to running an effective hiring process. A large part of having a hiring culture is about decisiveness. Loosing a fantastic Engineer because you spent 4 weeks making up your mind, or you didn't want to pay an extra $10K/year is just silly when it means that you won't hit your product roadmap goals.
  - RogerL 13y ago
    I think that is perhaps telling. Google doesn't interview you on your experience, but whether you can can solve network/graph algorithms taken from Knuth exercises. I would expect a random mess from that. It sounds like this person actually drilled down into what the candidate actually knew. I think it is far more useful to ask somebody about what they actually did. Are they excited and engaged by it, did they come up with novel thoughts about it, do they understand the CS literature in relation to it, and so on. Program a hash table on the whiteboard when you have been just fine using the built in one in your favorite library or language? I'm not sure what that predicts about my on job performance. If I had to do that I'd reread the literature and then whip one up, easy peasy. I can state that with confidence because I read the literature on other topics and do that every day. Have me explain the new numerical filtering technique I just implemented and I'll give you good interview. Ask me to implement a smart pointer? Not so much.
johnhess 13y ago
I'm always amazed by how confidently people can propose a hiring "rule" or even guideline. The article pays lip service to the fact "There's no way to A/B test hiring decisions" but then goes on to describe a handful of rules to "build an effective organization, regardless of size or resources." The thing is, there's no evidence that these are effective and definitely none to suggest they're generalizable to organizations of every size. There's an appeal to the author's authority as an Amazon VP (admittedly a hell of a credential), but that's not evidence of effectiveness. The fact is, hiring is tough to learn. It's made especially difficult because your feedback comes at least 6 months after you make the decision. I've had the good fortune to be in an organization which grew from ~10 employees this time last year to ~80 now, so I've gotten to see how a few dozen hires panned out in a short period of time. I learned a lot through that. Even so, as a professional data geek, I couldn't say with a straight face that I "know" anything about hiring. I do my best day in and day out, but saying you've got hiring all figured out is the professional equivalent of knowing "What (wo)men want." It's a sign of confidence, but not necessarily knowledge. Don't get me wrong, in the absence of quality data, I'll take the wise words of someone who has been there and done that. But I'd much rather some empirical proof or at least some "YMMV" humility from those who purport to know.
- vonmoltke 13y ago
  This topic has started taking on religious tones in the past year or so. It seems to be a long stream of anecdotes and opinion pieces presented with appeals to authority either of the author or someone the author is connected to.
coin 13y ago
Not related to the content, but why do sites like this disable pinchzoom? What value could it possibly add?
- freyr 13y ago
  Yesterday I needed to look at source code on GitHub using my iPhone. Horizontal scrolling AND pinch-to-zoom were disabled, making it impossible to read lines of code wider than my phone screen. Github. Why is this so hard?
- Daniel_Newby 13y ago
  For the same reason other sites use floats for sidebars: they hate you and want to become irrelevant.
evanjacobs 13y ago
In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's interview process is fatally flawed. Of course Amazon is still able to hire good candidates but I believe it is in spite of this process and not because of it. Here were the most frustrating aspects of interviewing for me: 1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area. 2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform. 3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript. 4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.
- WalterBright 13y ago
  > I think Amazon's interview process is fatally flawed. The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process. I suspect the real problem is the implication that this is even a solvable problem.
  - dylangs1030 13y ago
    I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions. Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with? We work with what we have, and take the benefits it gives us. Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
freyr 13y ago
"The anatomy of the perfect technical interview from a former Amazon VP" And more recently, a former Zynga VP. Surprising that such an expert at evaluating people chose to work for Zynga, of all places.
- hyperbovine 13y ago
  Money is a hell of a drug.
  - umsm 13y ago
    That's how you get the best talent. There is no secret question you can ask to find great talent. I've interviewed persons that did very very well during the interview process, but failed miserably on a day-to-day basis.
    - seunosewa 13y ago
      So money is the answer? Have you never observed a well paid developer who "failed miserably on a day to day basis"?
  - enraged_camel 13y ago
    Either that, or they really wanted a challenge.
jpalioto 13y ago
I appreciate the notion of having the whole team prepared for the interview process. I think this is a best practice. But, when I interview, I am looking for completely different things than the author. I really don't care about what you did at your last company or if you raised this by 50% or that by 30%. Rather, I want three things in a candidate: smart, talented and hungry. If I get someone who is smart, talented and hungry and I drop them into a situation where there is unlimited upside, my job then becomes to guide them to greatness. I also strongly disagree with the notion of not asking questions you do not know the answer to. Anyone can read an obscure section of the C++ language spec or a chapter on some data structure 5 minutes before an interview and make themselves feel superior to the candidate. What does this prove? Nothing. I often ask questions I don't know the answer to in order to determine whether or not the candidate can teach me something. I love having people on my team that have the ability to teach others. The author's point is very well taken -- interviewing is an important process and it should be conducted in a way that results in the best possible outcome. But, he's looking at the situation far too rigidly for my taste.
- umsm 13y ago
  I disagree with the hungry part (if you are referring to someone that needs the money). Being "hungry" just means that they will be stuck with a company but will not love what they do.
  - jpalioto 13y ago
    Less about needing money and more about being ambitious. Hungry for success I should say.
    - Terretta 13y ago
      Whether hungry for money or hungry for success, you're then missing hiring candidates who aren't "hungry" because they have a proven track record at having done very well financially from a series of successes. At this point, they don't need money or validation, they just figure they can help you solve things and are talking to you because they enjoy solving new things. Hungry is overrated when what you need is focus on solving the business' problems. This is one reason VC like to see founders take "shelter money" off the table early.
pdovy 13y ago
There seems to be a lot of criticism of the suggestions here, but I think they are good ones. It seems to pretty much boil down to: (1) Spend a lot of time discussing accomplishments at their current / previous employers. If you drill down on particular projects they should be able to explain, in great detail, exactly what their contribution was and how they worked with the rest of the team to get that done. (2) Pick a technical question that has a lot of depth so that you avoid false positives because people have just looked up the answer on the internet. I would also add that in my view the best questions have no tricks and require no "aha!" moments. It should be something the candidate can be reasoned through step by step so you can see their thought process. When they come up with an initial solution, then you can dive deeper by expanding the problem or adding additional constraints.
- mturmon 13y ago
  I'm also disappointed that so many comments above do not engage with the information in the article. I, too, found this to be a thoughtful and helpful piece. Along the parts I liked -- Using "calibrated" questions that you have used before and know how far people get with. Focusing on a specific accomplishment or area of strength of the candidate to probe. (As opposed to choosing something from the resume that, say, the interviewer was most comfortable with, or that the interviewer found interesting.) Paying attention to preparing the interviewers, so they know why the candidate is being considered, they know why they are on the interview list, and they are experienced enough with interviewing to give a useful opinion.
hobb0001 13y ago
> [he] believes every phase of the process needs to be meticulously designed to drill deep into skill sets, ... and leadership potential Ugh. I hate this. For a technical interview, why is leadership potential a priority? Speaking for myself, I've done the leadership thing and don't care for it. People tell me I'm good at it, but I still prefer dealing with algorithms over humans, and I actively decline offers for "promotion". (I already have my very own batch of little humans of to deal with, thank you very much.) Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. In terms of the corporate efficiency calculation of effectiveness/hour, I suppose that's still true. But for me, personally, that's not the equation that I care to maximize.
- dragonwriter 13y ago
  > For a technical interview, why is leadership potential a priority? Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate. > Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. The places where that is least true are also the places where evaluating leadership potential in technical workers is the most relevant to immediate job performance.
  - realspeech 13y ago
    but it is extremely hard (perhaps impossible) to accurately determine leadership potential in an interview
  - rhizome 13y ago
    Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate Does it work, by which does it produce the desired results or benefits? It seems somewhat likely that this would have a side-effect of pushing employees to asserting a dominating personality.
- RougeFemme 13y ago
  I agree re: the cultural mindset. And I definitely get preferring algorithms over humans. But I don't think leadership necessarily has to = management. Leadership can simply be about being persuasive, degree to which you deal with hurdles/stupidity/whatever without killing somone, able to facilitate a contentious discussion/meeting etc. - those soft skills that make you a good teammate, defacto team lead if/when necessary or whatever. Like you, I've been a manager and didn't like it (although I did like the fact that my input carried more weight than it would have otherwise). But I'm sure that leadership potential serves you well, even in non-management positions.
realspeech 13y ago
without any data and careful analysis of that data, his opinion on how to hire is worthless. it's somewhat disturbing that he likes to use the STAR framework, which normally leads candidates into artificial responses. is there any reason to believe that this framework leads to better results?
atopuzov 13y ago
You pass the interview process and then end up doing shit that monkey can be trained to do.
cell303 13y ago
Moving away from "the best available" towards "not the worst possible" (who could potentially destroy our company) would be a good idea.
tomjohnson3 13y ago
one of the most important things - something that is difficult to suss out in an interview - is how well you (or your team) would work with a person. (i think this is perhaps the most important quality - definitely more important than being able to solve brain teasers.) i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...
laichzeit0 13y ago
What's notably lacking from this perfect interview is mention of referrals. I've had the best of luck with referrals as the managers I've worked with have never felt the need to "burn" me and could give an accurate assessment of the roles and responsibilities I have in my current job.

220 points by jnazario 13y ago

I disagree with "In most cases, the “best and the brightest” already have jobs, so you’re really just on the lookout for the best available.". Just because one is employed does not mean one is taken. If you want the best you have to work at it. Appeal to their desires and they'll come to you. Looking for the "best available" is no better than settling. In my opinion you'll end up doing a disservice to both your company and the new hire.

walshemj 13y ago
Lets be honest Amazon is an online shop the best and brightest are either going to be working at tier 1 RnD or on wall street or the city.
- afandian 13y ago
  Let's be honest, Amazon has a massive infrastructure supporting a very large number of very diverse customers, tonnes of products and projects, and lots of active development on them.
- kenster07 13y ago
  Re Wall Street: It's unfortunate that some of our brightest are tempted into playing multi-million dollar poker, rather than focusing their energy on producing any significant long-term value. I believe software engineering wages will continue to rise over time, and will eventually reflect the true value the industry produces.
  - walshemj 13y ago
    Experience from other branches of engineering compared to other equivalent professions over the last 100 years suggests otherwise.
  - dylangs1030 13y ago
    I have no shame in admitting that I would rather use high-level mathematics and programming to play "multi-million dollar poker" on Wall Street than join half of the startups and big tech companies and have a boring job churning out insignificant apps at a desk. In fact, the way you phrased it, being a quant sounds really fun :) you don't want to play multi-million dollar poker? I mean, sure, meaning and existentialism and all that, but I bet it's at least somewhat exciting.
    - kenster07 13y ago
      That's the fundamental problem. The function of the finance system in a healthy economy is to ensure efficient allocation of capital. Playing poker has a neutral to negative effect on the system.
      - dylangs1030 13y ago
        Well you're the one who used the word poker, so I guess it depends on your definition here...but the financial system somewhat revolves around poker. Valuations of companies are only important to investors because of the inherent gambling. If you purchase capital at a single valuation that never rises or falls, what's the point? Even an index fund pursues gradual increases over time. ...Or am I building a strawman here? Is this not where you were going?
    - chris_mahan 13y ago
      Which is why I work for a large bank, in toxic assets resolution, where the stakes are still in the billions of dollars.
  - ryandrake 13y ago
    I think I'd rather use my tech skills to play multi-million dollar poker on Wall Street then create yet another beautiful, responsive, flexible, tweet-able, location-based, modular MVC library API component framework backbone architecture.js.
    - Zimahl 13y ago
      Also, who wouldn't want to play high-stakes poker with someone else's money!
      - deleted 13y ago
        [deleted]
    - dylangs1030 13y ago
      "It's beautiful! It's disruptive! It's agile and lean!" "What does it do?" "It revolutionizes the way you share pictures of your cats!" ...Yeah. Playing multi-million dollar poker on Wall Street sounds a lot sexier than the shiny startup veneer put on another photo sharing app.
- walshemj 13y ago
  down voted eh the truth hurts doesn't it there is a world of difference between working at JPL MIT etal and some me to start up using the latest shiny new thing copying some existing SM type solution of which there are dozens already.
  - jlgreco 13y ago
    cough http://aws.amazon.com/solutions/case-studies/nasa-jpl/ http://aws.amazon.com/solutions/case-studies/nasa-jpl/ cough Sorry, caught a nasty flu..
  - jmduke 13y ago
    I didn't downvote you, but calling Amazon a startup is pretty ridiculous -- and your spelling/grammar isn't lending your argument any favors. (Though, ironically, it does fit into PG's description of "a company optimized for growth.")
    - walshemj 13y ago
      I think it should have been obvious from the context that I was talking about a substantial number of the HN commentators who are working at me to start ups (nothing wrong with that but its not world changing rocket science) Soory if you have problems with reading as a literate dyslexic I had no problem with reading writing not so much :)
      - deleted 13y ago
        [deleted]
        pekk 13y ago
        > core tenant You mean "core tenet." Please don't be a dick to people about flaws that you share with them!
- lotsofcows 13y ago
  Have you heard of a little thing called Amazon Web Services?
vonmoltke 13y ago
If you have a situation, which I believe we do now, where the number of engineers who fit the "best and brightest" category as defined by the companies looking is less than the number required by these companies, somebody is going to have to either "settle" or work with a hole. It will also make some of these engineers overpaid. I don't see how this benefits your company. Yes, in this system some companies will be "winners" in the sense that they get an awesome team for a good price, but most will lose. I would not be surprised if the high rate of failure in the startup community is related to this. For the record, I do not see hiring the "best available" as settling unless they do not meet some minimum requirements of the business. The purpose of hiring people is to help execute a product, service, or whatever else your business is trying to make money off of. Anyone who can increase your chances of succeeding or increase your level of success is a good hire. Yes, that includes looking at intangibles like team and culture fit, work ethic, etc. But this person does not need to be the "best" to provide this boost to your business. In fact, it is effectively impossible to hire the best, because there will always be someone better.
- walshemj 13y ago
  "over paid" what is this concept that you speak off - there is no big guy with a big white beard sitting on a cloud who's going to reach down and say "you yes you with a third in media studies from Luton university should be paid X and those cheeky blue collar train drivers or miners who have scarce skills should be paid less than you". The PHB might think they are over paid but like it or not this is Laissez-faire capitalism pay is determined by scarcity and bargaining power. ps for those outside the UK Luton is the unfortunate University that props up the bottom of the university league table - though to be fair they do rank well for Nursing degrees.
  - saraid216 13y ago
    Paid more than one standard deviation higher than average for that level of competency, scarcity, and bargaining power.
    - humanrebar 13y ago

afandian 13y ago

"No matter how many times they say it, most still make decisions based on gut feel, basic credentials, GPAs, ivy league educations, flashy company names - even SAT scores. Roseman objects." What's a Roseman object? Is it some psychological classification of the means by which we just things? Of course, I mis-read it. Shame, that could have been an interesting topic.

lsb 13y ago
If it's confusing to parse, let me expand it: Neil Roseman strenuously objects to that assertion.
- afandian 13y ago
  I understood it perfectly about half a second after I misunderstood it. I just thought I'd share my experience, as I found it amusing.
saraid216 13y ago
I didn't know "just" was a verb. Is that when we do something but only barely competently?
- afandian 13y ago
  It's not a verb, it's a typo.
VikingCoder 13y ago
I think there should be something in science called the 'reindeer effect.' I don't know what it would be, but I think it'd be good to hear someone say, 'Gentlemen, what we have here is a terrifying example of the reindeer effect.' - Jack Handey
learc83 13y ago
>SAT scores Wait...what? Is this a thing? Are people actually putting this on their resumes? I had the highest SAT in my county (still just a 1360/1600), but I don't go around putting 2002 "STAR student"[1] on my resume. [1]Local award for highest SAT score. I think each school in Georgia awards this.

awakeasleep 13y ago

The one element no one ever accounts for is the interviewers. You're never going to have a perfect technical interview because you're an imperfect human. You're incapable of judging whether someone is a perfect cultural fit, and someone who doesn't fit the culture might help the company more than someone who does. That isn't even getting into the the conflict between whats good for the company and whats good for you.

Matt_Mickiewicz 13y ago
When Google pushed out their interview-results data this year, one of the big findings was that there is NO SUCH THING as a good interviewer. To quote the NY Times interview with Laszlo Black of Google - "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert." That being said, there are a lot of very disorganized interviewers out there, especially among seed stage companies. I've been amazed about how many people grumble about the "talent shortage", yet regularly miss interviews with "hot" candidates because of meetings, forget them altogether, or just lack a sense of urgency & speed when it comes to running an effective hiring process. A large part of having a hiring culture is about decisiveness. Loosing a fantastic Engineer because you spent 4 weeks making up your mind, or you didn't want to pay an extra $10K/year is just silly when it means that you won't hit your product roadmap goals.
- RogerL 13y ago
  I think that is perhaps telling. Google doesn't interview you on your experience, but whether you can can solve network/graph algorithms taken from Knuth exercises. I would expect a random mess from that. It sounds like this person actually drilled down into what the candidate actually knew. I think it is far more useful to ask somebody about what they actually did. Are they excited and engaged by it, did they come up with novel thoughts about it, do they understand the CS literature in relation to it, and so on. Program a hash table on the whiteboard when you have been just fine using the built in one in your favorite library or language? I'm not sure what that predicts about my on job performance. If I had to do that I'd reread the literature and then whip one up, easy peasy. I can state that with confidence because I read the literature on other topics and do that every day. Have me explain the new numerical filtering technique I just implemented and I'll give you good interview. Ask me to implement a smart pointer? Not so much.

johnhess 13y ago

I'm always amazed by how confidently people can propose a hiring "rule" or even guideline. The article pays lip service to the fact "There's no way to A/B test hiring decisions" but then goes on to describe a handful of rules to "build an effective organization, regardless of size or resources." The thing is, there's no evidence that these are effective and definitely none to suggest they're generalizable to organizations of every size. There's an appeal to the author's authority as an Amazon VP (admittedly a hell of a credential), but that's not evidence of effectiveness. The fact is, hiring is tough to learn. It's made especially difficult because your feedback comes at least 6 months after you make the decision. I've had the good fortune to be in an organization which grew from ~10 employees this time last year to ~80 now, so I've gotten to see how a few dozen hires panned out in a short period of time. I learned a lot through that. Even so, as a professional data geek, I couldn't say with a straight face that I "know" anything about hiring. I do my best day in and day out, but saying you've got hiring all figured out is the professional equivalent of knowing "What (wo)men want." It's a sign of confidence, but not necessarily knowledge. Don't get me wrong, in the absence of quality data, I'll take the wise words of someone who has been there and done that. But I'd much rather some empirical proof or at least some "YMMV" humility from those who purport to know.

vonmoltke 13y ago
This topic has started taking on religious tones in the past year or so. It seems to be a long stream of anecdotes and opinion pieces presented with appeals to authority either of the author or someone the author is connected to.

coin 13y ago

Not related to the content, but why do sites like this disable pinchzoom? What value could it possibly add?

freyr 13y ago
Yesterday I needed to look at source code on GitHub using my iPhone. Horizontal scrolling AND pinch-to-zoom were disabled, making it impossible to read lines of code wider than my phone screen. Github. Why is this so hard?
Daniel_Newby 13y ago
For the same reason other sites use floats for sidebars: they hate you and want to become irrelevant.

evanjacobs 13y ago

In my 10 years at Amazon, I interviewed lots of engineers and I think Amazon's interview process is fatally flawed. Of course Amazon is still able to hire good candidates but I believe it is in spite of this process and not because of it. Here were the most frustrating aspects of interviewing for me: 1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area. 2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform. 3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript. 4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.

WalterBright 13y ago
> I think Amazon's interview process is fatally flawed. The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process. I suspect the real problem is the implication that this is even a solvable problem.
- dylangs1030 13y ago
  I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions. Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with? We work with what we have, and take the benefits it gives us. Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.

freyr 13y ago

"The anatomy of the perfect technical interview from a former Amazon VP" And more recently, a former Zynga VP. Surprising that such an expert at evaluating people chose to work for Zynga, of all places.

hyperbovine 13y ago
Money is a hell of a drug.
- umsm 13y ago
  That's how you get the best talent. There is no secret question you can ask to find great talent. I've interviewed persons that did very very well during the interview process, but failed miserably on a day-to-day basis.
  - seunosewa 13y ago
    So money is the answer? Have you never observed a well paid developer who "failed miserably on a day to day basis"?
- enraged_camel 13y ago
  Either that, or they really wanted a challenge.

jpalioto 13y ago

I appreciate the notion of having the whole team prepared for the interview process. I think this is a best practice. But, when I interview, I am looking for completely different things than the author. I really don't care about what you did at your last company or if you raised this by 50% or that by 30%. Rather, I want three things in a candidate: smart, talented and hungry. If I get someone who is smart, talented and hungry and I drop them into a situation where there is unlimited upside, my job then becomes to guide them to greatness. I also strongly disagree with the notion of not asking questions you do not know the answer to. Anyone can read an obscure section of the C++ language spec or a chapter on some data structure 5 minutes before an interview and make themselves feel superior to the candidate. What does this prove? Nothing. I often ask questions I don't know the answer to in order to determine whether or not the candidate can teach me something. I love having people on my team that have the ability to teach others. The author's point is very well taken -- interviewing is an important process and it should be conducted in a way that results in the best possible outcome. But, he's looking at the situation far too rigidly for my taste.

umsm 13y ago
I disagree with the hungry part (if you are referring to someone that needs the money). Being "hungry" just means that they will be stuck with a company but will not love what they do.
- jpalioto 13y ago
  Less about needing money and more about being ambitious. Hungry for success I should say.
  - Terretta 13y ago
    Whether hungry for money or hungry for success, you're then missing hiring candidates who aren't "hungry" because they have a proven track record at having done very well financially from a series of successes. At this point, they don't need money or validation, they just figure they can help you solve things and are talking to you because they enjoy solving new things. Hungry is overrated when what you need is focus on solving the business' problems. This is one reason VC like to see founders take "shelter money" off the table early.

pdovy 13y ago

There seems to be a lot of criticism of the suggestions here, but I think they are good ones. It seems to pretty much boil down to: (1) Spend a lot of time discussing accomplishments at their current / previous employers. If you drill down on particular projects they should be able to explain, in great detail, exactly what their contribution was and how they worked with the rest of the team to get that done. (2) Pick a technical question that has a lot of depth so that you avoid false positives because people have just looked up the answer on the internet. I would also add that in my view the best questions have no tricks and require no "aha!" moments. It should be something the candidate can be reasoned through step by step so you can see their thought process. When they come up with an initial solution, then you can dive deeper by expanding the problem or adding additional constraints.

mturmon 13y ago
I'm also disappointed that so many comments above do not engage with the information in the article. I, too, found this to be a thoughtful and helpful piece. Along the parts I liked -- Using "calibrated" questions that you have used before and know how far people get with. Focusing on a specific accomplishment or area of strength of the candidate to probe. (As opposed to choosing something from the resume that, say, the interviewer was most comfortable with, or that the interviewer found interesting.) Paying attention to preparing the interviewers, so they know why the candidate is being considered, they know why they are on the interview list, and they are experienced enough with interviewing to give a useful opinion.

hobb0001 13y ago

> [he] believes every phase of the process needs to be meticulously designed to drill deep into skill sets, ... and leadership potential Ugh. I hate this. For a technical interview, why is leadership potential a priority? Speaking for myself, I've done the leadership thing and don't care for it. People tell me I'm good at it, but I still prefer dealing with algorithms over humans, and I actively decline offers for "promotion". (I already have my very own batch of little humans of to deal with, thank you very much.) Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. In terms of the corporate efficiency calculation of effectiveness/hour, I suppose that's still true. But for me, personally, that's not the equation that I care to maximize.

dragonwriter 13y ago
> For a technical interview, why is leadership potential a priority? Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate. > Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. The places where that is least true are also the places where evaluating leadership potential in technical workers is the most relevant to immediate job performance.
- realspeech 13y ago
  but it is extremely hard (perhaps impossible) to accurately determine leadership potential in an interview
- rhizome 13y ago
  Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate Does it work, by which does it produce the desired results or benefits? It seems somewhat likely that this would have a side-effect of pushing employees to asserting a dominating personality.
RougeFemme 13y ago
I agree re: the cultural mindset. And I definitely get preferring algorithms over humans. But I don't think leadership necessarily has to = management. Leadership can simply be about being persuasive, degree to which you deal with hurdles/stupidity/whatever without killing somone, able to facilitate a contentious discussion/meeting etc. - those soft skills that make you a good teammate, defacto team lead if/when necessary or whatever. Like you, I've been a manager and didn't like it (although I did like the fact that my input carried more weight than it would have otherwise). But I'm sure that leadership potential serves you well, even in non-management positions.

realspeech 13y ago

without any data and careful analysis of that data, his opinion on how to hire is worthless. it's somewhat disturbing that he likes to use the STAR framework, which normally leads candidates into artificial responses. is there any reason to believe that this framework leads to better results?

atopuzov 13y ago

You pass the interview process and then end up doing shit that monkey can be trained to do.

cell303 13y ago

Moving away from "the best available" towards "not the worst possible" (who could potentially destroy our company) would be a good idea.

tomjohnson3 13y ago

one of the most important things - something that is difficult to suss out in an interview - is how well you (or your team) would work with a person. (i think this is perhaps the most important quality - definitely more important than being able to solve brain teasers.) i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...

laichzeit0 13y ago

What's notably lacking from this perfect interview is mention of referrals. I've had the best of luck with referrals as the managers I've worked with have never felt the need to "burn" me and could give an accurate assessment of the roles and responsibilities I have in my current job.