9 ms·

How Effective are Technical Interviews?

95 points by jeanhsu 16y ago

russnewcomer 16y ago
I work for a small MSP in the Midwest. I do some network administration, some help desk, and some coding. The basic jack-of-all-trades, master of none type of position. The position was intended to be filled by someone with relatively low experience, i.e. recent college grad. (I had 5 years of development and IT Admin experience) When I was going through the hiring process, I got a list of questions that were specifically related to the kind of work I would do. "My e-mail doesn't work, how do I fix it?" "How would you write a script to check for number of a given event log events?" "Write a simple SQL statement to get this data out of this table." I was apparently the only candidate who even assumed that I was on the phone with someone whose email didn't work. That was largely what got me the job. Now, granted, this is only an anecdotal piece of evidence about one job in the Midwest. But the larger lesson that can be applied, I think, is the same one that Jean brings out in the article. The most effective technical interview is one that tests what the candidate will be doing. So, if you're developing Android apps, asking the candidate to develop an app is probably pretty effective. If you are looking for a software engineer, ask them to engineer. A java pop quiz has little worth.
larsberg 16y ago
How successful are the people you hire, and how well do you avoid bad hires? Ultimately, until you have a few years of data on your own decisions, it's hard to know how "effective" your interviewing process was. And even then, without wider baselines for comparisons within your corporation, you don't know how much better you could have done -- you only know how poorly you did by how many people you either had to terminate or did not turn into the high performers you expected. Most big tech companies probably do this -- if you had done a few years of hiring at Google, I'd bet recruiting could have told you how you did. At Microsoft, recruiting could tell you where your recommendations ended up, how you ranked as an interviewer, etc. The week on-site is probably a good substitute if you don't already have someone on your team with the proven ability to asses talent and fit rigorously and quickly.
tocomment 16y ago
I like interviews where you explain existing code you've written. I do well with those. I also do well if they give me a homework problem before the interview. True story. Interview at Google. They had me make this AJAX web application as homework before the interview. Three people told me they LOVED my solution and thought it was beautiful. I figured I was a shoe-in for the job at that point. But at the interview they never asked me any technical details about it. They then proceeded to ask me what happens when you add a '4'+4 in javascript. My mind kind of blanked. At which point they assumed I couldn't program :-( Even though I had created this web app they all loved. They sent me home after one 30 minute interview even though I had to take two days off of work and fly across the country. A very dismal experience it was.
- deleted 16y ago
  [deleted]
- lsb 16y ago
  It sounds like the AJAX web app was designed to test your sense of taste and your ability to get things done, and the Javascript question was to test whether you've ever done anything numerical in JS. ('4'+4)-4 is 40, whereas (4+4)-4 is 4. There are a lot of people who glue a few things together in Javascript and PHP and call it a day, and their filter against those is imperfect. Interviews as a whole are less informative than actually working with someone.
  - Silhouette 16y ago
    > and the Javascript question was to test whether you've ever done anything numerical in JS. It's a lousy test, then. The fact that someone can't tell you the answer to that question, off the top of their head under interview conditions, might mean they are a newbie claiming to know languages they really don't. However, it could also just mean that they have experience with several dynamically typed languages but, since they would never write something daft like '4'+4 in real code, they would have to look up which interpretation JS happens to use. In case anyone is wondering, in JavaScript, the answer is 44. However, in Perl, it is 8. In PHP, it is also 8. In Python, it is a type error without an explicit cast. You get the idea.
tomh- 16y ago
So I would be interested in knowing which companies have better, more successful alternatives to tech interviews. Is there any reasons why companies still insist that low level detailed knowledge of a language is a good indication that someone will do well on a job? I feel technical interviews are significantly harder than doing the actual job.
- tobych 16y ago
  I thought that, too, after the recent interview I described in my comment on the OP. When did I ever need to think this hard? Um, when I was writing proofs in my Semantics of Programming Languages course at university, I guess. Most of my work days are spent pondering gently, with - a few times each year - a trip to the library to hunker down and really crunch some x's and y's. But it did make me wonder whether I wasn't choosing sufficiently challenging work, in the sense of the amount of raw "think, think, think" effort required.
deleted 16y ago
[deleted]
jswinghammer 16y ago
I'm sure that the process I use rejects some people who could do good work but I've never let anyone through who I regretted hiring later. I always ask a basic programming question and ask some questions to gauge how much the person actually takes time to learn more about software development. That's usually enough data to figure out what's going on. 95% of people fail the programming question so I continue on the interview and get more information about why I'm going to reject them. Usually they fail every other portion of it as well.
- tomh- 16y ago
  What is the question I may ask?
  - jswinghammer 16y ago
    Implement: itoa
    - tomh- 16y ago
      I don't know the answer of this question. I could do a guestimate on how such thing would work, but I've never coded c/c++ extensively. Am I useless now? I don't really see how this proves someone has basic knowledge about programming. However I could be wrong and this is a function that c/c++ programmers write on a daily base and is indeed a good indicator if someone ever ran a c++ compiler.
      - NickPollard 16y ago
        It's not in itself something a C programmer would ever really find themself doing, but as an algorithmic question it's fine - it's testing knowledge of looping, radices, strings etc. In C, you normally have to output the string in reverse and then reverse it, so you need to be aware of that. It's also a problem that's quite easy to explain (programmers are familiar with numbers and strings), but due to it's library availability, most people won't have actually written it themselves so it's not just a case of having done it before. Coincidentally, I was asked this exact question in a phone interview this week. I didn't have to give exact code, I just walked through the principal of it (loop for each digit, dividing by the radix each time using the leftover specify the character (can be a lookup in an array), possible to use bit shifting in base 2/8/16 situations).
known 16y ago
interview != quiz
- cema 16y ago
  True, but it often feels like a quiz is part of it.
aplusbi 16y ago
I posted this in the comments on the blog, but I'm reposting it here for discussion: I've done a lot of interviewing and have been interviewed myself a few times so I think I have a good amount of experience in the process. There are a lot of factors involved in technical interviews which I think a lot of people miss. Tech interviews are usually only an hour, so the questions have to be short - this often leads to more academic questions, and questions not directly related to the work done at the company. Interviews only tell you so much. You won't really know much about a candidate until they've been working with you for awhile. As such, many technical interviews are more about weeding out bad candidates than finding good ones. People embellish, cheat and lie. Just because a resume is impressive doesn't mean the candidate is impressive. Just because they worked on a project doesn't mean they contributed anything worthwhile to it. Most people are honest but will still bend the truth sometimes. It's a lot harder to do that in person. I like the idea of having a candidate work with the company for a week, but that's not realistic for a majority of cases. Not all companies are going to be able to do this, and neither are all candidates. If a candidate already has a job, asking them to take a week off is asking a lot. I also like the idea of a "homework" assignment. However it's possible for the candidate to cheat. It's also asking a lot of the candidate who may have a job and a family life. These two approaches heavily favor college students and recent grads. It's far from a perfect system. However it's reasonable (or so it seems) and is easy for companies to implement.
_delirium 16y ago
It would be interesting if a large company that could afford to do so (say, Google) made an attempt to experimentally test that question. It'd take some thinking to design a useful study with some sort of blindness, but something along the lines of: have everyone do the technical interviews, but then for some subset of interviewees, the final decision-maker chooses whether to hire or not based on a coin flip instead of based on the interview. Then assess the performance of each group of hires a year or two later. Disclaimer: may or may not be legal; I'm no employment-law expert, so have no idea if "not hired due to coin flip" would be a grounds for suing.
dansingerman 16y ago
It is far worse to hire a bad person, than to fail to hire a good person. Especially for startups. Corollary: disgruntled tech interviewees who are good, but get passed on.
sown 16y ago
I have a sizable and growing portfolio of code I've written: some complicated algorithm stuff, some open source contributions; things like that. No one ever looks at it. I'm starting to wonder why linked lists are so important to employers.
- btipling 16y ago
  We look at stuff like that at Cloudkick/Rackspace.
- tobych 16y ago
  The "Linked Lists" chapter from "Programming Interviews Exposed" begins: "The deceptively simple linked list is the basis for a surprising number of problems regarding the handling of dynamic data. Problems about efficient list traversal, list sorting, and the insertion or removal of data from either end of a list are good tests of basic data-structure concepts... Their simplicity appeals to interviewers, who want to present at least two or three problems over the course of an hour-long interview... You can write a relatively complete implementation of a linked list in less than 10 minutes, leaving you plenty of time to solve the problem... In addition, there is little variation in the way linked lists are implemented, which means that an interviewer can simple say "linked list" and not waste time discussing and clarifying implementation details." http://www.piexposed.com/ http://www.piexposed.com/
- aplusbi 16y ago
  I always look at that stuff and I ask about it during interviews, but I still ask general technical questions. Sometimes the candidate's contribution to a project is nothing more than writing the README file.
  - deleted 16y ago
    [deleted]
msg 16y ago
I work for one of those big companies. After going over trouble spots in the resume and asking why they want to work here, I tend to ask one or two very similar questions that 1) force candidates to do nontrivial coding, algorithms, and design while writing a complete program, and 2) elicit a wide range of responses. I don't require a certain language or ding on syntax. We usually spend the rest of the hour on the question. In general, bad candidates get me hinky in the resume portion (or worse, don't really want to work here!) then crash and burn on the programming question. Borderline candidates do ok in resume but make me feel like they're not that comfortable programming. Great candidates usually do something clever during the problem. I believe that doing homework isn't convincing. It's too easy to cheat. That's probably why the commenter elsewhere on the thread got bounced from Google. Doing week-long trial runs would be ideal, but impossible at my company. We sometimes go contractor-to-hire though. Can you be a great programmer and fail my interview? I think it boils down to three things: skills, communication, and pressure. If you lose your skills and communication under pressure for an interview day, you probably won't survive in my environment over the long haul.
patio11 16y ago
I put tech interviews down fifty/fifty to "We don't know any better way of doing this" and "The unavoidable reality is that if a position is advertised openly then 98% of candidates are screamingly inappropriate for it and after applying a few filters to the resume we've gotten that to a more manageable 90% or so for interviews." Happily, there is a simple opt out mechanism. There are two ways to get into any company: through HR and the resume -> interview -> offer process, and the other way. (Convince a decisionmaker within the company that you, specifically, are the answer to their prayers. Get hired. Skip slush pile.) Pick door number two. If you get your jollies off of sorting linked lists you can do so after having received the moral equivalent of an offer (which should decrease the stress to the point where you can successfully sort a linked list).
- alanthonyc 16y ago
  Great answer, as always. However, it addresses the problem only from the point of view of the hiree. For companies looking to hire people, the typical interview process is equally a problem. I think the "dating" process described by the OP sounds like a good solution for the hiring company. It's also a good signal to the employee that the company is serious about hiring the best, using the best methods.
  - aplusbi 16y ago
    The "dating" process only works when both parties are available. If the candidate currently has a job asking them to take time off (or worse, quit without a guaranteed offer) is asking a lot.
- imajes 16y ago
  I've picked door number two consistently throughout my career. It's more effective and much more meaningful to establish a relationship with decision makers from the get go.
- btilly 16y ago
  I consider the existence of door #2 a red flag. If decision makers who are likely not technical themselves are making that kind of decision about technical hires, odds are that they are going to make some very bad hires that I won't like living with. That's not to say that there aren't a lot of companies where your advice works. But I'd prefer to work for the companies where it doesn't.
tastybites 16y ago
We hire people based on enthusiasm, general intelligence (you can usually tell after a few minutes) and a low level of demonstrated proficiency - we'll ask them to explain a code sample they wrote, or ask for a verbal description of how to solve a problem. We choose the person we like best, and if they don't work out after 2 weeks we fire them. Hot seat quizzing produces way too many false negatives. We are not google. We can't do 5+ rounds in search of yeoman geniuses. We have to spend time making money. Some companies don't seem to have this inconvenient little problem, but we do. And besides, most of our hires have come from existing networks where none of this is even necessary. Having an extensive professional network should be a feature of anyone you consider bringing on as founder or employee or contractor.
- shortlived 16y ago
  Are there laws that allow/prevent the 2 week trial period for a business?
  - tastybites 16y ago
    Not in California. It's called contract-to-hire usually.
  - chollida1 16y ago
    I'm Canadian, but here most, almost all, companies have a 3 month window in which a new hire can be let go without severance.
  - JoachimSchipper 16y ago
    This is allowed in the Netherlands, and I'd wager in most European countries.
  - evilduck 16y ago
    Most US states are at-will employment, allowing anyone to be fired at any time, for no reason at all. http://en.wikipedia.org/wiki/At-will_employment http://en.wikipedia.org/wiki/At-will_employment Fun fact, many anti-discrimination laws in the US don't apply to companies with less than 15 employees.
tomh- 16y ago
I would suggest people to filter out like riotgames does. They ask job applicants to answer some questions and implement a small project in Java. See http://pastebin.com/zhCxcRds http://pastebin.com/zhCxcRds for the questions and http://riotgames.com/careers/software-engineer-java-pvpnet-platform-stlouis-mo http://riotgames.com/careers/software-engineer-java-pvpnet-p... for the job post.
sophacles 16y ago
I have found that for technical interviews, the following pattern works very well (hiring side): 0. Explain to the candidate this algorithm and why. It isn't magic, there are no right or wrong answers, just testing "fit". This is a case where foreknowledge does not change the outcome! 1. Get a basic grasp of technologies both interviewer and candidate have in common, via say, just chatting. This is vital -- find an area where you are both somewhat competent. 2. State a problem you don't know the answer to, something you are currently working out for your own projects, in terms of the mutual tech. VERY IMPORTANT: you must not have the best solution in mind when you bring up the topic. 3. Brainstorm on it together. This provides more feedback in a valuable way than the classic brainteaser. You find out how the person thinks and problem solves -- you know, that thing you are hiring her to do. You find out how you and her work together. Even if no solution is reached, it is OK because there was no solution going it -- just a problem to be worked. Your job in the interview is then to make sure you steer the discussion towards various tech and deep problems, coding issues, etc. Afterwards you will have a pretty strong idea of how it is to work with the candidate, which is quite useful for deciding if you want to continue to work with the candidate as an employee. Further if you've done well at the steering of the discussion, you have learned about candidate's technical depth, without setting up test/interview anxiety situations.
- vitorbal 16y ago
  This hits the nail on the head for me, and I couldn't agree more. If I was interviewed this way for a position, even if I didn't get the job I would have huge respect for the company afterwards.
- aplusbi 16y ago
  This sounds like a great idea but I feel it has a couple of short-comings. Most importantly is that it can be counter to the idea of preferring smart, motivated candidates over candidates that are well versed in certain technologies. Additionally, this technique would probably work against college students and recent grads. It can also be hard to quantify the results and the questions will vary wildly between candidates. Of course none of those obstacles are insurmountable and I think that when this approach works, it probably works better than a more general tech interview.
pamelafox 16y ago
Four years ago, when I was interviewing for API Support Engineer in Developer Relations at Google, I was asked to complete a Technical Worksheet, with challenges like write an AJAX mashup, write an XML schema, write a letter to a disgruntled developer, explain HTTP verbs to a 5-year-old. It was basically a mini version of what I would later be doing on my job, it was really fun to do, and it was great preparation for the interview. Unfortunately, as much as I loved that worksheet, I hear that we do not usually give it out anymore to new interviewees. I guess that it offends or intimidates people that are more experienced or interviewing at multiple companies. (I was a college student when I did it, and it did take me a few days of ignoring my homework to complete). So now, we have to base our decision mostly on the technical interview, which as discussed, has its flaws - particularly if the candidate is a nervous interviewee. It does well at eliminating the folks we don't want, but I think a few times, it's eliminated people we would have benefited from. I think we'd do well by offering the worksheet as an option to all new interviewees, but not a requirement.
ylem 16y ago
I think technical competence has to be part of it. If you know the person's references and those references are honest, you can ask them. But, I had one friend who had the fortune of always being surrounded by pretty bright, motivated, and competent people. For her first hire, she did a phone interview (international candidate), but was thinking more about issues like "does this guy seem like a good fit for the project in terms of interest" and only found out when he arrived that his technical expertise was far lower than what one would expect given his resume...
yaakov34 16y ago
I hate to be a stick in the mud, but you have to ask algorithm questions, and you have to ask them to code an implementation. If you hire a lot of people, that is simply the only way to avoid making (some) horrible hires who later cost you dearly. Portfolio of code is not a substitute - you don't know exactly how much the candidate contributed to it himself, and you can't trust your own ability to evaluate a lot of code written by someone else, not unless you spend an unreasonable amount of time on it. Working with a person for a week is not realistic for most places that hire a lot of candidates. A lot of the time I decided to skip all that boring, old-fashioned tech interview stuff and hire someone who looks great on the basis of something else -- anything else, I've regretted it. I do admit things are different if you're in a tiny startup and basically looking for a co-founder rather than an employee in a big organization. But even then I'd really like to do a brief algorithm and coding run to be sure.
dkarl 16y ago
Many technical interviews are very theory-heavy, muddling the divide between Computer Science and Software Engineering. I've been faulted for this in my interviewing, but I think that you have to pose at least one problem that can be explained precisely in less than a minute and solved completely in less than an hour. Such a problem will inevitably have an abstract, theoretical feel. It's also good to ask more realistic questions to test domain knowledge, but there are candidates who can string the right words together in the right order all day long but fall apart when faced with a real problem. It's very frustrating to try to identify such people using a loosely-defined, open-ended problem, because they can speak competently and professionally while avoiding the aspects they don't understand, and many experienced candidates will exhibit the same blathering behavior even if they are technically strong, because it's a standard mode of speaking that everyone in the industry has to learn. I.e., if I faulted people for spouting BS in response to realistic, open-ended questions, I'd rule out strong candidates as well as posers. Asking candidates to completely and precisely solve a simple problem separates abstract thinkers from people whose skills are purely verbal.
- kscaldef 16y ago
  In my experience, people start complaining that an interview is "theory-heavy" at about the point that you expect them to know when and how to use a binary tree. Binary trees are not hard-core CS theory, they are freshman Intro to Data Structures. They are a very basic tool (almost) every programmer should be familiar with. Moreover, when I ask candidates questions about data structures and algorithms, while it's nice if they can ace it all without hesitation, what I'm really looking for is what happens when their first answer is wrong. When they pick an incorrect data structure for the problem and I probe to say, "so, using that data structure, how do you this task?", I want to see if they will realize that the task I'm asking for is hard to implement or has very poor performance using the data structure they have chosen, and whether they will step back and realize they made a mistake. The point is less what do you know, but do you know what you don't know and do you recognize when what you're trying isn't working.
  - strlen 16y ago
    > In my experience, people start complaining that an interview is "theory-heavy" at about the point that you expect them to know when and how to use a binary tree. Binary trees are not hard-core CS theory, they are freshman Intro to Data Structures. The other part about binary trees is by their very nature they involve pointers and recursion. Understanding pointers and recursion is the difference between being able to write code and being a programmer. The former is a fundamental skill any knowledge worker should have (scientists and financiers using Matlab or Python, statisticians using R); the latter takes passion, effort and talent. The other extreme I've seen is asking questions from computational geometry, combinatorics and complexity theory (for generalist software engineering positions). These are all relevant skills and for the former two a good cursory knowledge is important for a generalist: however, unlike pointers or recursion they're not as fundamental to programming.
orev 16y ago
Having been on the interviewing end of a number of interviews, I definitely have found that these types of "quiz" questions ARE EFFECTIVE, to a point. The number of people who claim to know something, but actually don't, are astounding. You'd like to think that people are honest, but they are not. The main issue is that of time. Interviewing people is an enormous time sync, and spending time to get to know them and their thought process is a complete waste if they don't know the material. Quiz questions can easily weed out the people who claim to know things but don't, and save you a lot of time. However, quiz questions should not be the only way to determine if someone is good for the job, and you always want to delve deeper with other kinds of questions as well. You can also use the quiz itself as a jumping off point for deeper discussions, which give you insight into thought process, etc...
- ojbyrne 16y ago
  "time sink"
agentultra 16y ago
I like the approach in this post. It sounds more reasonable. I dislike the term, "screening." It's not a good feeling for the interviewee to know that they're just being filtered before they get an interview. That the person interviewing them doesn't even believe that they can do their job properly. Screening reinforces the notion that the interviewee is just another rat in the race and that you, the interviewer, barely have the time to see them. It's an ugly, loaded term and I really don't like even hearing it. The traditional hiring process is a problem that goes both ways. It's nerve-wracking and emotionally draining for the person being interviewed and it's a time consuming risk for the person interviewing. The interviewee has to spend time crafting their CV, prepping themselves to answer a slew of random esoteric and useless trivia, and drag themselves to countless appointments. The interviewer has to sift through innumerable resumes and figure out which ones fit, sit through dozens of interviews, and then gamble that the person they've selected isn't going to waste a whole bunch of time and money. I think it's a search problem. At least for hiring programmers I think it's possible to filter incoming applications to your specific requirements. It should be possible to rank those submissions within your query. Then you should already have a good idea of how many interviews you will need to do and you can skip past the "screening" process. At least, I hope. I've been putting some back-40 hours into figuring out this problem. :)
michaelcampbell 16y ago
I haven't found them to be particularly useful except as a level 0 filter - you can get rid of some very shoddy people early, but finding the actual gems they do very little (in my experience, anway). I don't know who started this practice but it seems a lot of interviews recently are the brainteaser type; how many ping pong balls fit into a 1974 Super Beetle (hardtop), etc. Those are the worst.
fedd 16y ago
did someone mention http://codility.com/ http://codility.com/ ? i am not affiliated, just liked the site.
fleitz 16y ago
I prefer interviewing with koans. Setup a workstation with the tests and get the interviewee to make them pass. Have the interviewee check them into git / SCM on their own branch. It tests so much day to day knowledge and the ability to actually code / read code / and find / fix bugs. As well as all the various issues with your particular stack. Plus you can hide bugs all over the place, maybe the bug is in the database, etc. Maybe it's a real world task like 'speed up this page by 200%' type thing. There is a lot of flexibility and provides opportunity to for insight into how the developer actually solves real world problems. Maybe the increase the logic speed by 200%, maybe they just throw varnish on top of it.
stcredzero 16y ago
Sometimes it's the interviewer that ends up giving a bad impression. I remember one technical interview I was in: Interviewer: How would you do this? Me: Well, that depends. You could do it by doing polymorphism with subclasses, by passing in blocks, or through a Strategy. Interviewer: Well, actually, I'd do it like this. [proceeds to diagram polymorphic solution using subclasses as if that wasn't the first answer I gave] A good filter to use as a programmer interviewee: Ask to see their code. Ask to do some pair programming. If they blow you off or hem and haw about it, it's not a good place to work. If the interviewer reveals that they're actually in HR, just walk out.
jewgonewild 16y ago
When I interview candidates and get to the technical part, I usually start by discussing technologies and present a problem the team is currently working on and ask for a broad overview of how they would go about solving it. I start by asking them to talk me through their solution. This is where I filter out most candidates, because I know that if their problem solving skills aren't up to par I can't expect them to write any psuedo code to back up their problem solving skills. I generally give them another chance by offering a couple of hints, and if this doesn't work out I generally end it. If I like them, I ask them to talk to me about a challenging problem they have recently encountered and the solution to said problem. If I am pleased I ask them a few more questions and keep them in rotation. After this I ask for some simple psuedo code to back up the solution and that's about it. I am not big on asking any academic questions for the technical interview unless I know that the interviewee is a recent graduate and even then I don't weigh them that much. To finish off the interview I usually do some basic pair programming, just to see how the person does in a somewhat pressured situation. I find all of this to be pretty effective for the technical interview. Besides the technical side I strongly believe that a passionate person, that would make a good cultural fit is tremendously important. In my situation, I don't want someone who is smart but can't relate to the rest of the team.
makeramen 16y ago
direct link to the post, instead of the page of the blog: http://www.jeanhsu.com/?p=242 http://www.jeanhsu.com/?p=242

95 points by jeanhsu 16y ago

I work for a small MSP in the Midwest. I do some network administration, some help desk, and some coding. The basic jack-of-all-trades, master of none type of position. The position was intended to be filled by someone with relatively low experience, i.e. recent college grad. (I had 5 years of development and IT Admin experience) When I was going through the hiring process, I got a list of questions that were specifically related to the kind of work I would do. "My e-mail doesn't work, how do I fix it?" "How would you write a script to check for number of a given event log events?" "Write a simple SQL statement to get this data out of this table." I was apparently the only candidate who even assumed that I was on the phone with someone whose email didn't work. That was largely what got me the job. Now, granted, this is only an anecdotal piece of evidence about one job in the Midwest. But the larger lesson that can be applied, I think, is the same one that Jean brings out in the article. The most effective technical interview is one that tests what the candidate will be doing. So, if you're developing Android apps, asking the candidate to develop an app is probably pretty effective. If you are looking for a software engineer, ask them to engineer. A java pop quiz has little worth.

larsberg 16y ago

How successful are the people you hire, and how well do you avoid bad hires? Ultimately, until you have a few years of data on your own decisions, it's hard to know how "effective" your interviewing process was. And even then, without wider baselines for comparisons within your corporation, you don't know how much better you could have done -- you only know how poorly you did by how many people you either had to terminate or did not turn into the high performers you expected. Most big tech companies probably do this -- if you had done a few years of hiring at Google, I'd bet recruiting could have told you how you did. At Microsoft, recruiting could tell you where your recommendations ended up, how you ranked as an interviewer, etc. The week on-site is probably a good substitute if you don't already have someone on your team with the proven ability to asses talent and fit rigorously and quickly.

tocomment 16y ago

I like interviews where you explain existing code you've written. I do well with those. I also do well if they give me a homework problem before the interview. True story. Interview at Google. They had me make this AJAX web application as homework before the interview. Three people told me they LOVED my solution and thought it was beautiful. I figured I was a shoe-in for the job at that point. But at the interview they never asked me any technical details about it. They then proceeded to ask me what happens when you add a '4'+4 in javascript. My mind kind of blanked. At which point they assumed I couldn't program :-( Even though I had created this web app they all loved. They sent me home after one 30 minute interview even though I had to take two days off of work and fly across the country. A very dismal experience it was.

deleted 16y ago
[deleted]
lsb 16y ago
It sounds like the AJAX web app was designed to test your sense of taste and your ability to get things done, and the Javascript question was to test whether you've ever done anything numerical in JS. ('4'+4)-4 is 40, whereas (4+4)-4 is 4. There are a lot of people who glue a few things together in Javascript and PHP and call it a day, and their filter against those is imperfect. Interviews as a whole are less informative than actually working with someone.
- Silhouette 16y ago
  > and the Javascript question was to test whether you've ever done anything numerical in JS. It's a lousy test, then. The fact that someone can't tell you the answer to that question, off the top of their head under interview conditions, might mean they are a newbie claiming to know languages they really don't. However, it could also just mean that they have experience with several dynamically typed languages but, since they would never write something daft like '4'+4 in real code, they would have to look up which interpretation JS happens to use. In case anyone is wondering, in JavaScript, the answer is 44. However, in Perl, it is 8. In PHP, it is also 8. In Python, it is a type error without an explicit cast. You get the idea.

tomh- 16y ago

So I would be interested in knowing which companies have better, more successful alternatives to tech interviews. Is there any reasons why companies still insist that low level detailed knowledge of a language is a good indication that someone will do well on a job? I feel technical interviews are significantly harder than doing the actual job.

tobych 16y ago
I thought that, too, after the recent interview I described in my comment on the OP. When did I ever need to think this hard? Um, when I was writing proofs in my Semantics of Programming Languages course at university, I guess. Most of my work days are spent pondering gently, with - a few times each year - a trip to the library to hunker down and really crunch some x's and y's. But it did make me wonder whether I wasn't choosing sufficiently challenging work, in the sense of the amount of raw "think, think, think" effort required.

deleted 16y ago

[deleted]

jswinghammer 16y ago

I'm sure that the process I use rejects some people who could do good work but I've never let anyone through who I regretted hiring later. I always ask a basic programming question and ask some questions to gauge how much the person actually takes time to learn more about software development. That's usually enough data to figure out what's going on. 95% of people fail the programming question so I continue on the interview and get more information about why I'm going to reject them. Usually they fail every other portion of it as well.

tomh- 16y ago
What is the question I may ask?
- jswinghammer 16y ago
  Implement: itoa
  - tomh- 16y ago
    I don't know the answer of this question. I could do a guestimate on how such thing would work, but I've never coded c/c++ extensively. Am I useless now? I don't really see how this proves someone has basic knowledge about programming. However I could be wrong and this is a function that c/c++ programmers write on a daily base and is indeed a good indicator if someone ever ran a c++ compiler.
    - NickPollard 16y ago
      It's not in itself something a C programmer would ever really find themself doing, but as an algorithmic question it's fine - it's testing knowledge of looping, radices, strings etc. In C, you normally have to output the string in reverse and then reverse it, so you need to be aware of that. It's also a problem that's quite easy to explain (programmers are familiar with numbers and strings), but due to it's library availability, most people won't have actually written it themselves so it's not just a case of having done it before. Coincidentally, I was asked this exact question in a phone interview this week. I didn't have to give exact code, I just walked through the principal of it (loop for each digit, dividing by the radix each time using the leftover specify the character (can be a lookup in an array), possible to use bit shifting in base 2/8/16 situations).

known 16y ago

interview != quiz

cema 16y ago
True, but it often feels like a quiz is part of it.

aplusbi 16y ago

I posted this in the comments on the blog, but I'm reposting it here for discussion: I've done a lot of interviewing and have been interviewed myself a few times so I think I have a good amount of experience in the process. There are a lot of factors involved in technical interviews which I think a lot of people miss. Tech interviews are usually only an hour, so the questions have to be short - this often leads to more academic questions, and questions not directly related to the work done at the company. Interviews only tell you so much. You won't really know much about a candidate until they've been working with you for awhile. As such, many technical interviews are more about weeding out bad candidates than finding good ones. People embellish, cheat and lie. Just because a resume is impressive doesn't mean the candidate is impressive. Just because they worked on a project doesn't mean they contributed anything worthwhile to it. Most people are honest but will still bend the truth sometimes. It's a lot harder to do that in person. I like the idea of having a candidate work with the company for a week, but that's not realistic for a majority of cases. Not all companies are going to be able to do this, and neither are all candidates. If a candidate already has a job, asking them to take a week off is asking a lot. I also like the idea of a "homework" assignment. However it's possible for the candidate to cheat. It's also asking a lot of the candidate who may have a job and a family life. These two approaches heavily favor college students and recent grads. It's far from a perfect system. However it's reasonable (or so it seems) and is easy for companies to implement.

_delirium 16y ago

It would be interesting if a large company that could afford to do so (say, Google) made an attempt to experimentally test that question. It'd take some thinking to design a useful study with some sort of blindness, but something along the lines of: have everyone do the technical interviews, but then for some subset of interviewees, the final decision-maker chooses whether to hire or not based on a coin flip instead of based on the interview. Then assess the performance of each group of hires a year or two later. Disclaimer: may or may not be legal; I'm no employment-law expert, so have no idea if "not hired due to coin flip" would be a grounds for suing.

dansingerman 16y ago

It is far worse to hire a bad person, than to fail to hire a good person. Especially for startups. Corollary: disgruntled tech interviewees who are good, but get passed on.

sown 16y ago

I have a sizable and growing portfolio of code I've written: some complicated algorithm stuff, some open source contributions; things like that. No one ever looks at it. I'm starting to wonder why linked lists are so important to employers.

btipling 16y ago
We look at stuff like that at Cloudkick/Rackspace.
tobych 16y ago
The "Linked Lists" chapter from "Programming Interviews Exposed" begins: "The deceptively simple linked list is the basis for a surprising number of problems regarding the handling of dynamic data. Problems about efficient list traversal, list sorting, and the insertion or removal of data from either end of a list are good tests of basic data-structure concepts... Their simplicity appeals to interviewers, who want to present at least two or three problems over the course of an hour-long interview... You can write a relatively complete implementation of a linked list in less than 10 minutes, leaving you plenty of time to solve the problem... In addition, there is little variation in the way linked lists are implemented, which means that an interviewer can simple say "linked list" and not waste time discussing and clarifying implementation details." http://www.piexposed.com/ http://www.piexposed.com/
aplusbi 16y ago
I always look at that stuff and I ask about it during interviews, but I still ask general technical questions. Sometimes the candidate's contribution to a project is nothing more than writing the README file.
- deleted 16y ago
  [deleted]

msg 16y ago

I work for one of those big companies. After going over trouble spots in the resume and asking why they want to work here, I tend to ask one or two very similar questions that 1) force candidates to do nontrivial coding, algorithms, and design while writing a complete program, and 2) elicit a wide range of responses. I don't require a certain language or ding on syntax. We usually spend the rest of the hour on the question. In general, bad candidates get me hinky in the resume portion (or worse, don't really want to work here!) then crash and burn on the programming question. Borderline candidates do ok in resume but make me feel like they're not that comfortable programming. Great candidates usually do something clever during the problem. I believe that doing homework isn't convincing. It's too easy to cheat. That's probably why the commenter elsewhere on the thread got bounced from Google. Doing week-long trial runs would be ideal, but impossible at my company. We sometimes go contractor-to-hire though. Can you be a great programmer and fail my interview? I think it boils down to three things: skills, communication, and pressure. If you lose your skills and communication under pressure for an interview day, you probably won't survive in my environment over the long haul.

patio11 16y ago

I put tech interviews down fifty/fifty to "We don't know any better way of doing this" and "The unavoidable reality is that if a position is advertised openly then 98% of candidates are screamingly inappropriate for it and after applying a few filters to the resume we've gotten that to a more manageable 90% or so for interviews." Happily, there is a simple opt out mechanism. There are two ways to get into any company: through HR and the resume -> interview -> offer process, and the other way. (Convince a decisionmaker within the company that you, specifically, are the answer to their prayers. Get hired. Skip slush pile.) Pick door number two. If you get your jollies off of sorting linked lists you can do so after having received the moral equivalent of an offer (which should decrease the stress to the point where you can successfully sort a linked list).

alanthonyc 16y ago
Great answer, as always. However, it addresses the problem only from the point of view of the hiree. For companies looking to hire people, the typical interview process is equally a problem. I think the "dating" process described by the OP sounds like a good solution for the hiring company. It's also a good signal to the employee that the company is serious about hiring the best, using the best methods.
- aplusbi 16y ago
  The "dating" process only works when both parties are available. If the candidate currently has a job asking them to take time off (or worse, quit without a guaranteed offer) is asking a lot.
imajes 16y ago
I've picked door number two consistently throughout my career. It's more effective and much more meaningful to establish a relationship with decision makers from the get go.
btilly 16y ago
I consider the existence of door #2 a red flag. If decision makers who are likely not technical themselves are making that kind of decision about technical hires, odds are that they are going to make some very bad hires that I won't like living with. That's not to say that there aren't a lot of companies where your advice works. But I'd prefer to work for the companies where it doesn't.

tastybites 16y ago

We hire people based on enthusiasm, general intelligence (you can usually tell after a few minutes) and a low level of demonstrated proficiency - we'll ask them to explain a code sample they wrote, or ask for a verbal description of how to solve a problem. We choose the person we like best, and if they don't work out after 2 weeks we fire them. Hot seat quizzing produces way too many false negatives. We are not google. We can't do 5+ rounds in search of yeoman geniuses. We have to spend time making money. Some companies don't seem to have this inconvenient little problem, but we do. And besides, most of our hires have come from existing networks where none of this is even necessary. Having an extensive professional network should be a feature of anyone you consider bringing on as founder or employee or contractor.

shortlived 16y ago
Are there laws that allow/prevent the 2 week trial period for a business?
- tastybites 16y ago
  Not in California. It's called contract-to-hire usually.
- chollida1 16y ago
  I'm Canadian, but here most, almost all, companies have a 3 month window in which a new hire can be let go without severance.
- JoachimSchipper 16y ago
  This is allowed in the Netherlands, and I'd wager in most European countries.
- evilduck 16y ago
  Most US states are at-will employment, allowing anyone to be fired at any time, for no reason at all. http://en.wikipedia.org/wiki/At-will_employment http://en.wikipedia.org/wiki/At-will_employment Fun fact, many anti-discrimination laws in the US don't apply to companies with less than 15 employees.

tomh- 16y ago

I would suggest people to filter out like riotgames does. They ask job applicants to answer some questions and implement a small project in Java. See http://pastebin.com/zhCxcRds http://pastebin.com/zhCxcRds for the questions and http://riotgames.com/careers/software-engineer-java-pvpnet-platform-stlouis-mo http://riotgames.com/careers/software-engineer-java-pvpnet-p... for the job post.

sophacles 16y ago

I have found that for technical interviews, the following pattern works very well (hiring side): 0. Explain to the candidate this algorithm and why. It isn't magic, there are no right or wrong answers, just testing "fit". This is a case where foreknowledge does not change the outcome! 1. Get a basic grasp of technologies both interviewer and candidate have in common, via say, just chatting. This is vital -- find an area where you are both somewhat competent. 2. State a problem you don't know the answer to, something you are currently working out for your own projects, in terms of the mutual tech. VERY IMPORTANT: you must not have the best solution in mind when you bring up the topic. 3. Brainstorm on it together. This provides more feedback in a valuable way than the classic brainteaser. You find out how the person thinks and problem solves -- you know, that thing you are hiring her to do. You find out how you and her work together. Even if no solution is reached, it is OK because there was no solution going it -- just a problem to be worked. Your job in the interview is then to make sure you steer the discussion towards various tech and deep problems, coding issues, etc. Afterwards you will have a pretty strong idea of how it is to work with the candidate, which is quite useful for deciding if you want to continue to work with the candidate as an employee. Further if you've done well at the steering of the discussion, you have learned about candidate's technical depth, without setting up test/interview anxiety situations.

vitorbal 16y ago
This hits the nail on the head for me, and I couldn't agree more. If I was interviewed this way for a position, even if I didn't get the job I would have huge respect for the company afterwards.
aplusbi 16y ago
This sounds like a great idea but I feel it has a couple of short-comings. Most importantly is that it can be counter to the idea of preferring smart, motivated candidates over candidates that are well versed in certain technologies. Additionally, this technique would probably work against college students and recent grads. It can also be hard to quantify the results and the questions will vary wildly between candidates. Of course none of those obstacles are insurmountable and I think that when this approach works, it probably works better than a more general tech interview.

pamelafox 16y ago

Four years ago, when I was interviewing for API Support Engineer in Developer Relations at Google, I was asked to complete a Technical Worksheet, with challenges like write an AJAX mashup, write an XML schema, write a letter to a disgruntled developer, explain HTTP verbs to a 5-year-old. It was basically a mini version of what I would later be doing on my job, it was really fun to do, and it was great preparation for the interview. Unfortunately, as much as I loved that worksheet, I hear that we do not usually give it out anymore to new interviewees. I guess that it offends or intimidates people that are more experienced or interviewing at multiple companies. (I was a college student when I did it, and it did take me a few days of ignoring my homework to complete). So now, we have to base our decision mostly on the technical interview, which as discussed, has its flaws - particularly if the candidate is a nervous interviewee. It does well at eliminating the folks we don't want, but I think a few times, it's eliminated people we would have benefited from. I think we'd do well by offering the worksheet as an option to all new interviewees, but not a requirement.

ylem 16y ago

I think technical competence has to be part of it. If you know the person's references and those references are honest, you can ask them. But, I had one friend who had the fortune of always being surrounded by pretty bright, motivated, and competent people. For her first hire, she did a phone interview (international candidate), but was thinking more about issues like "does this guy seem like a good fit for the project in terms of interest" and only found out when he arrived that his technical expertise was far lower than what one would expect given his resume...

yaakov34 16y ago

I hate to be a stick in the mud, but you have to ask algorithm questions, and you have to ask them to code an implementation. If you hire a lot of people, that is simply the only way to avoid making (some) horrible hires who later cost you dearly. Portfolio of code is not a substitute - you don't know exactly how much the candidate contributed to it himself, and you can't trust your own ability to evaluate a lot of code written by someone else, not unless you spend an unreasonable amount of time on it. Working with a person for a week is not realistic for most places that hire a lot of candidates. A lot of the time I decided to skip all that boring, old-fashioned tech interview stuff and hire someone who looks great on the basis of something else -- anything else, I've regretted it. I do admit things are different if you're in a tiny startup and basically looking for a co-founder rather than an employee in a big organization. But even then I'd really like to do a brief algorithm and coding run to be sure.

dkarl 16y ago

Many technical interviews are very theory-heavy, muddling the divide between Computer Science and Software Engineering. I've been faulted for this in my interviewing, but I think that you have to pose at least one problem that can be explained precisely in less than a minute and solved completely in less than an hour. Such a problem will inevitably have an abstract, theoretical feel. It's also good to ask more realistic questions to test domain knowledge, but there are candidates who can string the right words together in the right order all day long but fall apart when faced with a real problem. It's very frustrating to try to identify such people using a loosely-defined, open-ended problem, because they can speak competently and professionally while avoiding the aspects they don't understand, and many experienced candidates will exhibit the same blathering behavior even if they are technically strong, because it's a standard mode of speaking that everyone in the industry has to learn. I.e., if I faulted people for spouting BS in response to realistic, open-ended questions, I'd rule out strong candidates as well as posers. Asking candidates to completely and precisely solve a simple problem separates abstract thinkers from people whose skills are purely verbal.

kscaldef 16y ago
In my experience, people start complaining that an interview is "theory-heavy" at about the point that you expect them to know when and how to use a binary tree. Binary trees are not hard-core CS theory, they are freshman Intro to Data Structures. They are a very basic tool (almost) every programmer should be familiar with. Moreover, when I ask candidates questions about data structures and algorithms, while it's nice if they can ace it all without hesitation, what I'm really looking for is what happens when their first answer is wrong. When they pick an incorrect data structure for the problem and I probe to say, "so, using that data structure, how do you this task?", I want to see if they will realize that the task I'm asking for is hard to implement or has very poor performance using the data structure they have chosen, and whether they will step back and realize they made a mistake. The point is less what do you know, but do you know what you don't know and do you recognize when what you're trying isn't working.
- strlen 16y ago
  > In my experience, people start complaining that an interview is "theory-heavy" at about the point that you expect them to know when and how to use a binary tree. Binary trees are not hard-core CS theory, they are freshman Intro to Data Structures. The other part about binary trees is by their very nature they involve pointers and recursion. Understanding pointers and recursion is the difference between being able to write code and being a programmer. The former is a fundamental skill any knowledge worker should have (scientists and financiers using Matlab or Python, statisticians using R); the latter takes passion, effort and talent. The other extreme I've seen is asking questions from computational geometry, combinatorics and complexity theory (for generalist software engineering positions). These are all relevant skills and for the former two a good cursory knowledge is important for a generalist: however, unlike pointers or recursion they're not as fundamental to programming.

orev 16y ago

Having been on the interviewing end of a number of interviews, I definitely have found that these types of "quiz" questions ARE EFFECTIVE, to a point. The number of people who claim to know something, but actually don't, are astounding. You'd like to think that people are honest, but they are not. The main issue is that of time. Interviewing people is an enormous time sync, and spending time to get to know them and their thought process is a complete waste if they don't know the material. Quiz questions can easily weed out the people who claim to know things but don't, and save you a lot of time. However, quiz questions should not be the only way to determine if someone is good for the job, and you always want to delve deeper with other kinds of questions as well. You can also use the quiz itself as a jumping off point for deeper discussions, which give you insight into thought process, etc...

ojbyrne 16y ago
"time sink"

agentultra 16y ago

I like the approach in this post. It sounds more reasonable. I dislike the term, "screening." It's not a good feeling for the interviewee to know that they're just being filtered before they get an interview. That the person interviewing them doesn't even believe that they can do their job properly. Screening reinforces the notion that the interviewee is just another rat in the race and that you, the interviewer, barely have the time to see them. It's an ugly, loaded term and I really don't like even hearing it. The traditional hiring process is a problem that goes both ways. It's nerve-wracking and emotionally draining for the person being interviewed and it's a time consuming risk for the person interviewing. The interviewee has to spend time crafting their CV, prepping themselves to answer a slew of random esoteric and useless trivia, and drag themselves to countless appointments. The interviewer has to sift through innumerable resumes and figure out which ones fit, sit through dozens of interviews, and then gamble that the person they've selected isn't going to waste a whole bunch of time and money. I think it's a search problem. At least for hiring programmers I think it's possible to filter incoming applications to your specific requirements. It should be possible to rank those submissions within your query. Then you should already have a good idea of how many interviews you will need to do and you can skip past the "screening" process. At least, I hope. I've been putting some back-40 hours into figuring out this problem. :)

michaelcampbell 16y ago

I haven't found them to be particularly useful except as a level 0 filter - you can get rid of some very shoddy people early, but finding the actual gems they do very little (in my experience, anway). I don't know who started this practice but it seems a lot of interviews recently are the brainteaser type; how many ping pong balls fit into a 1974 Super Beetle (hardtop), etc. Those are the worst.

fedd 16y ago

did someone mention http://codility.com/ http://codility.com/ ? i am not affiliated, just liked the site.

fleitz 16y ago

I prefer interviewing with koans. Setup a workstation with the tests and get the interviewee to make them pass. Have the interviewee check them into git / SCM on their own branch. It tests so much day to day knowledge and the ability to actually code / read code / and find / fix bugs. As well as all the various issues with your particular stack. Plus you can hide bugs all over the place, maybe the bug is in the database, etc. Maybe it's a real world task like 'speed up this page by 200%' type thing. There is a lot of flexibility and provides opportunity to for insight into how the developer actually solves real world problems. Maybe the increase the logic speed by 200%, maybe they just throw varnish on top of it.

stcredzero 16y ago

Sometimes it's the interviewer that ends up giving a bad impression. I remember one technical interview I was in: Interviewer: How would you do this? Me: Well, that depends. You could do it by doing polymorphism with subclasses, by passing in blocks, or through a Strategy. Interviewer: Well, actually, I'd do it like this. [proceeds to diagram polymorphic solution using subclasses as if that wasn't the first answer I gave] A good filter to use as a programmer interviewee: Ask to see their code. Ask to do some pair programming. If they blow you off or hem and haw about it, it's not a good place to work. If the interviewer reveals that they're actually in HR, just walk out.

jewgonewild 16y ago

When I interview candidates and get to the technical part, I usually start by discussing technologies and present a problem the team is currently working on and ask for a broad overview of how they would go about solving it. I start by asking them to talk me through their solution. This is where I filter out most candidates, because I know that if their problem solving skills aren't up to par I can't expect them to write any psuedo code to back up their problem solving skills. I generally give them another chance by offering a couple of hints, and if this doesn't work out I generally end it. If I like them, I ask them to talk to me about a challenging problem they have recently encountered and the solution to said problem. If I am pleased I ask them a few more questions and keep them in rotation. After this I ask for some simple psuedo code to back up the solution and that's about it. I am not big on asking any academic questions for the technical interview unless I know that the interviewee is a recent graduate and even then I don't weigh them that much. To finish off the interview I usually do some basic pair programming, just to see how the person does in a somewhat pressured situation. I find all of this to be pretty effective for the technical interview. Besides the technical side I strongly believe that a passionate person, that would make a good cultural fit is tremendously important. In my situation, I don't want someone who is smart but can't relate to the rest of the team.

makeramen 16y ago

direct link to the post, instead of the page of the blog: http://www.jeanhsu.com/?p=242 http://www.jeanhsu.com/?p=242