22 ms·

Technical interview performance is kind of arbitrary

260 points by leeny 11y ago

aprasad91 11y ago
This is why we at Lytmus believe the most effective way to assess a candidate is to see how they perform on a real work sample. We've built up a virtual machine based platform that allows candidates to showcase their skills in a real development environment with working code cases (web, mobile, data, systems, etc.). Most interviewing methods like algorithmic challenges often only provide signal into a discrete skill that can be acquired through practice, whereas what matters is whether or not you can actually work on real world projects, understand an existing code base and perform on the job as opposed to on an interview coding challenge. Google's SVP of People Ops Laszlo Bock also writes about the ineffectiveness of indirect tests and their weak correlation with on the job performance.
minimaxir 11y ago
The headline is "interview performance is kind of arbitrary," but the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary. I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?) The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place) That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.
- rewqfdsa 11y ago
  > the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary. This article's headline reveals the spin the HN community is trying to put on it. Many people in SF like to attack the very idea of meritocracy, because if you admit that meritocracy is a good idea, you're implicitly endorsing the idea that people have different levels of merit, and thus that between-group differences in representation might be due to something other than discrimination.
  - minimaxir 11y ago
    The real issue I have is that only a few of the other comments in this HN thread are looking at the data analysis presented in the article and are just taking the headline as gospel.
- leeny 11y ago
  Thanks, as always, for the thoughtful notes. We certainly don't mean to imply that this is gospel, and there are limitations to the work, and fwiw, we are thinking about potentially moving to a 5 star scale. That said, if click on the link in the footnotes to see the original data, you can get an idea of what we're working with. And lastly, the fontawesome thingy isn't plotly. It was built using http://blockbuilder.org/ http://blockbuilder.org/ by @enjalot
Xyik 11y ago
Not too surprising when you consider there isn't really a standardized guideline and every interviewer asks questions of varying difficulty. Sometimes interviewers don't even ask candidates the same question and tailor them based on the candidate's resume and experiences. I've had interviews as simple as writing a function that outputs if two strings are anagrams, and other interviews that test dynamic programming knowledge and other interviews that tested my knowledge of concurrency. At the end of the day, its luck of the draw which interviewer you get and what questions he decides to ask you.
deleted 11y ago
[deleted]
senekerim 11y ago
I have been to lots of interviews, on both sides of the table. I find most interviewers unprepared to evaluate the person for the role, and instead exercise their own biases, stroke their egos, etc. It's largely a voodoo practice that we'll look back and laugh at as a civilization at some point..
- colmvp 11y ago
  I wonder how many employ Kahneman's recommendation based on his book, "Thinking, Fast and Slow": > Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on. Don't overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call "very weak" or "very strong." > These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate add up the six scores ... Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better — try to resist your wish to invent broken legs to change the ranking. A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."
  - Someone1234 11y ago
    I like that. But just to be a little bit contrarian for the sake of conversation: Has it been shown that ignoring your gut instinct or even who you like most if inherently wrong? That kind of testing definitely removes a lot of bias, but it just assumes that biases are inherently bad. There certainly ARE bad biases (sexism, racism, etc). But you may also need to work with the individual you're recruiting, so how well you think you can get along with them or work with them is likely important too. Plus if this method was common, do you think candidates would prep specifically for those scores? Seems like something very easily gamed. Not to mention that scoring itself can be biased. As I said at the start, I actually like that, and think they're on the right track. But I'd likely want to combine it with something LESS subjective to get a fuller picture of a candidate.
staunch 11y ago
Most interviewers don't ask enough technical questions to have any idea what a candidate knows or doesn't know. If their one or two questions happen to be something the candidate knows well, they'll call them a genius. If they happen to not know, they'll label them an idiot. You can learn a lot more from 20+ rapid fire questions than forcing a candidate to eek out an answer to something they're not familiar with. And once you establish the areas they're familiar with, you can ask them truly useful questions. The key is to look for people who have strengths and not worry at all about gaps in their knowledge. Anyone who has earned genuine expertise in one area will be able to do so in other areas. The other big mistake most interviews make is forgetting about the "Curse of knowledge" https://en.wikipedia.org/wiki/Curse_of_knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge I've seen people research the answer to a question before an interview and then expect candidates to be equally informed without that advantage.
- TallGuyShort 11y ago
  Along these lines, I start out with very broad questions. Something like, "tell me how you'd troubleshoot a web service that's suddenly not accepting connections / suddenly performing badly." Different candidates will focus on different aspects of that problem depending on their background: low-level networking, cloud environments, application-level problems, databases etc. Based on their resume, I like to see if their expertise matches up with their experience. I like to see that they don't consider ONLY things within their expertise. I like to see if they can make reasonable guesses outside of their expertise without BS'ing me. I like to see if they have a good approach to exploring areas they're unfamiliar with and general problem-solving. I like to see if they recognize how valuable it is to have diagnostic / monitoring / change control tools in place and to have done pro-active testing. This can be tough to do with coding problems, but I still try give problems that should be familiar to experienced low-level C developers as well as high-level Python web devs and tailor my expectations to their experience. I'm more concerned with how well they've learned based on what they've done than if they've already learned what I'd like them to do.
  - 8873872782 11y ago
    This is a very good way for senior engineers to interview candidates. Newer engineers interviewing candidates will typically have difficulty doing this well.
LanguageGamer 11y ago
The problem is, whether or not interview performance is consistent, we still don't know how or when it's correlated with performance if hired, and that's the sort of thing you would need to actually help people making better hiring decisions. Does interviewing.io have any plans to collect employee performance metrics from companies that hire via their platform? Is that something companies would be willing to cooperate with?
- RogerL 11y ago
  We know big companies go through tons of people. Isn't stack ranking supposed to eliminate 10-20% a year. All this noise about avoiding bad hires while ignoring the elephant in the room - companies claiming that bounce as many or more people as the companies that don't use these foolish interviewing practices.
robodale 11y ago
Good, because I suck at technical interviews, but built my own SaaS offering (working the corporate job until my customer base is large enough).
richardwhiuk 11y ago
Isn't this exactly what you'd expect? e.g. if get a random cohort of developers who all of their peers would say are amazing, if I subject them to a series of tests, they will do differently well at the different tests? For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half? Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?
- dilemma 11y ago
  People do believe such things, yes, despite how absurd it is as you've shown.
dudul 11y ago
In the HN echo chamber, there isn't a day without some blog post/article describing how our interview process is BS, interview is broken, etc. I don't necessarily dispute this state of affaire, but does anyone know how it compares to other fiels/professions? How about interviewing a lawyer? Or a doctor? Or an account manager? Or a product marketer? Are developers the only one with a "broken" interview process?
- grillvogel 11y ago
  most other job interview processes don't involve solving a series of riddles
  - rewqfdsa 11y ago
    In most other jobs, candidates don't vary in skill by a factor of 10x or more
    - grillvogel 11y ago
      where did you get that idea?
    - thaumasiotes 11y ago
      Lawyers were mentioned directly in your grandparent comment. Sales and athletics also easily show at least that much variation from person to person. Business owners show much, much more variation than that, even if you restrict the field to very similar businesses. Researchers show variation similar to business owners (i.e. mind-bogglingly high). You can easily construct differences of larger magnitude for what you might think of as "simple" jobs like telephone receptionist; a fluent english speaker is going to be more than 10x as productive in that job, assuming they're supposed to do business in english, than someone who can barely get by, and probably billions of times more productive than someone who can't speak english at all. There are a lot of Chinese people out there whose highest ambition is to "work in an office". I've met some of them! Most will never achieve that goal, because they don't have the requisite skills. Most people who don't speak english won't apply for a job answering a telephone in english (but a lot of them will! This is exactly the kind of customer service representative everyone has grown to hate). I'd like to see some software job openings that specified a skill set, and were willing to hire people with that skill set. That doesn't seem to be the direction people are headed in, though...
grillvogel 11y ago
technical interviews are a joke. the majority of the time they exist so the interviewer can try to feel smart and subject the interviewee to whatever whimsical problem they found on the internet. how often do you do group coding in a whiteboard in your actual job? at one interview I was criticized for sitting and thinking about a problem for a minute without just blindly jumping into attempting to solve it. also tons of people are great at solving toy interview problems but can't debug their way out of a paper bag.
- marknutter 11y ago
  I once was called into an emergency meeting by the CEO of the company I was working for the time. When I entered the room, all of the top brass were seated around the table, some visibly agitated. The CEO proceeded to hand me a single black whiteboard marker as he stated "the fate of the company depends on you". The problem was outlined by a fellow engineer and it was explained that I had 10 minutes to solve it, or we might risk going out of business. With that, I took a deep breath and went to work furiously scribbling away on the board while everyone watched with anticipation. As I closed the final bracket, and stepped back to examine my work, audible gasps could be heard from around the room. The grizzled old CTO whom everyone was a little scared of broke the silence at last: "God dammit, Mark!" I caught a few nervous glances from some of the less technical folks. You could hear a pin drop when he continued: "you just saved the god damn company!" A slow clap started, and soon everyone joined in with a round of applause and cheering. Over the outburst of joy and adoration the CEO shouted to one of the developers who had gathered outside of the conference room, "get this code into production!" Soon the whiteboard was being whisked away by a group of smiling engineers as they navigated with great purpose through a sea of high fives and back slaps. And then, with a twinkle in his eye, the CEO shook my hand, and pulled a crisp $100 bill out of his wallet. As he handed it to me, he said "I knew from the moment I heard about your interview that you would be a great asset to this company, Mark. Go home and relax for the rest of the day and take your wife out for dinner tonight. You've earned it." So you see, being able to solve complicated problems on a whiteboard in a crunch time is a valuable skill.
  - turkishrevenge 11y ago
dms105 11y ago
The real weaknesses of technical interviews are: 1) They usually just measure the amount of effort a person has put into studying interview questions. Whether or not the ability to do this translates to being a better engineer is debatable. 2) An interviewer almost always exercise some form of personal bias, whether it be educational, personal, etc. This doesn't always show up in written feedback, but the interviewers with stronger personalities usually dominate interview debriefs, and often influence others into hire/no-hire decisions. This is especially prevalent in smaller startups where the process is more informal, things move quickly, and decisions are based more on gut feelings.
- iolothebard 11y ago
  I had a 800+ page study guide while living in SF for the ridiculous technical interviews they'd put you through. Then you'd see their code base and realize they've never actually done anything that would remotely resemble best practices. The best part though was realizing if you didn't answer that one question exactly how the "brilliant" person interviewing you wanted it answered, you were done. So at that point I'd quiz them on all their shortcomings that were evident based on what they'd told me. In my experience hiring it's not that difficult. Good attitude, good aptitude, genuine interest in the field (software engineering), ideally interest in the applied product (if not a software product), decent communicator and good hygiene. If they've done well in those areas, I've never had to let someone go.
  - stale2002 11y ago
    Ever thought about releasing this study guide that you've made? I would love to take a look at it/ send it to some friends!
    - iolothebard 11y ago
      Lol, no. It's mostly just an amalgamation of all the technologies and stupid fucking interview questions that you get. I have 10-12 different documents. CS/OO (basics for reminders with some simple algorithms), .Net, SQL (I always forget this shit, it's why I have data layer and use ORMs + LINQ), LINQ, jQuery, Vanilla JS, HTML5+CSS, Architecture/Patterns, ASP.Net vs MVC, Tuning, etc. Haven't updated them in a few years because I left the bubble. My interviews in the midwest are typically an hour or two at most with very little quizzing. At worst they ask you to make them a small simple app (which I find irritating but better than SF interviews).
brightball 11y ago
With programmers, the single easiest way to identify good candidates (in my experience) is sheer interest in what they do / desire to learn. This is a learn everyday field and if you're interested in what you're doing, you're going to do a lot better at it. It's hard to apply yourself mentally to something that you don't have a good level of interest in. Given that it's a learn everyday field, people with that level of interest will realistically be able to learn anything they need to do solve the problem you're hiring them to solve. The only real differentiating factor is your tolerance for ramp up time. I expect a programmer to be able to pick up a new language or database within a couple of weeks (tops) in most cases. If I'm hiring full time, that's something I'll tolerate. If I'm hiring a contractor, I'm going to be uneasy about paying high hourly rates for him to learn the job. The single most effective way that I've found to interview for "interest" is to just get them talking about something they've done before and ask them to go deep into the details. You get everything you need from watching somebody talk, with a smile on their face, about how they solved some problem in a creative way that makes them show some pride. Doesn't really matter what the problem was, if it was a business problem, code problem or hardware problem. The important thing is the level of attention to detail in addressing it. I've been using this technique for about 8 years now and while I don't make it the exclusive criteria for hiring, every person I've ever hired who has passed that part has ended up in my "great hire" category.
- jessedhillon 11y ago
  Interest is not enough, unfortunately. There are plenty of engineers who are attracted to the challenge and excitement of building new things, but have no appreciation for The Right Way to build things. Great engineers think beyond "how" and ask the "should" questions as well. Mediocre engineers glue things together in a haphazard way with little thought about what's the best way to write things. Caring about maintainability, comprehensibility and extensibility is partially a function of experience, but I've met plenty of experienced engineers who still write bad code and design systems poorly. It has no correlation to how much of a tinkerer and curious person they are, in my experience.
  - brightball 11y ago
kbd 11y ago
I had a really bizarre interview recently where, after the initial recruiter phone screen, I was rejected based on an in-person half-hour very simplistic paired coding exercise, only met with one person, and wasn't asked about my (imo very strong) resume once. I must have said something foolish at some point, which is on me, but the point is: interviews can be hit or miss. Fortunately you only need one hit.
- kybernetikos 11y ago
  It does sound like a bad interview experience, but I will say that I have very little interest in what a candidate writes on their resume. There are candidates with amazing resumes that can't do much and people without much that they can talk about publicly that are amazing. If the interview is low signal, the resume is even worse.
  - kbd 11y ago
    > There are candidates with amazing resumes that can't do much... That's certainly true. I understand not taking resumes at face value; I've interviewed developers (and DBAs) and I'd always pour over their resumes to look for both padding and meat to ask about. It can be insightful to ask about a big accomplishment someone claims as their own only to find out that they only actually affected a small part of it. However, In this case I'm honestly not sure whether my interviewer even read my resume.
- fapjacks 11y ago
  Interesting. I wonder if we interviewed at the same place! I had a very similar experience over a year ago. At least I was told they only select one person from the group to move forward from the pair-programming task, which I thought was weird, but hey, it's their company.
maxaf 11y ago
Take home tests FTW. The thinking goes as follows: 1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them. 2. If the candidate does reasonably well by completing the code test somewhat on time (with a fat margin allowed for them, well, having a life) and within parameters of the task, they're invited for a mostly non-technical onsite meet-and-greet. 3. During the meet-and-greet we make sure that the candidate isn't an axe murderer, is able to hold a quasi-technical conversation, and that both sides aren't immediately scared of each other. 4. The meet-and-greet can also include some low-key architecture discussion. Any nerds worth their salt will be able to conduct this line of questioning without making it obvious that an interview is taking place. Hopefully this isn't a critical step, as a good take-home code test will require the candidate to spend a little time designing or architecting their solution. After the above has taken place, it should be pretty clear whether the candidate in question is a fit or not. Note that this process is by design missing the useless traditional CS questioning component, contrived problem solving exercises, or a whiteboard code beatdown.
- kafkaesq 11y ago
  4 hours is a lot of time out of someone's day, and a near-dealbreaker for anyone gainfully employed and having some semblance of a life. Do you not realize this? Meanwhile, it really doesn't take long to find out if someone is "worth talking to." Ask them to provide a code sample, and talk to them about for 10 minutes or so. Or if you must use a coding exercise, really, just keep it short and sweet. A good coder can show you a lot about their sensibilities in their solution to even just a slightly non-standard sorting problem.
- tiglionabbit 11y ago
  You need to verify that the candidate actually did the take home test, and didn't hand it to their more talented friend.
  - Peroni 11y ago
    That's why you get them to talk you through their process during the face-to-face interview. It'll be immediately apparent if they didn't do the work themselves.
fecak 11y ago
Great post as always Aline. I'd be most curious as to how well anonymity was kept. Did interviewees identify their employers, schools, or any other information that might create bias while in the interview itself? I've been recruiting for a long time, and I'm rarely shocked about the result of an interview - maybe a few times a year. There are tons of possible explanations for that, and lots of possible explanations for your results as well. Keep up the great work.
- leeny 11y ago
  Most of our interviews didn't have people unmasking until after the feedback step.
pklausler 11y ago
I have learned that an (in)ability to program "in the small" correlates very well with an (in)ability to program in the large, and now ask mostly simply questions whose answers are things like one-line Boolean predicates to test for well-defined conditions. It is paradoxically easier for an inept candidate to fake his way through an algorithm design question than it is to fake the coding of a simple test for "determine whether two closed intervals [a,b] and [x,y] overlap each other".
- Ocerge 11y ago
  I've moved to doing something extremely simple: just give me pseudo-code for indexOf given a string and a character. If you can't write the 4-5 liner for linearly searching a string for a character, then there's not much point in moving further. It is shocking how many people get tripped up by it.
  - pklausler 11y ago
    I'm similarly shocked, but it's just how things are now. I'm not sure how I could set the bars any lower.
  - jcadam 11y ago
    Is it a sign of insecurity that I just popped open vi to make sure I could solve this? :) Where I work, we've been trying to fill a vacancy for a senior dev for months. Our one whiteboard question is similarly easy: find the largest integer in an array (The interviews have become easier since I was hired on). And yet... I've been watching an endless parade of people with decades of experience who apparently don't know how to write a for loop :(
    - pklausler 11y ago
      And yet they can somehow get through a telephone screening interview.
- hackinthebochs 11y ago
  Exactly this. I've been arguing for years that those simple string/array/link list/tree manipulation questions are simply a microcosm of the qualities that make for good developers in general. If those trip you up then you need to consider that there are serious holes in your abilities, not that the test is bad.
rjzzleep 11y ago
Constantly confused by this, so much arguing back and forth, and yet the single easiest way to deal with this is a stripped down real world problem and then giving it to a whole bunch of different candidates. Some people adopt this some people don't some people argue that it's meaningless, and go back to the standard silly interview patterns of algorithm questions and meaningless complex fizzbuzz alternatives. tptacek summarized this in his hiring post: http://sockpuppet.org/blog/2015/03/06/the-hiring-post/ http://sockpuppet.org/blog/2015/03/06/the-hiring-post/ My personal conclusion is that most companies don't want this for two reasons: 1. culture fit is more important for people in a rigid hierarchical structure, partly because an out of the box thinker could be dangerous for that structure. too much questioning authority, too much pointing out flaws. It's much easier to have a good worker bee than wondering why you need 40 employees to build an automated gif platform. 2. in most companies everyone is very reluctant to make decisions. for example management struggles with clear direction because it opens them up to the question of liability. if they make a decision and it's wrong they might get fired. HR works the same way, if HR passes a resume along they want it to hit a list of keywords, so they can cover their asses if he turns out to be a bad hire. Basically everyone is so scared to make a mistake that they make a lot more mistakes trying to avoid them. The opening of the cracking the coding interview she talks about how they don't really care about false positives and negatives, they just want those to stay below a certain threshold. But consider the hiring scale of google compared to a small company and suddenly those things matter. One bad hire can be toxic. And basing your hiring strategy on something a huge behemoth with infinite money does is kind of silly imho
kearneyandy 11y ago
Great post and awesome interactive graph! props I'm curious about the interviewer community. Specifically things like how are they vetted and how often they come to conduct interviews. It would be cool if there was a community of interviewers for the betterment of the process, but I could see their retention for conducting interviews to only be 1 or 2 before they drop out. I see in the appendix that there are those that do more, but no indication about what percent leave quickly. A better drinking game might be when a candidate offers a data structure they know nothing about. Would a red-black tree work here? No.. I guess not.
alive2007 11y ago
Minor question; is it just me or is the "Results of Interview Simulations by Mean Score" a bit difficult to parse? I understand that observing the behavior of any singular cohort involves looking at the endpoints of the cohort's curve at the horizontal line 'x=n', where n is number of simulations you wish to observe (the right point of the curve at x=n is P(fail) of the worst performer in the cohort at n simulations, the left point of the curve at x=n is the P(fail) of the best performer); which is why the gap between endpoints within a singular cohort decreases as n increases. But it seems kind of counterintuitive to observe any other kind of trend -- shouldn't the information be graphed as P(fail) being a function of # of simulations, as opposed to the other way around, seeing as the latter is the independent variable?
mentatseb 11y ago
The conclusion is misleading due to 2 wrong assumptions: 1. The population is heterogeneous: interviews test different skills. All interviews don't test the same set of skills, which is mandatory to compare interview scores because scores are aggregates of these skill tests. Different job opportunities means different skills to test, so it seems reasonable to assume that people evaluation vary for different job opportunities, and thus their scores vary for different interviews. 2. The observations are not statistically independent: past interviews may influence future interviews. People may get better at passing interviews or conducting interviews over time. This would impact their score. It would be good to study the evolution of individual scores over time. While (1) should strongly limit the conclusions of the study, the complete analysis may simply be irrelevant because of (2) if the statistical independence of observations is not demonstrated. Sorry guys but this is Statistics 101 introductory course.
- leeny 11y ago
  (1) We listened to most interviews on the platform to establish homogeneity. Interviews were across the board, language agnostic, and primarily algorithmic in nature. (2) We actually looked into this and noticed that time didn't really affect performance. Usually, people did their interviews over a pretty short time span and then found a job. Or, people were already experienced interviewers and had kind of hit a plateau. You can see the raw data and how it oscillates wrt time in the footnotes.
JasonCEC 11y ago
As a potential candidate, all of the standard complaints ring true - but once you're on the other side of the equation, and need to hire people... your ability to create new interviews is not nearly as wide or as clear as it would seem from the outside. 1) Take home test: OK for performance metrics, bad for "getting to know" the candidate, and terrible for selling the candidate on your company 2) Daylong interview: Expensive, requires interrupting our team, needs a fully planned and well executed itinerary - but is perfect for getting to know someone, getting the feel for their personality and interests, and is the best way to sell someone on the opportunity. 3) Work sample: we usually do this for interns[1] and pair it with a ~1 hour conversation (either before or after, doesn't really matter to us) on what the company is like and what they would be working on. Obviously, work samples suffer from the same deficiencies as a take home test for cultural fit and the like, but it's the best we can do for interns! [1] https://gastrograph.com/blogs/gastronexus/interviewing-data-science-interns.html https://gastrograph.com/blogs/gastronexus/interviewing-data-...
- rm_-rf_slash 11y ago
  I've said elsewhere on this thread, but my absolute favorite technical interview method is to set up a contract to pay the candidate to create a feature or fix a bug on your codebase, and have it merge and work as expected. It lets the candidate feel (literally) valued by your company, and you get to learn how they do the job you are hiring them to do.
  - hire_charts 11y ago
    Agreed that this is a great approach, but the major downside is that you are limiting yourself to candidates who either: a) have the time to do real work outside of their day job and aren't on a restrictive contract preventing them from moonlighting, or... b) are currently unemployed
    - rm_-rf_slash 11y ago
      I was a college student at the time. I was long on money and short on cash. I was also far less experienced than I was so there were a lot of new things for me then, like those fascinating "callback" thingies I had to figure out. Nowadays if I were given roughly the same assignment it would have taken me about 3-5 hours at best, which, given my graduated-and-employed hourly rate, is actually quite in line with what I was paid. So it paid as much as I would have charged for a Saturday hack.
marknutter 11y ago
Technical interviews are a form of hazing. Engineers often suffer from imposter syndrome, especially during an interview. Those who have already been hazed and accepted to the club will turn around and put potential candidates through the same humiliating process. And what's worse is that demonstrating you have superior capabilities in one area or another can be seen as a threat to the interviewer and they may give you a thumbs down based purely on their own insecurities. What ends up happening is that, just like in college fraternities, is that everyone ends up being similar, both culturally and in terms of abilities. If I had it my way I would do away with the interview process altogether and do something more akin to an internship. Potential employees could start their engagement with a company by working (for pay, mind you) on a very limited basis to solve actual problems that need solving (i.e. "write an algorithm that's 10% more efficient", "create a tooltip that's aware of the viewport in React", etc). Based on their output their engagement could be ramped up until they are brought on as a full time employee. That way it ends up being completely merit based. You can either solve these problems or you can't. And whether or not they ultimately end up becoming an employee doesn't end up mattering because both parties are compensated along the way. This would obviously put the burden on the company to boil its problems down into smaller, isolated efforts but that's something all companies should be trying to do anyways. In the end, they just want some code written that will end up solving some problem for their customer.
- kelvin0 11y ago
  Hazing? Really? Hazing is the practice of rituals and other activities involving harassment, abuse or humiliation used as a way of initiating a person into a group... https://en.wikipedia.org/wiki/Hazing https://en.wikipedia.org/wiki/Hazing
  - marknutter 11y ago
    Yeah, that sounds about right.
    - kelvin0 11y ago
      Maybe you should tell us which companies haze the potential candidates so we can steer clear of them? At least some hints?
deleted 11y ago
[deleted]
agentgt 11y ago
I haven't interviewed in some time but one thing I absolutely hate about technical interviews is "white board" coding. For some reason white boards intimidate me. I have terrible penmanship and complete lack of planning how much space I need for writing things. Then there is the fact the markers seem to have a hi failure rate when ever I use them. Perhaps I'm the only one that feels that way. I have even begged some interviewers that I would prefer them just watching me use a laptop but the offer is typically refused. Maybe things have changed now?
Gratsby 11y ago
Technical interviews are silly exercises IMO. You have a multi-month ramp up period and you have all kinds of environment specific things to deal with at just about every employer. Nobody in real life gets asked to program on stage or forced to answer esoteric questions as if your in some sort of math competition. If you don't have confidence in someone's ability based on their experience and their interview but you did like them, give them a task to accomplish offline. See if their results are anything like your results would be, and bring them in to see how they respond to feedback both negative and positive. I've seen too many interviews that go along the lines of "How would you rate your Java on a scale of 1-5?" "5" "So how would you fix the problem if your cache hit rate on SomeObscureCommercialProduct went from 94% to 82%?" "Forget that guy. Huge ego. Doesn't know anything." I did run into one company that had an interesting process for technical validation. They actually hire people for two weeks as contractors and have them work with the team. Then they hold a vote and decide whether to extend an offer.
mirceal 11y ago
3 things I search for: Fundamentals: CS basics. I don't nitpick on details. It's more around if you've heard about it or not and if you could figure out how and when to use it. Structure: I want to see a structured approach to problem solving. Doesn't matter if your code is perfect. Doesn't matter the programming language you want to use. Curiosity: You need to be curios about things. Asking the "why".
innertracks 11y ago
Just finished 2nd phone interview (it was somewhat technical) yesterday with a company and an online timed tech test this afternoon. First interview was with an internal recruiter and was non-technical. I've been impressed. They've been very straight forward regarding tech eval with no trick questions and respectful of time. Their interview process is selling me on the company.
uiri 11y ago
The data seems to show the opposite to me - despite scores being all over the place, the mean is very reliable. When a 2 or lower is considered a fail, those who consistently rate ~2.5 fail about half of their interviews while those who consistently rate ~3.0 fail only 10%. Of course, the probability that a candidate failed an interview approaches 1 as they are subject to more and more interviews. That the test has both false negatives and false positives does not invalidate the test. In fact, that the test is accurate despite the false positives and the false negatives ought to do the opposite. If a single bad interview invalidates a candidate for company A, that doesn't mean that the candidate won't go on to pass all of their interviews with company B.
jorgecurio 11y ago
Technical interviews are part of why I've moved away from engineering positions. I'm looking at product management jobs thinking that I could use what I learned past 6 years working on a SaaS alone. However, I found the exact same shit, even more technical interview questions that require whiteboard code writing. There might be some merit to why they are doing this but it's impossible for me to engage companies that discount real world product experience in favor of rote memorization. So far it's a pretty tough nut to crack, lot of product manager interviewers don't seem to know what they are doing, instead relying on law of large numbers and how great their fucking product is blah blah blah (it isn't). It's a bit worrying since some companies seem to be hiring product managers for some subjective end goal of an improved product and improved sales....they want one person to take the credit from, and the same person to take all the blame...another huge red flag when managers outright tell you they have no idea what to do so they just get someone else to outsource their thinking.
dilemma 11y ago
The problem with hiring is that hiring decisions are centralized, causing huge workloads for the decision maker. To reduce the load, arbitrary processes and voodoo tests are used, always with the same poor results. Instead, the team hiring should themselves interview candidates and make decisions on who to hire, because it requires personal knowledge that you can't get from tests.
lostcolony 11y ago
It may not be arbitrary, but if so it's buried deep in the data. I've turned down candidates who had impressive technical resumes, who had worked in startups that sold, who had been hired on as consultants at various places, etc, because they were unable to solve simple algorithms in a simple manner, and their code was atrocious. Does this mean they're "bad developers"? No. If we were a consulting firm or a startup they might well be worth it; where the important thing is getting code out the door quickly, and to have something that works, even if it's not easily maintainable. But I was hiring for a position that required someone who would keep solutions simple and maintainable ('craftsmanship' rather than productivity, if you will. Note that the former does not necessarily preclude the latter, but it's the trait that was necessary, and was lacking). Google optimizes for people with strong algorithmic knowledge. It's debatable whether they need everyone to have that, but certainly, many shops don't. Again, I've hired people with no formal CS background, because most of my job's problems don't require you to have deep algorithmic knowledge (the ones that do we can have others address, or work together on). We know that people can fail one technical interview, while being radiant in another, and the reality is that what we're looking for, and what others are looking for, are often different. That creates a lot of variance in the data when we compare them.
siliconc0w 11y ago
I dunno - if you look at the data, there are fairly clear clusters of people who are 'probably good' and 'probably not so good'. 'Programming' is necessary but not sufficient for product engineering and that is what most of these interviews are trying to tease out. Good companies will balance out 'programming' with other rounds like 'technical design' or 'pair programming' or even non-technical rounds with business analysts or product to gauge general ability.
macscam 11y ago
super true, and probably easy to bypass if you're a data viz wiz. It's more burdonsome for web devs who don't have the math / algorithms chops, but I know I have to learn it.

260 points by leeny 11y ago

This is why we at Lytmus believe the most effective way to assess a candidate is to see how they perform on a real work sample. We've built up a virtual machine based platform that allows candidates to showcase their skills in a real development environment with working code cases (web, mobile, data, systems, etc.). Most interviewing methods like algorithmic challenges often only provide signal into a discrete skill that can be acquired through practice, whereas what matters is whether or not you can actually work on real world projects, understand an existing code base and perform on the job as opposed to on an interview coding challenge. Google's SVP of People Ops Laszlo Bock also writes about the ineffectiveness of indirect tests and their weak correlation with on the job performance.

minimaxir 11y ago

The headline is "interview performance is kind of arbitrary," but the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary. I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?) The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place) That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.

rewqfdsa 11y ago
> the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary. This article's headline reveals the spin the HN community is trying to put on it. Many people in SF like to attack the very idea of meritocracy, because if you admit that meritocracy is a good idea, you're implicitly endorsing the idea that people have different levels of merit, and thus that between-group differences in representation might be due to something other than discrimination.
- minimaxir 11y ago
  The real issue I have is that only a few of the other comments in this HN thread are looking at the data analysis presented in the article and are just taking the headline as gospel.
leeny 11y ago
Thanks, as always, for the thoughtful notes. We certainly don't mean to imply that this is gospel, and there are limitations to the work, and fwiw, we are thinking about potentially moving to a 5 star scale. That said, if click on the link in the footnotes to see the original data, you can get an idea of what we're working with. And lastly, the fontawesome thingy isn't plotly. It was built using http://blockbuilder.org/ http://blockbuilder.org/ by @enjalot

Xyik 11y ago

Not too surprising when you consider there isn't really a standardized guideline and every interviewer asks questions of varying difficulty. Sometimes interviewers don't even ask candidates the same question and tailor them based on the candidate's resume and experiences. I've had interviews as simple as writing a function that outputs if two strings are anagrams, and other interviews that test dynamic programming knowledge and other interviews that tested my knowledge of concurrency. At the end of the day, its luck of the draw which interviewer you get and what questions he decides to ask you.

deleted 11y ago

[deleted]

senekerim 11y ago

I have been to lots of interviews, on both sides of the table. I find most interviewers unprepared to evaluate the person for the role, and instead exercise their own biases, stroke their egos, etc. It's largely a voodoo practice that we'll look back and laugh at as a civilization at some point..

colmvp 11y ago
I wonder how many employ Kahneman's recommendation based on his book, "Thinking, Fast and Slow": > Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on. Don't overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call "very weak" or "very strong." > These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate add up the six scores ... Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better — try to resist your wish to invent broken legs to change the ranking. A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."
- Someone1234 11y ago
  I like that. But just to be a little bit contrarian for the sake of conversation: Has it been shown that ignoring your gut instinct or even who you like most if inherently wrong? That kind of testing definitely removes a lot of bias, but it just assumes that biases are inherently bad. There certainly ARE bad biases (sexism, racism, etc). But you may also need to work with the individual you're recruiting, so how well you think you can get along with them or work with them is likely important too. Plus if this method was common, do you think candidates would prep specifically for those scores? Seems like something very easily gamed. Not to mention that scoring itself can be biased. As I said at the start, I actually like that, and think they're on the right track. But I'd likely want to combine it with something LESS subjective to get a fuller picture of a candidate.

staunch 11y ago

Most interviewers don't ask enough technical questions to have any idea what a candidate knows or doesn't know. If their one or two questions happen to be something the candidate knows well, they'll call them a genius. If they happen to not know, they'll label them an idiot. You can learn a lot more from 20+ rapid fire questions than forcing a candidate to eek out an answer to something they're not familiar with. And once you establish the areas they're familiar with, you can ask them truly useful questions. The key is to look for people who have strengths and not worry at all about gaps in their knowledge. Anyone who has earned genuine expertise in one area will be able to do so in other areas. The other big mistake most interviews make is forgetting about the "Curse of knowledge" https://en.wikipedia.org/wiki/Curse_of_knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge I've seen people research the answer to a question before an interview and then expect candidates to be equally informed without that advantage.

TallGuyShort 11y ago
Along these lines, I start out with very broad questions. Something like, "tell me how you'd troubleshoot a web service that's suddenly not accepting connections / suddenly performing badly." Different candidates will focus on different aspects of that problem depending on their background: low-level networking, cloud environments, application-level problems, databases etc. Based on their resume, I like to see if their expertise matches up with their experience. I like to see that they don't consider ONLY things within their expertise. I like to see if they can make reasonable guesses outside of their expertise without BS'ing me. I like to see if they have a good approach to exploring areas they're unfamiliar with and general problem-solving. I like to see if they recognize how valuable it is to have diagnostic / monitoring / change control tools in place and to have done pro-active testing. This can be tough to do with coding problems, but I still try give problems that should be familiar to experienced low-level C developers as well as high-level Python web devs and tailor my expectations to their experience. I'm more concerned with how well they've learned based on what they've done than if they've already learned what I'd like them to do.
- 8873872782 11y ago
  This is a very good way for senior engineers to interview candidates. Newer engineers interviewing candidates will typically have difficulty doing this well.

LanguageGamer 11y ago

The problem is, whether or not interview performance is consistent, we still don't know how or when it's correlated with performance if hired, and that's the sort of thing you would need to actually help people making better hiring decisions. Does interviewing.io have any plans to collect employee performance metrics from companies that hire via their platform? Is that something companies would be willing to cooperate with?

RogerL 11y ago
We know big companies go through tons of people. Isn't stack ranking supposed to eliminate 10-20% a year. All this noise about avoiding bad hires while ignoring the elephant in the room - companies claiming that bounce as many or more people as the companies that don't use these foolish interviewing practices.

robodale 11y ago

Good, because I suck at technical interviews, but built my own SaaS offering (working the corporate job until my customer base is large enough).

richardwhiuk 11y ago

Isn't this exactly what you'd expect? e.g. if get a random cohort of developers who all of their peers would say are amazing, if I subject them to a series of tests, they will do differently well at the different tests? For example, so I set a test where you have to write some Java, if half the candidates haven't written any Java, they'd all surely do worse on the test than the other half? Or is their a belief in the industry that there's some scale on which we can absolutely rank all developers - front end, back end, full stack, mobile, desktop, embedded? That sounds like a surprising belief which would require extra-ordinary evidence?

dilemma 11y ago
People do believe such things, yes, despite how absurd it is as you've shown.

dudul 11y ago

In the HN echo chamber, there isn't a day without some blog post/article describing how our interview process is BS, interview is broken, etc. I don't necessarily dispute this state of affaire, but does anyone know how it compares to other fiels/professions? How about interviewing a lawyer? Or a doctor? Or an account manager? Or a product marketer? Are developers the only one with a "broken" interview process?

grillvogel 11y ago
most other job interview processes don't involve solving a series of riddles
- rewqfdsa 11y ago
  In most other jobs, candidates don't vary in skill by a factor of 10x or more
  - grillvogel 11y ago
    where did you get that idea?
  - thaumasiotes 11y ago
    Lawyers were mentioned directly in your grandparent comment. Sales and athletics also easily show at least that much variation from person to person. Business owners show much, much more variation than that, even if you restrict the field to very similar businesses. Researchers show variation similar to business owners (i.e. mind-bogglingly high). You can easily construct differences of larger magnitude for what you might think of as "simple" jobs like telephone receptionist; a fluent english speaker is going to be more than 10x as productive in that job, assuming they're supposed to do business in english, than someone who can barely get by, and probably billions of times more productive than someone who can't speak english at all. There are a lot of Chinese people out there whose highest ambition is to "work in an office". I've met some of them! Most will never achieve that goal, because they don't have the requisite skills. Most people who don't speak english won't apply for a job answering a telephone in english (but a lot of them will! This is exactly the kind of customer service representative everyone has grown to hate). I'd like to see some software job openings that specified a skill set, and were willing to hire people with that skill set. That doesn't seem to be the direction people are headed in, though...

grillvogel 11y ago

technical interviews are a joke. the majority of the time they exist so the interviewer can try to feel smart and subject the interviewee to whatever whimsical problem they found on the internet. how often do you do group coding in a whiteboard in your actual job? at one interview I was criticized for sitting and thinking about a problem for a minute without just blindly jumping into attempting to solve it. also tons of people are great at solving toy interview problems but can't debug their way out of a paper bag.

marknutter 11y ago
I once was called into an emergency meeting by the CEO of the company I was working for the time. When I entered the room, all of the top brass were seated around the table, some visibly agitated. The CEO proceeded to hand me a single black whiteboard marker as he stated "the fate of the company depends on you". The problem was outlined by a fellow engineer and it was explained that I had 10 minutes to solve it, or we might risk going out of business. With that, I took a deep breath and went to work furiously scribbling away on the board while everyone watched with anticipation. As I closed the final bracket, and stepped back to examine my work, audible gasps could be heard from around the room. The grizzled old CTO whom everyone was a little scared of broke the silence at last: "God dammit, Mark!" I caught a few nervous glances from some of the less technical folks. You could hear a pin drop when he continued: "you just saved the god damn company!" A slow clap started, and soon everyone joined in with a round of applause and cheering. Over the outburst of joy and adoration the CEO shouted to one of the developers who had gathered outside of the conference room, "get this code into production!" Soon the whiteboard was being whisked away by a group of smiling engineers as they navigated with great purpose through a sea of high fives and back slaps. And then, with a twinkle in his eye, the CEO shook my hand, and pulled a crisp $100 bill out of his wallet. As he handed it to me, he said "I knew from the moment I heard about your interview that you would be a great asset to this company, Mark. Go home and relax for the rest of the day and take your wife out for dinner tonight. You've earned it." So you see, being able to solve complicated problems on a whiteboard in a crunch time is a valuable skill.
- turkishrevenge 11y ago

dms105 11y ago

The real weaknesses of technical interviews are: 1) They usually just measure the amount of effort a person has put into studying interview questions. Whether or not the ability to do this translates to being a better engineer is debatable. 2) An interviewer almost always exercise some form of personal bias, whether it be educational, personal, etc. This doesn't always show up in written feedback, but the interviewers with stronger personalities usually dominate interview debriefs, and often influence others into hire/no-hire decisions. This is especially prevalent in smaller startups where the process is more informal, things move quickly, and decisions are based more on gut feelings.

iolothebard 11y ago
I had a 800+ page study guide while living in SF for the ridiculous technical interviews they'd put you through. Then you'd see their code base and realize they've never actually done anything that would remotely resemble best practices. The best part though was realizing if you didn't answer that one question exactly how the "brilliant" person interviewing you wanted it answered, you were done. So at that point I'd quiz them on all their shortcomings that were evident based on what they'd told me. In my experience hiring it's not that difficult. Good attitude, good aptitude, genuine interest in the field (software engineering), ideally interest in the applied product (if not a software product), decent communicator and good hygiene. If they've done well in those areas, I've never had to let someone go.
- stale2002 11y ago
  Ever thought about releasing this study guide that you've made? I would love to take a look at it/ send it to some friends!
  - iolothebard 11y ago
    Lol, no. It's mostly just an amalgamation of all the technologies and stupid fucking interview questions that you get. I have 10-12 different documents. CS/OO (basics for reminders with some simple algorithms), .Net, SQL (I always forget this shit, it's why I have data layer and use ORMs + LINQ), LINQ, jQuery, Vanilla JS, HTML5+CSS, Architecture/Patterns, ASP.Net vs MVC, Tuning, etc. Haven't updated them in a few years because I left the bubble. My interviews in the midwest are typically an hour or two at most with very little quizzing. At worst they ask you to make them a small simple app (which I find irritating but better than SF interviews).

brightball 11y ago

With programmers, the single easiest way to identify good candidates (in my experience) is sheer interest in what they do / desire to learn. This is a learn everyday field and if you're interested in what you're doing, you're going to do a lot better at it. It's hard to apply yourself mentally to something that you don't have a good level of interest in. Given that it's a learn everyday field, people with that level of interest will realistically be able to learn anything they need to do solve the problem you're hiring them to solve. The only real differentiating factor is your tolerance for ramp up time. I expect a programmer to be able to pick up a new language or database within a couple of weeks (tops) in most cases. If I'm hiring full time, that's something I'll tolerate. If I'm hiring a contractor, I'm going to be uneasy about paying high hourly rates for him to learn the job. The single most effective way that I've found to interview for "interest" is to just get them talking about something they've done before and ask them to go deep into the details. You get everything you need from watching somebody talk, with a smile on their face, about how they solved some problem in a creative way that makes them show some pride. Doesn't really matter what the problem was, if it was a business problem, code problem or hardware problem. The important thing is the level of attention to detail in addressing it. I've been using this technique for about 8 years now and while I don't make it the exclusive criteria for hiring, every person I've ever hired who has passed that part has ended up in my "great hire" category.

jessedhillon 11y ago
Interest is not enough, unfortunately. There are plenty of engineers who are attracted to the challenge and excitement of building new things, but have no appreciation for The Right Way to build things. Great engineers think beyond "how" and ask the "should" questions as well. Mediocre engineers glue things together in a haphazard way with little thought about what's the best way to write things. Caring about maintainability, comprehensibility and extensibility is partially a function of experience, but I've met plenty of experienced engineers who still write bad code and design systems poorly. It has no correlation to how much of a tinkerer and curious person they are, in my experience.
- brightball 11y ago

kbd 11y ago

I had a really bizarre interview recently where, after the initial recruiter phone screen, I was rejected based on an in-person half-hour very simplistic paired coding exercise, only met with one person, and wasn't asked about my (imo very strong) resume once. I must have said something foolish at some point, which is on me, but the point is: interviews can be hit or miss. Fortunately you only need one hit.

kybernetikos 11y ago
It does sound like a bad interview experience, but I will say that I have very little interest in what a candidate writes on their resume. There are candidates with amazing resumes that can't do much and people without much that they can talk about publicly that are amazing. If the interview is low signal, the resume is even worse.
- kbd 11y ago
  > There are candidates with amazing resumes that can't do much... That's certainly true. I understand not taking resumes at face value; I've interviewed developers (and DBAs) and I'd always pour over their resumes to look for both padding and meat to ask about. It can be insightful to ask about a big accomplishment someone claims as their own only to find out that they only actually affected a small part of it. However, In this case I'm honestly not sure whether my interviewer even read my resume.
fapjacks 11y ago
Interesting. I wonder if we interviewed at the same place! I had a very similar experience over a year ago. At least I was told they only select one person from the group to move forward from the pair-programming task, which I thought was weird, but hey, it's their company.

maxaf 11y ago

Take home tests FTW. The thinking goes as follows: 1. If the candidate can't be bothered to complete a 2-4 hour (depending on claimed seniority) code test in the language of their choice, we can't be bothered to talk to them. 2. If the candidate does reasonably well by completing the code test somewhat on time (with a fat margin allowed for them, well, having a life) and within parameters of the task, they're invited for a mostly non-technical onsite meet-and-greet. 3. During the meet-and-greet we make sure that the candidate isn't an axe murderer, is able to hold a quasi-technical conversation, and that both sides aren't immediately scared of each other. 4. The meet-and-greet can also include some low-key architecture discussion. Any nerds worth their salt will be able to conduct this line of questioning without making it obvious that an interview is taking place. Hopefully this isn't a critical step, as a good take-home code test will require the candidate to spend a little time designing or architecting their solution. After the above has taken place, it should be pretty clear whether the candidate in question is a fit or not. Note that this process is by design missing the useless traditional CS questioning component, contrived problem solving exercises, or a whiteboard code beatdown.

kafkaesq 11y ago
4 hours is a lot of time out of someone's day, and a near-dealbreaker for anyone gainfully employed and having some semblance of a life. Do you not realize this? Meanwhile, it really doesn't take long to find out if someone is "worth talking to." Ask them to provide a code sample, and talk to them about for 10 minutes or so. Or if you must use a coding exercise, really, just keep it short and sweet. A good coder can show you a lot about their sensibilities in their solution to even just a slightly non-standard sorting problem.
tiglionabbit 11y ago
You need to verify that the candidate actually did the take home test, and didn't hand it to their more talented friend.
- Peroni 11y ago
  That's why you get them to talk you through their process during the face-to-face interview. It'll be immediately apparent if they didn't do the work themselves.

fecak 11y ago

Great post as always Aline. I'd be most curious as to how well anonymity was kept. Did interviewees identify their employers, schools, or any other information that might create bias while in the interview itself? I've been recruiting for a long time, and I'm rarely shocked about the result of an interview - maybe a few times a year. There are tons of possible explanations for that, and lots of possible explanations for your results as well. Keep up the great work.

leeny 11y ago
Most of our interviews didn't have people unmasking until after the feedback step.

pklausler 11y ago

I have learned that an (in)ability to program "in the small" correlates very well with an (in)ability to program in the large, and now ask mostly simply questions whose answers are things like one-line Boolean predicates to test for well-defined conditions. It is paradoxically easier for an inept candidate to fake his way through an algorithm design question than it is to fake the coding of a simple test for "determine whether two closed intervals [a,b] and [x,y] overlap each other".

Ocerge 11y ago
I've moved to doing something extremely simple: just give me pseudo-code for indexOf given a string and a character. If you can't write the 4-5 liner for linearly searching a string for a character, then there's not much point in moving further. It is shocking how many people get tripped up by it.
- pklausler 11y ago
  I'm similarly shocked, but it's just how things are now. I'm not sure how I could set the bars any lower.
- jcadam 11y ago
  Is it a sign of insecurity that I just popped open vi to make sure I could solve this? :) Where I work, we've been trying to fill a vacancy for a senior dev for months. Our one whiteboard question is similarly easy: find the largest integer in an array (The interviews have become easier since I was hired on). And yet... I've been watching an endless parade of people with decades of experience who apparently don't know how to write a for loop :(
  - pklausler 11y ago
    And yet they can somehow get through a telephone screening interview.
hackinthebochs 11y ago
Exactly this. I've been arguing for years that those simple string/array/link list/tree manipulation questions are simply a microcosm of the qualities that make for good developers in general. If those trip you up then you need to consider that there are serious holes in your abilities, not that the test is bad.

rjzzleep 11y ago

Constantly confused by this, so much arguing back and forth, and yet the single easiest way to deal with this is a stripped down real world problem and then giving it to a whole bunch of different candidates. Some people adopt this some people don't some people argue that it's meaningless, and go back to the standard silly interview patterns of algorithm questions and meaningless complex fizzbuzz alternatives. tptacek summarized this in his hiring post: http://sockpuppet.org/blog/2015/03/06/the-hiring-post/ http://sockpuppet.org/blog/2015/03/06/the-hiring-post/ My personal conclusion is that most companies don't want this for two reasons: 1. culture fit is more important for people in a rigid hierarchical structure, partly because an out of the box thinker could be dangerous for that structure. too much questioning authority, too much pointing out flaws. It's much easier to have a good worker bee than wondering why you need 40 employees to build an automated gif platform. 2. in most companies everyone is very reluctant to make decisions. for example management struggles with clear direction because it opens them up to the question of liability. if they make a decision and it's wrong they might get fired. HR works the same way, if HR passes a resume along they want it to hit a list of keywords, so they can cover their asses if he turns out to be a bad hire. Basically everyone is so scared to make a mistake that they make a lot more mistakes trying to avoid them. The opening of the cracking the coding interview she talks about how they don't really care about false positives and negatives, they just want those to stay below a certain threshold. But consider the hiring scale of google compared to a small company and suddenly those things matter. One bad hire can be toxic. And basing your hiring strategy on something a huge behemoth with infinite money does is kind of silly imho

kearneyandy 11y ago

Great post and awesome interactive graph! props I'm curious about the interviewer community. Specifically things like how are they vetted and how often they come to conduct interviews. It would be cool if there was a community of interviewers for the betterment of the process, but I could see their retention for conducting interviews to only be 1 or 2 before they drop out. I see in the appendix that there are those that do more, but no indication about what percent leave quickly. A better drinking game might be when a candidate offers a data structure they know nothing about. Would a red-black tree work here? No.. I guess not.

alive2007 11y ago

Minor question; is it just me or is the "Results of Interview Simulations by Mean Score" a bit difficult to parse? I understand that observing the behavior of any singular cohort involves looking at the endpoints of the cohort's curve at the horizontal line 'x=n', where n is number of simulations you wish to observe (the right point of the curve at x=n is P(fail) of the worst performer in the cohort at n simulations, the left point of the curve at x=n is the P(fail) of the best performer); which is why the gap between endpoints within a singular cohort decreases as n increases. But it seems kind of counterintuitive to observe any other kind of trend -- shouldn't the information be graphed as P(fail) being a function of # of simulations, as opposed to the other way around, seeing as the latter is the independent variable?

mentatseb 11y ago

The conclusion is misleading due to 2 wrong assumptions: 1. The population is heterogeneous: interviews test different skills. All interviews don't test the same set of skills, which is mandatory to compare interview scores because scores are aggregates of these skill tests. Different job opportunities means different skills to test, so it seems reasonable to assume that people evaluation vary for different job opportunities, and thus their scores vary for different interviews. 2. The observations are not statistically independent: past interviews may influence future interviews. People may get better at passing interviews or conducting interviews over time. This would impact their score. It would be good to study the evolution of individual scores over time. While (1) should strongly limit the conclusions of the study, the complete analysis may simply be irrelevant because of (2) if the statistical independence of observations is not demonstrated. Sorry guys but this is Statistics 101 introductory course.

leeny 11y ago
(1) We listened to most interviews on the platform to establish homogeneity. Interviews were across the board, language agnostic, and primarily algorithmic in nature. (2) We actually looked into this and noticed that time didn't really affect performance. Usually, people did their interviews over a pretty short time span and then found a job. Or, people were already experienced interviewers and had kind of hit a plateau. You can see the raw data and how it oscillates wrt time in the footnotes.

JasonCEC 11y ago

As a potential candidate, all of the standard complaints ring true - but once you're on the other side of the equation, and need to hire people... your ability to create new interviews is not nearly as wide or as clear as it would seem from the outside. 1) Take home test: OK for performance metrics, bad for "getting to know" the candidate, and terrible for selling the candidate on your company 2) Daylong interview: Expensive, requires interrupting our team, needs a fully planned and well executed itinerary - but is perfect for getting to know someone, getting the feel for their personality and interests, and is the best way to sell someone on the opportunity. 3) Work sample: we usually do this for interns[1] and pair it with a ~1 hour conversation (either before or after, doesn't really matter to us) on what the company is like and what they would be working on. Obviously, work samples suffer from the same deficiencies as a take home test for cultural fit and the like, but it's the best we can do for interns! [1] https://gastrograph.com/blogs/gastronexus/interviewing-data-science-interns.html https://gastrograph.com/blogs/gastronexus/interviewing-data-...

rm_-rf_slash 11y ago
I've said elsewhere on this thread, but my absolute favorite technical interview method is to set up a contract to pay the candidate to create a feature or fix a bug on your codebase, and have it merge and work as expected. It lets the candidate feel (literally) valued by your company, and you get to learn how they do the job you are hiring them to do.
- hire_charts 11y ago
  Agreed that this is a great approach, but the major downside is that you are limiting yourself to candidates who either: a) have the time to do real work outside of their day job and aren't on a restrictive contract preventing them from moonlighting, or... b) are currently unemployed
  - rm_-rf_slash 11y ago
    I was a college student at the time. I was long on money and short on cash. I was also far less experienced than I was so there were a lot of new things for me then, like those fascinating "callback" thingies I had to figure out. Nowadays if I were given roughly the same assignment it would have taken me about 3-5 hours at best, which, given my graduated-and-employed hourly rate, is actually quite in line with what I was paid. So it paid as much as I would have charged for a Saturday hack.

marknutter 11y ago

Technical interviews are a form of hazing. Engineers often suffer from imposter syndrome, especially during an interview. Those who have already been hazed and accepted to the club will turn around and put potential candidates through the same humiliating process. And what's worse is that demonstrating you have superior capabilities in one area or another can be seen as a threat to the interviewer and they may give you a thumbs down based purely on their own insecurities. What ends up happening is that, just like in college fraternities, is that everyone ends up being similar, both culturally and in terms of abilities. If I had it my way I would do away with the interview process altogether and do something more akin to an internship. Potential employees could start their engagement with a company by working (for pay, mind you) on a very limited basis to solve actual problems that need solving (i.e. "write an algorithm that's 10% more efficient", "create a tooltip that's aware of the viewport in React", etc). Based on their output their engagement could be ramped up until they are brought on as a full time employee. That way it ends up being completely merit based. You can either solve these problems or you can't. And whether or not they ultimately end up becoming an employee doesn't end up mattering because both parties are compensated along the way. This would obviously put the burden on the company to boil its problems down into smaller, isolated efforts but that's something all companies should be trying to do anyways. In the end, they just want some code written that will end up solving some problem for their customer.

kelvin0 11y ago
Hazing? Really? Hazing is the practice of rituals and other activities involving harassment, abuse or humiliation used as a way of initiating a person into a group... https://en.wikipedia.org/wiki/Hazing https://en.wikipedia.org/wiki/Hazing
- marknutter 11y ago
  Yeah, that sounds about right.
  - kelvin0 11y ago
    Maybe you should tell us which companies haze the potential candidates so we can steer clear of them? At least some hints?

deleted 11y ago

[deleted]

agentgt 11y ago

I haven't interviewed in some time but one thing I absolutely hate about technical interviews is "white board" coding. For some reason white boards intimidate me. I have terrible penmanship and complete lack of planning how much space I need for writing things. Then there is the fact the markers seem to have a hi failure rate when ever I use them. Perhaps I'm the only one that feels that way. I have even begged some interviewers that I would prefer them just watching me use a laptop but the offer is typically refused. Maybe things have changed now?

Gratsby 11y ago

Technical interviews are silly exercises IMO. You have a multi-month ramp up period and you have all kinds of environment specific things to deal with at just about every employer. Nobody in real life gets asked to program on stage or forced to answer esoteric questions as if your in some sort of math competition. If you don't have confidence in someone's ability based on their experience and their interview but you did like them, give them a task to accomplish offline. See if their results are anything like your results would be, and bring them in to see how they respond to feedback both negative and positive. I've seen too many interviews that go along the lines of "How would you rate your Java on a scale of 1-5?" "5" "So how would you fix the problem if your cache hit rate on SomeObscureCommercialProduct went from 94% to 82%?" "Forget that guy. Huge ego. Doesn't know anything." I did run into one company that had an interesting process for technical validation. They actually hire people for two weeks as contractors and have them work with the team. Then they hold a vote and decide whether to extend an offer.

mirceal 11y ago

3 things I search for: Fundamentals: CS basics. I don't nitpick on details. It's more around if you've heard about it or not and if you could figure out how and when to use it. Structure: I want to see a structured approach to problem solving. Doesn't matter if your code is perfect. Doesn't matter the programming language you want to use. Curiosity: You need to be curios about things. Asking the "why".

innertracks 11y ago

Just finished 2nd phone interview (it was somewhat technical) yesterday with a company and an online timed tech test this afternoon. First interview was with an internal recruiter and was non-technical. I've been impressed. They've been very straight forward regarding tech eval with no trick questions and respectful of time. Their interview process is selling me on the company.

uiri 11y ago

The data seems to show the opposite to me - despite scores being all over the place, the mean is very reliable. When a 2 or lower is considered a fail, those who consistently rate ~2.5 fail about half of their interviews while those who consistently rate ~3.0 fail only 10%. Of course, the probability that a candidate failed an interview approaches 1 as they are subject to more and more interviews. That the test has both false negatives and false positives does not invalidate the test. In fact, that the test is accurate despite the false positives and the false negatives ought to do the opposite. If a single bad interview invalidates a candidate for company A, that doesn't mean that the candidate won't go on to pass all of their interviews with company B.

jorgecurio 11y ago

Technical interviews are part of why I've moved away from engineering positions. I'm looking at product management jobs thinking that I could use what I learned past 6 years working on a SaaS alone. However, I found the exact same shit, even more technical interview questions that require whiteboard code writing. There might be some merit to why they are doing this but it's impossible for me to engage companies that discount real world product experience in favor of rote memorization. So far it's a pretty tough nut to crack, lot of product manager interviewers don't seem to know what they are doing, instead relying on law of large numbers and how great their fucking product is blah blah blah (it isn't). It's a bit worrying since some companies seem to be hiring product managers for some subjective end goal of an improved product and improved sales....they want one person to take the credit from, and the same person to take all the blame...another huge red flag when managers outright tell you they have no idea what to do so they just get someone else to outsource their thinking.

dilemma 11y ago

The problem with hiring is that hiring decisions are centralized, causing huge workloads for the decision maker. To reduce the load, arbitrary processes and voodoo tests are used, always with the same poor results. Instead, the team hiring should themselves interview candidates and make decisions on who to hire, because it requires personal knowledge that you can't get from tests.

lostcolony 11y ago

It may not be arbitrary, but if so it's buried deep in the data. I've turned down candidates who had impressive technical resumes, who had worked in startups that sold, who had been hired on as consultants at various places, etc, because they were unable to solve simple algorithms in a simple manner, and their code was atrocious. Does this mean they're "bad developers"? No. If we were a consulting firm or a startup they might well be worth it; where the important thing is getting code out the door quickly, and to have something that works, even if it's not easily maintainable. But I was hiring for a position that required someone who would keep solutions simple and maintainable ('craftsmanship' rather than productivity, if you will. Note that the former does not necessarily preclude the latter, but it's the trait that was necessary, and was lacking). Google optimizes for people with strong algorithmic knowledge. It's debatable whether they need everyone to have that, but certainly, many shops don't. Again, I've hired people with no formal CS background, because most of my job's problems don't require you to have deep algorithmic knowledge (the ones that do we can have others address, or work together on). We know that people can fail one technical interview, while being radiant in another, and the reality is that what we're looking for, and what others are looking for, are often different. That creates a lot of variance in the data when we compare them.

siliconc0w 11y ago

I dunno - if you look at the data, there are fairly clear clusters of people who are 'probably good' and 'probably not so good'. 'Programming' is necessary but not sufficient for product engineering and that is what most of these interviews are trying to tease out. Good companies will balance out 'programming' with other rounds like 'technical design' or 'pair programming' or even non-technical rounds with business analysts or product to gauge general ability.

macscam 11y ago

super true, and probably easy to bypass if you're a data viz wiz. It's more burdonsome for web devs who don't have the math / algorithms chops, but I know I have to learn it.