11 ms·
Ask HN: Has there been any academic research into software interviewing?
Some common interview types are algorithm problems, pair programming exercises, take-home assignments, etc.
Has there been any research into the predictive power of such assessments? Is there any evidence that a particular type of question correlates well with job success?
- sampo 7y agoTo correlate with job success, you need to define and measure job success. A sub-question: How's the academic research into measuring programmer job success?
- srfilipek 7y agoEasy: LOC / hour. </sarcasm>
- dominotw 7y agoI really think this is best we got so far.
- onion2k 7y agoSo the secret to being a 10X programmer is ":set tw=8" making your maximum line length 8 characters, forcing you be 10 times more productive than rival devs who're stuck at the old 80 character standard. Got it.
- RugnirViking 7y agoI think reading code deliniated into lines 8 characters code, written by somebody else, with no whitespace and single letter variable names is the punishment reserved in hell for the very worst of developers.
- d0mine 7y agoOr you can move around code creating large diffs and when asked why: just say "I'm familiarizing myself with the code" -- real story: my boss says about that person that he is the most productive programmer. It seems gaming trivial metrics works in this case.
- dominotw 7y ago1. no one will be allowed to do that if you work in a team environment. 2. No one has been able to come up with a better alternative. I wasn't implying its a prefect metric or even a reasonable one. If you really need a quantifiable metric then thats all we got.
- WrtCdEvrydy 7y agoHard Mode: Removed LOC / hour.
- d0mine 7y agoHardest mode: non-written code
- 0xfeba 7y agoSocial-engineering your PM to remove requirements, yep.
- WrtCdEvrydy 7y agoMy PM loves removing stuff from tickets. Most of the time, what people want isn't even what they want.
- organsnyder 7y agoOn an inverse scale, sure. Only the best coders have net-negative LOC/hour metrics.
- szatkus 7y agoThis would end up with a lot of code golf.
- organsnyder 7y agoReaching the objective as efficiently as possible... yeah, sounds about right.
- thanatropism 7y agoCode golf probably correlates with IQ.
- PorterDuff 7y agoHow about academic research into the value of academic research in programmer job success? I figure that eventually it'll be similar to the Voight-Kampff test and kept on file forever while being shared with all potential employers.
- arethuza 7y agoI think the baseline test from 2049 would be more appropriate: "Because they have to kill their own kind, they constantly need to be assessed as to whether their work is having some kind of moral impact on them'" https://bladerunner.fandom.com/wiki/Baseline_test https://bladerunner.fandom.com/wiki/Baseline_test
- eldridgea 7y agoGoogle publishes a lot of their work at https://rework.withgoogle.com/ https://rework.withgoogle.com/. Also, Google's former head of people (Lazlo Block) founded a company that is focusing on that area: https://humu.com/ https://humu.com/. While I wasn't always a fan of Google People/HR procedures while I was there, I did always appreciate the data-driven/research approach Lazlo and the People team took.
- hyperpape 7y agoHillel Wayne has been writing about this a bit on Twitter. Some threads: https://twitter.com/hillelogram/status/1120495752969641986 https://twitter.com/hillelogram/status/1120495752969641986 https://twitter.com/hillelogram/status/1119709859979714560 https://twitter.com/hillelogram/status/1119709859979714560 In general, there's a lot of research, though with some weird gaps, and a lot of results (though not all) are inconclusive.
- duxup 7y agoRelated, how do you define even the type of job and compare the person hired to the job? A noob could be successful hire and not ultra productive compared to say a more experienced hire. And that doesn't account for all the places hiring / think they need / hired "rockstar" developers .... to maintain their crud app.
- jrumbut 7y agoI've always thought that surveys of team members and managers could be a pretty cheap and effective way to do this (in a research setting, where you aren't basing pay or contract renewal on it). It may not give you much in an absolute sense, but should give you a good idea of the relative ranking of programmers. Of course software engineering research in general has always been a little unimpressive because what you really need is several teams completing projects, some using method A and others using method B. This could cost millions, and would only settle the bet between methods A and B. Lacking this, we're left with the publically available research that relies on unconvincing proxies for success and very small samples. It's better than nothing but far short of what we ought to have given how important software is.
- webmaven 7y ago> This could cost millions, and would only settle the bet between methods A and B. Even worse: it would only settle the bet between methods A and B in a particular context.
- JamesBarney 7y agoUsually they look at future reviews or just ask managers how well that individual is performing.
- deleted 7y ago[deleted]
- vthallam 7y agoI don't know of any academic research, but Triplebyte is trying to understand this better. Their interview process involves a little of algo problems and pairing on hangouts. Also, there's no one size fits all theory. There are companies where every ms of time you save by writing a better algorithm makes a big difference and most where it doesn't matter. Take home assignments, pair programming have all been tried and is still used by some companies, but majority still rely on white boards and it seems to work fine.
- umvi 7y agoAre a lot of jobs really algorithm development though? Every job I've had just used existing algorithm implementations for sorting, FFT, etc. I feel like to do well at software interviews I have to brush up on dynamic programming, sorting, greedy, etc. algo development even though I've almost never had to develop a serious performant algorithm on the job.
- EpicEng 7y ago>Are a lot of jobs really algorithm development though Probably not "a lot", but personally I have _only_ worked in companies where they are our bread and butter (Biotech and finance.)
- andrewmutz 7y agoCompanies tend to use algorithms for interviews for new college grads not because that's what the job entails, but because that reflects what the candidate's background was about (college). If someone is fresh out of a 4 year program that was all about data structures, algorithms, operating systems, etc, then demonstrating mastery of those topics tells an employer that a person is smart and can learn large amounts of complex technical material.
- spatley 7y agoI agree that algo questions are useful for recent grads. I find it more frustrating when I see those questions given to candidates for senior positions with 10 or more years of experience. Keeping fresh on the big-O of a bubble sort and rebalancing a binary tree ends up being in addition to building enterprise software solving real-world problems day to day.
- neilk 7y agoInterviewing.io is a startup that does some research into candidate evaluation. To their credit, they have even posted research with negative results. http://blog.interviewing.io/posts/ http://blog.interviewing.io/posts/
- jppope 7y ago^^ this. Aline learner runs a solid content strategy over there focusing on data journalism.
- aerovistae 7y agoLerner
- lemony_fresh 7y agoWhen I was self-studying social psychology I came across social facilitation theory and found it applicable to understanding interviewing. People have a hard time performing complex tasks while other people are watching because their working memory is monitoring the social situation. This varies for each individual. For tasks that have been rehearsed, like a musical or dance routine, they perform better. Politeness theory is applicable as well, people want to be liked and at the same time do not want their autonomy challenged. My pet theory is that the whiteboarding epidemic is a status hierarchy game essentially communicating "if you want to work here, you are going to do what I say."
- emaginniss 7y agoI disagree completely about your theory. I have been on both ends of the whiteboard interview an almost uncountable time during my career. What it has always come down to in my mind is a collaborative process to solve a problem where the one performing the task is trying to solve the problem, while the giver of the task makes suggestions and points out weaknesses before they get out of control. I'm sure there are sadists who just want to watch people squirm while trying to solve an extremely complicated problem, but I don't think I've ever run into that in an interview and it has never been what I've put people through.
- paulkon 7y agoIn that case, why use a whiteboard instead of a collaborative text editor like coderpad, repl.it, or even gnu screen? Testing in an environment completely different from 'production' isn't going to give a super strong signal on collaborative problem solving imo.
- sarbaz 7y ago> it has never been what I've put people through. How can you tell? Is this always based on how you experienced the interview, or do you ask candidates afterwards how they felt? (no insult intended, but) you made an empirical statement, and I'm curious where your evidence is from.
- mancerayder 7y ago
- Scoundreller 7y agoNot software development, but professional schools (eg medical schools) have published research about which admission selection methods can have predictive value both in-school and as a clinician. Lots of long follow-up. I’ve seen some programs do away with references, resumes and even interviews. The programs that I’ve seen remove interviews usually did replace them with standardized stations with different scenarios. E.g.: “Explain to someone how to wash their hands if they never have before”. If you can do that well, you can probably explain some health procedure you’ll later learn about and have to explain.
- ghaff 7y agoI haven't looked for any recent research in this area but when I saw some way back when it was, in one sense, somewhat depressing. I forget if it was for undergrad or MBA programs but, as I recall, the correlations were basically to SATs/GMATs and class rank/grades. Of course, your results will vary depending on how you define success but basically quantitative hard measures had the most predictive value.
- Scoundreller 7y agoBut what outcome were they measuring? Grades and rank in the MBA program? A good student will probably continue to be a good student. But how to select which students will become good practitioners may be a different story.
- ghaff 7y agoThis was a long time ago and I don't remember the details. One commonly-used quantitative workplace success metric is compensation/salary, which is not necessarily unreasonable for MBA programs but can be more problematic in other areas. And you're right. It's not really surprising that getting good grades/test scores at one level of school correlates reasonably well with the same thing at another level. Which I imagine is one reason universities/grad programs generally don't optimize purely on that metric because their objective isn't solely about cranking out students who study well in school. ADDED: This, by the way, is more or less just a variant of general intelligence measures being correlated to a lot of outcomes. The SAT and so forth aren't intelligence tests, but I'm sure they're very correlated.
- kaiju0 7y agoI think a prior study is needed in how job postings correlate to actual work. Once you have a stable initiating side from an academic standpoint you can begin to look at the receiving side of the equation.
- barry-cotter 7y agoI doubt there’s been too much on software interviewing specifically but interviewing/candidate selection for jobs is a large part of personnel psychology. A recent meta-analysis of what they know is below. Basically general mental ability (g, IQ) is the single best predictor, work sample tests work well, structured interviews can increase the performance of IQ tests or work samples and unstructured interviews are a trash fire. > The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years > This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed. https://www.semanticscholar.org/paper/The-Validity-and-Utility-of-Selection-Methods-in-%3A-Schmidt/235f263383afe7cda38ef9a49ceb5dc1090f4769 https://www.semanticscholar.org/paper/The-Validity-and-Utili...
- dcolkitt 7y agoCan't agree with this enough. The most consistent finding in industrial psychology is that general cognitive ability is by far the best predictor of job performance in virtually every role ever tested. Even jobs that don't seem cognitively demanding, like janitors or infantry. Higher IQ candidates almost universally do better, even if they start with much less experience. The takeaway as it applies to modern software hiring, is that skillset is way overemphasized. It's quite common to see job ads that heavily focus on the company's specific tech stack. ("Must have experience with Rails, React, Travis, and AWS") It's much better to cast a much wider net, and try to find the smartest people anywhere. High IQ people can easily re-tool their specific skillset. What's interesting is this is much closer to how the most successful tech firms, like Google, tend to hire.
- artemisyna 7y agoI know of at least one big tech company that has and it informs the ratio of different types of interviews for each level. I imagine it'd be really hard for an academic research institute to do so. Firstly, they'd have to focus on technical interviewing in specific. Secondly, they'd have to get some big tech companies to share information about trajectory and whatnot for employees once they've been hired -- I imagine the extra work of building those pipelines/anonymizing the data/etc probably wouldn't be worth it for a lot of companies.
- crispyambulance 7y ago"job success" is something that happens in the future relative to a "job interview". To be able to truly predict whether or not someone will be successful means that the interviewer would have to able to foresee how that person will interact with others in their workgroup , rise to challenges that don't yet exist, and be motivated to remain for a "long-enough" engagement with the company (whatever that means). Predicting the future is just damn hard, and it's even harder when nebulous desired outcomes such as "job success" are used. To make matters even more difficult, this problem also has "another half" to it which people tend to ignore: the employer doing the interviewing. These types of questions seem to take the point of view that an employer is presented with a bowl of fruit (candidates) and all they have to do is be able to select the best fruits. But it doesn't work that way. You may not get the fruit which you want, that fruit may want to go elsewhere or others might have already grabbed the fruit you would have wanted. The fruit which looks good now may end up rotten a short time later. Some fruit that looks undesirable today, may be awesome later. You may grow tired of apples and desire pears, but you've already filled your pantry with apples. What would happen if some organization was able to "figure this out" and truly maximally optimize their candidate selection using interview techniques? I am not so sure it would be easily distinguishable from what other similar competitive employers are doing. Predicting the future can only go so far.
- dudul 7y agoOne thing I've been wondering for a long time when I see the quasi-sado-masochist relationship between engineers and hiring processes is: are bad hires that 1) costly, 2) frequent ? In my 12+ years career, I've worked with so-so engineers, but never truly bad ones that would ruin a project. And even the ones that weren't great, what damages did they really cause? I've seen many more companies failing because of a bad product, bad sales strategies, bad market fit, bad business model than engineering teams using the wrong programming language, or not building microservices the right way. I find this constant obsession around identifying "rockstars", avoiding "bad" engineers at all cost to be truly unhealthy.
- gregcrv 7y agoYes, and these people: "bad product, bad sales strategies, bad market fit, bad business model" never get "tested" on anything during job interviews.
- mratzloff 7y agoI strongly agree. A good engineer can compensate for bad product management and occasionally bad leadership, but can't do much about all the other things you listed. There's no way to guarantee a successful engineering hire, but organizations try to because their culture is so blameful or fearful of conflict that all the incentives are misaligned. I believe most organizations copy the Google, Facebook, etc. hiring processes with the goal of adopting an "industry standard process" because they don't know how to conduct an interview process that's tailored to their own organization, and they don't want to think too deeply about it. Having a coherent interview process means dealing with your organizational baggage: understanding it and either resolving it or crafting the interview process to select for people who can integrate well with it. Usually it's the latter, since most of the issues start at the top. Google, Facebook, etc. have spent a lot of time and money crafting a good interview process for their organizations: specifically what they value and don't value (even if the individual interviewers aren't always fully aware of what the organization is selecting for). They can also support a lot of false negatives. But the copycats don't have those same needs or candidate pool, so it always comes across as a bit disconnected to me.
- rightbyte 7y ago
- bellevue 7y agoMade an account just for this: A professor I know from undergrad did research which could essentially evaluate, with some adaptation, whether or not people would work well in software engineering situations. More centered around whether team members would work well together, but relevant nonetheless: Malte Jung
- hguhghuff 7y agoAre you asking cause you are thinking of building a product around this idea?
- cm2012 7y agoTokenadult used to post the data on this on every hiring thread. Here's a link: https://news.ycombinator.com/item?id=4613543 https://news.ycombinator.com/item?id=4613543
- mratzloff 7y agoI haven't found any academic research about this specific area, although there is of course a wealth of research into interviewing in general. The closest analog to a technical exam is probably in the finance industry. My suspicion, however, is that these kinds of problems have limited predictive success and yield a lot of false negatives.
- azhenley 7y agoThis paper isn't about interviewing, but is very relevant: What makes a great software engineer? https://dl.acm.org/citation.cfm?id=2818839 https://dl.acm.org/citation.cfm?id=2818839 HN discussion: https://news.ycombinator.com/item?id=15892898 https://news.ycombinator.com/item?id=15892898
- simplify 7y agoIt's not academic, but we just published a guide that goes over the pros/cons of each type. https://medium.com/@cspa_exam/a-guide-to-building-your-own-technical-interview-95c32f1d43d2 https://medium.com/@cspa_exam/a-guide-to-building-your-own-t...
- tschwimmer 7y agoIt's unlikely that any academic research will happen. Companies view hiring pipeline data as proprietary because they fear competitors may learn from it and gain and advantage. Even if researchers agreed to anonymize data I doubt BigCos would agree to share because it still could be used to benefit a competitor. There is likely to be internal research at some companies.
- perfunctory 7y agoI don't think it's necessary. Good software engineers are so scarce you should just grab the first adequate one who comes your way. When I was in the position of interviewing candidates, 9 out of 10 couldn't solve the most trivial of programming tasks.
- bartread 7y ago> 9 out of 10 couldn't solve the most trivial of programming tasks. This has consistently been my experience. Nevertheless, the ability to code is necessary but not sufficient. You're going to want to dig into character at least a little. - How are they in a team context? - Do they treat others with respect? - Do they act with integrity? These kinds of attributes are just as important as whether or not they can code, and I've been badly bitten when they've been lacking.
- hsudheh 7y agoNo. All the reasons you’ll see below are due to bias of experience - including this one. It’s all a made up game to hire people who have similar lives to the people doing the hiring. If you’re from the ghetto no matter how smart you are you’re fucked. If you were born rich in Palo Alto no matter how dumb you are the world is your oyster.
- jkukul 7y agoI once bookmarked a blog post [1] which attempts to review existing research on the topic. The bottom line, I think, is that unstructured interviews have the least predictive power. So whatever is your interviewing process, make sure it's structured and similar for every candidate. [1] https://erikbern.com/2018/05/02/interviewing-is-a-noisy-prediction-problem.html https://erikbern.com/2018/05/02/interviewing-is-a-noisy-pred...
- saltvedt 7y agohttps://leanpub.com/leprechauns https://leanpub.com/leprechauns