12 ms·

Making software engineering interviews predictive of job performance

131 points by geoffroberts 7y ago

chillacy 7y ago
The cited data is from a paper The Validity and Utility of Selection Methods in Personnel Psychology https://www.researchgate.net/publication/232564809_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology https://www.researchgate.net/publication/232564809_The_Valid... in 1998 as a meta-analysis of other findings, but they are not specific to software development (one account is sourced from 32000 employees in 515 diverse civilian jobs in the 1980s). Maybe software is like any other job (sales, clerical, accounting) but maybe not.
- geoffroberts 7y ago
  Why would it not be?
  - chillacy 7y ago
    I have some guesses, but even if you take that it is, the authors were not very transparent that they made this assumption, in fact they did not point out that the research was done on all jobs while they spoke only of software. That is a bit misleading.
- jl2718 7y ago
  Oh wow I spent all that time reading with the assumption that this was a novel and relevant investigation over that year of being focused on the specific problem of software inteviews. Interview articles are the hot-button clickbait of Hacker News. Not necessarily the author’s fault, but this topic has a tendency to trend above the contribution of its content.
new_realist 7y ago
“General Mental Ability (GMA) tests (like the IQ test) are very predictive of future job performance, largely because people with high intelligence can learn the skills needed to be successful on the job more rapidly. However, due to legal concerns they’re not recommended for companies hiring in the US.” This is what a classic FAANG interview is: an IQ test which is disguised as a relevant skills test for legal reasons. And they work quite well.
- saagarjha 7y ago
  Do they?
  - deleted 7y ago
    [deleted]
  - new_realist 7y ago
    They’re the worst form of interview, except for all the other ones.
    - amznthrowaway5 7y ago
      Why not just give an actual IQ test?
      - jedmeyers 7y ago
        Not allowed by law.
        dominotw 7y ago
        > IQ tests (for interviewing) are typically spatial comparisons, some math, logic puzzles, reading comprehension, and pattern recognition I definitely got these in tech interviews. Thoughtworks gave me one last year.
        jedmeyers 7y ago
        It might have been a de facto IQ test, but they should have disguised it carefully.
      - bitwize 7y ago
        That opens you up to liability for race-based discrimination. The accepted wisdom is that IQ tests are inherently racist and favor white evaluees over black and brown ones.
tracker1 7y ago
In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied. Where I work now, we have a small pre-interview code challenge... it's not particularly difficult and in general should be resolvable in under an hour. I've seen questions on the other side, that I've literally spent 3+ hours on something that said an hour. I suggest those coming up with these things actually implement their challenge on a clean computer (only tools, no reference code) and time themselves. I now cut myself off at an hour and submit what I have stating as much. As to structured interviews, it will depend. It bugs me both ways when I see more generic, low level pattern questions in interviews only because the practical implications are usually very limited. On the interviewer side, it does help to ask some of the more quiz-like questions as it tends to give you a decent baseline of what the person being interviewed understands. In the end, personal questions along the lines of what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know. Culture, for as much of a B.S. thing that it is, is actually very important as well. A functional team or set of teams needs a few different types of personalities and worker types in order to do things well as a team. Being able to get along together is also imperitive. In the end.. it's a crap-shoot and understanding the needs, fill and fit are all varied a bit... trying to normalize doesn't help much. Also, business needs and talent pools will vary greatly.
- MaxBarraclough 7y ago
  > if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel And where you set the bar will depend on, roughly speaking, your prestige. Google can afford to be extremely selective when hiring technologists. Similar thing goes for Harvard and the Air Force. > what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know The usual counterpoint: nuclear engineers and surgeons tend to have hobbies with no bearing on nuclear engineering or surgery. Nothing wrong with a competent technology professional keeping it to work hours.
m3kw9 7y ago
To boil this interview thing, you are trying to make a very far prediction of someone based on very little data. The industry standard way to gather that data will probably get you to see if that person has done it before but not how well. Unless that person is sorta famous and even then he could be an asshol to the team and a disease
- itronitron 7y ago
  Furthermore, the unstated unknown that many interviewers have on their mind is whether the candidate will be successful in this specific company.
LifeLiverTransp 7y ago
I wondered, have any others been expected, to reverse the interview direction? As in ask the company manangment, sales and tech ceos, to answer your test scenarios? "You have developed a new product with broad market appeal, and sales already found you a huge first customer, which would secure you for years to come. In return they expect you to tailor the product to there needs. How do you react?" It sounds mad, but it seems somehow to be expected by some. Like they expect you to test your company for warning flags as thoroughly as they test you..
quanticle 7y ago
“In economic terms, the gains from increasing the validity of hiring methods can amount over time to literally millions of dollars,” writes Schmidt regarding the importance of valid assessment methods. “However, this can be viewed from the opposite point of view: By using selection methods with low validity, an organization can lose millions of dollars in reduced production. In a competitive world, these organizations are unnecessarily creating a competitive disadvantage for themselves. By adopting more valid hiring procedures, they could turn this competitive disadvantage into a competitive advantage.” The problem is that this misunderstands the purpose of the hiring process. The purpose of the hiring process is to get people who are qualified, while signalling that your company hires good people, and staying out of legal trouble along the way. I'm not sure I agree that an organization can gain "millions of dollars" by fine-tuning its hiring processes to the nth degree. While I absolutely agree that there are skill differences between programmers, I contend that the economic differences attributable to skill between programmers are dwarfed by the economic differences between any automation and no automation. Simply put, having a terrible programmer automate your business processes today is far better than sifting candidates to find the perfect programmer to automate your business process two years from now. Hiring is about satisficing, not optimizing.
- geoffroberts 7y ago
  I understand your point—it's not about optimizing your process above all else (at the expense of speed, for example). But you can and should optimize your hiring process just like you would your sales process, onboarding process, etc and the outcome you're looking for is hires that are successful once hired. It costs companies far more if they sacrifice in the name of speed and make poor hires.
chapium 7y ago
The best interview process is not going to account for bad corporate policy, terrible bosses, or other confounding factors.
- geoffroberts 7y ago
  No argument there!
- proverbialbunny 7y ago
  So true! It's sad it's so hard to tell if someone is going to be a terrible boss or if there are systemic issues. Some of the best interviews I've had, have had the worst bosses, and vice versa. Though, totally anecdotal, especially given I can only go off of the jobs I've accepted which is a sample set of the best interviews I've had.
fileyfood 7y ago
"Hard Interviews, Happy Workers". This note near the end assumes correlation shows causation, which it does not. You could also argue that more workers want to work in certain jobs, creating more competition and leading the company to make interviews harder. The workers in those jobs may be happier, but not because the interviews were harder.
kube-system 7y ago
Some of the points here are good, but I think some of them are too simplistic. This article makes a huge assumption: that performance is the only factor you're hiring for, and therefore unstructured interviews are worthless. That would make sense if you're interviewing for a factory of emotionless robots. But humans are social beings, and human performance as a team is more complicated than a sum of individual productivity. You need to take into account the affect that the hire would have on the entire team. Just about everyone has an anecdote about a coworker sometime during their career that was super productive, but completely demoralizing to the team. This is a very valid reason to give unstructured interviews, and if you ask smart people who give them, this is what they're looking for. I totally agree with the utility of work-sample tests. There is nothing better to see someone's performance than simply giving them a small but relevant chunk of work to do. It is very important to keep them short and flexible though. If you make them too lengthy, you'll skew the results against people who have better things to do with their time. What kind of person works for free anyway? Certainly not the caliber of people I want to work with. And if you make the test too specific to certain technologies, you skew the test against smart engineers who are less familiar with that tech. If you want a problem solver, give them a problem in plain words, and let their solution be open-ended. I don't agree with making tests particularly hard, though. Make it nuanced, yes, but of normal difficulty for a given assignment. You need the appropriate people to do the job. If you only hire top-tier algorithmic experts for your basic CRUD app, then you're going to have a very expensive CRUD app with a bored and demoralized team. If someone does sloppy work, you can spot this on a normal exercise as much as you can on an unreasonably difficult one.
- itronitron 7y ago
  >> What kind of person works for free anyway? I would expect everyone at some point has done something (worked) for free, either on speculation, as a favor, or for bragging rights. Open source wouldn't be a thing otherwise.
  - kube-system 7y ago
    Most work being done on significant open source projects is by people who are being paid for that work.
agentultra 7y ago
I think the important thing is to realize that you need to rely on realistic factors and many different signals to get useful data. I've been to plenty of interviews for "senior software engineers," where they used the standard gauntlet of algorithms/problem solving/data structures questions in rapid succession for eight hours. I tend to deliver most of my projects on time and of a high quality. I value things such as correctness and maintainability, and am conscious of other factors to consider such as viability and availability of time and capital. I spend a good amount of my time writing and seeking feedback from colleagues and think it's important to share our ideas and talk about our work. And when I practice my work I don't think I've ever been in a time-constrained scenario where I had to come up with an implementation of an algorithm while being judged and evaluated for how I think. I know enough about my work to understand when I need to use BFS or DFS and when to use a tree, a map, or a heap; but I also know that if I can't think of it off the top of my head I can walk over to my library and pull out a book or look it up online. I'm also a proponent of formal methods. I've written specifications to help solve race conditions and validate implementations of locking algorithms. I can write basic proofs. And I'm a great writer as a result. I spend a good amount of my time writing and communicating my ideas, seeking feedback from colleagues, and in general improving my understanding of problems. How does finding a cycle in a sorted array or reversing a linked list signal to you whether I would be a good fit for your company as a senior engineer? Does your company also value clear writing and good communication skills? Do you value engineers that can manage themselves and get work done autonomously? Modern interview practices at most places I've been through don't know how to get those signals. It's useful knowing that the people I work with are also familiar with when to use a heap versus an array but I also value other things at the senior level that most companies aren't really testing for. I always prefer data over gut instinct but I feel like the process of hiring people is not amenable to clean signals... it's a very messy process and I'm not hopeful that there will ever be a silver bullet.
- zygy 7y ago
  What does your interview process look like?
  - agentultra 7y ago
jamespollack 7y ago
so you're gonna pay us to do these "work samples" at market rate, right?
- kube-system 7y ago
  Sarcasm aside, this is exactly why we make ours as short as possible in duration. You don't want to filter out people with valuable time.
- geoffroberts 7y ago
  Actually, yes! Erik Bernhardsson, the CTO at Better.com who is cited in the article a few times, does exactly that for finalist candidates that they ask to provide a significant work sample. I think that's 100% appropriate. But that said, a work sample doesn't necessarily have to be something that takes hours on end either.
  - jamespollack 7y ago
    ayy, that's awesome if they pay candidates for significant work samples. i think there's some discussion to be had about what's significant. anything more than 1.5-2hours in my opinion. an hour or two is how long an interview might take in any profession. asking me to work unpaid for 4 or 8 hours (one project: 3 days!) is excessive. maybe call it the "half-day" test. is it a half-day of work? if so pay. if not, then you can do that once during an entire interview process.
- qudat 7y ago
  Aptible compensates for the amount of time we want you to spend on a work sample.
Havoc 7y ago
I don't think you can. Job performance is something that gets figured out on the fly. Unless you're working in a crazy standardized operation thats a moving target. e.g. We've got a ton of manager & they all sorta do their own thing...they gravitate towards their strong suit essentially. You can't really test for that in advance. Some people excel at leading 30 man teams on stable jobs. Others excel at being parachuted into technical shitshows and sorting it out. Very different personal attributes. It's a crap shoot as tracker1 says and will continue to be one
- tschiller 7y ago
  If you know what kind of company/situation you're in, you can certainly select for candidates that are more likely to thrive in that environment. As you mentioned, people gravitate toward their strong suits. So by asking behavioral questions, you can learn how they approached situations in the past and whether their strengths/weaknesses are the right mix for your needs.
  - thedance 7y ago
    If your company knows what situation it is in, and if that situation persists for more than a year, your company is already better than many. The company I just left had an endless parade of new missions, new "values", new career ladder descriptions, and the median tenure of managers in engineering was about 6 months. The idea that you can either rate the performance of your staff or predictively hire engineers who will succeed under these conditions is a joke.
    - geoffroberts 7y ago
      That's true.... that might not be a company you want to work for. If there's not clear strategic direction and things constantly shift, it's very hard to hire anybody effectively into the org.
    - tschiller 7y ago
      Yep, if your company is having leadership/strategy problems you're not going to go far. But as a hiring manager in that situation, you can select for team members who excel at dealing with ambiguity, internal locus of control, vision, etc. Additionally, you would look for people who value learning vs. psychological safety, etc. You're in a known situation, and that situation is constant change and churn
saviorand 7y ago
tl;dr: offer your applicants test exercises
- geoffroberts 7y ago
  Yes, but not generic tests—in order to design a hiring process that's actually predictive of future job performance, the "tests" have to be designed to be directly relevant to the real world work that the engineer will be tasked with.
hn_throwaway_99 7y ago
2 comments on this: 1. The authors of this paper are selling something. This is an advertisement. Not saying it's wrong or untruthful because of that, but it is an advertisement. 2. As another commenter said, the cited data is from a paper that doesn't look at software engineers specifically. Given that software engineering has one of the highest amounts of variability in productivity of any profession, that should be taken into close consideration.
- geoffroberts 7y ago
  The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically. What's your source in saying "software engineering has one of the highest amounts of variability in productivity of any profession?" That seems hugely unfounded.
  - hn_throwaway_99 7y ago
    > The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically. Lol, the article actually ends with a link to the product you're selling: > Ready to build a more predictive hiring process for software engineers? Schedule a 1-on-1 review of your hiring process or request a free trial of Qualified here.
phibz 7y ago
Interviews aren't just about talent and ability. Some fairly abstract concepts are important too. How the individual fits within the team is vital. You might have one of the smartest people on the planet but if they can't effectively communicate or understand and work within the cultural norms of the team you're going to have a problem. These norms and practices vary and shift between teams, countries, and even within the same team over time. For example hiring a Linus Torvalds for an intermediate, client facing, political position, is probably going to have poor results. These are difficult things to test for and are as much a decider of job performance as raw knowledge and technical experience.
lugu 7y ago
Just to share my experience. 1. We sucks at hiring, so we need to train juniors on how to hire early. 2. Programs live in the head of their creators. The code is just a draft left for computer to execute. Build a human culture where the aspiration is discussed. 3. We don’t replace people, we hand over the responsibility. Be ready to work differently. 4. I conduct interview by asking the candidates to explain me I piece of work she is is proud of. If I understand the what, how and why, she usually pass. 5. Remember: most issues are not technical but organizational. We can train people, but we can’t change them. 6. Set high quality standards for your team. Make them proud of their work. On boarding will be easier. So, it is less about how to hire the shinning star and more about how to welcome and how to create a culture around your values. My two cents.
mr_toad 7y ago
The elephant in the room is that we can’t accurately measure job performance by any objective metric either. Many things would be simpler if we could: performance reviews, promotions, incentives, salary equality, recruitment etc.
proverbialbunny 7y ago
Data Science interviews almost always have a take home interview as the technical challenge. The in person interviews tend to be cultural fit interviews. This is because DS work is larger than a bug fix, implementing a new feature, and even larger than writing a program from scratch. Even a small DS problem is a multi day problem. Most interviews allow one to take weeks to do a "quick" 2-3 day problem. This process may be out of necessity, but I feel it reduces noise. You really do have a better idea of knowing what you're getting. Software Engineers may groan at the idea of a take home project, but just throwing this out here: What if a company let the interviewer choose between a take home problem or an in person 30 min style white board interview? What kind of results would come from that? Would the people who take the take home interview work out better at the company, or would the whiteboard interviewers end up working out better?
- jl2718 7y ago
  There are a few ways to do a 2-3 day data science problem: 1. Random plug and chug / datarobot 2. Copy someone else 3. Expert feature engineering 4. Overly-documented EDA I don’t think any of this is relevant, but maybe it gives some indication that they can do anything at all. However, most good DS will pass because they‘ve been burned too many times by companies just looking for free help on a problem. If you’re a DS, don’t ever work on a company-relevant problem for free.
TSiege 7y ago
The best interview I've ever had was I was asked how I might design a real user feature for their company and review a sloppy PR with errors. For the first I came up with two options and weighed the best reason for each and which I might go with depending on the circumstances of the project. For the second I wrote PR comments I would leave any developer with suggestions for improvements, concerns, and praise for good work. That was representative of my actual job as a software engineer. The normal is taking some sort of tricky pop quiz styled questionnaire, which in my opinion was not representative of anything about me beyond I know how to use a for loop efficiently. Yet this is what I find these companies offer. They tout them as a quick fix cause its easy to standardize, analyze, and hard to do therefore good, and last but not least waste the candidates time but not the companies. Their the metrics are moot. Not only have they been proven to be ineffective by the big companies that spawned them, but I also find them demoralizing and a turn off. It seems like the industry is heading towards a self imposed SAT style quiz where the lucky winner of this startup war holds all the power and provides little benefit to either side of this process.

131 points by geoffroberts 7y ago

The cited data is from a paper The Validity and Utility of Selection Methods in Personnel Psychology https://www.researchgate.net/publication/232564809_The_Validity_and_Utility_of_Selection_Methods_in_Personnel_Psychology https://www.researchgate.net/publication/232564809_The_Valid... in 1998 as a meta-analysis of other findings, but they are not specific to software development (one account is sourced from 32000 employees in 515 diverse civilian jobs in the 1980s). Maybe software is like any other job (sales, clerical, accounting) but maybe not.

geoffroberts 7y ago
Why would it not be?
- chillacy 7y ago
  I have some guesses, but even if you take that it is, the authors were not very transparent that they made this assumption, in fact they did not point out that the research was done on all jobs while they spoke only of software. That is a bit misleading.
jl2718 7y ago
Oh wow I spent all that time reading with the assumption that this was a novel and relevant investigation over that year of being focused on the specific problem of software inteviews. Interview articles are the hot-button clickbait of Hacker News. Not necessarily the author’s fault, but this topic has a tendency to trend above the contribution of its content.

new_realist 7y ago

“General Mental Ability (GMA) tests (like the IQ test) are very predictive of future job performance, largely because people with high intelligence can learn the skills needed to be successful on the job more rapidly. However, due to legal concerns they’re not recommended for companies hiring in the US.” This is what a classic FAANG interview is: an IQ test which is disguised as a relevant skills test for legal reasons. And they work quite well.

saagarjha 7y ago
Do they?
- deleted 7y ago
  [deleted]
- new_realist 7y ago
  They’re the worst form of interview, except for all the other ones.
  - amznthrowaway5 7y ago
    Why not just give an actual IQ test?
    - jedmeyers 7y ago
      Not allowed by law.
      - dominotw 7y ago
        > IQ tests (for interviewing) are typically spatial comparisons, some math, logic puzzles, reading comprehension, and pattern recognition I definitely got these in tech interviews. Thoughtworks gave me one last year.
        jedmeyers 7y ago
        It might have been a de facto IQ test, but they should have disguised it carefully.
    - bitwize 7y ago
      That opens you up to liability for race-based discrimination. The accepted wisdom is that IQ tests are inherently racist and favor white evaluees over black and brown ones.

tracker1 7y ago

In the end, it's a crap shoot... Even their highest marker of validity is only about 50%. In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied. Where I work now, we have a small pre-interview code challenge... it's not particularly difficult and in general should be resolvable in under an hour. I've seen questions on the other side, that I've literally spent 3+ hours on something that said an hour. I suggest those coming up with these things actually implement their challenge on a clean computer (only tools, no reference code) and time themselves. I now cut myself off at an hour and submit what I have stating as much. As to structured interviews, it will depend. It bugs me both ways when I see more generic, low level pattern questions in interviews only because the practical implications are usually very limited. On the interviewer side, it does help to ask some of the more quiz-like questions as it tends to give you a decent baseline of what the person being interviewed understands. In the end, personal questions along the lines of what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know. Culture, for as much of a B.S. thing that it is, is actually very important as well. A functional team or set of teams needs a few different types of personalities and worker types in order to do things well as a team. Being able to get along together is also imperitive. In the end.. it's a crap-shoot and understanding the needs, fill and fit are all varied a bit... trying to normalize doesn't help much. Also, business needs and talent pools will vary greatly.

MaxBarraclough 7y ago
> if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel And where you set the bar will depend on, roughly speaking, your prestige. Google can afford to be extremely selective when hiring technologists. Similar thing goes for Harvard and the Air Force. > what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know The usual counterpoint: nuclear engineers and surgeons tend to have hobbies with no bearing on nuclear engineering or surgery. Nothing wrong with a competent technology professional keeping it to work hours.

m3kw9 7y ago

To boil this interview thing, you are trying to make a very far prediction of someone based on very little data. The industry standard way to gather that data will probably get you to see if that person has done it before but not how well. Unless that person is sorta famous and even then he could be an asshol to the team and a disease

itronitron 7y ago
Furthermore, the unstated unknown that many interviewers have on their mind is whether the candidate will be successful in this specific company.

LifeLiverTransp 7y ago

I wondered, have any others been expected, to reverse the interview direction? As in ask the company manangment, sales and tech ceos, to answer your test scenarios? "You have developed a new product with broad market appeal, and sales already found you a huge first customer, which would secure you for years to come. In return they expect you to tailor the product to there needs. How do you react?" It sounds mad, but it seems somehow to be expected by some. Like they expect you to test your company for warning flags as thoroughly as they test you..

quanticle 7y ago

“In economic terms, the gains from increasing the validity of hiring methods can amount over time to literally millions of dollars,” writes Schmidt regarding the importance of valid assessment methods. “However, this can be viewed from the opposite point of view: By using selection methods with low validity, an organization can lose millions of dollars in reduced production. In a competitive world, these organizations are unnecessarily creating a competitive disadvantage for themselves. By adopting more valid hiring procedures, they could turn this competitive disadvantage into a competitive advantage.” The problem is that this misunderstands the purpose of the hiring process. The purpose of the hiring process is to get people who are qualified, while signalling that your company hires good people, and staying out of legal trouble along the way. I'm not sure I agree that an organization can gain "millions of dollars" by fine-tuning its hiring processes to the nth degree. While I absolutely agree that there are skill differences between programmers, I contend that the economic differences attributable to skill between programmers are dwarfed by the economic differences between any automation and no automation. Simply put, having a terrible programmer automate your business processes today is far better than sifting candidates to find the perfect programmer to automate your business process two years from now. Hiring is about satisficing, not optimizing.

geoffroberts 7y ago
I understand your point—it's not about optimizing your process above all else (at the expense of speed, for example). But you can and should optimize your hiring process just like you would your sales process, onboarding process, etc and the outcome you're looking for is hires that are successful once hired. It costs companies far more if they sacrifice in the name of speed and make poor hires.

chapium 7y ago

The best interview process is not going to account for bad corporate policy, terrible bosses, or other confounding factors.

geoffroberts 7y ago
No argument there!
proverbialbunny 7y ago
So true! It's sad it's so hard to tell if someone is going to be a terrible boss or if there are systemic issues. Some of the best interviews I've had, have had the worst bosses, and vice versa. Though, totally anecdotal, especially given I can only go off of the jobs I've accepted which is a sample set of the best interviews I've had.

fileyfood 7y ago

"Hard Interviews, Happy Workers". This note near the end assumes correlation shows causation, which it does not. You could also argue that more workers want to work in certain jobs, creating more competition and leading the company to make interviews harder. The workers in those jobs may be happier, but not because the interviews were harder.

kube-system 7y ago

Some of the points here are good, but I think some of them are too simplistic. This article makes a huge assumption: that performance is the only factor you're hiring for, and therefore unstructured interviews are worthless. That would make sense if you're interviewing for a factory of emotionless robots. But humans are social beings, and human performance as a team is more complicated than a sum of individual productivity. You need to take into account the affect that the hire would have on the entire team. Just about everyone has an anecdote about a coworker sometime during their career that was super productive, but completely demoralizing to the team. This is a very valid reason to give unstructured interviews, and if you ask smart people who give them, this is what they're looking for. I totally agree with the utility of work-sample tests. There is nothing better to see someone's performance than simply giving them a small but relevant chunk of work to do. It is very important to keep them short and flexible though. If you make them too lengthy, you'll skew the results against people who have better things to do with their time. What kind of person works for free anyway? Certainly not the caliber of people I want to work with. And if you make the test too specific to certain technologies, you skew the test against smart engineers who are less familiar with that tech. If you want a problem solver, give them a problem in plain words, and let their solution be open-ended. I don't agree with making tests particularly hard, though. Make it nuanced, yes, but of normal difficulty for a given assignment. You need the appropriate people to do the job. If you only hire top-tier algorithmic experts for your basic CRUD app, then you're going to have a very expensive CRUD app with a bored and demoralized team. If someone does sloppy work, you can spot this on a normal exercise as much as you can on an unreasonably difficult one.

itronitron 7y ago
>> What kind of person works for free anyway? I would expect everyone at some point has done something (worked) for free, either on speculation, as a favor, or for bragging rights. Open source wouldn't be a thing otherwise.
- kube-system 7y ago
  Most work being done on significant open source projects is by people who are being paid for that work.

agentultra 7y ago

I think the important thing is to realize that you need to rely on realistic factors and many different signals to get useful data. I've been to plenty of interviews for "senior software engineers," where they used the standard gauntlet of algorithms/problem solving/data structures questions in rapid succession for eight hours. I tend to deliver most of my projects on time and of a high quality. I value things such as correctness and maintainability, and am conscious of other factors to consider such as viability and availability of time and capital. I spend a good amount of my time writing and seeking feedback from colleagues and think it's important to share our ideas and talk about our work. And when I practice my work I don't think I've ever been in a time-constrained scenario where I had to come up with an implementation of an algorithm while being judged and evaluated for how I think. I know enough about my work to understand when I need to use BFS or DFS and when to use a tree, a map, or a heap; but I also know that if I can't think of it off the top of my head I can walk over to my library and pull out a book or look it up online. I'm also a proponent of formal methods. I've written specifications to help solve race conditions and validate implementations of locking algorithms. I can write basic proofs. And I'm a great writer as a result. I spend a good amount of my time writing and communicating my ideas, seeking feedback from colleagues, and in general improving my understanding of problems. How does finding a cycle in a sorted array or reversing a linked list signal to you whether I would be a good fit for your company as a senior engineer? Does your company also value clear writing and good communication skills? Do you value engineers that can manage themselves and get work done autonomously? Modern interview practices at most places I've been through don't know how to get those signals. It's useful knowing that the people I work with are also familiar with when to use a heap versus an array but I also value other things at the senior level that most companies aren't really testing for. I always prefer data over gut instinct but I feel like the process of hiring people is not amenable to clean signals... it's a very messy process and I'm not hopeful that there will ever be a silver bullet.

zygy 7y ago
What does your interview process look like?
- agentultra 7y ago

jamespollack 7y ago

so you're gonna pay us to do these "work samples" at market rate, right?

kube-system 7y ago
Sarcasm aside, this is exactly why we make ours as short as possible in duration. You don't want to filter out people with valuable time.
geoffroberts 7y ago
Actually, yes! Erik Bernhardsson, the CTO at Better.com who is cited in the article a few times, does exactly that for finalist candidates that they ask to provide a significant work sample. I think that's 100% appropriate. But that said, a work sample doesn't necessarily have to be something that takes hours on end either.
- jamespollack 7y ago
  ayy, that's awesome if they pay candidates for significant work samples. i think there's some discussion to be had about what's significant. anything more than 1.5-2hours in my opinion. an hour or two is how long an interview might take in any profession. asking me to work unpaid for 4 or 8 hours (one project: 3 days!) is excessive. maybe call it the "half-day" test. is it a half-day of work? if so pay. if not, then you can do that once during an entire interview process.
qudat 7y ago
Aptible compensates for the amount of time we want you to spend on a work sample.

Havoc 7y ago

I don't think you can. Job performance is something that gets figured out on the fly. Unless you're working in a crazy standardized operation thats a moving target. e.g. We've got a ton of manager & they all sorta do their own thing...they gravitate towards their strong suit essentially. You can't really test for that in advance. Some people excel at leading 30 man teams on stable jobs. Others excel at being parachuted into technical shitshows and sorting it out. Very different personal attributes. It's a crap shoot as tracker1 says and will continue to be one

tschiller 7y ago
If you know what kind of company/situation you're in, you can certainly select for candidates that are more likely to thrive in that environment. As you mentioned, people gravitate toward their strong suits. So by asking behavioral questions, you can learn how they approached situations in the past and whether their strengths/weaknesses are the right mix for your needs.
- thedance 7y ago
  If your company knows what situation it is in, and if that situation persists for more than a year, your company is already better than many. The company I just left had an endless parade of new missions, new "values", new career ladder descriptions, and the median tenure of managers in engineering was about 6 months. The idea that you can either rate the performance of your staff or predictively hire engineers who will succeed under these conditions is a joke.
  - geoffroberts 7y ago
    That's true.... that might not be a company you want to work for. If there's not clear strategic direction and things constantly shift, it's very hard to hire anybody effectively into the org.
  - tschiller 7y ago
    Yep, if your company is having leadership/strategy problems you're not going to go far. But as a hiring manager in that situation, you can select for team members who excel at dealing with ambiguity, internal locus of control, vision, etc. Additionally, you would look for people who value learning vs. psychological safety, etc. You're in a known situation, and that situation is constant change and churn

saviorand 7y ago

tl;dr: offer your applicants test exercises

geoffroberts 7y ago
Yes, but not generic tests—in order to design a hiring process that's actually predictive of future job performance, the "tests" have to be designed to be directly relevant to the real world work that the engineer will be tasked with.

hn_throwaway_99 7y ago

2 comments on this: 1. The authors of this paper are selling something. This is an advertisement. Not saying it's wrong or untruthful because of that, but it is an advertisement. 2. As another commenter said, the cited data is from a paper that doesn't look at software engineers specifically. Given that software engineering has one of the highest amounts of variability in productivity of any profession, that should be taken into close consideration.

geoffroberts 7y ago
The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically. What's your source in saying "software engineering has one of the highest amounts of variability in productivity of any profession?" That seems hugely unfounded.
- hn_throwaway_99 7y ago
  > The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically. Lol, the article actually ends with a link to the product you're selling: > Ready to build a more predictive hiring process for software engineers? Schedule a 1-on-1 review of your hiring process or request a free trial of Qualified here.

phibz 7y ago

Interviews aren't just about talent and ability. Some fairly abstract concepts are important too. How the individual fits within the team is vital. You might have one of the smartest people on the planet but if they can't effectively communicate or understand and work within the cultural norms of the team you're going to have a problem. These norms and practices vary and shift between teams, countries, and even within the same team over time. For example hiring a Linus Torvalds for an intermediate, client facing, political position, is probably going to have poor results. These are difficult things to test for and are as much a decider of job performance as raw knowledge and technical experience.

lugu 7y ago

Just to share my experience. 1. We sucks at hiring, so we need to train juniors on how to hire early. 2. Programs live in the head of their creators. The code is just a draft left for computer to execute. Build a human culture where the aspiration is discussed. 3. We don’t replace people, we hand over the responsibility. Be ready to work differently. 4. I conduct interview by asking the candidates to explain me I piece of work she is is proud of. If I understand the what, how and why, she usually pass. 5. Remember: most issues are not technical but organizational. We can train people, but we can’t change them. 6. Set high quality standards for your team. Make them proud of their work. On boarding will be easier. So, it is less about how to hire the shinning star and more about how to welcome and how to create a culture around your values. My two cents.

mr_toad 7y ago

The elephant in the room is that we can’t accurately measure job performance by any objective metric either. Many things would be simpler if we could: performance reviews, promotions, incentives, salary equality, recruitment etc.

proverbialbunny 7y ago

Data Science interviews almost always have a take home interview as the technical challenge. The in person interviews tend to be cultural fit interviews. This is because DS work is larger than a bug fix, implementing a new feature, and even larger than writing a program from scratch. Even a small DS problem is a multi day problem. Most interviews allow one to take weeks to do a "quick" 2-3 day problem. This process may be out of necessity, but I feel it reduces noise. You really do have a better idea of knowing what you're getting. Software Engineers may groan at the idea of a take home project, but just throwing this out here: What if a company let the interviewer choose between a take home problem or an in person 30 min style white board interview? What kind of results would come from that? Would the people who take the take home interview work out better at the company, or would the whiteboard interviewers end up working out better?

jl2718 7y ago
There are a few ways to do a 2-3 day data science problem: 1. Random plug and chug / datarobot 2. Copy someone else 3. Expert feature engineering 4. Overly-documented EDA I don’t think any of this is relevant, but maybe it gives some indication that they can do anything at all. However, most good DS will pass because they‘ve been burned too many times by companies just looking for free help on a problem. If you’re a DS, don’t ever work on a company-relevant problem for free.

TSiege 7y ago

The best interview I've ever had was I was asked how I might design a real user feature for their company and review a sloppy PR with errors. For the first I came up with two options and weighed the best reason for each and which I might go with depending on the circumstances of the project. For the second I wrote PR comments I would leave any developer with suggestions for improvements, concerns, and praise for good work. That was representative of my actual job as a software engineer. The normal is taking some sort of tricky pop quiz styled questionnaire, which in my opinion was not representative of anything about me beyond I know how to use a for loop efficiently. Yet this is what I find these companies offer. They tout them as a quick fix cause its easy to standardize, analyze, and hard to do therefore good, and last but not least waste the candidates time but not the companies. Their the metrics are moot. Not only have they been proven to be ineffective by the big companies that spawned them, but I also find them demoralizing and a turn off. It seems like the industry is heading towards a self imposed SAT style quiz where the lucky winner of this startup war holds all the power and provides little benefit to either side of this process.