8 ms·

Three hundred programming interviews in thirty days

82 points by braythwayt 11y ago

dikaiosune 11y ago
I could have sworn I already read this, turns out the date on the article isn't correct, I think this was written in June '15: https://news.ycombinator.com/item?id=9766816 https://news.ycombinator.com/item?id=9766816 Which makes sense given that I tried out their online quiz this summer after reading this exact post. EDIT: Here's a link to when it was posted to LinkedIn, June 23rd, 2015: https://www.linkedin.com/pulse/three-hundred-programming-interviews-thirty-days-harjeet-taggar https://www.linkedin.com/pulse/three-hundred-programming-int...
- fweespeech 11y ago
  Yeah, that seems to be correct. I wonder why the date changed and it was reposted?
  - dikaiosune 11y ago
    Yeah, I'm not sure. It looks like maybe they changed their blogging backend from whatever in-house thing they used to Medium? Their blog (https://data.triplebyte.com/ https://data.triplebyte.com/) is now showing Medium stories with publish dates from last month. Maybe Medium doesn't let you set arbitrary dates when migrating?
    - whafro 11y ago
      This is certainly true. All migrated stories get their publication dates set to "now." In my case, I added a footer on each migrated post stating its original post date.
- braythwayt 11y ago
  My bad! I did an HN search for the link before submitting it, I find that gets around cases where people change the title. But I could have done a title search as well.
  - dikaiosune 11y ago
    Well it's clearly not that much of a problem, HN has such an appetite for what they're doing/writing about that it's back on the front page. Also for what it's worth even though I'd previously read the article I made it about halfway through before it clicked for me that I'd already seen it.
- ammon 11y ago
  We moved the blog to medium about a month ago. Since writing this, we've tested a bunch of new things, and now have about 20% better precision on the quiz (20% increase in the percentage that we predict will pass and go on to pass). There are some other interesting results (one is that easy programming problems are about as effective at screening as hard programming problems). I'll be writing more about this, with the new data/results in the next few months.
vdnkh 11y ago
>As soon as we started doing them however, I saw a problem. Almost everyone was passing. Our filter was not filtering. We tried extending the duration of the interviews to probe deeper and looking at code over Google hangouts. Still, the pass rate remained too high. To me, it seems like your team isn't that good at judging ability. To compensate, you rely on coding questions. This what the industry has been doing for years. What exactly is innovative here? I see the need for coding questions (and think that your method sounds pretty good) but too often culture fit & 'soft skills' are overlooked in favor of cramming as many technical questions as possible. One of the best experiences I had on an interview involved no coding on-site at all. I completed a project for them, they liked it, I got an interview. 3 hours of culture/team/work philosophy. There were no surprises - if I wasn't "technical enough", I wouldn't have been let in the door. It took time up front, but at least the interview itself wasn't waste of time, PTO, and money.
- paxcoder 11y ago
  So you bet your time. It's how I got my job too, but that should be the norm (unless there are guarantees). Companies should cover the expenses of finding employees, instead of crowdsourcing the unemployed.
  - paxcoder 11y ago
    I meant shouldn't be the norm. Screw these uneditable, permanent forums.
- ZanyProgrammer 11y ago
  3 hours of soft questions seems much more preferable to 3 hours of whiteboarding data structures. Hopefully your project wasn't a bunch of questions from an undergraduate data structures/algorithms class.
  - vdnkh 11y ago
    They were tough, insightful soft questions. And I asked a lot of them too. Gave me a good feel for how they worked and what they valued (which is what I really want to know). The challenge was pretty tough but it wasn't just graded on performance. The team lead came in with feedback from each member which was really good. Stuff like "you could have done this with less objects" or questions for me to resolve for them. I've used the feedback to improve my code since.
deleted 11y ago
[deleted]
minimaxir 11y ago
What the heck is going on with the statistical explanation? The chart's y-axis is labeled "correlation coefficient" but the text blurb analyzing it describes the definition of R^2 instead (% of variation of Y explained by X). The clarification at the end does not disambiguate this. Also, how do you numerically quantify "interview performance"/Y? Additionally 3 out of 6 of the variables have error bars crossing 0, which means that the correlation could be positive or negative but we don't know (it would handily fail a statistical test of significance at p = 0.05 significance). Why were they included at all? Many startups (including statistical startups!) have been playing hard and lose with statistical analysis lately and it's getting annoying.
- gragas 11y ago
  I completely agree. I also would not consider an r-squared value of 0.47 indicative of strong correlation between two variables. I guess the point is that the composite quiz score is more correlated with performance than traditional methods, but I still get the impression that someone is overselling something.
adambom 11y ago
I would applaud the author on trying to take a data-driven approach to discover the best hiring signals, but I must offer one major critique of the approach: These data are ultimately not too meaningful unless you correlate actual job performance with interview performance. By the author's own admission, interviews are often faulty. By correlating final-round interview performance with early-round interview performance, you are really only predicting how candidates will perform in the final-round interview, which is not necessarily a good predictor of long-term job performance.
- serge2k 11y ago
  Exactly. If you are serious about this then you should be interviewing candidates, deciding on whether you think they should get a job. Then giving them a job anyway and evaluating their performance after some time interval and see how accurate your interview process was in terms of weeding out those who would fail.
- rifung 11y ago
  In fairness, the author says at the end of the post that they do intend to rerun the analysis with actual job performance. It seems reasonable to optimize for hiring right now since otherwise nobody would go to them to find a job, and it'd also be impossible to measure job performance when the people you collect data can not get jobs.
- lqdc13 11y ago
  Sure, but this is still really useful for many companies in the interviewing space such as hackerrank, hackerearth etc. They're optimizing for hiring, not job performance.
  - vonmoltke 11y ago
    Which seems totally at odds with what their clients want and furthers the perception that tech interviewing has become a hazing ritual totally detached from evaluating the ability of the interviewee to actually do the job.
nordsieck 11y ago
Sadly, this is almost certainly illegal in the US due to the 4/5ths rule (http://adverseimpact.org/CalculatingAdverseImpact/Four-FifthsRule.htm http://adverseimpact.org/CalculatingAdverseImpact/Four-Fifth...).
- adambom 11y ago
  The language in that description refers to "protected groups". As long as the author is not calculating selection rates for groups of candidates based on some special criteria, such as race, gender, age, etc. he should be ok, right?
  - ThrustVectoring 11y ago
    "adverse impact" is designed to sniff out hiring criteria that are a proxy for discrimination based off criteria such as race, gender, and age. For instance, if you selected for college graduates for hiring firefighters, and that causes a lot of rejected applications among African American candidates, then it would be illegal based off of an adverse impact against a protected group. As an aside, suing employers over requiring a college degree is a seriously under-tapped and under-estimating way to charitably improve the lives of Americans (college is a positional good because college matters when employers are looking at candidates - if employers can't look at whether or not you have a college degree unless it's a bona-fide qualification, then many more people don't have to spend several years and tens of thousands of dollars)
  - nordsieck 11y ago
    That's not nearly enough. The 4/5ths rule is designed to identify "structural discrimination", or cases where no discriminatory act can be identified, but the outcomes are nonetheless considered discriminatory. As long as a lawyer can show that some protected group is disadvantaged by the quiz, that's enough for it to violate the rule.
    - danieltillett 11y ago
      I am not a lawyer, but I thought the defence was if you can show that the screening criteria correlated with on the job performance it is OK. No matter what you think of protected groups using a criteria that does not correlate with performance is not very smart anyway.
johnrob 11y ago
Why don't companies manage open source projects, and simply require a certain amount of contribution as a gateway to a final interview? This would allow candidates to prove their ability to write production ready code.
- branchless 11y ago
  Because they'd filter out all the good candidates who don't work for free (for for-profit companies)
  - johnrob 11y ago
    But they'll interview for free? They don't need to be large contributions (and if small changes require too much time, that becomes a natural filter).
droithomme 11y ago
This article is extremely interesting, but it seems to be equating success in interview decision with competent performance at the job, which has not actually been established yet. It is interesting though to see which components of pre- and early-interview evaluation ended up being related to the decision to hire.
pklausler 11y ago
I've conducted a lot of programming interviews over the past three decades. I think that it's easier to do at places like Google where the objective is to avoid a Type I (false positive) error and it's expected that a lot of time and money will be spent to identify the best hires. It's harder in other contexts where Type II errors matter more. I have a big portfolio of technical questions, but I usually stick to a small set that I consider to be well calibrated by posing them to smart coworkers and seeing what they do with them. I've found, counterintuitively, that the best indicators are the questions that deal with "programming in the small", such as coming up with a predicate expression to describe a test for a simple relationship between a few values, because either you can do it easily or you can't do it at all. And programming in the large depends on a lot of cases of programming in the small.
btilly 11y ago
I certainly hope that they don't ever assume that experienced candidates don't need a programming test. There is a particular kind of dysfunctional candidate who once was a programmer, graduated to being a lead programmer, and then became an architect. As an architect they stop programming, and their programming becomes rusty. Once that happens, the feasibility of their proposed "architectures" becomes steadily less practical. You can detect them by their first getting offended that they would be asked to program during an interview, and then failing to actually write working code. By contrast people who really do program have no objection to being asked to program in an interview. To the contrary being asked to actually program is seen as a positive sign that they won't have co-workers to deal with who can't actually program. (This is an unfortunate experience that most programmers try not to repeat.)
- collyw 11y ago
  I have declined some programming tests, when the employer wants yet another three hour challenge to build a mini app. That leaves not much time for actually thinking about a problem and rushing out a crappy solution. Not the way I like to write software. I have code online I can show them and talk to them about, and extend if they want to see me actually code. Bashing out code is usually the simplest part in writing a software solution to a problem.
  - btilly 11y ago
    There is a difference between "demonstrate that you can program" and "waste X hours on busywork".
- jkot 11y ago
  I write lot of code, but I refuse to do coding at interview. For me its sign of dysfunctional company. But I do not mind to do pair coding, and fix an actual bug. There are more reasons, in short it takes me couple of hours/days to warm up and grasp the problem. And secondly I refuse to waste my time, on half arsed throw-away code, with zero results.
  - 11y ago
raitom 11y ago
Why should I spend so much time with all these interviews when I can simply reply to one of the countless job offer I receive every week?
- to3m 11y ago
  Are these offers where you start next week if you sign the contact, or are they expecting to meet you first? If it's the former you should have no difficulty answering your question ;)
- JetSpiegel 11y ago
  You can start forwading those messages for a fee. Job Offers as a Service
Mc_Big_G 11y ago
"we’ve come up with a replacement for resume screens, and shown that it works well" No you haven't. You might have, but you'll have no way of knowing that until they've been on the job at least 3 months. Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development. College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time. Experience counts and if you disregard it then I'm firmly in the "Fuck you, I don't want to work for you" camp. Unfortunately, I've resigned myself to memorizing the typical interview questions or going into management. I'd love to stay a dev, but being ruthlessly quizzed about shit that you never use in real life makes changing jobs a pain in the ass.
- odonnellryan 11y ago
  > Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development. Aye. It's a hard problem: as an employer, you want to be pretty sure that you're getting someone on team that's going to bring you value. How do you do that? I guess ask them as much as you can. You definitely present a good problem, it is the same they're trying to solve over at triplebyte. > College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time. Yeah, you have to study for interviews. It sucks, but almost every software company asks the same questions. Just prepare, I guess, and it'll pay off with that higher salary. > makes changing jobs a pain in the ass. You might have nailed it here. Employee retention is good for everyone! I think the solution is you need connections. When someone works at the company and they know you're a good dev they aren't going to ask you questions they ask other developers.
  - hamitron 11y ago
    >> makes changing jobs a pain in the ass. >You might have nailed it here. Employee retention is good >for everyone! I'd say having a good work environment is good for employee retention, not arbitrary limitations to prevent people from switching jobs.
Ovid 11y ago
I have to take exception to this because part of what I do for a living is teach tech companies how to hire. The claims are: 1. Performance on our online programming quiz is a strong predictor of programming interview success 2. Fizz buzz style coding problems are less predictive of ability to do well in a programming interview 3. Interviews where candidates talk about a past programing project are also not very predictive Those claims may be true, but it misses a key point: no matter how well someone performs in an interview, it doesn't mean they'll perform well on the job. In particular, the post talks about hard skills (and does well there), but doesn't talk about soft skills. It's absolutely true that someone can talk a big game about hard problems they faced that they solved, but failed to mention their use of Google or StackOverflow. So unique coding tests can definitely tell us whether or not a candidate can really code (though the tests need to be tailored to the sort of real work the candidate will face). However, programming is a hard skill (one that is easy to teach), and not a soft skill (one that you tend to learn over time or have a natural inclination for). Case in point: early on we made the mistake of hiring a very talented programmer on the basis of his reputation and a brief interview. He was a disaster because he was completely unable to work without a very hard set of requirements. In the very rapidly changing, agile, environment we put him in, he flailed badly because he was good at implementing requirements, but he was very bad at projecting himself into the customer's shoes. Later, we put him on technically demanding tasks which required considerable talent, but had no ambiguity, and he was fine. In fact, he was able to implement tasks that would have been very difficult for other developers who were fine in ambiguous environments. Soft skills are tricky and most employers have no idea how to interview for them. They require an understanding of the actual role someone will be hired for and the set of skills necessary for them to succeed. Are there tight deadlines? If so, can the developer prioritize tasks and make a case for delaying less critical portions beyond the deadline? Is it a rapidly evolving environment which makes it hard to nail down exact behavior? Are you on a team with strong personalities who often disagree and can the dev handle that? Is the developer required to report to non-technical people who have no understanding of technology and can't appreciate what refactoring is? Or here's a fun one for you aspiring managers: we've had one project with a superstar developer who was so monumentally productive and so brilliant at his work that some very talented developers who worked with him nonetheless had negative productivity at times because they spent so much time playing "catch up" with the superstar and understanding how the project was evolving that they spent more time keeping abreast of the code than developing it. The code was "bus sensitive" because one brilliant developer was so competent. How do you manage that? These and other soft skills are part of what you really need to evaluate because depending on your work environment, you'll have these and other soft skills as critical skills that a dev will need, but you almost NEVER see interviewers check for these because very few people are trained in interviews. Pro tip: start reading up about structured interviews. They've been proven via multiple studies to have a strong correlation with employees being successful. Unfortunately, they're harder to conduct and require training to understand them. As a result, companies don't know about them or worse, ignore them, because it costs more money. But what costs you more money in the long run? Taking longer to interview or hiring the wrong person?
lewisl9029 11y ago
I gave the TripleByte application a try very recently, but this step wasn't a part of it: > 1. A fizzbuzz-like programming assignment. Applicants completed two simple problems. We tracked the time to complete each, and manually graded each on correctness and code quality. Instead, I was presented with a single, rather convoluted, 1-hour long programming problem that was most definitely not anywhere close to the difficulty level of a FizzBuzz test (the solution they're looking for probably involved dynamic programming, which I honestly haven't had much practice with). This was before I even spoke to any humans on the phone screen or interview steps of the process. I couldn't finish it in 1 hour (I kept going at it on my own time for another 2-3 hours before I finally arrived at a sub-optimal solution, at which point I had enough). I'm sure there are plenty of people who are much better at this than I am who can finish it in an hour. But I do want to question the wisdom of using something of this level of difficulty as a screening test. Your clients seem to value the "Product Programmer" very highly. Using a question like that as a screener will weed out a significant portion of these "Product Programmers" before they ever get a chance to showcase their product skills. Most "Product Programmers" I know (myself included) don't do enough convoluted dynamic programming problems in their spare time to be able to solve a dynamic programming problem like that within an hour with any level of certainty (and yes, solving problems like that quickly involves either extraordinary talent or lots and lots of practice). Instead, in their spare time, these "Product Programmers" like to build things, and learn new technologies and techniques to help them build things in a simpler, more maintainable, and more productive way.
lordnacho 11y ago
The best programmers I've worked with had this quality: They'd always heard of everything, and they had a good idea of what everything was, even though it was quite far from the current task to hand. For instance, plenty of people have not had the opportunity to work with functional languages, yet still have a good grasp of why they're useful, what languages there are, who uses them, and so on. They tend to see coding as a vast field where you can make many valid decisions, each with their own pros and cons. The kind of people I'd root out are the ones who merely see programming as a tool. Something along the lines of "I've got this data in a spreadsheet, I need to calculate this statistic, so here we go..." They almost always end up creating an unmaintainable mess, and they never mention any well known pattern. Quite hard to identify in an interview, though. Perhaps the lack of key words that suggest a deeper understanding? (Hey, that's an interesting thought. You could machine learn what words good devs use vs less good ones.) As for coding in the interview, I think this can only root out the absolute novices. It's quite hard to think of a short test that will demonstrate that someone can't write elegant code. Perhaps a take-home, open-ended test would work, but you're gonna lose people who don't want to spend a whole weekend doing it.
ap22213 11y ago
15 years ago you could get a software job by showing some passion, intelligence, a CS degree from a average university, and some knowledge of memory / space complexity of typical data structures and algorithms. And, if the hiring manager (usually a programmer) liked you? Job offer on the spot. Sure, you got paid 50-60k / year, but it was what it was. And, you actually got to work with an entire team of people under reasonable deadlines, 40 hours a week, with a real office, with clear objectives and scope, and an actual schedule. You got to actually use your brain and some creativity (not just google the 'best' answer or look on stack overflow or steal some code from github). Projects sometimes failed, almost always because of requirement failures, and sometimes the code was beautiful; most of the time it was average; sometimes the code was really bad. Today, you have to go through hours and hours of phone screens, technical interviews, team meet and greets, etc. You have to know how to implement a distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's such a time consuming process. And, the smart (or motivated) people know how to game the interview anyway (there are lots of resources out there on how to beat the interview process). And, even when you ace the interview it's still often difficult to get a straight answer on whether your hired or not (strangely more so if you're over 40). They say that software developers are in high demand... I say that people just hate the idea that software developers make a respectable wage. But, the funny thing is, after working in this field for over 15 years, after working with 100s of people and seeing millions of lines of code, the truth is that the projects are still failing, and the code is just as bad as it ever was. And, projects still fail most often because of bad requirements, poor vision, and bad management. In fact, I drink the koolaid: I hire people who ace the interview, and you know what I've found? There's a 50/50 chance that they're just as good as the person who I hired who failed parts of the interview (sometimes I'm desperate). And, those people don't bring down the project - it's the guy in the corner office. Luckily, I was smart enough to save a lot of money over the years. I'm just too old for this shit.

82 points by braythwayt 11y ago

I could have sworn I already read this, turns out the date on the article isn't correct, I think this was written in June '15: https://news.ycombinator.com/item?id=9766816 https://news.ycombinator.com/item?id=9766816 Which makes sense given that I tried out their online quiz this summer after reading this exact post. EDIT: Here's a link to when it was posted to LinkedIn, June 23rd, 2015: https://www.linkedin.com/pulse/three-hundred-programming-interviews-thirty-days-harjeet-taggar https://www.linkedin.com/pulse/three-hundred-programming-int...

fweespeech 11y ago
Yeah, that seems to be correct. I wonder why the date changed and it was reposted?
- dikaiosune 11y ago
  Yeah, I'm not sure. It looks like maybe they changed their blogging backend from whatever in-house thing they used to Medium? Their blog (https://data.triplebyte.com/ https://data.triplebyte.com/) is now showing Medium stories with publish dates from last month. Maybe Medium doesn't let you set arbitrary dates when migrating?
  - whafro 11y ago
    This is certainly true. All migrated stories get their publication dates set to "now." In my case, I added a footer on each migrated post stating its original post date.
braythwayt 11y ago
My bad! I did an HN search for the link before submitting it, I find that gets around cases where people change the title. But I could have done a title search as well.
- dikaiosune 11y ago
  Well it's clearly not that much of a problem, HN has such an appetite for what they're doing/writing about that it's back on the front page. Also for what it's worth even though I'd previously read the article I made it about halfway through before it clicked for me that I'd already seen it.
ammon 11y ago
We moved the blog to medium about a month ago. Since writing this, we've tested a bunch of new things, and now have about 20% better precision on the quiz (20% increase in the percentage that we predict will pass and go on to pass). There are some other interesting results (one is that easy programming problems are about as effective at screening as hard programming problems). I'll be writing more about this, with the new data/results in the next few months.

vdnkh 11y ago

>As soon as we started doing them however, I saw a problem. Almost everyone was passing. Our filter was not filtering. We tried extending the duration of the interviews to probe deeper and looking at code over Google hangouts. Still, the pass rate remained too high. To me, it seems like your team isn't that good at judging ability. To compensate, you rely on coding questions. This what the industry has been doing for years. What exactly is innovative here? I see the need for coding questions (and think that your method sounds pretty good) but too often culture fit & 'soft skills' are overlooked in favor of cramming as many technical questions as possible. One of the best experiences I had on an interview involved no coding on-site at all. I completed a project for them, they liked it, I got an interview. 3 hours of culture/team/work philosophy. There were no surprises - if I wasn't "technical enough", I wouldn't have been let in the door. It took time up front, but at least the interview itself wasn't waste of time, PTO, and money.

paxcoder 11y ago
So you bet your time. It's how I got my job too, but that should be the norm (unless there are guarantees). Companies should cover the expenses of finding employees, instead of crowdsourcing the unemployed.
- paxcoder 11y ago
  I meant shouldn't be the norm. Screw these uneditable, permanent forums.
ZanyProgrammer 11y ago
3 hours of soft questions seems much more preferable to 3 hours of whiteboarding data structures. Hopefully your project wasn't a bunch of questions from an undergraduate data structures/algorithms class.
- vdnkh 11y ago
  They were tough, insightful soft questions. And I asked a lot of them too. Gave me a good feel for how they worked and what they valued (which is what I really want to know). The challenge was pretty tough but it wasn't just graded on performance. The team lead came in with feedback from each member which was really good. Stuff like "you could have done this with less objects" or questions for me to resolve for them. I've used the feedback to improve my code since.

deleted 11y ago

[deleted]

minimaxir 11y ago

What the heck is going on with the statistical explanation? The chart's y-axis is labeled "correlation coefficient" but the text blurb analyzing it describes the definition of R^2 instead (% of variation of Y explained by X). The clarification at the end does not disambiguate this. Also, how do you numerically quantify "interview performance"/Y? Additionally 3 out of 6 of the variables have error bars crossing 0, which means that the correlation could be positive or negative but we don't know (it would handily fail a statistical test of significance at p = 0.05 significance). Why were they included at all? Many startups (including statistical startups!) have been playing hard and lose with statistical analysis lately and it's getting annoying.

gragas 11y ago
I completely agree. I also would not consider an r-squared value of 0.47 indicative of strong correlation between two variables. I guess the point is that the composite quiz score is more correlated with performance than traditional methods, but I still get the impression that someone is overselling something.

adambom 11y ago

I would applaud the author on trying to take a data-driven approach to discover the best hiring signals, but I must offer one major critique of the approach: These data are ultimately not too meaningful unless you correlate actual job performance with interview performance. By the author's own admission, interviews are often faulty. By correlating final-round interview performance with early-round interview performance, you are really only predicting how candidates will perform in the final-round interview, which is not necessarily a good predictor of long-term job performance.

serge2k 11y ago
Exactly. If you are serious about this then you should be interviewing candidates, deciding on whether you think they should get a job. Then giving them a job anyway and evaluating their performance after some time interval and see how accurate your interview process was in terms of weeding out those who would fail.
rifung 11y ago
In fairness, the author says at the end of the post that they do intend to rerun the analysis with actual job performance. It seems reasonable to optimize for hiring right now since otherwise nobody would go to them to find a job, and it'd also be impossible to measure job performance when the people you collect data can not get jobs.
lqdc13 11y ago
Sure, but this is still really useful for many companies in the interviewing space such as hackerrank, hackerearth etc. They're optimizing for hiring, not job performance.
- vonmoltke 11y ago
  Which seems totally at odds with what their clients want and furthers the perception that tech interviewing has become a hazing ritual totally detached from evaluating the ability of the interviewee to actually do the job.

nordsieck 11y ago

Sadly, this is almost certainly illegal in the US due to the 4/5ths rule (http://adverseimpact.org/CalculatingAdverseImpact/Four-FifthsRule.htm http://adverseimpact.org/CalculatingAdverseImpact/Four-Fifth...).

adambom 11y ago
The language in that description refers to "protected groups". As long as the author is not calculating selection rates for groups of candidates based on some special criteria, such as race, gender, age, etc. he should be ok, right?
- ThrustVectoring 11y ago
  "adverse impact" is designed to sniff out hiring criteria that are a proxy for discrimination based off criteria such as race, gender, and age. For instance, if you selected for college graduates for hiring firefighters, and that causes a lot of rejected applications among African American candidates, then it would be illegal based off of an adverse impact against a protected group. As an aside, suing employers over requiring a college degree is a seriously under-tapped and under-estimating way to charitably improve the lives of Americans (college is a positional good because college matters when employers are looking at candidates - if employers can't look at whether or not you have a college degree unless it's a bona-fide qualification, then many more people don't have to spend several years and tens of thousands of dollars)
- nordsieck 11y ago
  That's not nearly enough. The 4/5ths rule is designed to identify "structural discrimination", or cases where no discriminatory act can be identified, but the outcomes are nonetheless considered discriminatory. As long as a lawyer can show that some protected group is disadvantaged by the quiz, that's enough for it to violate the rule.
  - danieltillett 11y ago
    I am not a lawyer, but I thought the defence was if you can show that the screening criteria correlated with on the job performance it is OK. No matter what you think of protected groups using a criteria that does not correlate with performance is not very smart anyway.

johnrob 11y ago

Why don't companies manage open source projects, and simply require a certain amount of contribution as a gateway to a final interview? This would allow candidates to prove their ability to write production ready code.

branchless 11y ago
Because they'd filter out all the good candidates who don't work for free (for for-profit companies)
- johnrob 11y ago
  But they'll interview for free? They don't need to be large contributions (and if small changes require too much time, that becomes a natural filter).

droithomme 11y ago

This article is extremely interesting, but it seems to be equating success in interview decision with competent performance at the job, which has not actually been established yet. It is interesting though to see which components of pre- and early-interview evaluation ended up being related to the decision to hire.

pklausler 11y ago

I've conducted a lot of programming interviews over the past three decades. I think that it's easier to do at places like Google where the objective is to avoid a Type I (false positive) error and it's expected that a lot of time and money will be spent to identify the best hires. It's harder in other contexts where Type II errors matter more. I have a big portfolio of technical questions, but I usually stick to a small set that I consider to be well calibrated by posing them to smart coworkers and seeing what they do with them. I've found, counterintuitively, that the best indicators are the questions that deal with "programming in the small", such as coming up with a predicate expression to describe a test for a simple relationship between a few values, because either you can do it easily or you can't do it at all. And programming in the large depends on a lot of cases of programming in the small.

btilly 11y ago

I certainly hope that they don't ever assume that experienced candidates don't need a programming test. There is a particular kind of dysfunctional candidate who once was a programmer, graduated to being a lead programmer, and then became an architect. As an architect they stop programming, and their programming becomes rusty. Once that happens, the feasibility of their proposed "architectures" becomes steadily less practical. You can detect them by their first getting offended that they would be asked to program during an interview, and then failing to actually write working code. By contrast people who really do program have no objection to being asked to program in an interview. To the contrary being asked to actually program is seen as a positive sign that they won't have co-workers to deal with who can't actually program. (This is an unfortunate experience that most programmers try not to repeat.)

collyw 11y ago
I have declined some programming tests, when the employer wants yet another three hour challenge to build a mini app. That leaves not much time for actually thinking about a problem and rushing out a crappy solution. Not the way I like to write software. I have code online I can show them and talk to them about, and extend if they want to see me actually code. Bashing out code is usually the simplest part in writing a software solution to a problem.
- btilly 11y ago
  There is a difference between "demonstrate that you can program" and "waste X hours on busywork".
jkot 11y ago
I write lot of code, but I refuse to do coding at interview. For me its sign of dysfunctional company. But I do not mind to do pair coding, and fix an actual bug. There are more reasons, in short it takes me couple of hours/days to warm up and grasp the problem. And secondly I refuse to waste my time, on half arsed throw-away code, with zero results.
- 11y ago

raitom 11y ago

Why should I spend so much time with all these interviews when I can simply reply to one of the countless job offer I receive every week?

to3m 11y ago
Are these offers where you start next week if you sign the contact, or are they expecting to meet you first? If it's the former you should have no difficulty answering your question ;)
JetSpiegel 11y ago
You can start forwading those messages for a fee. Job Offers as a Service

Mc_Big_G 11y ago

"we’ve come up with a replacement for resume screens, and shown that it works well" No you haven't. You might have, but you'll have no way of knowing that until they've been on the job at least 3 months. Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development. College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time. Experience counts and if you disregard it then I'm firmly in the "Fuck you, I don't want to work for you" camp. Unfortunately, I've resigned myself to memorizing the typical interview questions or going into management. I'd love to stay a dev, but being ruthlessly quizzed about shit that you never use in real life makes changing jobs a pain in the ass.

odonnellryan 11y ago
> Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development. Aye. It's a hard problem: as an employer, you want to be pretty sure that you're getting someone on team that's going to bring you value. How do you do that? I guess ask them as much as you can. You definitely present a good problem, it is the same they're trying to solve over at triplebyte. > College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time. Yeah, you have to study for interviews. It sucks, but almost every software company asks the same questions. Just prepare, I guess, and it'll pay off with that higher salary. > makes changing jobs a pain in the ass. You might have nailed it here. Employee retention is good for everyone! I think the solution is you need connections. When someone works at the company and they know you're a good dev they aren't going to ask you questions they ask other developers.
- hamitron 11y ago
  >> makes changing jobs a pain in the ass. >You might have nailed it here. Employee retention is good >for everyone! I'd say having a good work environment is good for employee retention, not arbitrary limitations to prevent people from switching jobs.

Ovid 11y ago

I have to take exception to this because part of what I do for a living is teach tech companies how to hire. The claims are: 1. Performance on our online programming quiz is a strong predictor of programming interview success 2. Fizz buzz style coding problems are less predictive of ability to do well in a programming interview 3. Interviews where candidates talk about a past programing project are also not very predictive Those claims may be true, but it misses a key point: no matter how well someone performs in an interview, it doesn't mean they'll perform well on the job. In particular, the post talks about hard skills (and does well there), but doesn't talk about soft skills. It's absolutely true that someone can talk a big game about hard problems they faced that they solved, but failed to mention their use of Google or StackOverflow. So unique coding tests can definitely tell us whether or not a candidate can really code (though the tests need to be tailored to the sort of real work the candidate will face). However, programming is a hard skill (one that is easy to teach), and not a soft skill (one that you tend to learn over time or have a natural inclination for). Case in point: early on we made the mistake of hiring a very talented programmer on the basis of his reputation and a brief interview. He was a disaster because he was completely unable to work without a very hard set of requirements. In the very rapidly changing, agile, environment we put him in, he flailed badly because he was good at implementing requirements, but he was very bad at projecting himself into the customer's shoes. Later, we put him on technically demanding tasks which required considerable talent, but had no ambiguity, and he was fine. In fact, he was able to implement tasks that would have been very difficult for other developers who were fine in ambiguous environments. Soft skills are tricky and most employers have no idea how to interview for them. They require an understanding of the actual role someone will be hired for and the set of skills necessary for them to succeed. Are there tight deadlines? If so, can the developer prioritize tasks and make a case for delaying less critical portions beyond the deadline? Is it a rapidly evolving environment which makes it hard to nail down exact behavior? Are you on a team with strong personalities who often disagree and can the dev handle that? Is the developer required to report to non-technical people who have no understanding of technology and can't appreciate what refactoring is? Or here's a fun one for you aspiring managers: we've had one project with a superstar developer who was so monumentally productive and so brilliant at his work that some very talented developers who worked with him nonetheless had negative productivity at times because they spent so much time playing "catch up" with the superstar and understanding how the project was evolving that they spent more time keeping abreast of the code than developing it. The code was "bus sensitive" because one brilliant developer was so competent. How do you manage that? These and other soft skills are part of what you really need to evaluate because depending on your work environment, you'll have these and other soft skills as critical skills that a dev will need, but you almost NEVER see interviewers check for these because very few people are trained in interviews. Pro tip: start reading up about structured interviews. They've been proven via multiple studies to have a strong correlation with employees being successful. Unfortunately, they're harder to conduct and require training to understand them. As a result, companies don't know about them or worse, ignore them, because it costs more money. But what costs you more money in the long run? Taking longer to interview or hiring the wrong person?

lewisl9029 11y ago

I gave the TripleByte application a try very recently, but this step wasn't a part of it: > 1. A fizzbuzz-like programming assignment. Applicants completed two simple problems. We tracked the time to complete each, and manually graded each on correctness and code quality. Instead, I was presented with a single, rather convoluted, 1-hour long programming problem that was most definitely not anywhere close to the difficulty level of a FizzBuzz test (the solution they're looking for probably involved dynamic programming, which I honestly haven't had much practice with). This was before I even spoke to any humans on the phone screen or interview steps of the process. I couldn't finish it in 1 hour (I kept going at it on my own time for another 2-3 hours before I finally arrived at a sub-optimal solution, at which point I had enough). I'm sure there are plenty of people who are much better at this than I am who can finish it in an hour. But I do want to question the wisdom of using something of this level of difficulty as a screening test. Your clients seem to value the "Product Programmer" very highly. Using a question like that as a screener will weed out a significant portion of these "Product Programmers" before they ever get a chance to showcase their product skills. Most "Product Programmers" I know (myself included) don't do enough convoluted dynamic programming problems in their spare time to be able to solve a dynamic programming problem like that within an hour with any level of certainty (and yes, solving problems like that quickly involves either extraordinary talent or lots and lots of practice). Instead, in their spare time, these "Product Programmers" like to build things, and learn new technologies and techniques to help them build things in a simpler, more maintainable, and more productive way.

lordnacho 11y ago

The best programmers I've worked with had this quality: They'd always heard of everything, and they had a good idea of what everything was, even though it was quite far from the current task to hand. For instance, plenty of people have not had the opportunity to work with functional languages, yet still have a good grasp of why they're useful, what languages there are, who uses them, and so on. They tend to see coding as a vast field where you can make many valid decisions, each with their own pros and cons. The kind of people I'd root out are the ones who merely see programming as a tool. Something along the lines of "I've got this data in a spreadsheet, I need to calculate this statistic, so here we go..." They almost always end up creating an unmaintainable mess, and they never mention any well known pattern. Quite hard to identify in an interview, though. Perhaps the lack of key words that suggest a deeper understanding? (Hey, that's an interesting thought. You could machine learn what words good devs use vs less good ones.) As for coding in the interview, I think this can only root out the absolute novices. It's quite hard to think of a short test that will demonstrate that someone can't write elegant code. Perhaps a take-home, open-ended test would work, but you're gonna lose people who don't want to spend a whole weekend doing it.

ap22213 11y ago

15 years ago you could get a software job by showing some passion, intelligence, a CS degree from a average university, and some knowledge of memory / space complexity of typical data structures and algorithms. And, if the hiring manager (usually a programmer) liked you? Job offer on the spot. Sure, you got paid 50-60k / year, but it was what it was. And, you actually got to work with an entire team of people under reasonable deadlines, 40 hours a week, with a real office, with clear objectives and scope, and an actual schedule. You got to actually use your brain and some creativity (not just google the 'best' answer or look on stack overflow or steal some code from github). Projects sometimes failed, almost always because of requirement failures, and sometimes the code was beautiful; most of the time it was average; sometimes the code was really bad. Today, you have to go through hours and hours of phone screens, technical interviews, team meet and greets, etc. You have to know how to implement a distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's such a time consuming process. And, the smart (or motivated) people know how to game the interview anyway (there are lots of resources out there on how to beat the interview process). And, even when you ace the interview it's still often difficult to get a straight answer on whether your hired or not (strangely more so if you're over 40). They say that software developers are in high demand... I say that people just hate the idea that software developers make a respectable wage. But, the funny thing is, after working in this field for over 15 years, after working with 100s of people and seeing millions of lines of code, the truth is that the projects are still failing, and the code is just as bad as it ever was. And, projects still fail most often because of bad requirements, poor vision, and bad management. In fact, I drink the koolaid: I hire people who ace the interview, and you know what I've found? There's a 50/50 chance that they're just as good as the person who I hired who failed parts of the interview (sometimes I'm desperate). And, those people don't bring down the project - it's the guy in the corner office. Luckily, I was smart enough to save a lot of money over the years. I'm just too old for this shit.