29 ms·

Three hundred programming interviews in thirty days

• avodonosov 11y ago
  > The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. Yes, this is when you have something real.
• haversine 11y ago
  Gosh I hope they are controlling for time-of-day, day-of-week, and weather considering how closely packed together those 300 interviews were.
• jameshart 11y ago
  Correlation between testing and interviewing success tells you nothing about the correlation between either of them and actual work performance.
  • kohanz 11y ago
    From the article, in bold: The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results
    • ryandrake 11y ago
      How will they gather data about the job performance of the people who did not pass the interview? Without that, the study is only halfway done.
      • jia_min 11y ago
        That's true of any hiring system though: how do you gather data on the job performance of the people who don't pass a resume screen? A coffee date? A phone screen? Missing counterfactuals everywhere. To run a full analysis, you need to hire people randomly - both people who fail and pass your hiring process - and assess their subsequent performance. This never happens in real life. However, you can still run an informative statistical analysis based on the variability in interview scores and performance scores. For example, the people who scored 5/5 on the interview should perform better than the ones who scored 4/5.
  • protonfish 11y ago
    They said they would be correlating that at a later time. This is the beginning of what will hopefully be a much larger study.
  • evanpw 11y ago
    Even so, an automated quiz that could predict with high confidence that someone would fail an interview would be a big win, saving both interviewers and interviewees a lot of time.
• jasode 11y ago
  Imo, this post did not organize its data and findings into a coherent presentation. For example... >The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz. I read that paragraph several times and I don't understand what he's actually saying. If those candidates "dropped off" on the fizzbuzz, were they also still kept for further evaluation in the following extended coding session? A later paragraph says... >So we started following up with interviews where we asked people to write code. Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks. Conversely, people who spoke about very trivial sounding projects (or communicated so poorly we had little idea what they had worked on) were among the best at actual programming. For the fizzbuzz failures to be non-correlative and counterintuitive, it means he did not reject them for failing fizzbuzz and they later ended up doing spectacularly well in the larger coding sessions. If that's what happened, then yes, that is a very counterintuitive result. What were the topics of the larger coding sessions?
  • protonfish 11y ago
    I assumed it was just a quick summary, but yeah, I'd love to see more data too.
  • ammon 11y ago
    Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.
    • jasode 11y ago
      >Doing well on the fizzbuzz problems was not very correlated. If you mean "correlation" to only refer to the population that passed fizzbuzz, then it is to be expected that the final positive/accepted interview evaluations don't correlate. Fizzbuzz was never statistically designed for that. It was designed for early rejection and not for predicting ultimate success at the end of a multi-step interview cycle. >By dropoff I mean people who left during a step, and never logged back into our site. And the population mentioned in this sentence is what I first interpreted to be included in your "non-correlation". It looks like you don't include this population. The quitters that never logged back in were not further tested by you for later stage evaluations. That's where my confusion was and now it's resolved.
• michaelvkpdx 11y ago
  Without any demographic data about who the interviewees were, this is basically useless. Are they all under 30, white and Asian men? Where?
  • donall 11y ago
    I can see that that would be interesting information to have, but why is it absolutely necessary?
    • Xyik 11y ago
      Fresh graduates are often better at programming / algorithm interviews than older experienced programmers who have been out of the interviewing game for a while. It's actually pretty important to know which demographic of users this data applies to.
• protonfish 11y ago
  It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a full-time job and a full-time life outside of work. Spending that much time to qualify for a single job is too much to ask of anyone. If it were to pass a generic proficiency certification applicable to many positions, I would consider it, but this does not scale if a candidate is applying for multiple positions.
  • odonnellryan 11y ago
    I think it depends on the project. I'm looking to switch roles, and some projects are very small: solve a simple algo question. Should take less than an hour, which is less than the average interview, anyway! So no problem. Other projects are enormous. 40-80 HRs. of work, easily (write an entire web-app!) I did one of these... did not get the offer. I would not want to complete another.
  • eitally 11y ago
    My wife is in this position now, and it's not for a programming job. She works in pharmaceuticals, in pharmacovigilance & risk management. She does a lot of data analysis & writing (the latest report she wrote for Health Canada & the FDA was 461 pages). A company she's interviewing with asked her to do a sample writing task as part of her application/interview. She hasn't decided whether it's worth it or not. For someone who has 15 years experience in the field, excellent recommendations and credentials, spending (her estimate) 12-15 hours doing pro bono work just to apply somewhere is ridiculous.
    • tlrobinson 11y ago
      12+ hours is excessive even if you don't have 15 years experience.
    • akg_67 11y ago
      I work in data analysis and I decline any "test" requests for a job or project. If you don't value my time before hiring, you definitely are not going to value my time after hiring. Also, if you are incapable of judging a fit based on resume, interviews, and just talking with me, you are not mature enough, experienced enough, smart enough and intelligent enough and have failed the test for being my potential coworker and boss. Edit: The tests make perfect sense for entry level positions and entry level candidates but once someone requesting tests for experienced positions and candidates, it shows the immaturity of the company, group, team and people who are unable to assess suitability of candidate and place too much emphasis on technical skills and tests as a crutch. The best jobs in my career were those where coworkers and boss had good relationship dynamics. Technical excellence had nothing to do with how well we worked and delivered results.
• deleted 11y ago
  [deleted]
• majormajor 11y ago
  It seems like they're only evaluating phone screen methods against their pre-designed coding interview problem? But what if there are issues with that problem? There seems to be a big assumption that "our programming questions are going to be good and predictive, even if everyone else's are bad." What if being able to describe in-depth a past (real) project correlates just as well (or better) to on-the-job performance as being able to design and code one of their artificial ones? Or what if those artificial ones just don't correlate that well with on-the-job performance in the first place? It is definitely harder to BS-detect/grade, though. They want to re-run against actual job performance in the future, that's nice, but it seems like they're throwing ideas out awfully early, then.
• notacoward 11y ago
  Here's the part that really seems to matter the most. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases). But we have to start somewhere, and basing our evaluations on how people write actual code seems like a good place.The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.
  • ammon 11y ago
    Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters
    • notacoward 11y ago
      Cool. Might want to mention that in the article, since I'm unlikely to be the only person who thinks of that.
    • musesum 11y ago
      So, the jury is out until a followup. How to measure productivity without bias? Applying Demarco and Lister's "Coding War Games" approach may work between companies, but how to apply within a single company? What would be the standard error? A dozen new hires is a lot different than 300 interviewees. What kind of skill set? My previous company contracted with Galye Laakmann McDowell to coach the engineers before an acquisition due diligence. I thought it was unnecessary because we rarely used O() notation in our day-to-day. Since then, I'm designing a machine learning product and find sections in McDowell's book extremely relevant. My takeaway is that it all depends: are you hiring an architect or a hiring a construction crew? How do you measure such different skill sets? Or, how to measure with a job that combines both?
• choppaface 11y ago
  I've done over 1000 interviews and my experience agrees with their findings that talking about a project is not a good predictor of coding. I usually left detailed resume questions for the end since so many candidates would bomb the coding part of the interview. I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority. One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.
  • notacoward 11y ago
    I really don't mean to cast aspersions on your interviewing abilities, but the common factor in all of the interviews you've conducted is you, and how you talk to a candidate abut prior projects can really matter. At one extreme you have "tell me about your last project" free-form. At the other you have highly-specific drilling down on a relevant technical issue (e.g. flow-control mechanisms in Ethernet networks) or behavioral characteristic (e.g. openness to collaboration). Which of these have you tried? Have you observed any difference in effectiveness between them? My own experience, from interviewing people who conduct interviews themselves, is that people who say "asking about projects doesn't work" just don't know how to do it right. Again, I'm not saying that applies to you personally, but what information can you provide to refute the observation?
    • choppaface 11y ago
      I totally agree with you that the "ask about projects" question requires care. And, yes, I did see differing results depending on how the discussion went. When I did the question, I've run it as both a 1-minute quickie and a 15-minute conversation. I tried my best to dig an solid technical exposition out of the candidate if I could. Typical cases where the project discussion was not helpful: * Senior engineers could often talk in great detail about things they'd built, but then would most often bomb coding questions. * A lot of MS/PhD-level new grads could give great talks (about almost anything) but again would bomb coding stuff. * Lots of ESL candidates struggled tremendously to convey basic project details, but nevertheless were solid coders. From my TAing experience in grad school, I noticed that there are a lot of ESL CS students who have poor oral communication skills but are orders of magnitude stronger at written communication. (This made strong textbooks and written references crucial to the courses I TAed). * Sadly we hired quiet a few senior / PhD-level people who could give great talks, were highly responsible, but were eventually fired for poor coding. (There were also some management disasters associated with those cases, but stronger coding skills would have helped them nevertheless). Interviewing has both subjective and 'objective' parts. The most objective data one can collect are things like completion times, the actual code written, and (perhaps) raw contextual data like a measure of the candidate's mood (were they sweating like mad?). Project discussions rarely recover solid objective data. So, I'd definitely recommend against outsourcing those questions to a service like Triplebyte, and junior employees should probably also not be spending 15 mins on projects for /every/ candidate.
• mbesto 11y ago
  > In our first 30 days, we've come up with a replacement for resume screens, and shown that it works well. What's the metric that shows it works well? > The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. And how precisely do you measure job performance? If this is achievable, I've got a line of companies out my door that would love to pay for a service that systematically measures job performance.
  • ammon 11y ago
    Measuring job performance is hard. A good analogy is measuring intelligence. The IQ test is clearly bullshit (it does not come close to reflecting the complexity and high dimensionality of human intelligence). That said, it's a useful research tool. Across a large number of people in aggregate, it can be used to learn things (like that leaded gas was a really bad idea). We'll be doing the same sort of (very imperfect) measurement of job performance.
    • rudolf0 11y ago
      That is an excellent analogy. (Not being sarcastic.)
    • acbart 11y ago
      I'll be very interested in hearing what your measures are, and how you'll account for things like employees being in a bad environment/having bad bosses. It's a complicated thing to measure. Are you guys planning on publishing any of this material in a journal or research venue, or will you keep the results to blog posts?
  • nandemo 11y ago
    For this purpose, it shouldn't be too hard. You don't need the evaluation results for deciding raises or promotions, and you don't need to compare employees to each other. You just need to determine if the hiring decision was correct. We can do it with a binary score: 0. Oops, shouldn't have hired. 1. AAA would hire again. I suppose you could make finer distinctions (need to fire this person/we regret hiring but will work on it/meets expectations/exceeds expectations), but my point is that the golden standard for evaluating hiring decisions is whether the hire is adequate or not for the job. I mean, if someone gave you an oracle or a time machine that allowed you to evaluate the employee on the job for a while before hiring them, all your hiring problems would be solved. Now, if you cannot evaluate if someone is good enough for the job, then you have a bigger problem.
• phkahler 11y ago
  The chart is kind of dumb IMHO. It's nice that they show correlation between different factors and their final hiring decisions, that doesn't indicate the actual quality of the people hired. All it does is reveal an unwritten formula behind their hiring decisions. In other words, the time to complete a programming task is being ignored in their decision process and so should be removed from the process. That's independent of how well it indicates a good candidate. Or am I missing something?
• RogerL 11y ago
  "as soon as we started doing them however, I saw a problem. Almost everyone was passing." Why is that a problem? Maybe almost everyone is decently good (as evidenced by having a string of jobs, and presumably, references), and your interviews are creating tons of false negatives.Or heck, vice versa. You don't know. You are presuming your conclusions. You have no basis to make conclusions yet, you just have incomplete data. It's iteresting data, and I'm gleefully happy that somebody is looking at this in the context of programmers (too many studies are very broad, across many different career/job type, IMO). But I think all you have right now is data. Fishing for correlations at this point is nearly bound to lead you astray. With that aside, I'm very interested in the eventual correlation with test performance and job performance. I'm biased - I dislike IQ tests, but I must admit there is a lot of research on them out there. For me personally, I perform spectacularly on this sort of test, pretty poorly in whiteboard tests, so-so in pair program to get a job, and generally top of the heap in actual job performance. It would definitely help me personally if these tests were true. Yet, still, I wonder, do they measure "get things done"? Do they measure "don't piss off the CEO/customer" skills? There's a ton of things that I think are important beyond pure cognitive skills.
  • Scarblac 11y ago
    If they want to market themselves as being great at screening candidates so that they send only the best to interviews at client companies, then a test that 90% pass isn't of much use to them.
• brobdingnagian 11y ago
  I interviewed with TripleByte and it was different from a normal job interview in some ways, but fundamentally also the same. There is just no culture fit aspect to the interview. Keep in mind that they are not addressing the problem that coding interviews do not predict on-the-job success. They are basically addressing the needs of a broken system.
• kenferry 11y ago
  This is interesting, but are they using "performance on a sample coding task" as the truth? If we knew that was a good proxy for being a good hire, we'd already be done.
• xamuel 11y ago
  >Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks. I feel like this should be listed among the least surprising things in the world. Being a good programmer is about MAKING that hard project look easy, by approaching it in the correct way!
• Naomarik 11y ago
  I got a phone interview with Harj and wasn't considered for anything further. I'm not sure what kind of hackers they were looking for, but I've been directly involved with creating the infrastructure used in marketing campaigns with the likes of CNN, McDonalds, Infiniti, and more. I've turned an idea into a company with 8 full time employees and have investors seriously interested in one of my side projects. I'm currently involved with leading a project that integrates with a large bank. I'm a full stack ruby dev learning clojure in my spare time and heavily involved with self improvement. Anyone who watches me for a moment can see that I can solve problems very quickly. I didn't care much about being selected, I have a solid job and offers coming in. Would anyone who got selected by Triplebyte care to list their credentials/achievements? My main motivation was to see how I compare against others at my current level.
  • panorama 11y ago
    I fell through at that stage as well, which I think was surprising only because their value prop is connecting you to YC startups. I'm a former early employee at a YC startup having contributed to nearly half of its codebase and other YC startups constantly attempt to recruit me. Oh well, I'm at a non-YC startup now and very happy. I guess the discrepancy lies with Triplebyte selecting engineers that Triplebyte wants, not what is actually representative of what YC startups want. Overall it's still probably a decent signal—you'd have to expect some false negatives here and there (and I'm not a rockstar dev either).
  • Akkuma 11y ago
    I made it through the phone interview and honestly, you sound like a better candidate than me. Momma always said I should be a lawyer. I have 8 years experience and talked about my personal minimal JavaScript framework I built. I didn't pass the next interview, which in my opinion their provided reasoning was kind of atrocious. In fact, I got the "wrong" feedback sent to me, which for the most part I feel I refuted, only to then get a follow up that basically they sent the wrong feedback and I fiddled around to get things working too much for their liking, despite ultimately getting them both running.
• ccleve 11y ago
  I'd really like to see how references correlate with the other factors. Did they not call references?
• wheaties 11y ago
  The skill of the interviewer in interviewing candidates should correlate just as strongly with everything else. How are you capturing that? I ask because our #1 determinant after an at home code sample is the #3 thing you find are not predictive. Are they asking critical questions on what decisions and trade-offs were made? Their past projects, can they explain well the reasoning for choice of tools used? Can they talk about what types of improvements they wanted to see in the pipeline process of build-test-deploy? I'm just surprised that this question is singled out as "poor."
• lackbeard 11y ago
  Most candidates can speak well about interesting projects they'd worked on in the past, but a significant percentage of those can't pass a coding test. Also, not being able to speak well about a past project is highly correlated with doing well on a coding test. Sounds like one or the other should be thrown out. (Or maybe only the small percentage who do well on both will go on to do well on the job?)
  • avodonosov 11y ago
    One or other should be thrown out. Or both thrown out. Or neither thrown out. Maybe only the small percentage who do well on both will go on to do well on the job. Or maybe large percentage will do well on the job. Or maybe a percentage who do bad on both will do well on the job. In short: there is no data at all about doing well on the job, so we don't know what to throw out and whom to chose.
• jordigg 11y ago
  I really think technical, or any other kind of hiring is broken. As you showed best indicators are questions (like your quizzes) or by showing "live" you can rather than a pretty CV, certificate or a fancy university name (I'm talking about tech not medicine, construction engineering or others that really require those). I have been rejected many, many, many times because the first screening (CV check by non-technical recruiter). My last example was at a well know tech startup were I had to hack my way to get noticed in order to get the first interview. The funny thing is that I was the fasted candidate to get hired + I won a company-wide award for my work at the company just 4 months after joining. I haven't finished a degree because I thought was boring and I was learning things I already taught myself before, but this fact makes my resume go down the list very fast. Because interviewers don't have time to lose and thousands of candidates to check I'm sure they will find very useful the use of technology on getting those good prospects in front of everyone else. Something I've seen many times at my past jobs is having good technical applicants, some of them are even referred by one team member and are turned down later because culture. I don't know why but engineers and technical people are more likely to fail at those than others. The surprising thing is that they check culture as the last step because those who can run those type of interview are a few and can't become full-time culture keepers. This is an enormous waste of time and resources for the applicant, the interviewers and the company itself.
  • orkoden 11y ago
    For people without impressive CVs networking is key. Go to conferences, hackathons, and meetups to meet people. This will get your CV past HR almost always.
• hueving 11y ago
  Wait, how is this telling us anything? They don't seem to be predicting what determines the best candidate by looking at how he/she performed on the job. All this is doing is predicting how well they will do in a different part of the interview. Am I wrong?
• jbangert 11y ago
  Very interesting methodology, but it would be very nice to correlate this data with long-term job performance. Interview decisions (of which of course you get more than long-term results, and they are clearer to quantify) are hopefully, but not necessarily an indicator of whether an employee works out for your company. Otherwise you run the risk of optimizing the quiz/screening process around metrics that influence your interview (i.e. test for how you personally weigh performance indicators, not for how these performance indicators actually affect performance)
  • ammon 11y ago
    I 100% plan to do that. It's a very exciting idea
  • ConfuciusSay 11y ago
    This is exactly what I came here to say. It's all well and good that certain things (quiz) correlate more to performing well in a long-form remote coding task. But part of the problem of hiring is that these test-style code questions are themselves not necessarily good indicators of future job performance. The only way to truly analyze whether a given technique is working or not is to follow the candidates through into their jobs and see which ones actually become good long term employees.
• blue11 11y ago
  I found the article very interesting, but it seems to me that the metrics are the wrong ones. You cannot treat hire and no hire outcomes equally. Let's not forget that the goal of the interview process is to actually hire people. A successful hire is worth millions, a correct no-hire decision is just avoiding further losses of time and money. On the other hand, in terms of efficiency, a candidate that wasn't rejected early is the most expensive error. Well, not as expensive as actually hiring the wrong person, but it usually takes months to figure that out. Whereas the fact that a whole man-day was spent interviewing the wrong person on site becomes obvious in a few hours. Here are the important metrics in my opinion, in order of decreasing importance: - How many people were hired? Or what percentage of positions were filled? How long does it take to fill a position? Nothing in the article mentioned how many people were actually hired. - False positives (people making it to the most expensive stage of the interview, typically a day-long on site interview, and being rejected there). What percentage of people that went to on site interviews got offers? Personally, I have always advocated processes that eliminate as many false positives as possible, even if it comes at the cost of some false negatives. Of course, you have to be careful not to filter out people too aggressively, because then you're just not going to hire anyone. - False negatives (incorrectly rejecting good candidates early). By definition that's impossible to measure exactly. However, if you are not hiring fast enough, then maybe you have a problem with your screening process. At this point you could do an experiment and relax the screening process for half of the candidates and see what happens. But it could be just a sourcing problem, that is, you are not getting good candidates in the pipeline to begin with. It's very hard to tell whether you are being too harsh and not believing enough in people's abilities (or not willing to develop talent), or you are just not attractive to the kind of people that you want to hire. Of course, all of the above is from the employer's point of view. If you are also trying to provide job seekers with a good service, then you can devise other metrics for success and for efficiency from their perspective. (edited for formatting)
• dadrian 11y ago
  >Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks. This indicates to me that either the "simple programming tasks" are not well-designed, or the the discussion about the past projects was not long enough. It still sounds like this interview process is only identifying candidates who are good at coding while someone is watching over their shoulder. However, what I find to be the bigger issue with this article is that "success" is considered to be "passed the interview". Ultimately, all this article tells us are what currently correlates with qualities Triplebyte likes to see in candidates, not what correlates with good hires. To be fair, they do mention this at the end of the article.
• apexkid 11y ago
  OPEN LETTER ================ Hi, I had a reading of the conclusions you made and I felt as if you defined a process of hiring machines to code rather than humans. So I took a few moments to read your manifesto(https://triplebyte.com/manifesto https://triplebyte.com/manifesto) (the premise on which your entire conclusion is made), and here is my take on it. 1. /"Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code." Whiteboard coding show how someone really thinks. It illustrates the though process of the person and that helps the interviewer to judge him on his rational thinking and his logical approaches. Algorithms add to this by illustrating the problem solving ability. A person may not be able to solve an Algorithm actually, but the attempt on a whiteboards speaks more than his implementation on a online platform. 2./ "Candidates deserve a consistent experience and consistent evaluation". The entire USP of an interview is the diversity which allows the interviewer to judge if someone is able to adapt to new situations and come out of his comfort zone. What you are suggesting is to change the interview process into a GRE exam which will only in-turn develop the culture among developers to prepare for that exam for 2 years. 3./"Hiring decisions should be made using a clear scoring system, not gut feelings" Most of the companies have a 3round or 4 round interview process. It is obvious enough to remove the gut feeling factor. If you wanna argue that it may be possible that a candidate got selected based on the gut feeling of all 4 interviewers then my counter argument is that he is worth being selected if he could generate that gut feeling in so many people. 4./"The hiring process should be focused on discovering strengths, not uncovering weaknesses" Agree to the point. However, the irony is that you are trying to define a particular process to hiring. I wonder if it could actually perform the "discovery" part. 5./"Candidates should be told exactly what to expect in an interview, and be allowed to prepare in advance. " So basically you want to hire the guy who has studied the most over the smartest guy in the room. From my experience, I can surely say if companies like "Google" and "Fb" used to follow that practice, I wouldn't be even writing their name here. 6./"truthful feedback on how they did so they know how they can improve" Agreed. Something that should be adapted by all companies in their recruiting process. 7./"Good programmers come from all types of background." You enforce my point with this statement. Good programmers need not be just people who could quickly write a program for a search in a large of data using hash maps but can also be people who have brilliant problem solving ability and are slow in transforming that into code, or people who are amazing in thinking software design and scalability and probably cannot remember code syntax so well. A company needs a good blend of all these people. Then only a good ecosystem to growth is created rather than just making a team of 10 machines who could transform a pseudo code into Java in 10 minutes. 8./" The software industry needs to experiment more with hiring processes and figure out what really works." I think many are already doing that by doing Tech Hackathons, online challenges, weekend projects, open source contribution references etc. So, not something new which you guys figured out. 9./"Candidates are at a fundamental disadvantage in salary and equity negotiations" Not sure what kind of companies you have surveyed. I think most well known companies maintain clear standards of salary and compensation plan. Though people will surely be flattered reading this. :) 10./"Companies should not have to make recruiting a core competency" Now you are just trying to open the market for yourself by yourself. No comments. :P Would love to hear your counter arguments. Mail me. :-)
• gwbas1c 11y ago
  Many fields require serious professional certification. You can't become a doctor unless you go through a board certification that includes simulated patient interaction. Likewise, you can't build a bridge or a dam without engineering certification. IMO, this article demonstrates the need to certify software engineers; using a process similar to the interviewing process described. Therefore, when hiring, we can skip most of the "do you know how to code" and get down to cultural fit and mutual interest.
  • brudgers 11y ago
    Professional licencing processes establish minimum standards of competence, training and education. They do not guarantee excellence and a license represents that its holder is no worse than the worst legally allowable practitioner. Half of all surgeons are below average. You don't want one of them if you have a choice.
    • vonmoltke 11y ago
      Depends on what the average and standard deviation are. If the average is shitty in the absolute sense, I probably want one well above average. If the average is stellar, I probably don;t care as long as I am not getting bottom of the barrel.
    • Kapow 11y ago
      And if you had a choice between a below-average licensed surgeon and an unlicensed surgeon? There's no process that can guarantee above-average practitioners for everyone, it's a paradox. That doesn't mean minimum standards are pointless.
    • Retra 11y ago
      Half are below the average of practicing surgeons. But the average surgeon is much better because you weed out a lot of incompetent people who can't get certified.
      • brudgers 11y ago
        Surgeons are not number 10 heating oil commodities. The difference in quality between one surgeon and the next matters when it comes to an operation's outcome. Likewise - and this is the gist of my comment - licensing software engineers might raise the minimum bar, but it won't make hiring any easier because licensing doesn't establish relevant expertise or pertinent experience. Obstetricians not thoracic surgeons are what you want for Cesareans just as you don't want a J programmer writing your CMS.
• michaelvkpdx 11y ago
  Most of the problems of coding interviews can be solved by good reference checking. Code problems in isolation will never expose how a developer will do when faced with real-world issues, users, QA, and teammates. Why are people so afraid to pick up the phone and talk to references? I'm always happy to give out my references, and always delighted to talk about the good devs I've worked with, with specifics about what they've done. Standardized tests don't work for schools and don't work for jobs.
  • vonmoltke 11y ago
    > Why are people so afraid to pick up the phone and talk to references? First, references are chosen by the candidate, so they are almost universally expected to say good things. Second, many companies explicitly prohibit their employees from giving meaningful references about former employees. Relying on a process that needs people to violate company policies or to have moved on from the role wherein they worked with the candidate is not really a good idea.
    • poikniok 11y ago
      Not to mention that in the industry I am in people will tell you the exact opposite about people. It makes sense, you want your worst people to work for competitors, and your best people to not.
• omix 11y ago
  What about personality? Just because you can code, does not mean you can work well with others.
• obfk 11y ago
  One of the most enjoyable and mutually effective interviews I had included a one-hour pairing session which was language agnostic that required me to describe how I'd implement a basic data structure. The interviewer drove and implemented. This was then followed by a day long session of pairing on real problems with a number of different interviewers. Lunch was spent with some of the other team members, where we discussed basic things like culture, day-to-day affairs, and each of our histories. This was a great approach to me, because it didn't particularly focus on anything outside of the present. We worked on solving real problems, and contributing to the project. It's a great, low-stress, method of gauging if someone has the chops for what is typically the "day-to-day" life at the given shop.
• etrautmann 11y ago
  "The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results" I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them. The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success. I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.
  • ammon 11y ago
    Lack of insight into false negatives is (I think) the major flaw in most studies of hiring engineers. For the analysis in this blog post, we just passed everyone thorough the screening, to avoid the issue. Going forward, we'll be passing 10% of those who fail the screening, to be able to track our false negative rate. My dream is to reach a scale where can do the same thing with the final hiring decisions (well, less than 10%, clearly). Someone needs to do this.
• etrautmann 11y ago
  "The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results" I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them. The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success. I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.
• dikaiosune 11y ago
  Neat idea, especially for someone like me who's trying to get out of support and into development. From Triplebytes' FAQ: "When do I meet companies? If we decided to work together after giving you feedback post our technical interviews, we'll start introducing you to the companies and guiding you through their hiring process." So, just to be clear, first you quiz/screenshare/interview with Triplebyte, and then you still have to go through each company's search process? Or do companies partner with Triplebyte to fast-track candidates who've already been vetted?
• tokenadult 11y ago
  The blog post kindly shared here reports on a startup's experiments with offering a new kind of hiring screening service. "We launched Triplebyte one month ago, with the goal of improving the way programmers are hired. . . . Well, a little over a month has now passed. In the last 30 days, we've done 300 interviews. We've started to put our ideas into practice, to see what works and what doesn't, and to iterate on our process." They are currently validating what they are doing on the first steps just against what happens at the later steps in their process: "For now, we're evaluating all of our experiments against our final round interview decisions. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases)." I agree with the blog post author that current hiring processes mostly show that "too many companies are content to do what they've always done." And the idea of a standardized, automated quiz of programming knowledge sounds interesting. But what has to happen next is to an actual validation study and find out if programmers hired by this process do better as programmers in actual workplaces than programmers hired by some other process. Regular readers of HN are aware that I have a FAQ post about this topic of company hiring procedures.[1] Company hiring procedures are the research focus of industrial and organizational psychologists, who have almost a century of research to look back on with long-term, large n studies to provide data on what works and what doesn't work for hiring capable workers. It's a shame that most company human resource departments ignore basically all of that research. [1] https://news.ycombinator.com/item?id=4613543 https://news.ycombinator.com/item?id=4613543
• abalone 11y ago
  This is a noble project but I'm concerned that it may not be a methodologically valid blind study. Specifically, in order to be a blind study of the relationship between the screening exam and technical interview performance, the technical interviewers should not know the results of the screening exam before they make their decision. While they do not state this clearly, it seems possible that since the same 2 people were conducting all steps themselves, that they were not properly blinded. Thus we cannot rule out confirmation bias in the interviewers themselves, i.e. that they were impressed by good performance on the programming quiz, not that it was an independent predictor of good performance in the technical interview. Now, maybe one person did the screening and the other did the technical interview with no information sharing in every case, but this would need to be clarified.
• deleted 11y ago
  [deleted]
• m3mnoch 11y ago
  while this process feels like it's hitting the sweet spot for finding out who can write brilliant code, that's half or less of the battle in hiring people. personally (and as a hiring manager), i feel like a majority of the hiring process is dependent (obviously) on the environment you're hiring into. hiring for that small startup? you'll want multi-hat wearing people first, brilliant programmers second. hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second. that's not to say you should hire schleps, for sure. they should at least be competent programmers. i guess what i'm saying is (despite how it sounds), hiring someone who can program brilliantly is important, but not as important as hiring someone who can navigate your company's software-making requirements successfully. firing the brilliant engineer who thinks he's more talented than everyone else in the small company so he keeps demanding to be put in charge? yup, that's a thing. firing the brilliant engineer who fights tooth-and-nail over some inconsequential feature the product team wants to change? that's a thing too. assigning a brilliant engineer to crap, meaningless work because no one else on the team wants to work with them? yuppers -- seen it. in any organization, you are either the only one in charge or you're following someone else's orders -- both of which require different aspects around working well with others.
  • mVChr 11y ago
    Hear hear, the "no assholes" rule is always good to apply, regardless of programmer brilliance.
    • deleted 11y ago
      [deleted]
  • bjourne 11y ago
    > hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second. Aka, the reason why virtually all enterprise software is pure shit.
• jqm 11y ago
  "suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks." Unsurprising. Turns out talk really is cheap and doesn't indicate one can do. I've even seen people able to maintain jobs over a period of years by talk alone without ever really doing much. Or even being able to do much.
• jqm 11y ago
  "suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks." Unsurprising. Turns out talk really is cheap and doesn't indicate one can do. I've even seen people able to maintain jobs over a period of years by talk alone without ever really doing much. Or even being able to do much.
• mwfunk 11y ago
  Preamble: not middlebrow dismissal, I really like what they're doing here and will pay attention to them going forward. This is just picking a nit that my hypersensitive self just can't resist: "Fizz buzz style coding problems are less predictive of ability to do well in a programming interview" I'm sure this is 100% true, but I thought the point of fizzbuzz-type problems were to weed out people who couldn't program at all? It's not to identify good programmers or even competent ones, it's to identify blatantly incompetent ones, which are surprisingly common even when hiring in SV. I've never personally asked fizzbuzz when interviewing because my company's hiring process seems to do well enough to not require it. However, based on what I read here it's also very good for filtering out narcissistic divas (i.e., the occasional HN posters who pop a monocle when they get asked fizzbuzz: "how dare someone ask a dumb question that is beneath me?!? Needless to say, I walked out of the interview immediately! Harrumph!"). Maybe Triplebyte's article is using the term "fizzbuzz-type problem" to refer to any contrived programming problem, but in common usage fizzbuzz-type problems are bozo filters that serve no higher purpose than filtering out bozos.
  • ammon 11y ago
    Yeah, we're just using it to mean a asking a simple programming problem (of fizzbuzz difficulty).
    • mwfunk 11y ago
      I just can't imagine any hiring process in which someone could fail fizzbuzz (or something of that level of difficulty) and still be considered a good fit for any sort of programming job. Failing fizzbuzz means someone can't program professionally, but passing it doesn't mean anything other than meeting the lowest possible standard. Likewise for anything of that same level of difficulty.
    • mkozlows 11y ago
      But if you're not evaluating the ones who fail, then what's the point of it? you're just grading for style points? What I take away from your article is that being able to solve a FizzBuzz-style problem isn't correlated with being able to do a more intensive programming task later, which surprised me. What I take away from your comments here is that among people who were able to solve a FizzBuzz-style problem, some variety of style point scoring isn't correlated with being able to do a more intensive programming task, which... okay, that's totally plausible, and much less surprising/interesting.
• BrandonY 11y ago
  I should apologize to Triplebyte in the name of those of us, including me, who immediately created fake profiles and took the test just to see if we would do well against this new rubric.
• bjt 11y ago
  It's very nice to see hiring advice based on data rather than anecdotes. But I wonder if the process described in the article is pre-selecting for people who are out of work and desperate, rather than currently employed and casually looking for something better. From the article: > Our process has four steps: > 1. Online technical screen. > 2. 15-minute phone call discussing a technical project. > 3. 45-minute screen share interview where the candidate writes code. > 4. 2-hour screen share where they do a larger coding project. Then later: > ...we can't afford to send people we're unsure about to companies Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate.
  • fsk 11y ago
    >> ...we can't afford to send people we're unsure about to companies >Does every applicant in this system really have to go through four rounds of screening before even talking to someone who works at the actual company? I can't imagine doing that unless I was desperate. And from the candidates' viewpoint: That headhunter made me waste 3 hours on screening and didn't even get me a phone interview. Why am I wasting time on this? I realized another problem. You're trying to predict WHICH PEOPLE WILL GET HIRED BY YOUR CLIENT. That's a completely different outcome than trying to pick the people who are the best workers. If the employer's process is defective, your pre-screening is just reinforcing that bias (albeit improving your "efficiency" as headhunters).
• senorprogrammer 11y ago
  Where I work, we ask candidates to do a take-home task after they've passed the first in-person interview. The take-home task is trivial, something most developers with a few years experience should be able to knock out in less than an hour. All components of it are easily googleable. The advantage is that we're building a corpus of solutions to the same problem that we can compare against each other, which is interesting. More importantly, we're building a corpus of solutions that we can then pick from to have the candidate analyze in-person, and talk us through what they see, what they'd do differently, what they like/don't like, etc. In short, we familiarize them with the problem via their own answer, and then ask them to analyze someone else's (anonymized) answer. Our sample set so far is too small to draw definitive conclusions from, but it feels better than our old ways of doing things.
• seajosh 11y ago
  I personally hate the "take home test" approach to interviewing. I've had multiple such tests that take anywhere from 10-25 hours to complete because simply answering the question isn't enough; you need to give textbook correct answers and your code must be formatted perfectly with the requisite comments and documentation. In short, it's pretty similar to an upper-level college course's final exam; however, in college, you can get a good grade with a few mistakes; in interviewing, you get rejected for a few mistakes. I'm done giving a company 15 hours of my time just to get to a first interview; this is arrogant, condescending, and completely devalues my time. The reality of hiring is you're going to make mistakes, like every other part of running a business. Even in an extended "interview" such as dating for a potential life partner, people make mistakes so I'm not sure how the hiring process can be quantified to remove said error. The interview process is so excruciating these days I often hate the companies I'm talking with. While we're at it, the skills requirements listed with jobs today are astounding. My experience is that a company wants to hire a programmer with at least a journeymen's level of expertise in 6-8 skills. If you have 5 and are comfortable you can learn the other 3, you're dead in the water. Let's be honest, the latest Javascript framework isn't that complicated. The latest NoSQL database isn't that hard to learn. The truly hard parts of joining a new company are learning how projects are managed, getting the political lay of the land, finding a sherpa to answer your questions in the first couple of weeks, and learning where you fit within the organization.
  • fsk 11y ago
    If only there was some way that people could spend a couple of years proving they had basic competence, so they didn't have to prove basic skills on every interview. Why did I get a CS degree if every interview starts with the assumption that I'm an unqualified loser?
    • meowface 11y ago
      Because, unfortunately, a lot of people receive CS degrees but are completely technically incompetent.
    • mwfunk 11y ago
      Because lots of people with CS degrees don't have basic programming competence?
• deleted 11y ago
  [deleted]
• deleted 11y ago
  [deleted]
• dj_doh 11y ago
  I've had mixed reaction to this take home exercise. I've done that some 4 times and 3 of them resulted in getting called in for an onsite (Amazon). If problem is interesting and there's something to learn. I tend to go for it. But couple of them were lame. Neither it was a good use of my time, nor it was an indicative of my skills. In near future, I'll be requesting exercises that can be treated and used as a micro library. They are free to use it and I get to post it on github.
• oliverc2 11y ago
  Another approach: http://ortask.com/a-better-way-to-hire-developers-and-testers/ http://ortask.com/a-better-way-to-hire-developers-and-tester...
• fillskills 11y ago
  There are different kinds of jobs and different kinds of work environments. Grouping all of the situations together and trying to find just a single way of interviewing people flies in face of diversity in human nature. While this research is great and startups in this field are doing well, I see this huge gap that no one so far is trying to address.
• estomagordo 11y ago
  Great. So where can I do this quiz and these small programming tests? :)
• kriro 11y ago
  Maybe I glanced over it but who did the candidates interview for? Were those just simulated interviews or are they working with companies that actually hire people? Or are they hiring for themselves? I'm very confused by what "success" means. The last paragraph indicates that it wasn't actually used for a hiring process? I see a commenter on their site has the exact same reaction. Either way I applaud every afford to improve the hiring process. However I'm a tad bit skeptical. They should release the dataset (unless I missed it) because it's pretty convenient that the results seem to indicate that hiring can be improved by a quiz which they could build and sell. I'd be interested in the following screening filter: Have a programmer at the company read through the projects the candidates supplied (as a replacement to "read CV") and then come to a conclusion of yes/no. No projects = no job offer by default. You can always think of a different approach for people with no projects if you feel like you should hire from that group. Possibly have multiple programmers read the code and discuss it.