24 ms·

Lessons from 3,000 technical interviews

• henrik_w 10y ago
  I thought the most interesting finding was that completing Udacity or Coursera courses on programming/algorithms (for non-top school graduates) was highly predictable of strong interviewing performance.
  • cema 10y ago
    Well, those people were self directed and had a strong interest.
  • ubernostrum 10y ago
    I know some schools and bootcamps now have a separate "How to pass coding interviews" unit in their curriculum (since this is a separate skill unrelated to how to write good code on the job, sadly). If those courses include such a unit, it would explain the performance on stereotypical tech interviews.
  • jupiter90000 10y ago
    They took the course to pass the interview; is that surprising? :-)
• allThumbs 10y ago
  I feel like things are operating according to the following pattern: 1. Go to college: a. spend many semesters in lectures all of which tangentially brush upon the final exam based on the whims of the lecturer. b. cram for final exam last minute panic to crunch memory according to advice on content which was brushed upon during lectures. 2. Interview for job: a. cram for interview by going to coursera to crunch memory according to interview memes based on the whims of the interviewer. b. spend the rest of term of employment exercising skills which tend to be tangentially brushed upon during both interview and schooling while the majority of actual tasks are often googled and stack-overflowed into place based on arbitrary design decisions and politicized stack choices. 3. Results: a. good interviewees have learned appropriate memes to reassure interviewers. b. good students have learned obligatory cruft to reassure professors. c. actual necessities are tangential to many or most entry barriers. How accurate is this?
  • cema 10y ago
    Is 3a "good interviewees..."? Perhaps you are right. This is a usual argument against canned exams.
    • allThumbs 10y ago
      You are correct, sir. Updated accordingly. EDIT: Wow, this is downvoted? Really?
  • henrik_w 10y ago
    Very accurate in my opinion.
  • jmagoon 10y ago
    completely agreed, which is why it seems like years of experience has no correlation to technical interview performance, despite the likelihood that there would be some correlation between years of experience and actual job performance--because memetic technical interviews and what the job actually entails are generally miles and miles apart.
    • mattkrause 10y ago
      On top of that, I would assume that the bar is much higher for candidates with years of experience, either because the more experienced candidates self-select for more senior jobs or because it sets up some expectations from the the interviewer ("he's been a tech lead for 12 years and didn't know about quick sort!").
  • vdnkh 10y ago
    > the majority of actual tasks are often googled and stack-overflowed The more I've grown as a developer the less I've had to do this.
    • existencebox 10y ago
      Ah damn; and I had been doing so well mushing down my imposter complex :P I've found the exact opposite, the longer I dev the more I look for things. I've found it entirely impractical to keep the esoterica of dozens of different languages, frameworks and tools in my head at once. What little memory I have is typically reserved for thinking about abstractions, arch/features, and core concepts, and I let myself lean heavily on lookup resources for things that don't generalize. Just today I'm writing in py+full web stack+cpp+bash+powershell+sql(both PG and ms), and using full suites of associated libs. I at the very least am entirely unable to keep all of that in my head at once, I'd be curious if you find this same problem with bringing in a wide breadth of components. (My honest fear is that I've pushed too far to become a generalist and am sacrificing strength in depth because of this, although I am somewhat proud of the end to end results it lets me accomplish)
• jventura 10y ago
  It was a very good reading, but I wonder how interviewing performance relates to job ("real") performance?
  • chewyshine 10y ago
    If the interviews are "structured", the correlation between interview performance and job performance is probably around .3 based on meta-analyses of research findings. If the interviews are "unstructured", the correlation will be within sampling error of zero.
    • JamesBarney 10y ago
      Do you have a source for this? I totally believe you, but I've been trying to find good empirical studies and authors to read on interviewing.
      • ryanworl 10y ago
        Not the person who originally replied to you, but the HR class I took in college broke down the correlation of interview performance to job performance into a few finer-grained buckets than just structured and unstructured. Sorry, no sources here, as I've since gotten rid of that textbook. Structured vs. unstructured are the fairly general categories, but that correlation number is accurate. The best predictor I can recall (~0.6 IIRC?) is a "work sample test". You can Google that and find all kinds of HR resources about it. That is HR jargon for, in this case, writing code to solve a real problem under normal conditions for that company. That is, on an actual computer, not on a whiteboard, and not being constantly scrutinized throughout the process.
        • JamesBarney 10y ago
          Yeah from what I've read the two most effective predictors of employees were work-sample and GMA tests(basically IQ).
      • chewyshine 10y ago
• chewyshine 10y ago
  Top school is probably serving as a proxy for intelligence in this analysis...a well known predictor of both interview and actual job performance.
  • nickbauman 10y ago
    Top school is also serving as a proxy for wealth and privilege. Good grades are also a proxy for wealth. Kids who have rich parents get private tutoring.
    • FT_intern 10y ago
      The privilege to value education and to spend time studying, yes. Almost all resources can be found online nowadays. If you've ever been to an SAT tutoring class, you know that the only benefit the kids have is the benefit of being forced to take practice tests. A top school is a good signal for how much time someone spent studying in high school, except for affirmative action students who get into top schools with much worse scores and GPA.
      • _w6l2 10y ago
        > A top school is a good signal for how much time someone spent studying in high school, except for affirmative action students who get into top schools with much worse scores and GPA. For the truly competitive schools, the affirmative action students still need very high scores and GPA, they just tend to be given a little more of a pass on the extracurricular activities and essays in the application. I'd argue that for the most part, students of any demographic have to do some pretty crazy things to get the attention of top schools. The probabilities of an affirmative action student being granted admittance are just much higher than for an equivalent non-affirmative action student, but that doesn't mean that affirmative action students can get away with having poorer grades and scores. I'd argue they just have to be less well-rounded.
      • rhizome 10y ago
        A top school is a good signal for how much time someone spent studying in high school Assuming all high schools are the same.
• Bahamut 10y ago
  It should be noted that these technical interviews are biased to a particular style, so the data only really is of relevance for these types of interviews.
• boha 10y ago
  Sad to see so much detail paid to the data, and so little to the setup of the experiment itself. It shouldn't be surprising that an online technical screen favors candidates who've participated in a MOOC, but is blind, say, to years of experience. A screen like this is timed-performance-at-a-distance, which resembles MOOC participation. The full spectrum of qualities that comprise a Good Hire might incorporate the other signals from the post, but this type of interview won't test them. (I'll be the first to admit I'm biased against performative coding in engineering interviews. Tech screens like this are often necessary, though, so they have their place.)
• bhntr3 10y ago
  I wonder if "took courses" could be a stand in for "prepared heavily". It seems like people with all the other attributes might think they didn't need to study. People without them might think they did and took courses to "catch up". In my experience, preparation is the key driver of performance in these types of interviews. It seems reasonable that a person who took a MOOC might have prepared in other ways as well while people who didn't probably didn't prepare much at all (since watching a few Algo lectures seems the most accessible refresher.)
• lintiness 10y ago
  "I’m excited to see that what mattered way more than pedigree was the actions people took to better themselves." so a degree from a top school is not earned (nor are admissions i guess), but rather conferred at birth? i beg to differ. the commentary on the "disutility" masters degrees is even worse.
  • adrenalinelol 10y ago
    I don't think it's entirely that black and white. If an undergraduate degree from a tier 1 school didn't cost approx. a quarter of a million dollars, along with the fact that your "peers" with rich parents will have attended a private school + tutoring + usually had a decent home-life; these things stack up, you can't claim it's a level playing field. Of course you need to be smart to get there in the first place (in most cases). But it's a far cry from a pure meritocracy.
    • lintiness 10y ago
      you've described a range and complex combination. the author of the piece stated -- almost word for word -- that degrees from top schools are not earned.
    • jonathankoren 10y ago
      See also: legacy admissions.
    • nsp 10y ago
      For what it's worth, financial aid at top tier schools is generally fantastic
• collyw 10y ago
  Interesting and surprising, especially the experience thing. I think I am a significantly better engineer than earlier in my career, so I assumed experience would count for a fair bit. Then again I have inherited projects from experienced guys who make crap high level architecture decisions and the code is way more difficult to work with than it ought to be. But then this article seems to be measuring interview performance, not actual ability on the job. So is any of it actually relevant at all?
  • leeny 10y ago
    Hey, author here. You're absolutely right. We hope that with time and data, the interview process can get better at predicting on-the-job performance -- one of the things we're really interested in is seeing how predictive different interview questions can be. To do that, we're also working on collecting data around what happens after interviewing.io users find jobs.
    • techman9 10y ago
      What are some metrics that you expect to use to measure on-the-job performance?
  • hibikir 10y ago
    Most of the things that make us better during our career have nothing to do with programming faster: If anything, they can make us program slower, and do worse in interviews. For instance, an interview I took last year came with a premise that I found downright bonkers. It was based on code the company had in production, but it stopped being a good problem to look at years before. I was having trouble coming up with the right tradeoffs for the implementation because all my experience was telling me that the entire approach was misguided in the first place, so the problem should not be solved. I passed, but it was a far rougher performance than I would have liked. There's also how being far from college makes the least important knowledge gained fade away, and the least important thing I studied was memorizing algorithms. I write new algorithms at work sometimes, and I implement off the shelf ones too, but I don't have to recall them off the top of my head. Nobody has to implement distributed consensus algorithms under time pressure, or write HyperLogLog from memory. So ultimately, there's what easy to measure, and then there's what is valuable and important. We go with what is easy, and those are things that are taught in college. Understanding the right level of testing or designing a system for observability are far more valuable in the long run: It's crazy how much downtime in well known companies comes from people not learning those things in college. But since we are bad at measuring those things, and kids right out of school don't know them, we don't interview for that. And sadly this is why we all end up hiring by network so much: We can't tell if someone is good in a day, and we can't really ask people to dedicate two weeks to work with us in a probationary period if they have real jobs, but we sure can recall quality former coworkers and ask them to join in.
• faitswulff 10y ago
  It's rather shocking how much effect Udacity/Coursera had on interview performance - more than graduating from a top school or being employed at a top company: "...only 3 attributes emerged as statistically significant: top school, top company, and classes on Udacity/Coursera."
  • edblarney 10y ago
    It's not just 'impact' it's 'correlation' (or both). Universities don't teach programming, so it's not surprising you don't get applied skills out of the box with a degree. 'Good courses' are focused directly on teaching specific programming skills. I think they are a good idea because learning programming by 'osmosis' (as you go along) can result in 'not knowing' a lot of key things, even when they are right in front of your face the whole time. Most importantly - people who are going to take the time to specifically learn new skills, are displaying the kind of conscientiousness that you want in your talent - learning the actual skills is another benefit of that.
    • user5994461 10y ago
      > Universities don't teach programming, so it's not surprising you don't get applied skills out of the box with a degree. I disagree with this statement. Some universities teach programming, some not so much, some only need a Java 101 to graduate. There is a huge variance. We'd need a statistical analysis accounting for the quality of education given.
  • chewyshine 10y ago
    It would be interesting to see if all 3 predictors of performance held up in a simultaneous regression. Most likely, the predictors are highly intercorrelated. If so, then only one or none would be significantly related to interview performance in a multiple regression analysis.
  • Alex3917 10y ago
    > It's rather shocking how much effect Udacity/Coursera had on interview performance Having done Udacity and also taking a couple CS classes at Cornell, this doesn't surprise me at all. People who take Udacity classes are doing so voluntarily on their own time, so they are going to generally be smarter and of higher socioeconomic status. If you look at people who take CS classes over the summer at college when they don't have to be and when there are no student loans, you're going to get a similar population.
• blazespin 10y ago
  I am perplexed why anyone would think that interview performances has any interesting statistical relevance. Much more interesting would be how successful the candidate was after receiving a job at the company.
  • munchbunny 10y ago
    For two reasons: 1. From the perspective of a job seeker, the interview is what gets you the job, so it's in your interests as a job seeker to learn how to look better as a candidate based on data about how interviewers judge candidates. 2. The purpose of an interview is to be a predictor for job performance, so while this article doesn't address this part of the question, you want interviews to predict job performance, and how to make interviews do that is interesting.
  • throw_away_777 10y ago
    To get a job at a company you have to be given the opportunity to interview, and to succeed at the interview. If you only evaluate performance at the job level, you don't evaluate performance of anybody who failed the interview. This is one of the fundamental challenges in hiring - you have a biased sample when trying to determine how effective the hiring process is at screening candidates.
  • crispyambulance 10y ago
    You're right. It is weird that so much effort is spent on interviewing and so much effort is spent on performance reviews. But AFAIK, hardly any organization tries to use performance evaluation of current employees to inform their hiring decisions.
  • inopinatus 10y ago
    Most hiring managers would hope that interview performance is at least partially correlated to subsequent job performance. And once experience is gained as an interviewer and team manager, it is not an unreasonable hope.
    • JamesBarney 10y ago
• ditonal 10y ago
  Very unsurprising for me. You are measuring your ability to solve algorithm puzzles. Most engineers don't actually do many algorithm puzzles in day-to-day work, especially the types of algorithms that interviews tend to focus on like sorting and dynamic programming. So "years of experience" is not measuring experience in what you're actually being tested on. On the other hand, you do exactly those types of things in many CS classes, and in Coursera classes, algorithms are exactly what you practice. So it makes sense it correlates. Top company is a predictor for the obvious reason - it's selection bias for people who already passed those interviews at the company. You're not good at the interview because you worked at the company, you work for the company because you're good at the interview. Master's degrees seem like largely for international students needing visas, career switchers, etc so not surprised they are not a strong predictor. And if anything the course material moves past the intro data structures stuff the whiteboard interviews tend to test. The only huge surprise for me here is that Coursera is a stronger predictor than top company and top school. I would have predicted top company > top school > Coursera. The post that I would be much more interested in is correlating performance reviews to interview performance. That gets suggested as a possible future post.
  • leeny 10y ago
    Yup! We'd love to close that loop and are working to collect as much on-the-job performance data as we can. Please stay tuned.
  • semicolon_storm 10y ago
    Coursera might top them because it's a good indicator that they're actively trying to round out what they know, which means they likely studied up on algorithms right before the interview.
  • marcosdumay 10y ago
    Those riddles are second semester problems on CS, thus anybody that isn't an student spent at least the last 4 years not working on them. The same probably applies to top companies, people probably interviewed there more than 2 years ago. But Coursera stuff that gets into a CV is almost certainly recent.
• geebee 10y ago
  Interesting bit on the MS degree. I followed the link, and I'm not quite as surprised that the correlation is poor, or even negative, given the way the data was collected and analyzed. Absolutely agree that some MS degrees are pretty much less rigorous cash cows by now, that allow students to skip the fundamentals such as data structures, operating systems, and compilers. However, many CS MS degrees actually do require this as a background, to the point where some programs have emerged to prepare non-CS majors for MS degrees, kind of like those post-bac premed programs. It's hard to believe that those MS degrees, which require a decent GPA in those core courses, along with high GRE scores (sorry, but we are talking about interviewing skill, which may be more related to exam taking ability than job performance), wouldn't result in a similar profile to people with CS degrees from top schools. This is fully acknowledged in the text of the article referenced in a link, but unless people follow it, I do think the message may be a bit misleading. That's an aside, though. The value may very well be in the prep for these degrees (ie., the post-bac CS coursework required for admissions to a reputable MS program). If you can get that through online courses (udacity or coursera) through genuinely rigorous self-study? Yeah, that might do it, for far less money. I've audited a few of them, and they're the real deal, that's the real coursework there.
  • sporkenfang 10y ago
    I'm curious what would happen if they looked at, as separate categories, people who mastered out of a doctoral program, people who did the 5-year BS/MS, and people who obtained a doctoral degree as well as a master's.
    • adjkant 10y ago
      Agreed completely. I would suspect that BS/MS and Ph.D. interviewees would be very distinct from a straight MS, but this was a missed opportunity to look at that, even if it is just interview performance.
    • geebee 10y ago
      That could be useful. The article mentioned rhat there was no distinction made between CS masters degrees and Info Systems degrees. My guess is that pure MS would correlate positively from CS programs that require a background in more advanced math and algorithms. Info sys degrees typically don't. I'm not knocking those degrees, keep in mind we're measuring performance on a technical interview, which often leans very heavily toward algorithms, trees, sometimes graphs, and occasionally numerical analysis or computing. If the MS degree data includes a lot of programs that dont include this background, then this woukd not be a surprising result.
• analog31 10y ago
  Something I wonder is how the participants in these interviews were selected from the general population of job candidates. Painting with a broad brush, the best workers might not even be candidates, because they've already been hired. And the best candidates might be the least likely to seek coding interview practice.
• fecak 10y ago
  Thanks for writing this Aline. As a recruiter for almost 20 years, I wish I had access to all my data and then the time to compile it, and anecdotally I'd expect the finding about MOOCs would be similar. The most selective of my hiring clients over the years tended to stress intellectual curiosity as a leading criterion and factor in their hiring decisions, as they felt that trait had led to better outcomes (good hires) over the years. MOOCs are still a relatively recent development and new option for the intellectually curious, but it's not much different than asking someone about the books on their reading list. Unfortunately, demonstrating intellectual curiosity often takes up personal time, so someone with heavy personal time obligations and a non-challenging day job is at a significant disadvantage. One could assume that those who have the time to take MOOCs also have time to study the types of interview questions likely favored by the types of companies represented in this study. Thanks for continuing to share your data for the benefit of others.
  • moxious 10y ago
    > Unfortunately, demonstrating intellectual curiosity often takes up personal time, so someone with heavy personal time obligations and a non-challenging day job is at a significant disadvantage. This is true, and one of the things that tends to stack disadvantages against people who are older in the industry. There's a good correlation between free time and age in some ways, as people get to mid-career they tend to marry, have kids, and so on. Maximum free time comes during college and immediately after, for many people. Valuing intellectual curiosity also tends to mean that this curiosity is valued over direct experience, which is what the older more time-strapped people have to offer relative to younger people who are doing the MOOCs.
    • fecak 10y ago
      I see this pattern time and time again in both my recruiting work and in my resume business. Someone gets a rather non-challenging job that pays market rate or better, starts a family, pays little attention to any tech outside their day job, and suddenly finds that their marketability outside the current job has suffered greatly. As developers get older, they need to be conscious of the relevance of their work in regards to current trends, and if the day job is largely irrelevant they may need to find access to projects (often unpaid) that will serve to impress new employers. I don't think that the value of experience has truly diminished, but I think what is now classified as 'experience' may have shifted a bit.
• serge2k 10y ago
  That graph is about the best evidence I've seen that interviews are garbage. Years of experience doesn't matter at all? Coursera courses are the best thing ever?
• forrestbrazeal 10y ago
  The author draws a hard distinction between Udacity/Coursera MOOCs (good) and traditional master's degrees (bad). I'll interject that with Georgia Tech's Online Master's in Computer Science program [0], which is delivered via Udacity and insanely cheap [1], you can get the best of both! (Their "Computability, Complexity and Algorithms" class is one of the top Udacity courses cited in the article.) Keep in mind that a traditional degree program does have a huge advantage over a strict MOOC: accountability. It sounds good to say that anybody can go push themselves through one of these courses. Try pushing yourself through ten, and actually writing all the papers and implementing all the code, while working full time and having a family. That grade looming at the end of the semester really does wonders for your motivation. Plus you can get help from live professors and TAs, and the Piazza forums for OMSCS are full of smart, curious students who love talking about the subject at hand. There's a richness to the degree experience that I don't think you get with scattered classes. (Obvious disclaimer: I'm a current OMSCS student) [0] http://omscs.gatech.edu http://omscs.gatech.edu [1] https://www.omscs.gatech.edu/program-info/cost-payment-schedule https://www.omscs.gatech.edu/program-info/cost-payment-sched...
  • lordCarbonFiber 10y ago
    Jumping in as another OMSCS student to second your post. The accountability really makes a difference finding the balance between work/life and the schooling (which can be nontrivial considering the courses are certainly rigorous).
    • nkozyra 10y ago
      Same here. There's much more at stake and it helps push you. Grades and money are extrinsic motivators, but they get you from point a to b.
  • jonathanfoster 10y ago
    $6,600 [1] for a master's from a top 10 computer science program [2] is an absolute steal! My master's program costs that per semester. [1] https://www.omscs.gatech.edu/prospective-students/faq https://www.omscs.gatech.edu/prospective-students/faq [2] http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-science-schools/computer-science-rankings http://grad-schools.usnews.rankingsandreviews.com/best-gradu...
• k2xl 10y ago
  Interviewing is a funny thing. I remember when I graduated from a "Top School" and interviewed at "hot startups" from the valley. I aced a lot of the interviews - why? Because I had just taken classes on LinkedLists, Binary Trees, HashMaps, etc... So when they asked me to whiteboard a "shortest path algorithm" it was just rehashing what I did in school. Years later, looking back, I fail to see the relevance in most of the technical questions. In fact, if I had to do those questions over again today I would probably fail miserably. Yet, I have been in the industry for a while now and have worked with countless more technologies and have accomplished far more than my younger self. Just because someone performs well in a technical interview doesn't mean they will do a good job. That is the data that really matters. I've interviewed hundreds of candidates as a hiring manager for some big startups, and from my experience technical interviews are not a great indicator of success. I'm saying this coming from someone who has gone to a "Top School" and done multiple Coursera/Udacity/etc classes. Yes, someone might be able to whiteboard a random forest or write a merge sort, but do they know how to engineer a system? Can the candidate: > Communicate well with others in a group? > Solve unique technical problems? > Research and learn new technologies effectively? > Understand how to push back to product owners if there's scope creep? etc... These are all things that are not really analyzed in many technical interviews. As I'm reading this analysis all I can think of is that it is pretty useless - if not dangerous for the industry. What I've found is that it is critically important that someone knows how to code at some basic level. But their ability to code and explain algorithms on the fly, while probably relevant in academia/research, is such a minor part of the day-to-day of a programmer - At least from my experience.
  • phd514 10y ago
    That has been my experience, too. I remember deriving the optimal solution for an interview question that was a variant of a radix sort and impressing the interviewer for one of the hot startups at the time. That was partly the result of my ability to recognize the similarities between the problems but even more a consequence of having recently studied radix sorts. Twenty years later, I'm a much better developer than I was then and have shipped many projects with significant degrees of complexity, but I couldn't answer that interview question or ones like it as well today as I did then.
• ggggtez 10y ago
  Interestingly, they suggest that if you attend a to school, the effect of Udacity is negligible. I'd argue that Udacity is this filling in gaps of a poor education.
• ma2rten 10y ago
  Until recently I worked at a startup as Machine Learning Engineer/Data Scientist. There I got some experience interviewing people and looking at their resumes. In my experience, which is very limited compared to this post, people who put an MOOC on their resume are usually less qualified compared to people who don't. There is nothing wrong with MOOCs, but they are almost always beginner-level. If you put them on your resume it kindof implies you don't have a lot of experience beyond that. Putting the Coursera Machine Learning course on your resume would be the equivalent of putting Java 101 on your resume for a Software Engineer. I would recommend anyone to put projects on your CV instead. Even if you don't have a lot of work experience, just put side-projects and school projects on there.
  • jghn 10y ago
    I think the point here are folks taking MOOCs for enrichment, not as a gateway to get a job. So for instance someone taking the ML course just for knowledge, not to try to get in the door as a data scientist
    • ma2rten 10y ago
      Yeah, like I said, there is nothing wrong with MOOCs, but I would not put them on my resume as a candidate nor would I value them as a positive signal from an employer's point of view when I see them on a resume.
    • pmiller2 10y ago
      I don't think we can assume that, unless the participants were specifically asked this.
  • zump 10y ago
    Elitism.
  • deleted 10y ago
    [deleted]
  • stolsvik 10y ago
    That's absurd. Everything in a masters degree is "beginner" when you've worked 10 years in a field. When I took my MSc in computer science, there hardly was any "big data" and deep learning around. That you actually would fail me for taking a course in this field is utterly beyond me.
• conqrr 10y ago
  Slightly Off-topic, but does anyone have an invite for this platform? I have been trying to get one since ages.
• Apocryphon 10y ago
  Not to harp on the "technical interviews are disconnected from actual work!" angle too much, but I'm reminded of a comment from a thread about the creator of Homebrew failing a Google interview. Someone pointed out that it goes to show that it's possible to create widely-used software without an intimate knowledge of CS. I wonder if that's a disconcerting fact for some employers to grapple with.
  • edblarney 10y ago
    To be fair, the creator of a bit of software that is widely used may not necessarily be a good fit for Google. For many reasons. I find some creators put a lot of ego into their products, not that they are egoists so much, but doing something on your own can be emotionally challenging. I don't mean 'ego' in a negative way - just that when you are on your own you have to make so, so many decisions. APIs, compatibility, etc. etc. you're often a little overwhelmed, you often have to 'just do it' and 'go with your gut' in many areas. Which may or may not be a good practice/attitude for a Google employee. They might have a stronger sense of the 'outcome' (i.e. what it does) than the capacity for the underlying code. Maybe it 'does the job' but is poorly written? Which again, might not work in Google environment. ... and working on a very large team is night-and-day from working on your own. I think we all want to 'feel it is unfair' that such an important contributor somehow 'can't get a job at Google' ... maybe he's worthy, maybe not, but I can totally grasp someone mighn't be a good contributor at Google. Maybe it's easier to see it from a non-technical standpoint: it's often very difficult for founders of their own companies to get along in a place like Google. I think that's easier for us grasp. The same thing can apply in the technical domain.
  • jorblumesea 10y ago
    Google's specific interview standards only select for academic rigor, not just successful software skills. Look at how many failed or poorly designed products and libs Google has released.
    • Apocryphon 10y ago
      I've always said there seems to be a weird relationship in tech between "the industry" and academia. On one hand, there's a growing disconnect between the everyday API gluing-frameworks galore-trudging through someone's code instead of rolling your own nature of day to day work vs. the clean algorithms of school. On the other, there's a youthful, even immature, vibe of all-nighters and casualness for the sake of casualness and Silicon Valley perks. Maybe this is what happens when an industry is built around worshipping whiz kids. If CS in SV was about venerating NASA/aerospace code, less flashy but very very stable, well-engineered stuff that lives are dependent upon, it'd look a lot more like EE and other more traditional engineering fields.
• lgleason 10y ago
  While I do think that interviewing is broken, I would love to see the raw data with this. For example, did Udacity courses have other related traits associated with them, IE: did these candidates that also have a certain number of years of experience, degree etc.? 3000 is a small sample size and I'm wondering if there is some sampling bias here.
• AlexCoventry 10y ago
  If you know me, or even if you’ve read some of my writing, you know that, in the past, I’ve been quite loudly opposed to the concept of pedigree as a useful hiring signal. With that in mind, I feel like I owe clearly acknowledge, up front, that we found this time runs counter to my stance. Did the interviewers have access to the applicant's resume? If so, to what extent do these results simply reflect the interviewers' bias for top schools and famous companies?
• closed 10y ago
  Interesting article! Some minor statistical pet peeves: 1. Setting non-significant bars to 0 seems fishy. Leaving them and putting confidence intervals on everything would let them speak for themselves. 2. Calling something effect size is ambiguous. That's like saying you measured distance in units (and the wiki article on effect size linked makes clear there are a billion measures of effect size). I'm guessing their measure of effect size were the beta coefficients in a multiple regression?
  • eanzenberg 10y ago
    Not minor at all. It's very easy to mislead intentionally or not with statistics and that's why rigorousness is needed. No details are given for the analysis, even basic things like definition of what "top university" or "top company job" even means. It's just amazing to see how many positive comments a post like this gets without even the hint of methodology.
• janeandrews 10y ago
  If you have infidelity issues like I had, I strongly suggest you hire a pro hacker with 100% success, he helped me out with several deals and I am sure he can help you out as well.He is truly a genius at what he does. It was worth the fee. You can reach him via his eMail: vietnamguru@programmer.net
• pmiller2 10y ago
  Was there any kind of statistical correction applied when the data were partitioned into MOOC + top school vs MOOC + no top school?
• Eridrus 10y ago
  Huh, I hadn't bothered to list MOOCs on my resume since I didn't think employers would be interested, maybe data like this will make employers more interested in the courses, which would probably get more people to shell out for the certificates.
• deleted 10y ago
  [deleted]
• marsian 10y ago
  Is this guy a paid shill for academic friends trying to boost enrollments and overcome the disillusionment of the younger people who realize too much emphasis is placed on academics and not enough on practical application? The world needs more vocational schools and trade schools and technical schools than it does colleges and universities.
• grogenaut 10y ago
  Unless I missed it in the article the data is all about passing the interview not acutally seeing if any of these things correlate to the employees working out in the 1,3,5 year time spans. With this data you're just biasing towards people who interview well, which, I don't think you actually care about. Well I mean I guess you do if you're a recruiter (if you're a moral recuiter you care about both), but not really if you're an employer.
• sundvor 10y ago
  Searched the article and the comments here for "Pluralsight", with zero hits. So what makes Udacity/Coursea preferable? TLDR, I'm asking this as Pluralsight was a significant contributor to my landing my latest role after redundancies. The long version: I recently landed a role after some time off, having changed from mainly back end Php/Coldfusion to C# in the last year. I was able to make the switch in my last role. For me, moving to C# was a big transition; as well as guidance from a (fantastic) mentor, I used Pluralsight to learn C#, asp.net and DDD - e.g. from Jon Skeet, Scott Allen and Julie Lerman, to mention but a few. Being completely burnt-out on the old stacks, I was set on making my next role a C# one. I've come to love what Microsoft are doing with Core, open sourcing etc, as well as the strictly typed C# language and ability to use NCrunch with live unit tests. So I signed up for a year after relinquishing my corp subscription, kept doing their courses, and found the training material highly accessible with great quality content. Each interview was a learning process, when I didn't know something from a test, I'd go and study it so that I'd be better prepared for the next role. One of these was the study of data structures and basic computer algorithms, where I was lacking. I might not have had years of experience, but the experience I had was mostly best practice. During my search, I typically got great feedback on the fact that I was doing Pluralsight courses, and it was a significant factor in being hired for the new role - it showed cultural fit, in addition to passing their tech tests (which happened to involve structures). My company had interviewed a lot of candidates, struggling to find the right talent. Just possessing technical skills is one thing, having the right attitude towards learning is another. At any rate, I'll keep using Pluralsight to raise my proficiency in my new stack - even as an old timer, I am having a newfound level of enthusiasm towards my whole profession which I haven't felt since I coded in assembly on the good old Amigas. I would be interested in knowing why Coursera / Udacity might be better or more accepted in the marketplace though.
• bootload 10y ago
  "We got this data from looking at interviewees’ LinkedIn profiles." Verification of completion and award id? There are a lot of individuals who will add a degree regardless of attendance, completion or award. Who validates the assertions?
• graffic 10y ago
  "Whether passing an algorithmic technical phone screen means you’re a great engineer is another matter entirely and hopefully the subject of a future post." This sentence plus the inverse correlation between experience and "interview performance" shown there. Makes a big smell about how biased are those interviews to themselves and not to real technical interviews. From the data it looks like the questions asked using that service are the ones you might learn in university and after many years not using them, that knowledge fades away because you're not using it. This is reinforced by MOOCs being the 101 of the subject they're dealing with. It would be interesting to see if there are trivia questions from 101 courses. The most obvious bias is in the clickbait title. Those 3K interviews are in a specific platform, meaning they're done in a specific way. So after checking their results it seems that interviews done using that service benefit people with fresh university or 101 lessons knowledge. What worries me more is the lack of improvement and perhaps the moral superiority of ending the article with a "these findings have done nothing to change interviewing.io’s core mission". It feels like the entire statistics game shown there was to feed back what they already knew.
  • wheaties 10y ago
    Yes, because we all know that your ability to implement a red-black tree from memory has a direct correlation with your ability to implement some random business logic or CRUD app. Oh wait, they don't.
    • blindhippo 10y ago
      I used to think along these lines. Then I started doing 10+ interviews a month and realized a very clear reality: basic CS knowledge and problem skills is far more important to me and my team then knowing how to slap together some semblance of a working CRUD system. I ask "algorithmic" questions, normally expressed as a legitimate business case (invent a real world problem, solution is implement some algorithm or use specific data structure). My warm up question typically is a simplistic "find the subset in a given collection that matches this specific criteria" (with a subtle implication of "do it efficiently"). The average coder should be able to solve this type of thing, on their own, in about 10 minutes max, 15 with some feedback on improvements. Yet, 80% of my candidates take nearly 45 minutes and cannot deliver a workable solution without massive handholding, and I don't even get to my higher order, "real questions". The scenario of a coder who can't solve my warm-up being let loose on code I actively maintain makes my stomach churn. Until I see the average bar for problem solving go up, I'm going to keep asking basic CS questions in my coding interviews. The job is to solve complex, typically ambiguous problems. Coding is one of the tools - and I want peers who understand the theory behind using those tools. (I should note, I tend not to pay attention to credentials on a resume. I care more about ability to do the job then past history - though if a candidate has a masters in some field of CS, I might delve into it a bit out of curiosity... they are an expert afterall)
• ordinaryperson 10y ago
  The master's in CS can be useful if: 1. You have an undergrad degree in liberal arts 2. You pay as little tuition as possible 3. You take no time off and continue to work FT These apply to me -- my undergrad was in English, I paid 6k total (27% of the 21k total cost) and went to school at night over 4 years while my career continued to progress. Most of the people in my program couldn't write a FOR loop if their life depended on it, they viewed it (incorrectly) as a jobs program while the school needed the $$ to keep the dept afloat, so I'm not surprised they fared poorly in technical interviews. But that doesn't mean the degree isn't useful. If you're already a programmer, it helps get your foot in the door at many places. HR managers/recruiters feel more confident forwarding on your résumé, they can't parse your GitHub repos. The degree is icing on the cake, it's not going to magically turn you into the Cinderella of Programming if you have no real-world experience. I got my master's with a QA and a paralegal and today? They're still a QA and a paralegal. That being said, timed technical interviews are almost universally asinine, IMHO. When in real life do you have 10 minutes to figure out a problem? Or are prevented from Googling the answer? The measure of successful programmers is how efficient and professional they are in problem solving, not how much useless information they can keep in their head. Things I've never had to do in 'real' life: -Never had to split a linked list given a pivot value -Never had to reverse a string or a red/black tree -Never written my own implementation for Breadth First Search etc etc Personally I'd rather see take-home assignments that roughly approximate the type of work you'd do, which in my career has been churning out new features or applications. Does knowing the time-complexity of radix sort vs heap sort really have a material impact on your effectiveness as a programmer? No.
  • popcorncolonel 10y ago
    > The measure of successful programmers is how efficient and professional they are in problem solving They are not asking you to split a linked list given a pivot value or reverse a string without using a builtin function because that's what you going to be doing on your job. They're doing that so they can see how efficient you can be given a new problem. That's why companies have interviewers sign NDA's - so their interview questions escape as little as possible, so it's not just problem memorization.
• sytelus 10y ago
  yes, this is absolutely startling: For people who attended top schools, completing Udacity or Coursera courses didn’t appear to matter. (...) Moreover, interviewees who attended top schools performed significantly worse than interviewees who had not attended top schools but HAD taken a Udacity or Coursera course. Possible explanation might be that people going through regular degree typically spread themselves thin over many subjects (digital electronics, compiler design, OS theory, networking etc) while MOOC folks sharply focuses on exactly the things for interviews (i.e. popular algorithms). Its like interval training for one specific purpose vs long regime for fully rounded health. The problem here is not academic system but how we measure performance in interviews. I highly doubt if results would be same if interviewers started asking questions from all these different subjects instead of just cute algorithm puzzles.
• Futurebot 10y ago
  The takeaway from this is that those who do best are those with: - the wealthiest/most financially supportive parents/relatives - upbringings that are conducive to academic success - the most free time as those are the ones who, by a large margin, attend top schools, work at top companies, and have time to spend on self-learning. Another data point of confirmation of a well-studied idea. Assortative mating: http://www.economist.com/news/briefing/21640316-children-rich-and-powerful-are-increasingly-well-suited-earning-wealth-and-power http://www.economist.com/news/briefing/21640316-children-ric... Few poor at rich schools even all these years later: https://www.nytimes.com/2014/08/26/education/despite-promises-little-progress-in-drawing-poor-to-elite-colleges.html?_r=0 https://www.nytimes.com/2014/08/26/education/despite-promise... Why people care about elite schools: https://medium.com/@spencer_th0mas/why-america-cares-about-elite-schools-411798386ba7#.yream7cae https://medium.com/@spencer_th0mas/why-america-cares-about-e...
  • mgraczyk 10y ago
    I did not see in the data set where the interviewee's family wealth or upbringing were mentioned. Would you mind providing a link?
    • Futurebot 10y ago
      "Having attended an elite school" is a proxy for that. Info available at 2nd link.
      • morgante 10y ago
        It's not that good of a proxy though. There are still tons of poor students at elite universities.
        • jacalata 10y ago
          From the article >In 2006, at the 82 schools rated “most competitive” by Barron’s Profiles of American Colleges, 14 percent of American undergraduates came from the poorer half of the nation That doesn't sound like 'tons'.
• maverick_iceman 10y ago
  This is a very poorly done analysis. At a minimum she needs to define top school/top company. Also I'd like to see the confidence intervals around the effect sizes. In addition, looking up MOOC information from LinkedIn may result in a lot of false negatives. (She doesn't mention if MOOC courses in non-CS subjects count.) Did all the interviewees have CS degrees? What about the Masters degrees, is she including non-CS ones? Is the sample of interviewees representative or there's any selection bias that we should be aware of? A study which doesn't answer so many basic methodological questions is garbage.
• shanwang 10y ago
  I'm not surprised that MOOCs are a big factor, people like me who have left school years ago have forgotten how to write a BFS, we need something to brush up those knowledge. If you run statistics against using sites like careercup, you may find that being the top factor.
• memracom 10y ago
  I think you are seeing the effect of people who have decided for themselves to pursue lifelong learning. The Udacity/Coursera thing just clusters these people in a way that you notice them in the stats. But remember that statistics do lie. You need to dig into the reality behind the numbers, and question whether you are measuring all the right indicators. My experience comes from several decades developing software and from time to time, hiring people. The people that worked out best, either as colleagues or hires, always seemed to be learning new things and were ahead of the curve trying out new techniques or tools before they became popular. If you understand how a tool/technique becomes popular as the mass of software developers wrestle with new problems and finally find a way to master them, then it makes sense that constant learning makes some people stand out of the crowd. They happen to be the first ones to learn the new tool/technique and if they do not introduce it to their development team, then when management does make the decision to introduce it, the folks who know how to drive it have a chance to excel and appear to be rocket scientists.
• acjohnson55 10y ago
  On the master's front, I went down a slightly unusual path. I enrolled in a master's program in music technology at NYU [1]. I already had a master's in engineering from Princeton [2], but after time away from the software world, I wanted to retool for a return to engineering, but with a focus on applications that actually mattered to me. It turned out to be a very expensive, but very fulfilling decision, and it paved a route for a very successful past four years. Compared to my first master's, it was less theoretical and much more project-based. In that sense, it was fantastic preparation for career work, because every semester, I had to conceptualize and ship 4-5 different projects in all sorts of subject areas. The value of that shouldn't be underestimated. It also directly led me to cofounding a startup that had a brief lifetime, but effectively converted me to a full-stack engineer. Today, I don't use much of the subject matter I learned in my day-to-day, but I draw on the creativity, problem-solving skills, and work patterns every day. My Princeton program was great too, but I thought I'd share about the NYU program, as that was the more outside-the-box choice. There's something special to be said for a master's degree, when it's interdisciplinary and let's you focus on the intersection of engineering skills and subject matter expertise. [1] http://steinhardt.nyu.edu/music/technology http://steinhardt.nyu.edu/music/technology [2] http://ee.princeton.edu/graduate/meng-program http://ee.princeton.edu/graduate/meng-program
• chvid 10y ago
  I have been through an interview process many times; I have never been asked a technical question / asked to do on blackboard or on computer problem solving. I guess that that form of interviewing is simply not common in my neck of the woods (Denmark). I am curious to what sort of questions / tasks are actually given to the interviewee? And are they in any way biased towards more textbook/academic ones? (I.e. "implement bubble sort" rather than "create a blue button").
  • ascorbic 10y ago
    Out of interest, what sort of questions do you get asked in interviews for Danish tech jobs? To answer your question, when I have interviewed people the tasks I have set have always been job-related. I set pre-interview at-home tasks, as well as smaller tasks during the interview. I did this for the same reason I avoided whiteboard exercises: I wanted it to be realistic. One example I used was to implement an app that shows sunrise and sunset times. They were allowed to do this in any way they wanted, using open source libraries if they wanted, and add whichever features they wanted.
    • chvid 10y ago
      I do a lot of short term consulting contracts. Typically for 6 months or so. My role is either Java/JavaScript senior developer or architect. In the general the interviews runs for 30 minutes. The first 15 minutes, the client presents themselves and the particular project they are working on. Then I spent typically 5-10 minutes running thru my resume; focusing on any relevant previous projects. If there is any technical talk it will be based on the projects in my resume or the client's project. The most technical it will get would be pros and cons of particular technology choices. I have never been asked a "quiz question" - that is a question where the interviewer knew the correct answer in advance.
• sgt101 10y ago
  When you are interviewing for a specialist post (and most posts are specialist to some degree) you are looking for evidence that the candidate can do that particular job. Therefore a course that indicates that they have the particular skills required is highly desirable!
• kenoyer130 10y ago
  We really need a further correlation between people who pass the interviews and job performance a year later. I do a lot of interviewing at my current job and we have found no strong correlation at all between CS skills and actual ability to "get things done". We toned down the CS type questions since they tend to take too long. We still ask a few basic tree and string manipulation questions to weed out the people who have no idea how to program and get insight into how the person thinks. I still feel at the end of the day we could flip a coin on accepting an interview candidate once they have shown basic competency and have the same results. I have been telling candidates that a public github repo with a nice commit history carries much more weight with me then a CS degree since we have been burned so many times before.
• pklausler 10y ago
  I conducts lots of tech interviews for SWE positions, and as everybody's boning up on algorithmic trivia, I've learned that I can get a stronger hiring signal by asking simpler questions that people with an aptitude for programming will succeed on and people with an aptitude for memorizing the implementations of algorithms will not. (Simple example: given two closed intervals [a..b] and [c..d], how do you compare the four values to determine whether or not the intervals overlap? You may laugh, but it defeats about 50% of candidates in the first minute of an interview because they just don't understand simple relationships and Boolean expressions.)
• clark-kent 10y ago
  Basically a "bad" programmer that can't write maintainable code that prepares for technical interviews by brushing up on algorithms and whiteboard style questions will do better than a very good programmer with lots of years experience.
• Mister_Y 10y ago
  One of the reasons I got hired by Airbnb is that I took MOOCs, but I also believe that most of my knowledge comes from reading books and that's a thing I didn't put on my CV. So, even if showing interest in learning opens for you a huge amount of opportunities, I think you actually have to go deeper than just enrolling on a couple of MOOCs.
• deleted 10y ago
  [deleted]