10 ms·
We looked at how a thousand college students performed in technical interviews
- sologoub 9y agoCharts don’t have the same scale. As the result, it’s hard to visually compare these - for example, for juniors, the shapes are similar for score 4, but top tier school students are above 20 mark, while the rest are below. Interviewing is expensive, especially if you hire the wrong person, and if a given population tends to cluster around higher scores, that what you’d pick first.
- starpilot 9y agoYeah, why didn't she normalize results and combine charts for easy comparison?
- lifeisgood99 9y agoThe graphs/images do not show for me. Chrome 64.0.3282.140. Win 10
- acconrad 9y agoAfter running over a hundred interviews within the last few years, this doesn't surprise me at all. Interviewing is a skill separate from school, and if you train for it, you can become good at it, regardless of your pedigree. I actually think studying for the SATs is a better comparison - it's a test that doesn't exactly translate to real-world performance, but has huge bearing on prospects, and often the best advice for acing both (tech interviews and the SATs) is to simply do as many practice tests/problems as possible in the areas you're weakest in.
- montyf 9y agoI was always under the impression that SAT scores correlate strongly with IQ, not with practice. I had almost a perfect score[0] on my SATs, and I only used about half the time available, and I did zero with a capital Z practice problems beforehand -- many other kids took practice tests every weekend and still performed abominably. [0]: I only got two questions wrong on the entire test, both math problems. Since I was practically unstoppable at math back then, I always suspected that there were errors in grading.
- stale2002 9y agoSAT questions, especially the math questions, follow an extremely predictable formula that is told to the world in advance. Can you practice to get good at quickly solving basic algebra problems? Yes. Which means that you can practice to get better at the SAT.
- opportune 9y agoFor the record, while I agree that evidence suggests that SAT correlates strongly with IQ, your comment is a great case study in how SAT scores correlate poorly with tact
- montyf 9y agoTactful people are boring. :)
- mlazos 9y agoFor the record, I practiced math the most, writing second and literature last and scored exactly in that order. I also tracked my progress before and after practice and I got dramatically better. I really don’t think the SAT measures anything useful other than how well someone can solve SAT problems.
- misja111 9y agoI don't see how it would be possible that SAT scores would not correlate with practice. The fact alone that SAT correlates with IQ is proof, because IQ test scores themselves can be improved by practicing.
- wfo 9y agoA large portion of the SAT is knowledge of English vocabulary which obviously has nothing to do with IQ. The mathematics questions nearly all use the same basic rules and are presented in the same format, and can easily be learned and trained. SAT prep courses are ubiquitous and expensive because they work. Do people without training do well? Sure. Do people with training still do poorly? Sure. But a good SAT question would be "does this mean that training doesn't help or that there isn't a correlation between training and success?" because the obvious answer is no.
- coldcode 9y agoI find training for interviews to be funny. Passing an interview is not correlated to doing well in a job. The purpose of an interview is to determine if you will improve the team and whatever the team is doing, not prove to be a whiteboard genius. If you design the interview to get whiteboard geniuses that's all you will hire.
- jayess 9y agoJust goes to confirm my own experience with grads that the difference between schools is largely cohort, not quality of students.
- blueplastic 9y agoLooks like the server is overloaded... images in middle of the article aren't loading for me.
- ggggtez 9y agoOn the other hand, they discount self selection bias, and the bias of "qualifying". Plus the charts don't even show up, probably due to load. edit: And the bias of the interviewers who use their platform, who may not be able to attract top tier talent, and are happy to get even barely competent people... It's impossible to say since the charts don't show up and we know nothing about the methodology and motivations of the different actors.
- leeny 9y agoGraphs fixed. Just added static image fallback for when Plotly runs into issues. Sorry about that, and thanks for calling it out! Re qualification bias, we call it out explicitly in the article. Without the coding assessment before the real interview, the result may very well have been different. Lastly, re the bar, our interviewers are coming from Google, Facebook, Dropbox, AWS, and so on.
- supersas 9y agoThere is huge selection bias. The talent that already perform well in technical interviews most likely spend less time practicing (and using interviewing.io)
- leeny 9y agoWould it be helpful to update the post with school tier distribution?
- darpa_escapee 9y agoI'd be interested in seeing this.
- bdhess 9y agoI’d actually assume the opposite—- that like in most life situations, those who “wing it” will generally do worse than those who practice.
- stevespang 9y agoYeah, I used Chrome and Firefox, none of the charts show up, so these people don't even check their webpages before posting.
- InclinedPlane 9y agoThe charts were moderately cool when one of them loaded the first time I opened the page, but they're not really superior to static images, especially when they don't load at all. And when they are elements that your whole article hinges on, well...
- leeny 9y agoFixed! Thanks for calling it out.
- capocannoniere 9y ago> What this means is that top-tier students are achieving the same results as those in no-name schools This is an misleading conclusion that ignores a HUGE selection bias. I doubt top MIT CS students, for example, would feel the need to practice coding interviews on interviewing.io
- leeny 9y agoWe have a bunch of MIT students. The sad truth is that everyone is scared of technical interviews.
- rcheu 9y agoI went to MIT and knew a good number of people who were confident in technical interviews. The problems in interviews were much easier than what I got in class, or even CTY CS classes in middle school. While there's definitely good people at non-elite universities, selection bias is at play here too. For example, in the senior year case, you're just looking at people who were concerned enough about interviews to go to interviewing.io during their senior year. Most people I knew at MIT had a full time offer they were happy with from their Junior year internship by their senior year. You can actually see the impact of this in your graphs too, notice that the % of people with a 1 or a 2 in elite schools increases for senior year vs junior year. People who have been doing well in technical interviews and have good return offers won't spend as much time preparing.
- akhilcacharya 9y ago>CTY CS classes in middle school. The hell? What sort of problems are you getting? Were you doing edit distance and word break and shortest palindrome in middle school?
- rcheu 9y agoWas a long time ago so I don't remember them all, but some examples I do remember: * Print pascal's triangle up to the n-th level, with proper spacing (so that it looks like a triangle). * Make the game Go for 2 players. I got a sub problem of this during my Google interview (determine whether a piece is captured).
- akhilcacharya 9y agoI think this is a bit specious, at least from anecdotal evidence. I know bunches of people at my school (Top 50) that have interviewed at companies that ask leetcode-style questions yet very few get offers or even make the 2nd round or onsite. If you look at LinkedIn you'll see the vast majority work at Cisco, IBM, SAS, Amazon etc and not Google, Jane Street, or Facebook. The quality of the cohort and, to a lesser extent, the quality of their DS&A classes matters quite a bit.
- Ancalagon 9y agoIs the Amazon interview process no longer considered a top-tier difficult interview?
- akhilcacharya 9y agoAs someone that has gone through it, no. It is possible to get intern and new grad full time offers by passing one online debugging test and one coding test (in rare cases a single one!). Also the comp is a league below the others I mentioned.
- akhilcacharya 9y ago*with that said, Amazon does work on a lot of really revolutionary stuff that I definitely wouldn’t be able to work on in 90% of other companies. Really enjoyed my internship there.
- readams 9y agoMaybe what they've proven is that technical interviews are not adding much information to the hiring process. This pretty much matches with my experience that quality of interview is at best only very loosely correlated with actual ability as a software engineer. Much more relevant is direct experience working with them. School quality is a strong signal.
- Tade0 9y agoI'm leaning towards blaming this on the interviewing process. Last year I spent some time interviewing job candidates and what I learned from this is that the recruitment process in IT, in general, is broken. A typical candidate is first faced with some kind of task to weed out the lower 20% of applicants. After that comes the proper interview during which the candidate is asked various questions like what was their major or how is the abbreviation "SOLID" expanded. In my experience, none of this correlates highly with future performance and this may be the reason why regardless of the school they attended those students performed roughly the same. Time and time again the one thing that seperates the best from the rest is their ability to perform code reviews. I have yet to find somebody who's a poor developer, but a great reviewer.
- dbcurtis 9y agoInteresting. Do you have candidates review code?
- owyn 9y agoI have done this. Print out a page of code and review it with a candidate. Have them explain what it does, or how they might go about figuring that out, and what kinds of improvements they might make. Include some glaring bugs or code style problems if you want to ask those sorts of questions. I feel like small fragments of code work well for refactoring questions, whole programs tend to have a bunch of boiler plate code, but going over a bigger program to find the real core features quickly is also a valuable skill. I've done this with SQL too. Printed out a real statement from an application and ask about performance, optimization, what the application might be doing, etc. Or load a 100,000k line application in their favorite editor and ask "okay, try to find the frobzing subsystem, I want to know how it frobz bazzes". (I haven't done this yet...)
- rowyourboat 9y agoI have and it has, so far, never mislead.
- Zxian 9y agoWe have a sample repository with a simple, but broken, web application. We ask our candidates to review the code as they would if a coworker had submitted it as a merge request. it gives quite a bit of insight about what aspects they latch onto as issues of note. Do they catch the insidious, subtle issues or just the low hanging fruit? Do they understand what stdlib function arguments expect as input? Do they just pass your review through a linter and call it a day?
- BigChiefSmokem 9y agoPeople have forgotten how to train their core social and psychological skills like resilience and confidence. Unfortunately, these are not things we are born with or can code our way out of. These are things we have to train very hard at similar to how martial artists or athletes approach their training. The thing most programmers and knowledge workers in general are missing is true belief in their own skills (you have been doing this how long? come on snap out of it) and the resilience that comes from rejection after rejection, and failure after failure. After a certain point in your experience you should come to understand that raw skills are not really the end all, be all. If you are not the guy then you are not the guy no matter how many degrees you have. Sometimes the guy with the PhD from MIT is not the right guy because he doesn't connect with the rest of the team, or the vision, or whatever, simple as that. Also, if you're testing only for specific skill sets rather than aptitude and interest then you already failed as a hiring manager.
- chrisweekly 9y agoAmen. Also "hire for strengths, not to avoid weaknesses".
- jeff571 9y agoPeople who bother to practice for coding interviews (using the same website) and successfully land an interview do similarly well on coding interviews... That's a lot of selection bias.
- metaphor 9y ago> What this means is that top-tier students are achieving the same results as those in no-name schools. When NYU and Arizona State are "Top 50" while Michigan State and Vanderbilt are classified as "no-name schools", I question the meaningfulness of this blog's baseline and what it means for an interview to be technical.
- chrisweekly 9y agoWait - are you talking about basketball?
- kenhwang 9y agoI think the bucketing of schools by overall rank is about as good as arbitrary. Why not rank by strength of CS program? If we use PayScale's "Computer Science Majors by Salary Potential"[1] as an indicator for interview success, it looks very much like the CS rankings. [1] https://www.payscale.com/college-salary-report/best-schools-by-majors/computer-science https://www.payscale.com/college-salary-report/best-schools-...
- bradleyjg 9y agoThat's not very far off from general prestige. Harvard is too low, but only UCSD, UW, and maybe CMU wouldn't be on the list if it were that instead. And flipping it around missing from that list are University of Chicago, Johns Hopkins, and maybe Northwestern. (Also Caltech but it's tiny.) For Phd programs there are some real surprises, but for undergraduates there's not much in the way of major specific strength differences. That's largely because the strength of an undergraduate program is mostly a function of the strength of the matriculants not faculty.
- metaphor 9y ago> Why not rank by strength of CS program? If we use PayScale's... CS alone is far from representative of tech industry in the aggregate. Furthermore, PayScale's list clearly doesn't control for locality bias--e.g. consider Purdue #191, UW Madison #138, UM Ann Arbor #88, UT Austin #79 to name a few; or what Duke #10 while UNC Chapel Hill #67 even means?? How do top programs which feed industry around tech hubs like Austin and Research Triangle meaningfully compare to the Bay Area and NYC if unadjusted gross salary serves as the sole basis of value?
- elgenie 9y agoGarbage in, garbage out. What is used in divvying up a self-selecting cohort of undergrads performing on technical interviews in 2018 are the US News & World report rankings for graduate schools composed in 2014 "based on a survey of academics at peer institutions" (https://www.usnews.com/best-graduate-schools/top-science-schools/computer-science-rankings https://www.usnews.com/best-graduate-schools/top-science-sch... … and I don't think the lack of that link in the article is an accident). Having a selective grad school doesn't mean much if anything for the standards and teaching in that school's undergrad program, especially for state schools with giant undergrad populations and relatively small grad programs. For example, Illinois' grad school is selective and highly thought of by professors and thus it's treated as an "elite" school and students of UCSD's undergrad CS program are classified as "top 15". Regardless, my alternate hypothesis would be "students of a specific level of confidence in their ability to pass a technical interview use interview.io for a limited period of time in which it provides value to them … that population of students receives a certain distribution of scores".
- ericmcer 9y agoThis really makes no sense. Even if every school had identical curriculum and teachers, students who manage to get into an MIT or Stanford have to have work ethic and prior experience that places them well above the average CS student. The fact that these students have already been vetted by a respected school is one reason why I think companies prioritize them. I am all for big data analysis disproving common conceptions, but this feels off.
- rxhernandez 9y ago> get into an MIT or Stanford have to have work ethic and prior experience that places them well above the average CS student. You're comparing apples to oranges. The worst at MIT is not necessarily any better than the average at some state school. I've seen more than a few people with bachelor's degrees from UCSD and UCLA be below the class average at Cal Poly for their second bachelors or masters degree. If there's any metric you can rely on it's that their average is usually better than the average of another school.
- bitmadness 9y agoOne data point: I was at Georgia Tech for undergrad, now at Caltech for a CS PhD. Both are good schools, but Caltech is definitely considered more "elite". As a Caltech student, I definitely attract attention that I just didn't get at a Georgia Tech student.
- ordu 9y agoThere is one problem with conclusions, they assume that interview score is the same as real world success of a candidate. It is not so, interview result correlates with future success but there is no strict determinism. So companies prefer students that successful in interview AND come from good school, rather than just rely on interview results. It allows them to maximize probablilty of the finding a good employee.
- smt88 9y agoThis tells us nothing about the talent or intelligence of the candidates because these tests are bullshit. I wish companies like this didn't exist to enable these types of interviews.
- samueldavid 9y agoi wish i can upvote this more, bless this comment to the top
- GreaterFool 9y agoWe all know that interviews are terrible. Wouldn't that be the problem? Then students from top-tier universities achieving the same interview score as student from lower-tier universities is meaningless. The interviews fail to capture what matters. The best jobs I had and also jobs I performed the best I got through credentials and experience, not scribbling on a whiteboard. If you went to top 10 school in the world, you worked hard (at least it used to be this way). I don't need to look at specifics of what you did but I know you needed grit, good work-ethic, personal time sacrifice, etc to get through. Unless you're super smart. In either case it adds to your personal brand. I think that's what perpetuates the system. Given unlimited time I'd be happy to give everyone a shot. But time is precious so stick to what you know? I know how things are in the Uni I went to. If a CV landed on my desk form someone who did the same course, I'd put them on top of my list.
- stmw 9y agoThis sounds reasonable enough, but on the other side, so is the observation that often people who get into and get through top 10 schools are actually ones that did not work hard. The point of using interviewing.io online technical interview as the first step of the process is that it makes your "I'd be happy to give everyone a shot" take far less time than alternatives. Still can't get unlimited time or perfect fairness, but it's better.
- opportune 9y agoI went to a "top 10 school." I'd say that only maybe 20% of the students (in CS) were people that I would want to hire if I were running a company. A lot of top school students get accepted because a) mom and dad are rich, sent kid to feeder high school, pushed kid to do ECs/do tutoring/get help with essays b) they fit into some admissions "bucket" like {has inspirational story} or {huge volunteering effort} or {also amazing artist}. The worst students were always the ones that went to feeder schools and had wealthy parents, which are incidentally the majority of students at top schools
- stmw 9y ago
- didibus 9y agoI started wondering why CS interviews aren't more based around reputation and experience, like other jobs. I've seen a lot of bad devs get hired into places with supposedly high bars, or devs being let go and then ending up at Google next. While some places often let go of good devs, I'm talking of cases where I believe it was justified.
- _pastel 9y ago> Indeed, statistical significance testing revealed no difference between students of any tier when it came to interview performance. Ah, the classic confusion between not finding enough evidence to prove there is a difference, and finding enough evidence to prove there is no difference. The juniors from elite schools in particular have fewer 1s and 2s and more 3s and 4s than the other juniors. Really, you found statistical evidence that they aren't from different distributions? I'd love to see it.
- _xczx 9y agoI would love to see more recruiting at state schools and community colleges etc from tech companies. There are a lot of hard working people who could be great to work with but didn't have the family background they needed to be noticed.
- kriro 9y agoI suppose a bit of methods nitpicking is in order. Wouldn't it make a lot more sense to use equivalence testing if one wants to write a statement like "Indeed, statistical significance testing revealed no difference between students of any tier when it came to interview performance." AFAIK statistical significance testing cannot reveal this at all. Of course there is no real explanation of the method that was used besides the fact that it was some sort of "statistical significance testing". Equivalence testing makes more sense to me if one wants to essentially say that MIT is Aspirin but Ohio State is a generic drug that is similar enough to work just as well (for a reasonable definition of similar enough).
- auganov 9y agoThe graphs are super confusing. Without reading the story it's hard to figure out if the point is their sameness or if you should keep trying to figure out the deltas. Given that sameness was the point - just make a single graph with 4 colored lines.
- erdo 9y agoOh come off it, for an ounce of credibility can they maybe keep the scale of their graphs the same? And the link to the "statistical significance testing" is nothing of the sort. I'm calling bullshit.
- blackflame7000 9y agoI failed my first 2 tech interviews and Google and Snapchat leaving me feeling cheated by the dumbass whiteboard. Whiteboard tests are tailor-made for the inside the book thinkers. The ones that stop learning when the chapter ends and soon forget when the next subject begins. Computers are the thing that gets you to the thing. You want people who can see that the software they are working has a broader impact beyond 1s and 0s. PS. 5 Years later, I'm now the CT0 of my company managing 50 people. So srs, not srs Googs you missed out =P
- falcor84 9y agoNot that I think that whiteboard interviews are the best approach, but I'd say you're being quite too harsh. In particular, the type of student that you mentioned who soon forget every topic they ever learned would surely not do so well on a whiteboard interview. Or if they were able to "relearn" every aspect of algorithmics in the week before the interview then they can't be that bad. In any case, a good(!) whiteboard interview absolutely can focus on all of that broader impact you mentioned and get the interviewee to discuss the UX/efficiency/maintainability/etc of what they're creating.
- blackflame7000 9y agoWhile I can see your point, I am of the mindset that Computer Science is too vast to expect someone to memorize it all. Furthermore, it's not even a memory test its a form of hazing, to be honest. It has little indication of how well someone will do in a role as dynamic as software engineering. Im certain I could make you look like a fool in front of a whiteboard and im sure you could do the same to me. But have we actually proven anything about who is smarter or more capable? No. I look for people who know where to find the answer even if they don't know it off the top of their heads. At least that way I know I won't spend half my day answering their questions if I hirer them. The kids who spend their time in college depending on the book and the professor to hold their hands aren't the ones who succeed in the long term. It takes ingenuity and creativity for that. That's who I want on my team.
- solomatov 9y agoAs far as I understand, the interviews they conduct are preliminary interviews, and questions on such interviews are much easier than questions on on-site interviews. So, basically they test for simple CS/programming tasks which any person who studies CS should be able to perform.
- lifeisstillgood 9y agoA serious question- is there a syllabus of interview questions ? That is something that says the most important subject areas are X, and this puzzle covers X (I am not sure how one decides if finding shortest substrings is important or not ... perhaps most common usage?)