8 ms·
Programming competitions correlate negatively with being good on the job
- splike 12y agoHow did they quantify "being good on the job"?
- emerongi 12y agoHave other people look at your code? If it's easy to understand, to modify etc - it's good. Otherwise, less good.
- collyw 12y agoassuming it works in the first place.
- fridek 12y agoThat's Google, so probably quarterly review scores.
- wyldfire 12y agoBased on the research I've read, one of the most valued skills for workers is communication/teamwork. I'll speculate wildly that the loners who perform well at these tasks don't have the leadership/teamwork skills that are so highly desirable.
- victorhn 12y agoAlmost all of the programming competitions are individual, one of the most prestigious (ACM ICPC) though is a team based programming contest.
- Beltiras 12y agoThe team can be lone-wolf organized thou since (for on-site competitions) each team only has one computer. The one I did we were 3 in the team. We divvied up the problems and each took turns hacking out the solution. Yes we consulted each other on the solution we were doing but mostly we worked on our own problem. This might have been a defective strategy since we missed the top ranked team by one problem, but had 3 nearly finished solutions when the buzzer rang.
- hatmatrix 12y agoThis was just a sensationalist title.
- shin_lao 12y agoAnother way to see it is that there is a world of difference between writing a program that has to work once for a competition and writing a program that has to work and be maintained for years.
- svisser 12y agoComing up with clever solutions is indeed very different from coming up with maintainable solutions.
- codingdave 12y ago... and many of the programmers I find in the corporate world do neither. But that is probably for another discussion.
- venomsnake 12y agoWhy code when you can go full dilbert on corporate politics.
- anotherThrowawy 12y agoThere's a lot to gain by respecting your coworkers and developing the communication skills required to tactfully ask about their work.
- fridek 12y agoThat's not a surprise really. A typical competition may be won by a programmer scoring 90/100 on four out of five tasks and 20/100 on the last one. Such performance at work would likely result in a quick termination.
- emerongi 12y agoI'm pretty sure the competition winners are not one-trick ponies either. They probably can crank out a solution to the last one as well (if given enough time -- like on a job). That's probably not the problem.
- Ruberik 12y agoFor the majority of these competitions, the programmer you described would have zero points. If you don't get 100/100 on a task, you get no points. The IOI is a notable exception.
- fsk 12y agoI don't see how it would be negative correlation or even zero. Winning a competition shows a certain amount of raw talent. For someone who only programmed for competitions and never worked on a serious project I could see it being a negative, but with some experience the competition winner should (on average) do better than average.
- brudgers 12y agoProgramming contests are winner take all. Contemporary professional practice emphasizes collective success, teamwork, and group ownership of code. Two processes that select for opposite traits will often have negative correlation.
- _delirium 12y agoThis is looking at a correlation among people who were hired by Google, which is a fairly specific population, and one that's actually in part selected by the same variable being studied, which complicates things further. There are a lot of possible mechanisms by which the correlation might've been produced, if it's valid. One is that winning programming competitions is anti-predictive of job performance (the interpretation this summary takes). But another is that Google puts (or previously put) too much positive weight on winning programming competitions in their hiring vs. other factors. If Google were, for example, more willing to overlook other weaknesses in people who had won programming competitions, or treated them more leniently in interviews, that would be another mechanism for producing a population of hires where those who won programming competitions were worse at their jobs.
- fsk 12y agoI see what you're saying. The programming contest winner will do better on the algorithm brainteaser questions, so is more likely to get hired, even if he's lacking the ability to do better after he's hired. A 30 minute interview question is more like a programming contest question than a serious project. It sounds his conclusion is measuring a defect in their hiring process more than "programming contest winners tend to make worse employees".
- 12y ago
- bajsejohannes 12y agoThat's very interesting, especially as someone who did pretty well on those competitions. I don't doubt the finding, but I do think that it possible to put your mind in different modes. I frequently think about "running" vs "going slow" when solving problems at work. I use the former for whipping up a single use script and sketching solutions, and the latter for permanent solution, especially in hard domains. I think knowing both modes can be useful, and knowing only one can be limiting, no matter which one it is.
- zk00006 12y agoPeople participating in competition and winning (or not) are in my opinion self driven individuals that are more suited to build their own company as opposed to be a good employee.
- skwirl 12y ago"Here are the exact specifications of the format of input and output into a console application, and the exact specifications of how the output must relate to the input. You have one hour to do this and your program must finish running within X seconds of being invoked on our test input." I can't see how being talented at this would have any connection to talent at building a company, in fact I wouldn't be surprised if it was negatively correlated. I'm not really sure talent at winning programming competitions could be positively linked to other than talent at winning programming competitions. Valuable industry tasks differ substantially from what is done in a programming competition.
- zk00006 12y agoI agree, if you consider such question I don't see any relation either. What I meant was a general competition (not strictly programming). If you participate, it means that you want to achieve something, hence self motivation. Of course self motivated employees are great for the company, no doubt about it. But I feel that too much of it may conflict with the interest of the company.
- namelezz 12y agoDriven by what? If it is driven by title then you have something like this http://www.nextbigwhat.com/indian-developers-accused-of-cheating-in-google-code-jam-297/ http://www.nextbigwhat.com/indian-developers-accused-of-chea...
- kenferry 12y agoThe hard part of really building something is keeping to it over long periods of time, though. That seems quite different from being up for a competition.
- PythonicAlpha 12y agoI think, software development is most often more like a marathon run and less like a sprint run. For a marathon run exactly this behavior is negative, that makes you win a sprint run. When you start fast and are eager to have fast successes, you are likely to loose the marathon. In my experience, software development (in larger scale) needs a lot patience, planning and great staying power.
- yodsanklai 12y agoI'd like to have more information on this. But I don't find this surprising. I imagine people that win this type of competitions capable to work obsessively on what they like, but also to be prone to ignore other things they find less challenging.
- bnegreve 12y agoSince job interviews and competitions are somewhat similar, I wonder if this also applies to job interviews. I mean, job interviews are very competitive, they are time constrained, and they result in a binary decision made by some external authority, just like competitions.
- deleted 12y ago[deleted]
- xxxyy 12y agoInterviews are incomparably easier than good programming competitions.
- cloakandswagger 12y agoYou can draw the relation to some extent. When I conduct interviews for software developers it's a 50/50 split between technical acumen and cultural fit. I've often brought on people who flunked the technical part of the interview just because they have good energy and a likable personality.
- Ruberik 12y agoThe conclusion this article offers is drawn from a badly flawed study from over a decade ago, which Google decided neither to publish nor to use the conclusions from. A machine learning system was trained at Google as an attempt to do "scientific hiring." As their ground truth, the team used the performance reviews of people who had already been hired at Google. To optimistically give it the benefit of every doubt, this study says that Google put too much emphasis on success in programming competitions when it was making hiring decisions in the early 2000s. To look at it more pessimistically, the possibility of correlations between the features cast doubt on that conclusion as well.
- orborde 12y agoCitation?
- kentonv 12y agoRuberik ran Google Code Jam for several years. (Or maybe someone registered his handle on HN just to forge this comment, but that seems like a silly thing to do.) EDIT: To be clear, Ruberik is NOT an impostor, and it's sad that someone is now going around flagging his comments.
- dang 12y ago> it's sad that someone is now going around flagging his comments. No user flagged those comments. Some comments (e.g. by noob accounts from Tor IPs) are moderated by default because of past activity by trolls. These eventually get unflagged. We're going to open the unflagging part to the community soon, but that work isn't done yet.
- kentonv 12y agoI see. Thanks for the clarification. (Seems as usual, I underestimated the complexity of HN's algorithms...) Looks like his comments are un-flagged now.
- 12y ago
- fsniper 12y agoWorking conditions and competition conditions are really different and the psychological implications of a competition has it's toll on the applicant. Target applications, time limits, thinking, documentation, help usage differs in both conditions and better results for business can not be accomplished with artificial/inorganic restrictions.
- yodsanklai 12y agoI thought that Google interview process was similar to this type of programming competitions.
- deleted 12y ago[deleted]
- hmate9 12y agoHaha, exactly! That's what I find strange too.
- deleted 12y ago[deleted]
- jroseattle 12y agoTwo different, mis-aligned objectives and outcomes from the same origination point. And optimization for one doesn't equate to optimization for the other. You don't say?
- oskarth 12y agoIt's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street, I really want that contest winner, I got to take him every time. Or her. But if it's somebody who passed the hiring bar, then they are all pretty much at the same level. And maybe the ones used to winning the contest, they are really used to going really, really fast and cranking the answer out and moving on to the next thing. And you perform better on the job if you are a little more reflective. And go slowly and make sure you get things right. -- Peter Norvig.
- gwern 12y agoYeah, this might be an instance of http://en.wikipedia.org/wiki/Berkson%27s_paradox http://en.wikipedia.org/wiki/Berkson%27s_paradox - correlations after selection/conditioning may be different than before. Not well known, but relevant to a lot of the stuff about Google's hiring in particular (eg I wasted a lot of time explaining to people 2 years ago or so that Google's HR guy saying that intelligence didn't matter to their hiring decisions != intelligence doesn't matter to job performance (it does, as has been demonstrated many times), because Google was already selecting for <1% of the population.)
- stretchwithme 12y agoIntelligence SHOULD have mattered when that HR guy was hired. Perhaps he means it in the Carol Dweck sense, that how we view our failures is what determines our success. IQ tests only measure what we've learned already, not our attitude towards learning.
- diminoten 12y agoNo, you're missing the whole point. Intelligence doesn't matter because everyone is smart who applies to Google.
- diminoten 12y agoThis just smacks of defensive justification for lack of participation. "I don't do this and I don't want to do this, so I'm going to bash it and the people who do it so I can pretend I'm making a choice instead of just being lazy". It's a little bit like the folks who say, "I'm saving myself for marriage" when in reality they've never been on a date in their lives. Pretending like people can't think in more than one mindset is an inaccurate model. I've never done an honest-to-goodness coding competition, but I can imagine someone who does code competitively to think very differently when they're competing vs. when they're designing/implementing a solution as a professional endeavor. Someone would have to explain to me in much more detail how this "negative correlation" is established before I believe it.
- zodiac 12y ago> Someone would have to explain to me in much more detail how this "negative correlation" is established I think it's just a statistical correlation they found in the hiring data.
- diminoten 12y agoI'd need to know more about the hiring data, the actual correlation method, how they collected the data, what was the sample size... Basically, like I wrote, I would need to know more.
- collyw 12y agoI have refused to do a couple of time pressure hacker rank challenges as part of a job interviews recently, as it is not how I work on a day to day basis. I was good at these sort of things fresh out of university 15 years ago. I have never had to implement any algorithm under time pressure in my professional career.
- diminoten 12y agoWhy must it be "how you work on a day to day basis"? Why must interviews only be about exactly what you're being hired to do?
- butwhy 12y agoNah. That's how Facebook hired interns in The Social Network, so it must be a great strategy.
- fatjokes 12y agoSadly, IIRC, the ICPC/IOI/TCO/GCJ/etc don't have drinking components.
- namelezz 12y agoGoogle keeps proving itself as a king of hyping. Their employees periodically make sound statement regarding to winning programming competitions as not being a good indication of effective employees but they only consider hiring those who can crack programming competition questions during their short interview session. Good for you if you want to judge a fish by its ability to climb a tree.
- sukilot 12y agoGoogle interviews are quite different from programming contests. In a programming contest, there is little time to think or talk, success is about instantly pattern matching problem to solution , copy pasting library Code, and pounding out a solution. Interviews are 30min of discussion and 20 lines of code.
- fatjokes 12y agoConsidering that playing office politics is a key part of being "good on the job", I believe the sensationalized title. The top programming contest folk I knew were as socially awkward as they were brilliant. Based on my (admitted small) social circle, this means they excel at algo trading firms. A lot of trading teams I know of are headed by former Putnam/ICPC/IOI/IMO/etc. top rankers. The politics there are simple: did you make money?
- _yosefk 12y agoA different angle - from someone averse to competitions and trading of most kinds: both have simple cost functions, unlike most of the rest of life. Yeah, it will tend to simplify politics which I think is a good thing. But in other areas a large part of one's effort goes into figuring out what to do - what the cost function actually is, into expanding the understanding of what it is. Incidentally, much of human intelligence and talent in many areas manifests in understanding what to do; it's what machine intelligence is worst at right now; and it's the thing without which machines often beat people (board games are a simple example.) So there are downsides to "simple politics" or rather simple cost functions; I for one don't care about such problems very much.
- DanAndersen 12y agoIt is funny how our industry enjoys building competitive hoops for us to jump through (see also programming interviews that you have to study for). We reject employers judging devs on how many WPM they can type, why do we do this to ourselves?
- tedsanders 12y agoNot to be gratuitously negative, but... All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias. Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Google. So when Google says things like "GPA is not correlated with job performance" what you should hear is "Given that you were hired by Google, GPA is not correlated with job performance." In general, when you have some thresholding selection, it will cause artificial negative correlations to show up. Here's a very simple example that I hope illustrates the point: Imagine a world where high school students take only two classes, English and Math, and they receive one of two grades, A or B. Now imagine a college that admits students with at least one A (AB, BA, or AA) and that rejects everyone without an A (BB). Now imagine that there is absolutely zero correlation between Math and English - performance on one is totally independent of the other. However, when the college looks at their data, they will nonetheless see a stark anticorrelation between Math and English grades (because everyone who has a B in one subject always has an A in the other subject, simply because all the BBs are missing from their dataset). The bottom line is that whenever you have some score that is some positive combination of input variables and then you threshold your observed data on a minimum total score (as is the case in hiring at Google or in college admissions), then you will see much stronger negative correlations between your inputs than exists in real life. And really, whenever you run some selection algorithm, you should hope that (on the margin) there are no correlations between your selection decision and your inputs. If there still is a correlation post-selection, that means your algorithm has left money on the table. So when Google says that programming competitions are negatively correlated with performance and GPA is uncorrelated with performance, what that likely means is that Google's hiring overvalues programming competitions and fairly values GPA. In fact, if we did a randomized controlled study (the gold standard of proving causation), I think we'd see the expected results. Just imagine - if you grabbed two random programmers from the entire population, one who had won a competition and one who had not, do you really think the competition winner would be the inferior programmer? Edit: Many other posts here are coming up with plausible sounding narratives to fit the data. "Competition winners are one-trick ponies or loners or write awful code." I encourage everyone to think critically about the data analysis itself. Edit2: From Gwern's comment, this phenomenon is apparently called Berkson's paradox: http://en.wikipedia.org/wiki/Berkson%27s_paradox http://en.wikipedia.org/wiki/Berkson%27s_paradox
- malisper 12y ago> being a winner at programming contests was a negative factor for performing well on the job The title is misleading. Being a winner of a programming competition is very different from being good at programming competitions.
- natch 12y agoI'm someone who would probably suck at programming competitions. I also suck at office politics. Yet my code is decent, maintainable, performs well, and is said by those who review it to be high quality (including well respected and competent team leads who are also hands-on coders and architects of large systems). And I have made, just my own contribution though it's in a context where other people also contribute, probably hundreds of millions of dollars for companies I have worked for. By creating creative, effective solutions to real-world problems. In some cases my not being hung up on algorithms like some colleagues has led to me see solutions that they were blind to. Not better algorithms, but completely out of the box solutions. And these solutions have worked well, and made it into products used daily by hundreds of millions of people, with price markups attributable directly to the solution. All this is to say that from the admittedly biased perspective of a non-competition winner, my experience tends to validate the implied corollary of this headline.
- rebootthesystem 12y agoWhat none of these tests, contest or not, can filter for is long term work ethic, dedication and creativity. Of course, these may not be important for all jobs. It is easy for some personality tyles to get excited and focused in the context of an event. It is a very different story to dedicate yourself intensely to a project or a codebase for years. I know people who suck at tests who blow away just about anyone due to the intense dedication to the job they can deliver on a consistent basis.
- victorhn 12y agoA contest winner may be either very naturally talented implying great creativity but not saying anything about work ethic or dedication. Or the person can be not that naturally talented, but had to work a lot (work ethic and dedication) to get good at it and win the competitions. At the highest levels though, i would think you would need all 3 to win an international contest.
- JacobEdelman 12y agoI'd be interested to know what type of programming competition is being talked about here. There is a world of difference between Hackathons and Algorithmic style competitions.
- nicolasp 12y agoThey're talking about the algorithmic competitions like ACM ICPC or TopCoder. Google have their own (Google Code Jam), which they use as a recruiting tool.
- cagilerbas 12y agoYou do good in competitions when you 'know' the question range. It certainly helps but that alone doesn't make one a good engineer.
- benhamner 12y agoThis is flat out wrong. It disagrees with every other data point I have, so I'm very skeptical with both the methodology (which is opaque) and the conclusion. From all of my experience at Kaggle (we run machine learning competitions), with our community, and from being close to programming competition sites & understanding their communities, doing great at competitions is an unambiguously positive signal. (It's worth noting that doing great at competitions is only a positive signal - the lack of competitions is by no means a negative signal). Many of our customers have found that their best hires have come from competitions. In a lot of cases, this surfaces candidates that would normally be completely overlooked because they don't fit the "top tier CS school" mold that recruiters commonly overfit to. Several companies have had a successful recruiting strategy built on poaching our top users (https://www.kaggle.com/users https://www.kaggle.com/users). Peter Norvig's criticism that "programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right" is specific to programming competitions with very short time durations (vs. the machine learning competitions that I'm used to running, which typically last months and incentivize solutions that generalize well). However, we've seen that many programming competition winners also do well on machine learning competitions, and the same qualities that aid in competitive programming (creativity, efficiency, tenacity, fluidity with tools, and the ability to build something that works) help win machine learning competitions.
- asdfologist 12y agoI think you're conflating positive correlation with getting hired vs positive correlation with doing well on the job (once hired). See oskarth's comment at the top. No one disputes that doing great at competitions is a positive predictor for getting hired.
- deleted 12y ago[deleted]
- sukilot 12y agoKaggle is new and runs deeper contests than TopCoder and CodeJam of the 2000s.
- tokenrove 12y agoI can imagine how this could happen, even though, as others have pointed out, there's no concrete evidence presented here. I was somewhat obsessed with programming competitions as a teenager, and the first time I made it to a national-level competition I was shocked to discover that most of the people at that level were more "math people" than "coding people". Most of them had no interest in building real software. (This is not to slight them; they were also way smarter than I was.) One of the reasons I stopped doing competitions is that I realized that, once you have your bag of tricks (especially things like dynamic programming, network flow, and the usual set of algorithms), programming competitions in the style of the IOI have nothing to do with programming. However, I bet winners of contests like the IOCCC or the ICFP programming contest would be strong programmers with good on-the-job performance, though. (Or even winners of voluntary programming "contests" like Ludum Dare.)
- stretchwithme 12y agoCould it be because those who win are often those who game the competition the most, even preparing their apps beforehand?
- skj 12y agoWhat's going on is that practicing for programming competitions (which is needed to perform well in that venue) gives you a leg up on Google-style interviewing skills, but not a leg up on Google-style day-to-day demands. As a consequence, among those who do get the job, those who were good at competitions have a correlation to not being quite as good at the day-to-day, since there are some in that category who would not have got the job without the competition experience.
- ScottBurson 12y agoInteresting. My only anecdatum runs directly counter to Norvig's hypothesis. Some years ago, in my last job, we hired into my group a former TopCoder winner. He turned out to be the most careful, patient programmer I have ever met. It would seem to take him a long time to get things done, and sometimes I got a little frustrated with this. But he worked for us for about two years, and wrote a significant though not massive amount of code; and to my knowledge, he never shipped a bug. Not one. I think tedsanders has the right answer.
- PetrSavelev 12y agoIf you are not careful while solving problems in programming contests you are screwed. Because if you have limited time you don't have time too find problems.