9 ms·
Being good at coding competitions correlates negatively with job performance
- WrtCdEvrydy 6y agoBut is the opposite true, and can you prove it by binary tree optimization?
- OpieCunningham 6y agoHow much time do I have?
- logicslave 6y agoDoubt it. Big companies use competition coding problems as a proxy for IQ, which is the strongest predictor of success at work and in life. Theres alot of data internally at FAANGs that the leetcode questions work.
- Ensorceled 6y agoBeing good at coding problems, even leetcode questions, is a different thing than being good at coding competitions. This isn't a "dunk".
- Raidion 6y agoDoubt it. Big companies using coding problems as a proxy for knowledge of coding, which turns out to not be a strong predictor of success at work and in life. There is a lot of data internally at FAANGs that being good at coding competitions correlates negatively with job performance.
- logicslave 6y ago:)
- ska 6y agoThe first two statements you make are incorrect. Well, I'll give you that you might find a big company who thinks of it this way (but they are confused). There is a lot of data at FAANGS about hiring, but it will be structurally very hard to infer anything about broader activity from it, even if you had all the data.
- zaptheimpaler 6y agoMaybe it was a proxy for IQ back when leetcode type interview questions (and websites to help you prepare for them) were less popular. Now that they are so common, its just another specific skill that candidates can grind, so it measures preparation as much/more than IQ.
- deleted 6y ago[deleted]
- Barrin92 6y agoHaving participated in competitive programming and comparing it to development work it feels to me comparing chess tactic puzzles to classical chess. If you're a good classical player you're probably reasonably good at puzzles, but the opposite is not necessarily true. Competitive coding, despite superficially involving typing code into an editor, has almost nothing to do with working on large pieces of software. It's a lot of rote memorisation, learning algorithms, matching them onto very particular problems, and so on, it's more of a sport. Just like playing too much bullet chess can be bad for your classical chess I can honestly see how it gets into the way of collaborative work.
- cupofcoffee 6y agoThat's not an ideal comparison. Chess is 90% tactics, whereas development job isn't 90% competitive programming.
- permo-w 6y ago"Chess is 90% tactics" is a pretty subjective statement
- deleted 6y ago[deleted]
- cupofcoffee 6y agoI didn't come up with the quote appears in Chess circles a lot. I think a chess master said it.
- jmchuster 6y agoHikaru Nakamura's take on the statement https://www.youtube.com/watch?v=IrN16c3DTzE&feature=youtu.be&t=4m50s https://www.youtube.com/watch?v=IrN16c3DTzE&feature=youtu.be...
- colmvp 6y agoThat seems to ring true. At lower levels like where I'm at, players are prone to mistakes and blunders, so having a good eye for tactics allows you to take advantage of those moments in the game as well as prevent yourself from getting into a bad situation. But at elite levels, tactics have less importance (as he says in the video he estimates it drops to 50%) as every player at that level is extremely solid.
- 737min 6y agoCool headline but not really data or a serious article. Obviously it’s a mix of factors, but some of the best I know are also very good at coding interviews, contests, etc.
- sicromoft 6y agoImagine replying to a post based on a huge dataset, saying it's "not really data", then countering with an opinion based on a personal anecdote.
- _underfl0w_ 6y agoNow imagine that the originator of the aforementioned "not really data" came to this very thread to admit that the proposed correlation disappeared once the dataset became more "huge" (c.f. Norvig's actual comment)
- Godel_unicode 6y agoIn what way is this not data? Are you saying that because they didn't show you the data that it doesn't count? Alternatively, is it possible that this is an instance where your local experience doesn't generalize.
- jaredtn 6y agoThis is Berkson's Paradox. Even if coding competition performance correlates positively with job performance in the general population (which it certainly does, given that most people can't code), selecting for this attribute in the hiring process leads to a negative correlation among those hired. Great write-up by Erik Bernhardsson, CTO of Better, here: https://erikbern.com/2020/01/13/how-to-hire-smarter-than-the-market-a-toy-model.html https://erikbern.com/2020/01/13/how-to-hire-smarter-than-the....
- ummonk 6y agoYup exactly.
- adamnemecek 6y agoHe’s not the CTO of Spotify, he worked at Spotify and he’s the CEO of a company called Better.
- jaredtn 6y agoThanks, fixed!
- dcolkitt 6y agoSimple analogy. There is no correlation between height and salary across NBA players.[1] The naive conclusion would be that height has nothing to do with basketball ability. The real answer is that markets are efficient and are already correcting one important feature against other predictors. Steph Curry wouldn't even be in the NBA if had the shooting ability of Gheorghe Mureșan. [1] https://rpubs.com/msluggett/189114 https://rpubs.com/msluggett/189114
- Alex3917 6y agoWhy doesn't the NBA just lower the height of baskets by a foot? They would get much better athletes, so surely the games would be more entertaining and the league would make more money, no?
- thom 6y agoAha, now I can feel good about my late but completely unit tested Advent of Code solutions.
- atom_arranger 6y agoproject structure + unit tests > coding fast + clever solutions
- taeric 6y agoI would have more trust in this if a) it reflected on the style of interview they gave and/or b) they released data.
- amelius 6y agoYou can't cheat in coding competitions. In a real life job, however, you can.
- optimalsolver 6y agoThe winners of the Kaggle/Facebook million dollar contest were disqualified for cheating.
- k_sze 6y agoCan you define cheating? Does looking on Stack Overflow for solutions to real life job problems count as cheating? (That was a partly rhetorical question. My point is that what is considered cheating in coding competitions is pretty much normal in real life jobs. Somebody who knows to look for solutions on Stack Overflow, I call them "resourceful", not cheater. Being resourceful is really important in real life jobs.)
- SrslyJosh 6y agoITT: Lots of folks misunderstanding correlation. =)
- elihu 6y agoIn the linked video, Peter Norvig attaches a big caveat which is that this analysis was done on people who were hired and therefore met Google's various other hiring criteria. So the context isn't random people off the street, but rather that among people who are already (presumably) competent at what they do, winning coding competitions correlates inversely with job performance for some reason.
- temac 6y agoSo I would even not just say "the context isn't random people off the street", but that "the context isn't random employed/employable programmers". It's people working at Google. Deducing anything from that with high certainty is hard. Not only it could be inversed in other companies, but even if not it is hard to apply statistics to individual candidates.
- cupofcoffee 6y agoFocus on coding competitions has negatively impacted my software engineering journey. The only benefit is you are better at interviewing and they are fun to do but the cons far outweigh the pros.
- ta988 6y agoCould you define what you mean by focus? Do you mean you were training, practicing and down ng many events?
- altdatathrow 6y ago> and made sure things were right. Every coding competition I’ve participated in required correctness as a first criteria for basic acceptance, and then speed as a secondary scoring criteria.
- bawolff 6y agoDid you loose significant points for incorrect solutions? Otherwise that still discourages verifying over guessing.
- Tehdasi 6y agoFor Advent of Code, you lose a minute of time for every wrong answer. Also in most how-to-leaderboard guides it's heavily noted that making mistakes (ie. bugs) is going to cost you the most times (anecdotally, I'd agree with this). Those who leaderboard have to write bug free code.
- ta988 6y agoAnd just as a reference the difference between two performers is usually between 1 and 15s so one minute is a lot. On current days, if you are fast 1 minute may still get you leaderboard as the delta between number 1 and 100 is 10 min for day 1 it was 34s.
- vlovich123 6y agoGiven that companies regularly struggle to evaluate performance well or even accurately, an equally valid conclusion could be “Coding and playing politics to be well-ranked by peers are negatively correlated”. Now the question becomes is coding or playing politics what you value more organizationally.
- stevecalifornia 6y agoKnowledge does not equal wisdom. It's as simple as that.
- SulfurHexaFluri 6y agoI hate this kind of comment where people take synonyms and say they are the different things. Knowledge and wisdom are the same thing despite whatever bullshit definitions people apply to try to make them seem different. Perhaps what you mean is knowledge in artificial crafted problems is not the same as knowledge in practical tasks that you would perform in a real job on a real world application.
- mythrwy 6y agoKnowledge is knowing what happened. Wisdom is realizing what is likely going to happen.
- SulfurHexaFluri 6y agoThe dictionary definition does not make any reference to this. At most you could say the difference is knowledge can be purely theoretical while wisdom is knowledge + practical experience. But nothing about a coding challenge is purely theoretical. Its mostly experience with the specific set of problems that come up in these challenges which is a different set of problems that come up in a commercial situation.
- mythrwy 6y agoKnowledge is knowing what the dictionary says. Wisdom is knowing what that definition implies :) Sorry, just ribbing you a bit. I believe the words do have subtle but distinct differences in meaning. Agreed on coding challenges being a different set of problems then most commercial applications.
- gdulli 6y agoThey're not even close to being the same thing. The point here is that wisdom is more akin to judgment than mere possession of facts. For example, sufficient wisdom may give one the humility to reflect on the completeness of their understanding before submitting an inflammatory reply.
- linuxhansl 6y agoIn my hiring for software engineers and architects I always consider coding/design skill to be merely a necessary condition - something you just briefly test for, mostly to detect b*llshit on the resume. What I found is far more important are social skills. Can the person work as member of a team? How do they respond to feedback? Or when something is hard? When they get stuck? How do they communicate a design? Rally a team around them? Deal with disputes? React to changing situations? Can they take the initiative or need to be told what to do? Etc, etc. Together with actual coding/design skills - and with proper management - these are the necessary conditions. All my humble opinion, of course.
- ta988 6y agoHow do you test for those social skills?
- linuxhansl 6y agoIMHO, part of the technical questions is to evaluate reactions. Perhaps I disagree with a solution. Or I do not give all the information needed to solve a problem so the person has to ask. Or I give hints on the way and see how receptive the person is. And so on. After that it's just part of the conversation. Probing the resume. And asking some of these question flat out. I also discuss real practical problems/challenges we have at work, most of which require a team to work together. We also (used to pre-covid) take the person for lunch and decidedly talk not about work at that time. Of course it's different between an engineer fresh from college and a seasoned software engineer or architect. In the end, I think interviewing is itself a skill one needs to train. Just having a person solve some brain-teasers doesn't cut it. I have _way_ more teams, co-founders, etc, fail due to social issue rather than technical skills. And I am by no means presuming that I'm good at it, just that these are things I look for. An important part I forgot to mention in the first post that the person is interviewing the company as much as we interview the person. So we leave a significant amount of time for asking questions about the company... and try to make a good impression. (Sorry for the essay.)
- tensor 6y ago
- jupp0r 6y agoI wouldn't be surprised if interview performance also correlated negatively with job performance, but I guess we'll never find out.
- ta988 6y agoThat's possible, extremely high performance at interview may just mean that person trained really well for the interview process not for the job.
- toby 6y agoI think you'll find a good sample of people who did poorly on their interviews but were still hired due to stellar references, corporate politics or nepotism.
- renewiltord 6y agoBerkson's, right? A perfect interview process would result in a population where none of the people who are hired as a result of it have any attributes that could be correlated with job performance. i.e. all the information has been 'used up' by the selection filter. If you have a correlation, then you can improve the selection filter, so it can't be perfect. This can have interesting outcomes. For instance, when Triplebyte published their blog post about which environments get the most hires⁰, it revealed the areas they haven't yet entirely accounted for in their quest to increase matching performance. 0: https://triplebyte.com/blog/technical-interview-performance-by-editor-os-language https://triplebyte.com/blog/technical-interview-performance-...
- temac 6y agoHard to call that a "perfect interview process", because from individual candidate point of view, some with unusual characteristics could be unfairly disadvantaged (and other unfairly advantaged), while reaching your overall "neutral" distribution at the end. Short of being an omniscient interview process, I'm not sure this can absolutely be avoided, so in practice you have to be very careful with the kind of correlation presented here. Even if that ends up not being Berkson, but a property of all programmers.
- titanomachy 6y agoI don't follow the reasoning. Even if you have a reliable prediction of performance, what prevents you from hiring some candidates who are exemplary and some who are just well-qualified? Or are we assuming that the best candidates would be given a higher-tier job for which they just meet the requirements?
- renewiltord 6y agoIt's a spherical cow conversation. The cows you hire can be placed in a smooth manifold of job difficulty and you have a predictable prediction of the job performance distribution.
- titanomachy 6y ago
- permo-w 6y agoCoding competitions don't require any of the soft skills that a real job does
- pseudalopex 6y agoOr many of the hard skills.
- deleted 6y ago[deleted]
- kumarvvr 6y agoWell yeah. Coding competitions are like an F1 race. High speed, high skill, high stress. Building products and services is like driving an 18-wheeler cross country. Needs patience, dedication, long drawn effort, some degree of talent, team work, guidance, and what not.
- k_sze 6y agoI kinda like this analogy. Coding competitions are also like F1 races in that the problem space usually is very well-defined and very narrow. You'll only run in this track, you know pretty much exactly who your opponents are, you know exactly how much budget you have. The only things that are unpredictable are the weather on the day of the race, the accidents that may happen, a team member getting sick. Building actual products and services is like driving an 18-wheeler in that the road is much more open and you don't know what other vehicles and/or reckless drivers you'll come across, the weather variation over a very long distance, traffic, road works and detours. The driver also needs to stay awake for much, much longer lengths of time.
- AlexAltea 6y agoAnecdotal, but I have noticed a similar correlation between achievements in Capture The Flag (CTF) competitions and performance in the cybersecurity industry. As some people have pointed out, this effect appears when observing just people within the field, not the general population. Disclaimer: I don't have any data to back this up.
- hyper_reality 6y agoI've seen the opposite. Many of the very best security researchers are also top performers in CTFs (to give one example, Orange Tsai). It's hard for me to see how your correlation works unless "cybersecurity industry" is defined extremely narrowly.
- AlexAltea 6y agoCorrelations don't have to "work" as they don't imply causality. Orange Tsai is indeed quite talented at both competitions and research (but it would prove as little as my own anecdotes). If I'm biased, it might be because I defined the "cybersecurity industry" too broadly, not too narrowly: One can acquire certain skills from competing in CTFs/competitions, e.g. hard skills related to reverse engineering and vulnerability research... but I believe in most cases further skills are additionally needed to succeed in the industry, e.g. software engineering, and softer skills such as communication, planning and negotiation (useful for other jobs as well). Overly optimizing skills to win CTFs while neglecting other matters can be harmful, like badly assigned character points in an RPG. :-)
- WClayFerguson 6y agoJust like writing code on a whiteboard with people watching, it's almost like activities that have nothing to do with what you'd do on the actual job actually aren't a great predictor of job performance. haha.
- npsomaratna 6y agoIOI gold medalist here. Most coding competitions tend to assess a specific set of skills: puzzle solving ability; algorithmic knowledge; and being able to code fast. All of these skills are useful in "real life" programming. However, since the code you write will be thrown away post-competition, your focus is on churning out solutions that "just work" — proper engineering practices and maintainability isn't relevant. All your code needs to do is to generate the correct outputs. Does competing turn you into a strong coder? Absolutely. Does this equate to being a strong engineer? Nope. Software engineering isn't just about coding fast. This is anecdotal, but from what I've seen (as a trainer and friend of several IOI medalists): some of them appreciate that coding != engineering and proceed to develop their engineering skills. Others don't and remain stuck at the "I'll come up with a fast solution" mindset. Whether one or the other happens very much depends on the person, plus, I believe, whom they end up working with. After all, we've all heard about the "10x" programmer – and when your colleague or subordinate appears to code at 10x speed, you just might think twice about whether you're qualified to advise or guide them. That results in their keeping any bad habits they might have.
- neolog 6y ago> proceed to develop their engineering skills Is there a way to focus practice on that?
- robjan 6y agoIt's very hard to assess automatically (and therefore cheaply) since "engineering skill" is largely qualitative rather than quantitative.
- frog_baxter 6y agoWrite a lot of programs. Reflect on how they can be better. Repeat. Program with people. Learn from them. Exchange ideas. Get code reviews and give them. Look at program source code on github for inspiration. And just keep writing programs.
- httpsterio 6y agoVoluntarily feature creep. Instead of writing simpler one-off projects, try to keep plugging at the same project for a longer time and expand the product in ways you couldn't have guessed at the beginning. You have to learn how to write maintainable code and plan ahead.
- 908B64B197 6y agoI can already see folks commenting out that this is the reason we should stop whiteboard interviews. But let's take a step back. Whiteboard weeds out candidates that cannot code period. I wouldn't believe the legend of the "senior engineer" who couldn't FizzBuzz until I met him. I've personally conducted interview where a simply word count function (take a string, count the words) couldn't be implemented in 30 minutes. And the resume listed proficiency in several programming languages. Now this study tells us that competitive programmers aren't that great among the candidates that were hired, not among the pool of candidates. That's very different.
- TT3351 6y agoI have very little frame of reference for other people's technical skills when they enter the profession since I'm entirely self-taught and still move dirt for a living. Thank you for sharing that story, it drove me to try my hand, it took less than 15 minutes.
- norvig 6y agoI regret causing confusion here. It turns out that this correlation was true on the initial small data set, but after gathering more data, the correlation went away. So the real lesson should be: "if you gather data on a lot of low-frequency events, some of them will display a spurious correlation, about which you can make up a story."
- Tehdasi 6y agoAlso, for any given title, ppl on HN will come up with anecdata to support it's assertion.
- etrautmann 6y agoAh, I remember hearing this from you in person in 2011 and have repeated it occasionally since. Thanks for the update!
- gwern 6y agoYou and/or Google HR were also prominently quoted as saying, IIRC, that GPA, standardized test scores (and interview ratings?) had no observable correlation with job performance either. I always wrote that off as just range restriction/Berkson's paradox, but did those also go away?
- titanomachy 6y agoInterview ratings would be a surprising one, hard to imagine not overhauling the system after discovering that.
- KMag 6y agoI remember reading something about Google interview ratings being uncorrelated to job performance, but that's clearly a conditional correlation based on the person being hired. For all we know, the ratings might be highly correlated with job performance up until the hire/no-hire cutoff. After all, their primary purpose is to make that binary hire/no-hire decision. Hopefully, the scoring system is hyper-optimized to be a good signal right around the hire/no-hire boundary, as the scores themselves aren't that useful for obvious hires and obvious no-hires: the scores are a decision-making tool. In order to really get a good assessment of if the interview ratings were effective, they'd need to also hire some random unbiased sample of those who fail the interview process. There are alternative ways of slicing the data to help give insight, such as looking at only those who barely passed the interview process, or looking only at the bottom 10% of performers. However, when you're looking at such a highly biased sample (only the small-ish percentage of people hired), it's hard to say what the correlation is across the entire interview population. At the risk of repeating myself, we don't particularly care the predictive power of the scores across the whole range, only their predictive power across those who aren't obvious no-hires and those who aren't obvious hires. That's the range where the power of the interview scores as a decision-making tool is most important. Also, if two metrics disagree, it's not clear which one is problematic. It's possible that a poor correlation indicates that there's a problem with the performance rating system.
- warabe 6y agoI wonder, is it true for DataScience too? Does being good at kaggke competions negatively correlates with actual job performance?
- angry_octet 6y agoKaggle comps don't, to my knowledge, involve tasks like: - Convincing someone to give you data. - Convincing them to spend great effort into changing their data collection and labelling practices. - Explaining why a particular technique was used and why it is correct. - Explaining why they can't expect magic from 'big data'. - Making models that are robust and easily maintained, vs fragile spaghetti. But I don't think being good at kaggle implies being bad at data science soft skills. Technical skills are probably weakly correlated -- that's my prior, it would be good to see a study. I did find a paper examining the performance of TopCoder participants: https://doi.org/10.1007/s10664-019-09755-0 https://doi.org/10.1007/s10664-019-09755-0
- phkahler 6y agoMy takeaway from this was that you can't have too many things you're judging a candidate on. If you over emphasise one trait you're going to screw yourself over in finding another. I was also happy to see that ignoring things that many companies over-value is a good strategy if you think you know what's over valued. My own 3 hiring criteria are still not talked about much and were very effective last I used them. Now I understand why a little better.
- Badfood 6y agoThe first developer I ever had to let go was good at coding competitions. He was a gambling addict and I feel like the coding competition thing had similar motivations... Get rich quick somehow.
- rixed 6y ago> Peter added that programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right. Alternative explanation: good job performance, especially in a big companies where such studies can be conducted, requires some consideration for corporate politics that correlates negatively with interest in programming?
- caidehen 6y agobeing excellent at algorithms and mathematics is the base to solve unsolved computer science problems. though most companies just need to implement projects, top ones will try to push the boundaries. all projects and companies will eventually die someday, algorithms and mathematics will stay.