6 ms·

Positive correlation between Competitive Programmer’s Handbook and software engineer interviews? Yes. Positive correlation between being a strong competitive pr

Positive correlation between Competitive Programmer’s Handbook and software engineer interviews? Yes. Positive correlation between being a strong competitive programmer and a strong software engineer? Doubtful.

mck- 10y ago
Agreed. These kinds of competitions or coding interviews may cause over-fitting.
- nkozyra 10y ago
  Is there really risk in being particularly adept at algorithm design?
  - aswanson 10y ago
    No, but by definition, any non-trivial software product (or academic CS paper, for that matter) is the outcome of a collaborative process, not an artificially time-constrained hack-a-thon/competition. You have to put things like this in their proper place, and let them be what they are. (I downloaded the pdf, btw).
    - deleted 10y ago
      [deleted]
  - closeparen 10y ago
    The risk is that you might be less useful than someone who is particularly adept at system design. It seems most problems are actually not sorting, searching, or finding the optimal whatever. Maybe it's just the bubble I work in, but from my perspective it seems that most programmers aren't addressing a problem of the form, "the obvious solution to this well-defined problem is too slow, please have a clever insight that leads to a faster one." The problems we work on are instead of the form, "please model this sprawling and subtle domain with reasonable fidelity and in a way that'll handle future changes to the domain." "Please satisfy these five dozen individually trivial requirements in a way that gets every corner-case interaction exactly right, and won't turn into a nightmare when there are a dozen more next quarter." "Please decompose this problem in such a way that 10 different people can work on it in separate parts of the codebase in parallel." "Please take this problem that's solved for one machine and make it work over an arbitrary number of machines, and make it reliable under all the weird and abusive scenarios that a few years of usage in production can manage to throw at you." "Please design a monitoring and dashboarding strategy that will identify all outages immediately while not overwhelming the oncall with false alarms, and provide first-class instrumentation, debugging, and remediation tools so that someone new to the codebase can find out exactly what went wrong and fix it in the middle of the night. It's not at all uncommon to deliberately ignore the optimal algorithm in favor of the readable algorithm. We usually try to keep cleverness behind the curtain of abstraction (RDBMS, standard library, etc). Of course, someone has to build them, but then they are widely reusable. Of course, people who can do all of the above while interacting fluently with the code for the optimal algorithm are so incredibly highly paid in the Bay Area that they can buy houses.
    - apatters 10y ago
      This comment is so good that I think just being able to write it would qualify someone for a midlevel programming job!
      - bencoder 10y ago
        Senior
  - throwaway29292 10y ago
    Think of it this way: there's only a certain amount of productive hours available to people for side programming, and either you spend a majority contributing to open source/building prototypes/learning new technologies/getting your hands dirty; or you spend a majority of the time solving artificial problems to get a leg up on programming interviews. It's a catch-22 if ever there was one.
kazarov 10y ago
I'm pretty sure there is a positive correlation, just because people who tend to be good at competitions usually do them because they like computer science in general, so they learn other stuff too (and they are likely smart). And this is also a reason why this side is tested so much in interviews - it is easy to test and the result is somewhat significant. Now, most good software engineers would probably not be very good at competitions, because competitions require certain specific skills that are not useful outside of competitions. And that's why good interviews don't put too much weight on algorithmic skills.
- goostavos 10y ago
  It sounds like it may have been awhile since you've interviewed. The modern "good" interview is all data structure and algorithm white boarding nonsense. For me, who is not a competitive programmer, the standard interview prep is spending a couple of hours on Hacker Rank every day and making sure I can delete a node in a binary tree on a white board without so much a missing semicolon. For some reason that signals "good developer" more than a catalog of work on github, big name companies, or experience in general.
  - rs86 10y ago
    From my experience i'd rather work with programmers with strong CS skills. They put out better code
    - seanp2k2 10y ago
      I'd rather work with the ones who actually care about the people they're solving problems for vs the ones who can write bubble sort with no reference material, which happens in the real world precisely never. Employers pay developers to solve problems, not necessarily to write the most efficient code possible. I get that it's a craft and it's good to understand what happens under the hood, but you can get pretty damn far solving actual business problems for people without having to care which brand of sort ES2016 uses. These skills in micro-optimizations / textbook CS don't help much with understanding customers and solving their problems at the end of the day, so they're really not that valuable for many business scenarios. I think that a lot of engineers in this industry forget that they're not generally paid to write the most efficient data structures and algorithms for 40 hours a week, and that ironically, attempting to do so would very likely not be the most efficient means of increasing shareholder value. Can't see the forest for the B-trees, if you will. https://github.com/poteto/hiring-without-whiteboards https://github.com/poteto/hiring-without-whiteboards has a list of companies who don't participate in this particular flavor of shenanigans.
SamReidHughes 10y ago
Not doubtful at all. You can expect a positive correlation between being an good driver and a good software engineer. Definitely some random person with the CS fundamentals, coding fluency, and creative problem solving ability to be good at competitive programming is more likely to be good at software engineering than some random developer. You're basically saying, high intelligence isn't correlated with being a strong software engineer. Or maybe you think people doing competitive programming are damaged. This is nuts.
- thr0waway1239 10y ago
  >> Or maybe you think people doing competitive programming are damaged. This is nuts. I don't see how that is implied by GP's comment. It is one thing to read between the lines, but now you are reading between the characters.
  - anon263626 10y ago
    Yup. More mental "shortcuts" that seek magic "signals" to avoid both evidence and pragmatic interviewing effort testing both business problem-solving critical thinking and cultural fit. There are no shortcuts. Furthermore, "competitive" programming has little to do with the real world, and it's more about showing off and ego inflation.
  - boltzmannbrain 10y ago
    Reading between the lines: Problems that are typical of software engineering interviews and in competitive programming are poor evaluation of one's capabilities as a professional software engineer.
jimbokun 10y ago
Maybe not, but just skimming through the topics looks like there is a lot of overlap between competitive programming and popular job interview questions.
sidusknight 10y ago
You have a very weak understanding of what the word 'correlation' means.
jodooshi 10y ago
Being good at programming competitions correlates negatively with being good on the job - Peter Norvig http://www.catonmat.net/blog/programming-competitions-work-performance/ http://www.catonmat.net/blog/programming-competitions-work-p...
- icc97 10y ago
  Very interesting. Although I think he's speaking specifically about competition winnners, not just being good at them.
- mining 10y ago
  This is relative to other people who had been hired at Google - there's probably still a positive correlation between those two variables amongst the general population.
- boltzmannbrain 10y ago
  Oh cool, I was hesitant to say "positive correlation" without data. Thanks for the link!
- minwcnt5 10y ago
  This has been circulated around HN and Reddit several times, and it's disappointing that someone of Norvig's stature would present the data in such a misleading way. Here's a good explanation posted by "tedsanders" the last time this came up on HN: """ All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias. Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Google. So when Google says things like "GPA is not correlated with job performance" what you should hear is "Given that you were hired by Google, GPA is not correlated with job performance." In general, when you have some thresholding selection, it will cause artificial negative correlations to show up. Here's a very simple example that I hope illustrates the point: Imagine a world where high school students take only two classes, English and Math, and they receive one of two grades, A or B. Now imagine a college that admits students with at least one A (AB, BA, or AA) and that rejects everyone without an A (BB). Now imagine that there is absolutely zero correlation between Math and English - performance on one is totally independent of the other. However, when the college looks at their data, they will nonetheless see a stark anticorrelation between Math and English grades (because everyone who has a B in one subject always has an A in the other subject, simply because all the BBs are missing from their dataset). When Google says that programming competitions are negatively correlated with performance and GPA is uncorrelated with performance, what that likely means is that Google's hiring overvalues programming competitions and fairly values GPA. """ I've also heard people involved in Google's Code Jam competition say that Norvig's study was done a long time ago, and no longer really applies.
kybernetikos 10y ago
I observed a correlation between competitive programmers and those who come across as just hacking at it until it works in interviews. If your interview system is looking for people who think about things then it probably already biases against many competitive programmers correctly.
deleted 10y ago
[deleted]
catears 10y ago
I have had a lot of experience with competitive programming (competed in NWERC and NCPC multiple times) and this idea that competitive programming makes you worse at software engineering comes up every now and then. From what I've noticed your statement holds a bit of truth, but I just wanted to add from my own experiences. To me it seems like those that have done a little bit of competitive programming are good coders. They can code a FizzBuzz in 2 minutes without blinking, they are people that will reach the magic 10K hours of coding quicker than most. However, those that excel at competitive programming can code FizzBuzz in 30 seconds but you will have a hard time understanding what they have actually written. I think that there is a point in competitive programming you pass where you actively have to sacrifice the understandability of your code (and your algorithm library) for increased implementation speed. I also think that in order to excel at competitive programming you have to devote a lot of time to learning algorithmic theory and training on problems which can be solved in <1K lines of code. (Most solutions I've written are less than 300 lines) The hypothesis I have is that in order to be a great competitive programmer you will have to routinely write bad code and put a lot of effort into solving problems which you will later throw away the solution to and never look at again. Two properties that do not help with software engineering skills. However, if you are only moderately devoted to competitive programming, then you can earn the practical experience (hours of coding) without reducing the quality of your code.
- mafribe 10y ago
  write bad code [...] which you will later throw away Implementing a quick & dirty prototype, throwing it away and reimplementing it from scratch is not such a bad software engineering techique: first time to understand the problem, second time to make the implementation nice. A lot of bad software comes from not throwing the prototype away, but insteat trying to refine it into a product. Result = spaghetti.
  - hashhar 10y ago
    I agree. I got into the practice after I picked up F#. I used to create simple solutions in the interactive REPL as mainly proof of concepts. Once I had the basics down, I sat down and wrote a list of possible edge cases, expected user inputs, the format of output and the command line interface for the tool. Then I started coding from scratch and had a good and clean working solution ready. Helped me a lot.
IvanVergiliev 10y ago
Your second assumption is only valid for people who conflate the two disciplines. If you don't recognize that competitive programming and software engineers have completely different goals and requirements, then being trained in competitive programming may very well lead you to bad programming practices - e.g. single-letter names are usually fine for competitions, while at the same time usually horrible for engineering. However, if you recognize the differences and use the skills you learn in the two as complementary, you can be better at both.

mck- 10y ago

Agreed. These kinds of competitions or coding interviews may cause over-fitting.

nkozyra 10y ago
Is there really risk in being particularly adept at algorithm design?
- aswanson 10y ago
  No, but by definition, any non-trivial software product (or academic CS paper, for that matter) is the outcome of a collaborative process, not an artificially time-constrained hack-a-thon/competition. You have to put things like this in their proper place, and let them be what they are. (I downloaded the pdf, btw).
  - deleted 10y ago
    [deleted]
- closeparen 10y ago
  The risk is that you might be less useful than someone who is particularly adept at system design. It seems most problems are actually not sorting, searching, or finding the optimal whatever. Maybe it's just the bubble I work in, but from my perspective it seems that most programmers aren't addressing a problem of the form, "the obvious solution to this well-defined problem is too slow, please have a clever insight that leads to a faster one." The problems we work on are instead of the form, "please model this sprawling and subtle domain with reasonable fidelity and in a way that'll handle future changes to the domain." "Please satisfy these five dozen individually trivial requirements in a way that gets every corner-case interaction exactly right, and won't turn into a nightmare when there are a dozen more next quarter." "Please decompose this problem in such a way that 10 different people can work on it in separate parts of the codebase in parallel." "Please take this problem that's solved for one machine and make it work over an arbitrary number of machines, and make it reliable under all the weird and abusive scenarios that a few years of usage in production can manage to throw at you." "Please design a monitoring and dashboarding strategy that will identify all outages immediately while not overwhelming the oncall with false alarms, and provide first-class instrumentation, debugging, and remediation tools so that someone new to the codebase can find out exactly what went wrong and fix it in the middle of the night. It's not at all uncommon to deliberately ignore the optimal algorithm in favor of the readable algorithm. We usually try to keep cleverness behind the curtain of abstraction (RDBMS, standard library, etc). Of course, someone has to build them, but then they are widely reusable. Of course, people who can do all of the above while interacting fluently with the code for the optimal algorithm are so incredibly highly paid in the Bay Area that they can buy houses.
  - apatters 10y ago
    This comment is so good that I think just being able to write it would qualify someone for a midlevel programming job!
    - bencoder 10y ago
      Senior
- throwaway29292 10y ago
  Think of it this way: there's only a certain amount of productive hours available to people for side programming, and either you spend a majority contributing to open source/building prototypes/learning new technologies/getting your hands dirty; or you spend a majority of the time solving artificial problems to get a leg up on programming interviews. It's a catch-22 if ever there was one.

kazarov 10y ago

I'm pretty sure there is a positive correlation, just because people who tend to be good at competitions usually do them because they like computer science in general, so they learn other stuff too (and they are likely smart). And this is also a reason why this side is tested so much in interviews - it is easy to test and the result is somewhat significant. Now, most good software engineers would probably not be very good at competitions, because competitions require certain specific skills that are not useful outside of competitions. And that's why good interviews don't put too much weight on algorithmic skills.

goostavos 10y ago
It sounds like it may have been awhile since you've interviewed. The modern "good" interview is all data structure and algorithm white boarding nonsense. For me, who is not a competitive programmer, the standard interview prep is spending a couple of hours on Hacker Rank every day and making sure I can delete a node in a binary tree on a white board without so much a missing semicolon. For some reason that signals "good developer" more than a catalog of work on github, big name companies, or experience in general.
- rs86 10y ago
  From my experience i'd rather work with programmers with strong CS skills. They put out better code
  - seanp2k2 10y ago
    I'd rather work with the ones who actually care about the people they're solving problems for vs the ones who can write bubble sort with no reference material, which happens in the real world precisely never. Employers pay developers to solve problems, not necessarily to write the most efficient code possible. I get that it's a craft and it's good to understand what happens under the hood, but you can get pretty damn far solving actual business problems for people without having to care which brand of sort ES2016 uses. These skills in micro-optimizations / textbook CS don't help much with understanding customers and solving their problems at the end of the day, so they're really not that valuable for many business scenarios. I think that a lot of engineers in this industry forget that they're not generally paid to write the most efficient data structures and algorithms for 40 hours a week, and that ironically, attempting to do so would very likely not be the most efficient means of increasing shareholder value. Can't see the forest for the B-trees, if you will. https://github.com/poteto/hiring-without-whiteboards https://github.com/poteto/hiring-without-whiteboards has a list of companies who don't participate in this particular flavor of shenanigans.

SamReidHughes 10y ago

Not doubtful at all. You can expect a positive correlation between being an good driver and a good software engineer. Definitely some random person with the CS fundamentals, coding fluency, and creative problem solving ability to be good at competitive programming is more likely to be good at software engineering than some random developer. You're basically saying, high intelligence isn't correlated with being a strong software engineer. Or maybe you think people doing competitive programming are damaged. This is nuts.

thr0waway1239 10y ago
>> Or maybe you think people doing competitive programming are damaged. This is nuts. I don't see how that is implied by GP's comment. It is one thing to read between the lines, but now you are reading between the characters.
- anon263626 10y ago
  Yup. More mental "shortcuts" that seek magic "signals" to avoid both evidence and pragmatic interviewing effort testing both business problem-solving critical thinking and cultural fit. There are no shortcuts. Furthermore, "competitive" programming has little to do with the real world, and it's more about showing off and ego inflation.
- boltzmannbrain 10y ago
  Reading between the lines: Problems that are typical of software engineering interviews and in competitive programming are poor evaluation of one's capabilities as a professional software engineer.

jimbokun 10y ago

Maybe not, but just skimming through the topics looks like there is a lot of overlap between competitive programming and popular job interview questions.

sidusknight 10y ago

You have a very weak understanding of what the word 'correlation' means.

jodooshi 10y ago

Being good at programming competitions correlates negatively with being good on the job - Peter Norvig http://www.catonmat.net/blog/programming-competitions-work-performance/ http://www.catonmat.net/blog/programming-competitions-work-p...

icc97 10y ago
Very interesting. Although I think he's speaking specifically about competition winnners, not just being good at them.
mining 10y ago
This is relative to other people who had been hired at Google - there's probably still a positive correlation between those two variables amongst the general population.
boltzmannbrain 10y ago
Oh cool, I was hesitant to say "positive correlation" without data. Thanks for the link!
minwcnt5 10y ago
This has been circulated around HN and Reddit several times, and it's disappointing that someone of Norvig's stature would present the data in such a misleading way. Here's a good explanation posted by "tedsanders" the last time this came up on HN: """ All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias. Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Google. So when Google says things like "GPA is not correlated with job performance" what you should hear is "Given that you were hired by Google, GPA is not correlated with job performance." In general, when you have some thresholding selection, it will cause artificial negative correlations to show up. Here's a very simple example that I hope illustrates the point: Imagine a world where high school students take only two classes, English and Math, and they receive one of two grades, A or B. Now imagine a college that admits students with at least one A (AB, BA, or AA) and that rejects everyone without an A (BB). Now imagine that there is absolutely zero correlation between Math and English - performance on one is totally independent of the other. However, when the college looks at their data, they will nonetheless see a stark anticorrelation between Math and English grades (because everyone who has a B in one subject always has an A in the other subject, simply because all the BBs are missing from their dataset). When Google says that programming competitions are negatively correlated with performance and GPA is uncorrelated with performance, what that likely means is that Google's hiring overvalues programming competitions and fairly values GPA. """ I've also heard people involved in Google's Code Jam competition say that Norvig's study was done a long time ago, and no longer really applies.

kybernetikos 10y ago

I observed a correlation between competitive programmers and those who come across as just hacking at it until it works in interviews. If your interview system is looking for people who think about things then it probably already biases against many competitive programmers correctly.

deleted 10y ago

[deleted]

catears 10y ago

I have had a lot of experience with competitive programming (competed in NWERC and NCPC multiple times) and this idea that competitive programming makes you worse at software engineering comes up every now and then. From what I've noticed your statement holds a bit of truth, but I just wanted to add from my own experiences. To me it seems like those that have done a little bit of competitive programming are good coders. They can code a FizzBuzz in 2 minutes without blinking, they are people that will reach the magic 10K hours of coding quicker than most. However, those that excel at competitive programming can code FizzBuzz in 30 seconds but you will have a hard time understanding what they have actually written. I think that there is a point in competitive programming you pass where you actively have to sacrifice the understandability of your code (and your algorithm library) for increased implementation speed. I also think that in order to excel at competitive programming you have to devote a lot of time to learning algorithmic theory and training on problems which can be solved in <1K lines of code. (Most solutions I've written are less than 300 lines) The hypothesis I have is that in order to be a great competitive programmer you will have to routinely write bad code and put a lot of effort into solving problems which you will later throw away the solution to and never look at again. Two properties that do not help with software engineering skills. However, if you are only moderately devoted to competitive programming, then you can earn the practical experience (hours of coding) without reducing the quality of your code.

mafribe 10y ago
write bad code [...] which you will later throw away Implementing a quick & dirty prototype, throwing it away and reimplementing it from scratch is not such a bad software engineering techique: first time to understand the problem, second time to make the implementation nice. A lot of bad software comes from not throwing the prototype away, but insteat trying to refine it into a product. Result = spaghetti.
- hashhar 10y ago
  I agree. I got into the practice after I picked up F#. I used to create simple solutions in the interactive REPL as mainly proof of concepts. Once I had the basics down, I sat down and wrote a list of possible edge cases, expected user inputs, the format of output and the command line interface for the tool. Then I started coding from scratch and had a good and clean working solution ready. Helped me a lot.

IvanVergiliev 10y ago

Your second assumption is only valid for people who conflate the two disciplines. If you don't recognize that competitive programming and software engineers have completely different goals and requirements, then being trained in competitive programming may very well lead you to bad programming practices - e.g. single-letter names are usually fine for competitions, while at the same time usually horrible for engineering. However, if you recognize the differences and use the skills you learn in the two as complementary, you can be better at both.