4 ms·

No, overfitting is a real thing. Overfitted learning algorithms are generally worse at generalizing their ability to broader examples and new situations. The t

by p4wnc6 11y ago

No, overfitting is a real thing. Overfitted learning algorithms are generally worse at generalizing their ability to broader examples and new situations. The types of candidates who spend the time necessary to memorize algorithm trivia for the sake of passing these exams are exactly like overfitted learning algorithms. What they happen to know is unlikely to generalize well. Of course you could get lucky and hire someone like that who can generalize, but that's rare. More often, since hiring is political, you pat yourself on the back for how "good" the candidate is (based on some trivia) and make excuses when their on-the-job performance isn't what you'd hoped, and find ways to deflect attention from that so that you, as an inefficient hirer, won't be called out on it. Willingness to waste time overfitting yourself to algorithm trivia absolutely predicts worse later-on performance than candidates with demonstrated experience and pragmatism (e.g. I'm not wasting my time memorizing how to solve tricky things that rarely matter. I will look them up / derive them / figure them out when/if I need them). If given the choice between hiring a math/programming olympiad winner vs. a Macgyver/Edison-like tinkerer who may not be able to explain how to convert between Thevenin and Norton circuit forms, but who took their family radio apart and put it back together, Macgyver/Edison wins every time (unless you're hiring for bullshit on-paper prestige, and of course many places are while proclaiming loudly that they aren't).

microtherion 10y ago
But in original author's interview recaps, it's pretty evident that he's NOT comfortable "generalizing his ability to new situations". A BFS is not a super exotic, highly specialized algorithm. It's pretty much a bread-and-butter application of general principles (linear search of children + storing away hypotheses for later).
serge2k 11y ago
> Overfitted learning algorithms are generally worse at generalizing their ability to broader examples and new situations. Source? or just trying to justify your own shortcomings?
- p4wnc6 11y ago
  Here, let me Google that for you. [0] < https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting > > Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model that has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data. [1] < https://en.wikipedia.org/wiki/Generalization_error#Relation_to_overfitting https://en.wikipedia.org/wiki/Generalization_error#Relation_... > > The concepts of generalization error and overfitting are closely related. Overfitting occurs when the learned function f_S becomes sensitive to the noise in the sample. As a result, the function will perform well on the training set but not perform well on other data from the joint probability distribution of x and y. Thus, the more overfitting occurs, the larger the generalization error. Shall I fetch a ruler?
  - serge2k 11y ago
    I apologize. > The types of candidates who spend the time necessary to memorize algorithm trivia for the sake of passing these exams are exactly like overfitted learning algorithms. I should have asked for a source for that portion. It's the part that is ridiculous. Again, my bad.
    - p4wnc6 11y ago
      Sigh. I wish instead of the sarcasm, you actually introspected on the toxic and needlessly aggressive way you're defending rote memorization of algorithm trivia. For example, you saying > Source? or just trying to justify your own shortcomings? is clearly, unequivocally unprovoked and needlessly antagonistic (what do my shortcomings have to do with my point ... either you engage with the claim or not, but use ad hominem insults is not a valid discussion tactic. Yet you seem to assume no responsibility for your completely unprovoked hostility, and continue on with sarcasm, even sarcasm about your own ridiculousness.) In terms of data, obviously no study is perfect and we should not simply base everything on a single study, but the 2012 PISA results offer at least some evidence < http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-i.htm http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volum... >. The particular sections of the actual study that cover poor performance of those who focus on memorization is not available in the free sample (Chapter 2), but it was reported on, e.g. here: < http://hechingerreport.org/memorizers-are-the-lowest-achievers-and-other-common-core-math-surprises/ http://hechingerreport.org/memorizers-are-the-lowest-achieve... >, including this: > The U.S. has more memorizers than most other countries in the world. Perhaps not surprisingly as math teachers, driven by narrow state standards and tests, have valued those students over all others, communicating to many other students along the way – often girls – that they do not belong in math class. > The fact that we have valued one type of learner and given others the idea they cannot do math is part of the reason for the widespread math failure and dislike in the U.S. There is also the discussion from Google that test scores and brainteasers do not predict later-on job success < http://www.newyorker.com/tech/elements/why-brainteasers-dont-belong-in-job-interviews http://www.newyorker.com/tech/elements/why-brainteasers-dont... > (and many algorithm interviews are absolutely the same kind of brainteaser nonsense). From the NY article: > The major problem with most attempts to predict a specific outcome, such as interviews, is decontextualization: the attempt takes place in a generalized environment, as opposed to the context in which a behavior or trait naturally occurs. Google’s brainteasers measure how good people are at quickly coming up with a clever, plausible-seeming solution to an abstract problem under pressure. But employees don’t experience this particular type of pressure on the job. What the interviewee faces, instead, is the objective of a stressful, artificial interview setting: to make an impression that speaks to her qualifications in a limited time, within the narrow parameters set by the interviewer. What’s more, the candidate is asked to handle an abstracted “gotcha” situation, where thinking quickly is often more important than thinking well. Instead of determining how someone will perform on relevant tasks, the interviewer measures how the candidate will handle a brainteaser during an interview, and not much more. The other thing we have to fight against is self-selection. It's not necessarily in everyone's interest to publicize that their riddles and algorithm hazing process isn't working. Especially not for start-ups which need investors to feel like they are crammed to the brim with stereotypical nerds or something. So if you go looking along the lines of "well, my company uses riddles and we have hired well" you're already done. You haven't hired well. Your company (if it's a startup) probably isn't even remotely proven yet, even if it's well-funded, and the verdict is out on whether the people you rejected really should have been.
consz 11y ago
Totally disagree with the final sentence. Most math/programming olympiad winners are way more than capable of handling anything the Macgyver/Edison type would be good at. At least in my industry, every olympiad winner has been a consistently spectacular performer, and I have absolutely no qualms heavily biasing myself towards that credential.
- p4wnc6 11y ago
  Having worked with some such folks in a quant-focused field, I can say that their performance was no better than average and in some cases worse. After a few hires like this, my boss actually pulled me aside and asked me to take over one of the main hiring projects because he and a few other managers were unhappy with the way the emphasis on this type of paper credential had failed to produce good enough analysts, and they wanted a process more focused on probing someone's experience and ingenuity. I didn't enjoy it at the time, but that 9-month project focused on that team's hiring needs (which took me away from some of my technical projects for more time than I liked) ended up teaching me a ton about what candidates in general look like (at least for that type of firm). But I grant this is reasoning just from the anecdata that I have. I can believe that winners perhaps represent a higher degree of skill, but then we're talking about an extremely small number of people. Generally you're facing a tradeoff where you have to choose between a sort of rustic self-reliance skill set versus a bookworm skill set. People from either group can learn the other over time, but you can't predict how well by testing them solely on trivia that constitutes their current main group. My preference is to hire for self-reliance and learn bookworm stuff later. I used to believe the opposite (e.g. hire someone good at math because they can always learn to be an effective programmer later) but my job experience made me believe the opposite (e.g. actually it's pretty easy to teach people stochastic processes, machine learning, or cryptography, but it's incredibly hard to teach people how to be good at creative software design).

by p4wnc6 11y ago

microtherion 10y ago

But in original author's interview recaps, it's pretty evident that he's NOT comfortable "generalizing his ability to new situations". A BFS is not a super exotic, highly specialized algorithm. It's pretty much a bread-and-butter application of general principles (linear search of children + storing away hypotheses for later).

serge2k 11y ago

> Overfitted learning algorithms are generally worse at generalizing their ability to broader examples and new situations. Source? or just trying to justify your own shortcomings?

p4wnc6 11y ago
Here, let me Google that for you. [0] < https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting > > Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model that has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data. [1] < https://en.wikipedia.org/wiki/Generalization_error#Relation_to_overfitting https://en.wikipedia.org/wiki/Generalization_error#Relation_... > > The concepts of generalization error and overfitting are closely related. Overfitting occurs when the learned function f_S becomes sensitive to the noise in the sample. As a result, the function will perform well on the training set but not perform well on other data from the joint probability distribution of x and y. Thus, the more overfitting occurs, the larger the generalization error. Shall I fetch a ruler?
- serge2k 11y ago
  I apologize. > The types of candidates who spend the time necessary to memorize algorithm trivia for the sake of passing these exams are exactly like overfitted learning algorithms. I should have asked for a source for that portion. It's the part that is ridiculous. Again, my bad.
  - p4wnc6 11y ago
    Sigh. I wish instead of the sarcasm, you actually introspected on the toxic and needlessly aggressive way you're defending rote memorization of algorithm trivia. For example, you saying > Source? or just trying to justify your own shortcomings? is clearly, unequivocally unprovoked and needlessly antagonistic (what do my shortcomings have to do with my point ... either you engage with the claim or not, but use ad hominem insults is not a valid discussion tactic. Yet you seem to assume no responsibility for your completely unprovoked hostility, and continue on with sarcasm, even sarcasm about your own ridiculousness.) In terms of data, obviously no study is perfect and we should not simply base everything on a single study, but the 2012 PISA results offer at least some evidence < http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-i.htm http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volum... >. The particular sections of the actual study that cover poor performance of those who focus on memorization is not available in the free sample (Chapter 2), but it was reported on, e.g. here: < http://hechingerreport.org/memorizers-are-the-lowest-achievers-and-other-common-core-math-surprises/ http://hechingerreport.org/memorizers-are-the-lowest-achieve... >, including this: > The U.S. has more memorizers than most other countries in the world. Perhaps not surprisingly as math teachers, driven by narrow state standards and tests, have valued those students over all others, communicating to many other students along the way – often girls – that they do not belong in math class. > The fact that we have valued one type of learner and given others the idea they cannot do math is part of the reason for the widespread math failure and dislike in the U.S. There is also the discussion from Google that test scores and brainteasers do not predict later-on job success < http://www.newyorker.com/tech/elements/why-brainteasers-dont-belong-in-job-interviews http://www.newyorker.com/tech/elements/why-brainteasers-dont... > (and many algorithm interviews are absolutely the same kind of brainteaser nonsense). From the NY article: > The major problem with most attempts to predict a specific outcome, such as interviews, is decontextualization: the attempt takes place in a generalized environment, as opposed to the context in which a behavior or trait naturally occurs. Google’s brainteasers measure how good people are at quickly coming up with a clever, plausible-seeming solution to an abstract problem under pressure. But employees don’t experience this particular type of pressure on the job. What the interviewee faces, instead, is the objective of a stressful, artificial interview setting: to make an impression that speaks to her qualifications in a limited time, within the narrow parameters set by the interviewer. What’s more, the candidate is asked to handle an abstracted “gotcha” situation, where thinking quickly is often more important than thinking well. Instead of determining how someone will perform on relevant tasks, the interviewer measures how the candidate will handle a brainteaser during an interview, and not much more. The other thing we have to fight against is self-selection. It's not necessarily in everyone's interest to publicize that their riddles and algorithm hazing process isn't working. Especially not for start-ups which need investors to feel like they are crammed to the brim with stereotypical nerds or something. So if you go looking along the lines of "well, my company uses riddles and we have hired well" you're already done. You haven't hired well. Your company (if it's a startup) probably isn't even remotely proven yet, even if it's well-funded, and the verdict is out on whether the people you rejected really should have been.

consz 11y ago

Totally disagree with the final sentence. Most math/programming olympiad winners are way more than capable of handling anything the Macgyver/Edison type would be good at. At least in my industry, every olympiad winner has been a consistently spectacular performer, and I have absolutely no qualms heavily biasing myself towards that credential.

p4wnc6 11y ago
Having worked with some such folks in a quant-focused field, I can say that their performance was no better than average and in some cases worse. After a few hires like this, my boss actually pulled me aside and asked me to take over one of the main hiring projects because he and a few other managers were unhappy with the way the emphasis on this type of paper credential had failed to produce good enough analysts, and they wanted a process more focused on probing someone's experience and ingenuity. I didn't enjoy it at the time, but that 9-month project focused on that team's hiring needs (which took me away from some of my technical projects for more time than I liked) ended up teaching me a ton about what candidates in general look like (at least for that type of firm). But I grant this is reasoning just from the anecdata that I have. I can believe that winners perhaps represent a higher degree of skill, but then we're talking about an extremely small number of people. Generally you're facing a tradeoff where you have to choose between a sort of rustic self-reliance skill set versus a bookworm skill set. People from either group can learn the other over time, but you can't predict how well by testing them solely on trivia that constitutes their current main group. My preference is to hire for self-reliance and learn bookworm stuff later. I used to believe the opposite (e.g. hire someone good at math because they can always learn to be an effective programmer later) but my job experience made me believe the opposite (e.g. actually it's pretty easy to teach people stochastic processes, machine learning, or cryptography, but it's incredibly hard to teach people how to be good at creative software design).