9 ms·

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularit

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying to leverage that correlation for hiring purposes probably also made it go away. Now you've got a whole lot of people practicing hard on these sorts of problems, spending huge chunks of their free time grinding away on Project Euler and Advent of Code and HackerRank. That muddies the quality of this stuff as a proxy for what it was originally trying to detect: natural aptitude. I'm guessing having time to level grind like that also correlates inversely with other traits that are desirable in a programmer.

dfilppi 8y ago
The tests have become a pure filter, screen out those who dont study heavily prior to taking them. They are uncorrelated from the candidates ability at every day tasks.
- chillacy 8y ago
  It’s like an industry hazing ritual. How bad do you want it?
  - bunderbunder 8y ago
    Hmm. Y'know, that's a great point. I could build a whole conspiracy theory off of that idea: If I'm a FAANG, I'm simply not using my normal interview process to hire for the really interesting jobs. I reserve those ones for people who got the job by virtue of their publication history in the academic literature, or because they built some well-known cool thing, or because they got promoted into the position. Those people get shunted over into the "you didn't come to us, we came to you" interview process. The seats I'm looking to fill with the more public interview process are mostly seats for the grunt coders who work under those people. My ideal candidate for that position isn't some rock star creative genius; it's a workaholic who is resistant to boredom. And what's something a workaholic who's resistant to boredom would be really good at? Grinding away on programming interview questions, of course.
    - geoka9 8y ago
      But they are not supposed to be boring? At least to me they never are, it's just that they are rarely high on my list of important (or fun) things to do on my self-improvement time.
    - isoskeles 8y ago
      > or because they built some well-known cool thing What, like homebrew? :)
      - whatever_dude 8y ago
        > What, like homebrew? :) Context for everybody else: https://twitter.com/mxcl/status/608682016205344768?lang=en https://twitter.com/mxcl/status/608682016205344768?lang=en TL;DR: author of Homebrew interviews at Google, but doesn't get hired because he couldn't/wouldn't invert a binary tree on a whiteboard.
        illumin8 8y ago
        Wow, this is an incredible answer from Max on Quora as well: https://www.quora.com/Whats-the-logic-behind-Google-rejecting-Max-Howell-the-author-of-Homebrew-for-not-being-able-to-invert-a-binary-tree/answer/Max-Howell?share=100e0bb6&srid=unBJ9 https://www.quora.com/Whats-the-logic-behind-Google-rejectin...
    - louisswiss 8y ago
      This isn't a conspiracy theory - it's literally exactly what business schools do... Teach a bunch of people to think in a certain way, speak a certain language and respect authority. Someone who excels at the repetitive mundanity of business school will be a perfect junior marketing manager at BigCo. It's basically taking the way the Army trains new recruits and applying it to white collar jobs.
      - Gibbon1 8y ago
        Old quote I've forgotten the source but it's over 40 years old now. 'Business schools turn out well trained, amoral yet obedient clerks.' Can likely say the same about most CS programs. I know you can say that about engineering.
  - chooseaname 8y ago
    This is exactly what it is. Which Frat house do you want into? Google? FB? Microsoft? Amazon?
- oarabbus_ 8y ago
  Surely, this must eventually lead to negative results? It's curious because it doesn't seem to be having any negative impact as of now.
on_and_off 8y ago
It is a perverse effect on the profession as a whole. In the silicon valley I have a fair amount of colleagues that expect of any dev to spend a fair amount of their free time grinding on even more dev. No surprise that there is a lack of diversity in the profession as a result.
- whatshisface 8y ago
  Except for people over 50 who have probably realized there is more to life, why would some demographics be willing to grind less than others? Edit: By "over 50," I mean age demographics in general, which is the main thing that raises your family obligations. The only demographic division that I can think of that would reduce someone's willingness to abandon their personal life would be age.
  - sunir 8y ago
    They have kids. Developing a sentient being is a higher priority and more difficult than developing React Redux.
    - whatshisface 8y ago
      Wouldn't that just be another expression of age demographics?
      - jgamman 8y ago
        some people have kids in their 20s (gasp!)
    - wolfgke 8y ago
      > They have kids. Developing a sentient being is a higher priority and more difficult than developing React Redux. This is like saying: "I have a time-intense hobby that I prefer over working too much for you.".
      - 8y ago
soihrbfsodihbv 8y ago
I think it's wrong to narrowly conceive of interview questions as a test of "natural aptitude", as though we're just trying to measure the CPU specs of the candidate's brain. I've done really well in coding interviews by taking a few weeks to practice when I start looking for new jobs. I spend a few hours a night studying, put up a whiteboard in my house, and then interview for a dozen low-priority jobs to get really comfortable before going after the big guns. This has resulted in a near-perfect success rate in the application -> offer funnel. At the beginning of the process the idea of interviewing is anxiety-inducing. By the end, I'm completely in my element, it's fun and rewarding and I often blow the interviewers' minds. Now, hearing this story might piss you off if you think interviews are supposed to be measuring "natural aptitude" for software engineering. I posit that there is no such thing. The personal characteristics that actually exist are commitment and determination. And you need to be decent at communicating in English and a programming language or two. These characteristics that make someone succeed at interviews will also make them succeed at work. You need core knowledge about how to code, the ability to play well with others, and the willingness to apply yourself, to condition your brain to succeed at the task at hand. If you can do these things, making software is a piece of cake. In short, if you're too afraid of adversity to get good at interviewing, how well can you fare in the jungle that is a software company?
ohyes 8y ago
What’s frustrating to me is that we haven’t found a way to teach programming that doesn’t rely on natural aptitude. It’s really a travesty that we can’t teach it the same way that we teach maths or natural languages. The end result is we’re left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.
- trhway 8y ago
  using literacy model - we're testing people to write haiku according to all the rules of style and form when the job we're testing for is basically just low level clerical like copying documents (using copy machine), sometimes taking notes and filling simple forms.
- gizmo686 8y ago
  As someone who has studied both math and computer science (and is now a professional programmer), I have no idea what you are talking about. We are far better at teaching people to program than we are at teaching them to do math. Its just that most people self select out of math [0], so, due to selection bias, it appears that we are great at teaching those that remain. As an aside, as someone who also studied linguistics [1] (and a couple of foreign languages at a beginner level), I am very confident in saying that our approach to teaching natural languages relies almost entirely on natural aptitude. It is just that, absent a serious mental disorder, all humans have a very large natural aptitude for natural languages [0] Probably because we are absolutly terrible at teaching it. [1] I assume not what you mean by teaching natural languages, but the topic of language acuisition (including in adults) does come up; plus it gives some perspective on how teaching language would look if we didn't rely on natural aptitude.
  - Gibbon1 8y ago
    I took math as part of my engineering degree. And that was 30 years ago. The older I get the more I think the way it was taught was terrible. Same reason I think teachers are bitching about the US's fetish for academic testing. Testing pressures teachers to teach students to mechanically solve problems. But with shallow understanding.
Balgair 8y ago
I'm not sure if HackerRank has updated itself recently, but the last time I poked my head in there (years ago), all the answers were in the "Talk about this challenge" section. You'd just go in there and copy paste the code in, maybe change the variable names/order a bit. Ever since I learned that, it's never been that 'wowzers' for me.
- meko 8y ago
  Isn't the whole point of the challenge to figure out how to do the problem? What you say is literally just copy/pasting others' answers. Is that not looked down on?
  - Izkata 8y ago
    Given the name of the site (I've never used it), I always assumed it was about proving your ability to others (like maybe future employers). Easily-found answers totally destroys the trust that's needed for such a thing to work.
    - Balgair 8y ago
      Bingo. Years back, in what small circles I run in, the score was something to toot your horn about. When I discovered that you could 'cheat' on the scoring, it took away all the prestige from that ranking score. Honestly, other than practice, what use is it?
- pedrosorio 8y ago
  "Ever since I learned that, it's never been that 'wowzers' for me." I am not sure I follow. For people who want to, there is a section where they can find answers even if they have not solved the problems. What was 'wowzers' about HackerRank before you learned that?
  - Balgair 8y ago
    Well, some folks would brag about their score (years ago, dunno about now) and use that ranking as an indicator of their coding prowess. When I learned that the whole thing was easy to 'cheat' (whatever that mean in this context), then that rank lost all prestige for me.
bitwize 8y ago
The companies have caught on to this and nowadays, even if you passed the technical bits with flying colors, they look to rule you out or rank you based on softer criteria, like how well will you fit with our team? IOW, how much like us are you?

dfilppi 8y ago

The tests have become a pure filter, screen out those who dont study heavily prior to taking them. They are uncorrelated from the candidates ability at every day tasks.

chillacy 8y ago
It’s like an industry hazing ritual. How bad do you want it?
- bunderbunder 8y ago
  Hmm. Y'know, that's a great point. I could build a whole conspiracy theory off of that idea: If I'm a FAANG, I'm simply not using my normal interview process to hire for the really interesting jobs. I reserve those ones for people who got the job by virtue of their publication history in the academic literature, or because they built some well-known cool thing, or because they got promoted into the position. Those people get shunted over into the "you didn't come to us, we came to you" interview process. The seats I'm looking to fill with the more public interview process are mostly seats for the grunt coders who work under those people. My ideal candidate for that position isn't some rock star creative genius; it's a workaholic who is resistant to boredom. And what's something a workaholic who's resistant to boredom would be really good at? Grinding away on programming interview questions, of course.
  - geoka9 8y ago
    But they are not supposed to be boring? At least to me they never are, it's just that they are rarely high on my list of important (or fun) things to do on my self-improvement time.
  - isoskeles 8y ago
    > or because they built some well-known cool thing What, like homebrew? :)
    - whatever_dude 8y ago
      > What, like homebrew? :) Context for everybody else: https://twitter.com/mxcl/status/608682016205344768?lang=en https://twitter.com/mxcl/status/608682016205344768?lang=en TL;DR: author of Homebrew interviews at Google, but doesn't get hired because he couldn't/wouldn't invert a binary tree on a whiteboard.
      - illumin8 8y ago
        Wow, this is an incredible answer from Max on Quora as well: https://www.quora.com/Whats-the-logic-behind-Google-rejecting-Max-Howell-the-author-of-Homebrew-for-not-being-able-to-invert-a-binary-tree/answer/Max-Howell?share=100e0bb6&srid=unBJ9 https://www.quora.com/Whats-the-logic-behind-Google-rejectin...
  - louisswiss 8y ago
    This isn't a conspiracy theory - it's literally exactly what business schools do... Teach a bunch of people to think in a certain way, speak a certain language and respect authority. Someone who excels at the repetitive mundanity of business school will be a perfect junior marketing manager at BigCo. It's basically taking the way the Army trains new recruits and applying it to white collar jobs.
    - Gibbon1 8y ago
      Old quote I've forgotten the source but it's over 40 years old now. 'Business schools turn out well trained, amoral yet obedient clerks.' Can likely say the same about most CS programs. I know you can say that about engineering.
- chooseaname 8y ago
  This is exactly what it is. Which Frat house do you want into? Google? FB? Microsoft? Amazon?
oarabbus_ 8y ago
Surely, this must eventually lead to negative results? It's curious because it doesn't seem to be having any negative impact as of now.

on_and_off 8y ago

It is a perverse effect on the profession as a whole. In the silicon valley I have a fair amount of colleagues that expect of any dev to spend a fair amount of their free time grinding on even more dev. No surprise that there is a lack of diversity in the profession as a result.

whatshisface 8y ago
Except for people over 50 who have probably realized there is more to life, why would some demographics be willing to grind less than others? Edit: By "over 50," I mean age demographics in general, which is the main thing that raises your family obligations. The only demographic division that I can think of that would reduce someone's willingness to abandon their personal life would be age.
- sunir 8y ago
  They have kids. Developing a sentient being is a higher priority and more difficult than developing React Redux.
  - whatshisface 8y ago
    Wouldn't that just be another expression of age demographics?
    - jgamman 8y ago
      some people have kids in their 20s (gasp!)
  - wolfgke 8y ago
    > They have kids. Developing a sentient being is a higher priority and more difficult than developing React Redux. This is like saying: "I have a time-intense hobby that I prefer over working too much for you.".
    - 8y ago

soihrbfsodihbv 8y ago

I think it's wrong to narrowly conceive of interview questions as a test of "natural aptitude", as though we're just trying to measure the CPU specs of the candidate's brain. I've done really well in coding interviews by taking a few weeks to practice when I start looking for new jobs. I spend a few hours a night studying, put up a whiteboard in my house, and then interview for a dozen low-priority jobs to get really comfortable before going after the big guns. This has resulted in a near-perfect success rate in the application -> offer funnel. At the beginning of the process the idea of interviewing is anxiety-inducing. By the end, I'm completely in my element, it's fun and rewarding and I often blow the interviewers' minds. Now, hearing this story might piss you off if you think interviews are supposed to be measuring "natural aptitude" for software engineering. I posit that there is no such thing. The personal characteristics that actually exist are commitment and determination. And you need to be decent at communicating in English and a programming language or two. These characteristics that make someone succeed at interviews will also make them succeed at work. You need core knowledge about how to code, the ability to play well with others, and the willingness to apply yourself, to condition your brain to succeed at the task at hand. If you can do these things, making software is a piece of cake. In short, if you're too afraid of adversity to get good at interviewing, how well can you fare in the jungle that is a software company?

ohyes 8y ago

What’s frustrating to me is that we haven’t found a way to teach programming that doesn’t rely on natural aptitude. It’s really a travesty that we can’t teach it the same way that we teach maths or natural languages. The end result is we’re left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.

trhway 8y ago
using literacy model - we're testing people to write haiku according to all the rules of style and form when the job we're testing for is basically just low level clerical like copying documents (using copy machine), sometimes taking notes and filling simple forms.
gizmo686 8y ago
As someone who has studied both math and computer science (and is now a professional programmer), I have no idea what you are talking about. We are far better at teaching people to program than we are at teaching them to do math. Its just that most people self select out of math [0], so, due to selection bias, it appears that we are great at teaching those that remain. As an aside, as someone who also studied linguistics [1] (and a couple of foreign languages at a beginner level), I am very confident in saying that our approach to teaching natural languages relies almost entirely on natural aptitude. It is just that, absent a serious mental disorder, all humans have a very large natural aptitude for natural languages [0] Probably because we are absolutly terrible at teaching it. [1] I assume not what you mean by teaching natural languages, but the topic of language acuisition (including in adults) does come up; plus it gives some perspective on how teaching language would look if we didn't rely on natural aptitude.
- Gibbon1 8y ago
  I took math as part of my engineering degree. And that was 30 years ago. The older I get the more I think the way it was taught was terrible. Same reason I think teachers are bitching about the US's fetish for academic testing. Testing pressures teachers to teach students to mechanically solve problems. But with shallow understanding.

Balgair 8y ago

I'm not sure if HackerRank has updated itself recently, but the last time I poked my head in there (years ago), all the answers were in the "Talk about this challenge" section. You'd just go in there and copy paste the code in, maybe change the variable names/order a bit. Ever since I learned that, it's never been that 'wowzers' for me.

meko 8y ago
Isn't the whole point of the challenge to figure out how to do the problem? What you say is literally just copy/pasting others' answers. Is that not looked down on?
- Izkata 8y ago
  Given the name of the site (I've never used it), I always assumed it was about proving your ability to others (like maybe future employers). Easily-found answers totally destroys the trust that's needed for such a thing to work.
  - Balgair 8y ago
    Bingo. Years back, in what small circles I run in, the score was something to toot your horn about. When I discovered that you could 'cheat' on the scoring, it took away all the prestige from that ranking score. Honestly, other than practice, what use is it?
pedrosorio 8y ago
"Ever since I learned that, it's never been that 'wowzers' for me." I am not sure I follow. For people who want to, there is a section where they can find answers even if they have not solved the problems. What was 'wowzers' about HackerRank before you learned that?
- Balgair 8y ago
  Well, some folks would brag about their score (years ago, dunno about now) and use that ranking as an indicator of their coding prowess. When I learned that the whole thing was easy to 'cheat' (whatever that mean in this context), then that rank lost all prestige for me.

bitwize 8y ago

The companies have caught on to this and nowadays, even if you passed the technical bits with flying colors, they look to rule you out or rank you based on softer criteria, like how well will you fit with our team? IOW, how much like us are you?