6 ms·

I wonder if this could act as a negative filter for good programmers. Who worth their salt is going to invest 3 hours of their time just to submit a resume? Wou

by abalone 13y ago

I wonder if this could act as a negative filter for good programmers. Who worth their salt is going to invest 3 hours of their time just to submit a resume? Wouldn't you want to talk to the employer first to see if they're a good fit? In a tight labor market, people willing to do this are more desperate than others. So it's going to filter for desperate developers who passed a 3 hour basic skills test. Old fashioned prior experience and education is probably still a better first filter. This would probably make more sense specifically for junior hires right out of school or with no experience.

balajis 13y ago
Well, empirically we're getting quite a lot of people attempting it! :) Three hours is pretty lenient (you could blaze through it in less time), though the last question is challenging. We think the questions are kind of fun as well. Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. This process is more fair and (IMO) allows someone without a college degree, or from outside the US, to compete on a level playing field.
- jay_kyburz 13y ago
  Do you mind if people do the test with no interest in the job?
  - balajis 13y ago
    Sure, go ahead! We just ask that you not share the questions with others after completing it.
- 7Figures2Commas 13y ago
  > Well, empirically we're getting quite a lot of people attempting it! > Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. So instead of hundreds or thousands of unstructured resumes in different formats, you have hundreds or thousands of completed programming tests. Even assuming that you automatically filter out submissions that fail testing, given that there are a lot of ways to solve a problem, you'd still be stuck evaluating potentially hundreds of correct submissions. More importantly, a test doesn't seem to be aligned with your top priority: > ...but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Are you assuming that all of the folks most accomplished in the real world are going to complete your programming test successfully? Since you ultimately seem to be interested in folks who can build dashboards, I would think you'd be far better off asking candidates to build a dashboard prototype using sample data you provide. There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.
  - dlss 13y ago
    > You'd still be stuck evaluating potentially hundreds of correct submissions. I'm guessing the MVP of hackerrank was the web code editor + lint/findbugs/whatever for filtering. A lot of people have spent a lot of time writing open source projects that convert code to metrics, so making a reasonably prioritized list is probably fairly easy. > There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application. Agree! I think this is a great critique of HackerRank as a business idea -- while they are new, it's just a hurdle that weeds the busier coders out of your hiring pool... if they become established there will be books on cramming for it, and will no longer separate the wheat from the chaff. (ie https://web.archive.org/web/20130114163457/https://raganwald.posterous.com/i-dont-hire-unlucky-people https://web.archive.org/web/20130114163457/https://raganwald... meets http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law)
    - lifeisstillgood 13y ago
      To be honest I think it's the other way round - if you can do programming tests all day long you will succeed at most coding jobs. However there are a lot of developers who can't tackle an interview that with a nights sleep and no pressure can easily code a intuitive and effective architecture. Sorry - programming tests raise far more false negatives than false positives (and this from a person who considers himself a false negative in most tests)
      - dlss 13y ago
        I'm not sure what kind of coding work you do, but I assure you the vast majority of web development work has little resemblance to coding questions. You are almost never designing an algorithm: that was already done for you by whoever designed the library you are using. Majority of the tasks IME: - code reading & refactoring - figuring out why two libraries aren't playing nice - bug triangulation - optimization (not the algorithmic type, but rather the fix-the-auto-generated-SQL-query / frontend asset packing kind) - repetitive/simple/obvious bits of jQuery Maybe there's some kind of online test that could exercise those skills, but if you ask me the main performance changers for modern web development are, they're: really knowing the stack (problem anticipation, what to use, how to use the various tools to quickly get search keywords), google fu, and most importantly having done it a couple hundred times (muscle memory).
- abalone 13y ago
  > Well, empirically we're getting quite a lot of people attempting it! I think you missed my point. It's that the people submitting to a multi-hour exam just to submit a resume are more desperate / value their time less than those that can stand on their experience. The fact that there a "quite a lot" of submissions only worsens the probability that the multi-hour effort will be wasted.
- zodiac 13y ago
  >> allows someone ... from outside the US, to compete on a level playing field. Just curious, why do you say that the normal process doesn't allow people outside the US to compete on a level playing field? Higher barrier to on-site interviews?
- emsy 13y ago
  The questions were pretty good, because they were close to the real world and you couldn't google them so easily. The programming tasks were rather unbalanced. The first two were super easy, while you can easily get stuck on the last one, even as a good programmer. I knew that I wouldn't be able to answer it today, so I skipped it after a short while. If I'd plan such a test, I'd try to get an example of the structure and comprehensibility of my future employees work.
lordmax 13y ago
3 hours is upper bound and extremely friendly one to allow for all but hopelessly incompetent candidates. I did the test in less than 90 min. First 27 questions should take a competent hacker less than 20 min; rest of the time is for last 3 coding assignments.
deleted 13y ago
[deleted]
gcb0 13y ago
I'm saving it for the weekend. Will read about all the virtualization techniques for those spoj like sites and then have a go in 3hours trying to break that one :-)
jheriko 13y ago
one of the best programmers i know refuses to continue with a job application if there is a programming test... its a bit eccentric but valid given that his experience and past achievements say more than a programming test ever could.
- danielbarla 13y ago
  I have come across this opinion as well from some excellent developers, but I can't say that I agree with it. All in all, a well-executed* technical test can be an excellent filter and basis for further interviews. * By well-executed, I mean - relatively language agnostic, lacking in trick questions and meaningless trivia, and being open to multiple good answers. The intention is to not only check basic ability to solve simple problems, but the general approach and programming style of the individual interviewed. Past experience is often vaguely stated in CVs and looks great on paper, but can be disingenous. It takes a thorough two-way discussion which is customised to each individual to reveal this, however. Some people have excellent and highly visible past achievements, but not everyone does. I wouldn't throw out a CV simply because the individual doesn't have any open source code to show me, nor do they have any awards from companies they have worked for (which is often more about self-marketing than actual ability). I guess your mileage may vary based on the position that's in question and the companies involved, but I've seen some shocking discrepancies between CVs and actual programming ability. Even worse than the usual "fizz-buzz" statistics that get thrown around.
  - ukd1 13y ago
    Agreed re discrepancies: I've had CS Majors / Masters fail, yet my self-taught CEO passes. This amazes me. [edit; the cs majors amaze me]
    - jheriko 13y ago
      why does it amaze you? my experience of 'self taught' individuals in the general case is that they have a rich understanding of the subject by necessity. conversely my experience of education is that it trains you to solve education style problems - which are essentially absent from the real world - where being able to invent and implement a slightly broken solution in an hour with 30 seconds notice is much more valuable than being able to solve a problem you know about far in advance and are given loads of information and guidance to help in preparation for it. on the other hand if i wanted someone to do research then a CS major might be a better choice. :)

by abalone 13y ago

balajis 13y ago

Well, empirically we're getting quite a lot of people attempting it! :) Three hours is pretty lenient (you could blaze through it in less time), though the last question is challenging. We think the questions are kind of fun as well. Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. This process is more fair and (IMO) allows someone without a college degree, or from outside the US, to compete on a level playing field.

jay_kyburz 13y ago
Do you mind if people do the test with no interest in the job?
- balajis 13y ago
  Sure, go ahead! We just ask that you not share the questions with others after completing it.
7Figures2Commas 13y ago
> Well, empirically we're getting quite a lot of people attempting it! > Without something like this we'd just have 100s-1000s of unstructured resumes in different formats to rank against each other. So instead of hundreds or thousands of unstructured resumes in different formats, you have hundreds or thousands of completed programming tests. Even assuming that you automatically filter out submissions that fail testing, given that there are a lot of ways to solve a problem, you'd still be stuck evaluating potentially hundreds of correct submissions. More importantly, a test doesn't seem to be aligned with your top priority: > ...but the most important requirement is significant independent programming experience as demonstrated by your GitHub account, personal projects, academic publications, or startup success. Are you assuming that all of the folks most accomplished in the real world are going to complete your programming test successfully? Since you ultimately seem to be interested in folks who can build dashboards, I would think you'd be far better off asking candidates to build a dashboard prototype using sample data you provide. There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application.
- dlss 13y ago
  > You'd still be stuck evaluating potentially hundreds of correct submissions. I'm guessing the MVP of hackerrank was the web code editor + lint/findbugs/whatever for filtering. A lot of people have spent a lot of time writing open source projects that convert code to metrics, so making a reasonably prioritized list is probably fairly easy. > There are lot of really smart developers out there with advanced degrees who could solve programming quizzes all day long but couldn't design and implement a useful dashboard application. Agree! I think this is a great critique of HackerRank as a business idea -- while they are new, it's just a hurdle that weeds the busier coders out of your hiring pool... if they become established there will be books on cramming for it, and will no longer separate the wheat from the chaff. (ie https://web.archive.org/web/20130114163457/https://raganwald.posterous.com/i-dont-hire-unlucky-people https://web.archive.org/web/20130114163457/https://raganwald... meets http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law)
  - lifeisstillgood 13y ago
    To be honest I think it's the other way round - if you can do programming tests all day long you will succeed at most coding jobs. However there are a lot of developers who can't tackle an interview that with a nights sleep and no pressure can easily code a intuitive and effective architecture. Sorry - programming tests raise far more false negatives than false positives (and this from a person who considers himself a false negative in most tests)
    - dlss 13y ago
      I'm not sure what kind of coding work you do, but I assure you the vast majority of web development work has little resemblance to coding questions. You are almost never designing an algorithm: that was already done for you by whoever designed the library you are using. Majority of the tasks IME: - code reading & refactoring - figuring out why two libraries aren't playing nice - bug triangulation - optimization (not the algorithmic type, but rather the fix-the-auto-generated-SQL-query / frontend asset packing kind) - repetitive/simple/obvious bits of jQuery Maybe there's some kind of online test that could exercise those skills, but if you ask me the main performance changers for modern web development are, they're: really knowing the stack (problem anticipation, what to use, how to use the various tools to quickly get search keywords), google fu, and most importantly having done it a couple hundred times (muscle memory).
abalone 13y ago
> Well, empirically we're getting quite a lot of people attempting it! I think you missed my point. It's that the people submitting to a multi-hour exam just to submit a resume are more desperate / value their time less than those that can stand on their experience. The fact that there a "quite a lot" of submissions only worsens the probability that the multi-hour effort will be wasted.
zodiac 13y ago
>> allows someone ... from outside the US, to compete on a level playing field. Just curious, why do you say that the normal process doesn't allow people outside the US to compete on a level playing field? Higher barrier to on-site interviews?
emsy 13y ago
The questions were pretty good, because they were close to the real world and you couldn't google them so easily. The programming tasks were rather unbalanced. The first two were super easy, while you can easily get stuck on the last one, even as a good programmer. I knew that I wouldn't be able to answer it today, so I skipped it after a short while. If I'd plan such a test, I'd try to get an example of the structure and comprehensibility of my future employees work.

lordmax 13y ago

3 hours is upper bound and extremely friendly one to allow for all but hopelessly incompetent candidates. I did the test in less than 90 min. First 27 questions should take a competent hacker less than 20 min; rest of the time is for last 3 coding assignments.

deleted 13y ago

[deleted]

gcb0 13y ago

I'm saving it for the weekend. Will read about all the virtualization techniques for those spoj like sites and then have a go in 3hours trying to break that one :-)

jheriko 13y ago

one of the best programmers i know refuses to continue with a job application if there is a programming test... its a bit eccentric but valid given that his experience and past achievements say more than a programming test ever could.

danielbarla 13y ago
I have come across this opinion as well from some excellent developers, but I can't say that I agree with it. All in all, a well-executed* technical test can be an excellent filter and basis for further interviews. * By well-executed, I mean - relatively language agnostic, lacking in trick questions and meaningless trivia, and being open to multiple good answers. The intention is to not only check basic ability to solve simple problems, but the general approach and programming style of the individual interviewed. Past experience is often vaguely stated in CVs and looks great on paper, but can be disingenous. It takes a thorough two-way discussion which is customised to each individual to reveal this, however. Some people have excellent and highly visible past achievements, but not everyone does. I wouldn't throw out a CV simply because the individual doesn't have any open source code to show me, nor do they have any awards from companies they have worked for (which is often more about self-marketing than actual ability). I guess your mileage may vary based on the position that's in question and the companies involved, but I've seen some shocking discrepancies between CVs and actual programming ability. Even worse than the usual "fizz-buzz" statistics that get thrown around.
- ukd1 13y ago
  Agreed re discrepancies: I've had CS Majors / Masters fail, yet my self-taught CEO passes. This amazes me. [edit; the cs majors amaze me]
  - jheriko 13y ago
    why does it amaze you? my experience of 'self taught' individuals in the general case is that they have a rich understanding of the subject by necessity. conversely my experience of education is that it trains you to solve education style problems - which are essentially absent from the real world - where being able to invent and implement a slightly broken solution in an hour with 30 seconds notice is much more valuable than being able to solve a problem you know about far in advance and are given loads of information and guidance to help in preparation for it. on the other hand if i wanted someone to do research then a CS major might be a better choice. :)