9 ms·

At my previous company, I was the hiring manager and gave a homework assignment that was targeted a little shorter than you describe, at right about 2-3 hours.

by lbatx 10y ago

At my previous company, I was the hiring manager and gave a homework assignment that was targeted a little shorter than you describe, at right about 2-3 hours. I was clear upfront that it would take about that long, though if you wanted to take longer, you could. Within reason, I didn't hold the time against the person if they took longer. The crazy thing was that over half of the applicants that passed a resume screen and phone interview could not complete the homework assignment. Over half. Some couldn't get it at all (either gave up or had compile errors), some got it to run but it didn't work, a few got it to mostly work but their programs had serious bugs, and the last few actually got it right. The assignment was to read a file containing a list of numbers (some formatted incorrectly, so there was some very simple parsing logic involved), call an API using each correctly formatted number as a parameter, and store what the API returned to a file. I'm still shocked over half of developers couldn't accomplish that task. Makes me wonder what they were accomplishing in their existing roles.

vinhboy 10y ago
Thanks for sharing. I think this is a good data point to prove that most people will not cheat on their take home. There is no incentive to cheat. Why would you want a job you are not qualified to do. From your experience, it sounds like these are good tests, because it's screening for qualified candidates, unlike the phone interview.
- gambogi 10y ago
  > Why would you want a job you are not qualified to do. money.
  - gaius 10y ago
    Also, credentialling. Even if you can't do the work and only stay a few months, Google or Goldman or another big name on your CV will still open many doors.
    - wolfgke 10y ago
      In Germany there is a culture of not changing jobs very often (at most every few years, but even this should not happen too often). So if you have a "well-known" name such as Google or Goldman Sachs on your CV, but worked there only for few months, it is a clear sign that you were fired and about every future employer will be very cautious.
      - blisterpeanuts 10y ago
        For medium to large companies, this is true in the U.S. as well. Six months at Facebook, Google, or Bank of America would be a huge red flag.
        gaius 10y ago
        That person will always be the "Goldman alumni" or the "ex-Googler". Couple that with a semi-prestigious degree and they'll be set.
      - user5994461 10y ago
        What if you joined Goldman Sacchs and was poached by JP Morgan after 6 months? What if you joined Google and was poached by Facebook after 1 year?
        wolfgke 10y ago
        1. What is most important in your CV is the last employer. If it seems that you were fired, this can cause problems with getting a (decent) job afterwards. If you were poached, it is because the other employer wants you badly. So in your example your new "last" employer becomes JP Morgan or Facebook. So this kind of problem does not occur here. This does not contradict the fact that it still leaves/can leave a bad impression on impression if your were willing to be poached without a very good reason, since can be considered as a strong sign that your are not loyal to your employer and only hunting for money. 2. Poaching is rather uncommon in Germany. As I outlined in 1 being willing to be poached multiple times is a sign that you are not very loyal to your employer. Thus there is a strongly reduced incentive for employees to be poached. Also for poaching to work, you have to offer lots of money - I'd be really surprised if there are lots of employers that are willing to start such an "arms race with money". Why not simply offer decent money in your regular job advertisements and get good people without any arms race? Poaching might make sense if there you want to get "rockstars" that are not available on the "free job market", but can only be poached. But I'm also not aware of any industry sector in Germany where there exist employees that are treated as rockstars. :-(
- koolba 10y ago
  Are you kidding me? People will cheat just for fake internet points. With real cash on the line, they'd do just about anything.
  - mathgeek 10y ago
    This. If people didn't lie (or stretch the truth) or cheat, we would only need resumes to know whether someone can do the technical aspects of any job.
- matwood 10y ago
  I don't even care if the person 'cheats' because I'll figure it out when we discuss the solution.
  - avn2109 10y ago
    This has happened at my company. We give a similarly near-trivial hw problem, and sometimes people come in and literally "cannot remember" any single thing about the assignment, including the names of the columns in the data or the approximate objective of the HW. This is after performing plausibly well on a phone screen etc.
  - vinhboy 10y ago
    Thanks for clarifying this. This is what I meant when I said "Why would you want a job you are not qualified to do." I am not so good with words sometimes... Haha...
- dottedmag 10y ago
  From my recruiting experience people cheat even during live-coding interview. I was pretty amused seeng a person whispering to a buddy outside of the video pickup area trying to get _any_ code written. Why do they do it? I have no idea.
pryelluw 10y ago
What kind of values did you have to validate from the file? Im tempted to do this myself right now just for the fun of it. Can you provide a sample?
mdip 10y ago
> I'm still shocked over half of developers couldn't accomplish that task. Makes me wonder what they were accomplishing in their existing roles. I once asked a candidate to write a for loop in C#[0], using my laptop and Visual Studio and watched for a solid 5 minutes while he struggled -- And I don't mean jitters around syntax, I mean, failing to declare a variable properly and while he knew what a for loop was, he had no idea how to write one in C#. His resume touted a series of projects that we estimated put him a little above a Junior developer and the only language he referenced was C#. I don't normally do these kinds of "on the spot tests" because I, personally, hate them, but I was cheering when I read Coding Horror's "Fizz Buzz" article[1] on the subject. This was the most foul example and my only experience is with developer and ops/security positions, but if you're getting half of developers who are in the same time zone as the title[2], you're doing well. Does this happen in other parts of the business world? Some of the folks we get in would be like a Walmart cashier applying to run a finance department because "they've handled money before". And what if they got the job, do they think they'd come close to being successful? I'm all for the "fake it until you make it" and I'm very willing to overlook weaknesses around experience/education if the candidate is really passionate and I can get enough of a comfort level that they'll be a quick study and love the job, but you have to be realistic about your deficiencies. A lot of the problem, I've found (being on both sides of this in the past) is with recruiters, as well. They tend to operate on a "throw darts at the wall, blindfolded" approach, with the assumption that if they send out a million candidates for a million jobs without regard for qualifications, that they'll get lucky, one will land somewhere and they'll get a commission from sheer volume on bad odds. This serves nobody well -- it causes companies to put up multi-step gates that involve people who aren't close enough to the field to discern a good candidate from a bad one via resume performing a filter operation that disproportionately disqualifies the really passionate but "not quite a perfect fit on the qualifications side" and it causes job seekers to take a bunch of interviews for positions they won't like/aren't qualified for and won't land -- causing them to have a miserable experience interviewing (and probably resulting in some dropping out of the job market entirely). [0] The exercise was, specifically, write a for loop with an int value that is incremented by one, adds that int value to another int variable, and multiplies the result against the counter, assigning the value to a third variable. Print the result to the console. This was before "Fizz Buzz" became all the rage, I did my best to give a blindingly simple problem to just validate that the person could do something in the language. [1] https://blog.codinghorror.com/why-cant-programmers-program/ https://blog.codinghorror.com/why-cant-programmers-program/ [2] Never mind the details "Must have 4 years of experience in Angular", nonsense -- a topic for another post, but the way we write job requirements is about stupid. I mean, I've seen experience requirements that exceed the length of time the technology was in existence, and I'd rather see a candidate demonstrate an ability to adapt and learn new things than be overly concerned with whether or not they've used a framework that is really similar to a bunch of other frameworks.
swalsh 10y ago
When you first started talking about a "homework assignment" my stomach started turning. But reading a file, doing some parsing adjustments, and calling an API sounds like a small amount of python code... so I guess not so bad :\
- jasikpark 10y ago
  I'm just impressed that this is a 2-3 hr task.
  - seanp2k2 10y ago
    same. Seems like: for num in $(egrep --only-matching '^[0-9]+' input-file); do curl -vv -X GET "http://example.com/api/v1/foo?num=${num}" -o "resp-${num}"; done edit: assuming "parameter" means query param
    - elmigranto 10y ago
      Okay. Now you're sitting with an interviewer to talk about your home asssigment. Turns out there were some changes in a system and you program needs to evolve accordingly: actual format is protobuff struct and API is asynchronous — you send a bunch of tasks, server queues them up and gives you ids, after some time, results are pushed to a client as they come up. You must store data in SQLite and account for HTTP or REST errors (retry logic, etc.). Your one liner is neat and all, but now you have to do 2-3 hours of initial work + w/ever time it takes for adjustments in 2 hours scheduled for an interview. You could've thought out this toy architecture up front with initial simple implementations or at least discuss what they expect you to provide, but instead you wrote a shell script for some nefarious reason, even though you saw obvious disconnect between estimated time and time you actually spent.
      - deleted 10y ago
        [deleted]
      - swalsh 10y ago
        Nah, i'd hire the guy with the 1 liner over the guy with the robust architecture any day. A robust architecture is great and all, but I think it should be built after a need for it has been proven. When I was a junior developer, I was just trying to get code to work. When I learned design patterns, and had some bad experiences, then I started architect "robust" architectures for everything. After perhaps 6 or 7 years of doing this, I started to question my insanity. Sure there were several instances where I was like "Damn, really glad I built it like that" but there were an equal amount where it just ended up turning a simple problem into a complex problem. There's no good excuse for that. So now, I go high school simple first, and college hard later. If it means I have to do some rework, fine. I'm okay with a little rework, because in the end it aggregates to a simpler platform that took less time to build.
deepGem 10y ago
Just curious, what was the size of your file and did you propose any latency limitations for the API calls ?
oh_sigh 10y ago
Considering how few 'developers' could pass FizzBuzz, it shouldn't surprise you any more.
hinkley 10y ago
The last two Homework assignments I got took about 50% longer than the hiring person suggested they would take. I'm perfectly competent at writing complex code. In fact I'm usually the guy who fixes bugs or perf issues that nobody else believes exist. A problem you came up with on your own is going to seem simpler to you than to everyone else. Why? Because you thought you picked it at random but your subconscious picked the one it thought was good. In other words, you picked the one you were already primed to answer, and it seemed simple. So either people are going to work at it longer, the timid will give up (there's nothing wrong with timid coworkers as long as they have someone to listen to them), and the brash will get it done in the time allotment by cutting corners. Are those the people you were trying to hire? Because that's who you're gonna get.
- chillydawg 10y ago
  I'm sorry but the problem is a trivial one. Even if you've never read a file, never made a http call in code or written to a file, these are all things a 5 minute stack overflow away. I absolutely would not hire someone who failed that test. Your defence of it seems bizarre to me.
  - kevin_nisbet 10y ago
    chillydawg, I take hinkley's assertion to be more of a warning to be careful about biases that may have inadvertently been introduced through the testing methodology. I think it's impossible for us to know without seeing the test you use, but it's always possible that the requirements aren't as clear as you think they are, or potentially what you deem as trivial may be biased from recent direct experience on a problem that you've then turned into a test. I think these tests are a great tool, but like anything we all just need to be aware and careful of the biases introduced. Also, we need to be mindful that really good developers who are employed, have families etc, may not have an over abundance of time to spend on these, so if 10 companies each send a challenge that takes 4 hours, and it's unclear if the assessment will lead to further interviews, some better candidates may give up on the process or be selective about the ones they complete. I think this creates an additional bias, that the people who complete your assessments may be the ones with the most time to invest in the process (ie the currently unemployed).
throwaway_374 10y ago
I'm going to assume "couldn't accomplish the task" means couldn't get it entirely correct. Do you provide a testing baseline that determine the expected behaviour? With TDD being the norm and correct way to test your program behaviour, I'm absolutely not going to go through every edge case with a fine tooth comb to check your "correctly formatted number" calls the API as you expect for a homework assignment. Provide a testing baseline or clearly demarcate the various possible edge cases as would be discussed in real life.
deleted 10y ago
[deleted]
caseymarquis 10y ago
What language was this in, out of curiousity? That timeframe seems very long for the task, but only because I tend to work with languages which have nice standard libraries.
deleted 10y ago
[deleted]
bbernoulli 10y ago
Can you provide anymore details about this? This doesn't seem like it would take 2-3 hours. I'm going to have to assess come applicants soon and have been trying to come up with something appropriate, which is why I'm interested.

by lbatx 10y ago

vinhboy 10y ago

Thanks for sharing. I think this is a good data point to prove that most people will not cheat on their take home. There is no incentive to cheat. Why would you want a job you are not qualified to do. From your experience, it sounds like these are good tests, because it's screening for qualified candidates, unlike the phone interview.

gambogi 10y ago
> Why would you want a job you are not qualified to do. money.
- gaius 10y ago
  Also, credentialling. Even if you can't do the work and only stay a few months, Google or Goldman or another big name on your CV will still open many doors.
  - wolfgke 10y ago
    In Germany there is a culture of not changing jobs very often (at most every few years, but even this should not happen too often). So if you have a "well-known" name such as Google or Goldman Sachs on your CV, but worked there only for few months, it is a clear sign that you were fired and about every future employer will be very cautious.
    - blisterpeanuts 10y ago
      For medium to large companies, this is true in the U.S. as well. Six months at Facebook, Google, or Bank of America would be a huge red flag.
      - gaius 10y ago
        That person will always be the "Goldman alumni" or the "ex-Googler". Couple that with a semi-prestigious degree and they'll be set.
    - user5994461 10y ago
      What if you joined Goldman Sacchs and was poached by JP Morgan after 6 months? What if you joined Google and was poached by Facebook after 1 year?
      - wolfgke 10y ago
        1. What is most important in your CV is the last employer. If it seems that you were fired, this can cause problems with getting a (decent) job afterwards. If you were poached, it is because the other employer wants you badly. So in your example your new "last" employer becomes JP Morgan or Facebook. So this kind of problem does not occur here. This does not contradict the fact that it still leaves/can leave a bad impression on impression if your were willing to be poached without a very good reason, since can be considered as a strong sign that your are not loyal to your employer and only hunting for money. 2. Poaching is rather uncommon in Germany. As I outlined in 1 being willing to be poached multiple times is a sign that you are not very loyal to your employer. Thus there is a strongly reduced incentive for employees to be poached. Also for poaching to work, you have to offer lots of money - I'd be really surprised if there are lots of employers that are willing to start such an "arms race with money". Why not simply offer decent money in your regular job advertisements and get good people without any arms race? Poaching might make sense if there you want to get "rockstars" that are not available on the "free job market", but can only be poached. But I'm also not aware of any industry sector in Germany where there exist employees that are treated as rockstars. :-(
koolba 10y ago
Are you kidding me? People will cheat just for fake internet points. With real cash on the line, they'd do just about anything.
- mathgeek 10y ago
  This. If people didn't lie (or stretch the truth) or cheat, we would only need resumes to know whether someone can do the technical aspects of any job.
matwood 10y ago
I don't even care if the person 'cheats' because I'll figure it out when we discuss the solution.
- avn2109 10y ago
  This has happened at my company. We give a similarly near-trivial hw problem, and sometimes people come in and literally "cannot remember" any single thing about the assignment, including the names of the columns in the data or the approximate objective of the HW. This is after performing plausibly well on a phone screen etc.
- vinhboy 10y ago
  Thanks for clarifying this. This is what I meant when I said "Why would you want a job you are not qualified to do." I am not so good with words sometimes... Haha...
dottedmag 10y ago
From my recruiting experience people cheat even during live-coding interview. I was pretty amused seeng a person whispering to a buddy outside of the video pickup area trying to get _any_ code written. Why do they do it? I have no idea.

pryelluw 10y ago

What kind of values did you have to validate from the file? Im tempted to do this myself right now just for the fun of it. Can you provide a sample?

mdip 10y ago

> I'm still shocked over half of developers couldn't accomplish that task. Makes me wonder what they were accomplishing in their existing roles. I once asked a candidate to write a for loop in C#[0], using my laptop and Visual Studio and watched for a solid 5 minutes while he struggled -- And I don't mean jitters around syntax, I mean, failing to declare a variable properly and while he knew what a for loop was, he had no idea how to write one in C#. His resume touted a series of projects that we estimated put him a little above a Junior developer and the only language he referenced was C#. I don't normally do these kinds of "on the spot tests" because I, personally, hate them, but I was cheering when I read Coding Horror's "Fizz Buzz" article[1] on the subject. This was the most foul example and my only experience is with developer and ops/security positions, but if you're getting half of developers who are in the same time zone as the title[2], you're doing well. Does this happen in other parts of the business world? Some of the folks we get in would be like a Walmart cashier applying to run a finance department because "they've handled money before". And what if they got the job, do they think they'd come close to being successful? I'm all for the "fake it until you make it" and I'm very willing to overlook weaknesses around experience/education if the candidate is really passionate and I can get enough of a comfort level that they'll be a quick study and love the job, but you have to be realistic about your deficiencies. A lot of the problem, I've found (being on both sides of this in the past) is with recruiters, as well. They tend to operate on a "throw darts at the wall, blindfolded" approach, with the assumption that if they send out a million candidates for a million jobs without regard for qualifications, that they'll get lucky, one will land somewhere and they'll get a commission from sheer volume on bad odds. This serves nobody well -- it causes companies to put up multi-step gates that involve people who aren't close enough to the field to discern a good candidate from a bad one via resume performing a filter operation that disproportionately disqualifies the really passionate but "not quite a perfect fit on the qualifications side" and it causes job seekers to take a bunch of interviews for positions they won't like/aren't qualified for and won't land -- causing them to have a miserable experience interviewing (and probably resulting in some dropping out of the job market entirely). [0] The exercise was, specifically, write a for loop with an int value that is incremented by one, adds that int value to another int variable, and multiplies the result against the counter, assigning the value to a third variable. Print the result to the console. This was before "Fizz Buzz" became all the rage, I did my best to give a blindingly simple problem to just validate that the person could do something in the language. [1] https://blog.codinghorror.com/why-cant-programmers-program/ https://blog.codinghorror.com/why-cant-programmers-program/ [2] Never mind the details "Must have 4 years of experience in Angular", nonsense -- a topic for another post, but the way we write job requirements is about stupid. I mean, I've seen experience requirements that exceed the length of time the technology was in existence, and I'd rather see a candidate demonstrate an ability to adapt and learn new things than be overly concerned with whether or not they've used a framework that is really similar to a bunch of other frameworks.

swalsh 10y ago

When you first started talking about a "homework assignment" my stomach started turning. But reading a file, doing some parsing adjustments, and calling an API sounds like a small amount of python code... so I guess not so bad :\

jasikpark 10y ago
I'm just impressed that this is a 2-3 hr task.
- seanp2k2 10y ago
  same. Seems like: for num in $(egrep --only-matching '^[0-9]+' input-file); do curl -vv -X GET "http://example.com/api/v1/foo?num=${num}" -o "resp-${num}"; done edit: assuming "parameter" means query param
  - elmigranto 10y ago
    Okay. Now you're sitting with an interviewer to talk about your home asssigment. Turns out there were some changes in a system and you program needs to evolve accordingly: actual format is protobuff struct and API is asynchronous — you send a bunch of tasks, server queues them up and gives you ids, after some time, results are pushed to a client as they come up. You must store data in SQLite and account for HTTP or REST errors (retry logic, etc.). Your one liner is neat and all, but now you have to do 2-3 hours of initial work + w/ever time it takes for adjustments in 2 hours scheduled for an interview. You could've thought out this toy architecture up front with initial simple implementations or at least discuss what they expect you to provide, but instead you wrote a shell script for some nefarious reason, even though you saw obvious disconnect between estimated time and time you actually spent.
    - deleted 10y ago
      [deleted]
    - swalsh 10y ago
      Nah, i'd hire the guy with the 1 liner over the guy with the robust architecture any day. A robust architecture is great and all, but I think it should be built after a need for it has been proven. When I was a junior developer, I was just trying to get code to work. When I learned design patterns, and had some bad experiences, then I started architect "robust" architectures for everything. After perhaps 6 or 7 years of doing this, I started to question my insanity. Sure there were several instances where I was like "Damn, really glad I built it like that" but there were an equal amount where it just ended up turning a simple problem into a complex problem. There's no good excuse for that. So now, I go high school simple first, and college hard later. If it means I have to do some rework, fine. I'm okay with a little rework, because in the end it aggregates to a simpler platform that took less time to build.

deepGem 10y ago

Just curious, what was the size of your file and did you propose any latency limitations for the API calls ?

oh_sigh 10y ago

Considering how few 'developers' could pass FizzBuzz, it shouldn't surprise you any more.

hinkley 10y ago

The last two Homework assignments I got took about 50% longer than the hiring person suggested they would take. I'm perfectly competent at writing complex code. In fact I'm usually the guy who fixes bugs or perf issues that nobody else believes exist. A problem you came up with on your own is going to seem simpler to you than to everyone else. Why? Because you thought you picked it at random but your subconscious picked the one it thought was good. In other words, you picked the one you were already primed to answer, and it seemed simple. So either people are going to work at it longer, the timid will give up (there's nothing wrong with timid coworkers as long as they have someone to listen to them), and the brash will get it done in the time allotment by cutting corners. Are those the people you were trying to hire? Because that's who you're gonna get.

chillydawg 10y ago
I'm sorry but the problem is a trivial one. Even if you've never read a file, never made a http call in code or written to a file, these are all things a 5 minute stack overflow away. I absolutely would not hire someone who failed that test. Your defence of it seems bizarre to me.
- kevin_nisbet 10y ago
  chillydawg, I take hinkley's assertion to be more of a warning to be careful about biases that may have inadvertently been introduced through the testing methodology. I think it's impossible for us to know without seeing the test you use, but it's always possible that the requirements aren't as clear as you think they are, or potentially what you deem as trivial may be biased from recent direct experience on a problem that you've then turned into a test. I think these tests are a great tool, but like anything we all just need to be aware and careful of the biases introduced. Also, we need to be mindful that really good developers who are employed, have families etc, may not have an over abundance of time to spend on these, so if 10 companies each send a challenge that takes 4 hours, and it's unclear if the assessment will lead to further interviews, some better candidates may give up on the process or be selective about the ones they complete. I think this creates an additional bias, that the people who complete your assessments may be the ones with the most time to invest in the process (ie the currently unemployed).

throwaway_374 10y ago

I'm going to assume "couldn't accomplish the task" means couldn't get it entirely correct. Do you provide a testing baseline that determine the expected behaviour? With TDD being the norm and correct way to test your program behaviour, I'm absolutely not going to go through every edge case with a fine tooth comb to check your "correctly formatted number" calls the API as you expect for a homework assignment. Provide a testing baseline or clearly demarcate the various possible edge cases as would be discussed in real life.

deleted 10y ago

[deleted]

caseymarquis 10y ago

What language was this in, out of curiousity? That timeframe seems very long for the task, but only because I tend to work with languages which have nice standard libraries.

deleted 10y ago

[deleted]

bbernoulli 10y ago

Can you provide anymore details about this? This doesn't seem like it would take 2-3 hours. I'm going to have to assess come applicants soon and have been trying to come up with something appropriate, which is why I'm interested.