7 ms·

> Use homework assignments to assess the candidate’s engineering skills while homework problems can be better than in-person interviews, they can also be a lot

by rguzman 10y ago

> Use homework assignments to assess the candidate’s engineering skills while homework problems can be better than in-person interviews, they can also be a lot worse. for one, because they are NOT time-bound. secondly, they are much more difficult to setup and evaluate: do you provide a spec? code to modify? third, asking people to do work for free in order to explore the potential relationship is a big ask. lastly, they are very time-consuming for the company. in grad school i learned the hard way to always ask for in-class exams instead of take-homes :-). time and scope constraints can be favorable both for the company and the candidate. > Enough with the graph traversal problems. the point of asking graph-traversals and similar algorithms questions is not that they are the relevant job skill. it is that they are a way to gain information about the candidate's problem-solving ability in a limited amount of time. unavoidably, these questions test for speed and familiarity along with problem-solving. that makes them a lot less than perfect, but i've generally found them valuable.

jplasmeier 10y ago
The issue with tricky algorithms questions is that they often become tests of how well you can keep your cool. I've (and I'm sure others have also) bombed relatively easy questions due to nervousness compounding. I've also done much better than expected on other interviews because I remained calm and kept the interview flowing smoothly. This is the real skill under test, IMO. Of course, your ability to solve problems is still important, and a subset of what's being tested. The issue is just that there's a lot of (possibly inherently random) factors that cloud that signal.
- base698 10y ago
  My favorite example of this is when an interviewer asked me about an algorithm. In the course of the problem it occurred to me, what I needed was to break down the input integer into two parts--two equal parts that could be multiplied together. I kept referring to these two parts as such. After the interview he said I was clearly weak on math because I didn't know the definition of a square root. Something I did know (10 or so college math courses in linear algebra, calc etc) and was describing, but was at a loss for words.
saghm 10y ago
> for one, because they are NOT time-bound I find this point interesting, as to me, this would be an argument for, not against. Time constraints on homework assignments are a lot closer to the constraints you'd have programming for a job than "whiteboard coding" would be. I'd hazard a guess that doing a good job on an in-depth, open-ended problem given a few days to a week is a better indicator of ability to be a successful software engineer than being able to solve a logic puzzle on a whiteboard in fifteen minutes.
- _delirium 10y ago
  If the interview time is paid, sure. But if it's not, agreeing to do unbounded unpaid work for a chance of being hired for a paid gig, especially when it's not even clear the chance is particularly strongly correlated to effort put in, gets worse as a proposition the longer you spend on it.
  - deleted 10y ago
    [deleted]
  - saghm 10y ago
    I was speaking more from the side of the company; from the side of the interviewee, it's less clear-cut; if you're the type that doesn't tend to do well in interviews, it might be preferable, but if you don't feel like you'll do a better job on a homework-like problem, then it's definitely worse.
  - deleted 10y ago
    [deleted]
  - jpttsn 10y ago
    By design. In the candidate pool, there will be some who see the odds as too steep. For others the cost of completing assignments is lower, making the proposition better. Not good, but comparatively better (less bad) than for the former kind. And that's how we know whom to hire.
    - 10y ago
enraged_camel 10y ago
>>while homework problems can be better than in-person interviews, they can also be a lot worse. for one, because they are NOT time-bound. Um, this is a good thing, not a bad thing. Research shows that stress significantly reduces cognitive ability, and time pressure definitely adds to stress - sometimes substantially. What this means is that when you ask whiteboard coding questions during the interview, you're not measuring the candidate's on-the-job ability. You're simply measuring how well they perform under severe pressure. If that's fine with you, as in it's representative of your work environment where people have to churn out code around the clock, fine. Otherwise, homework questions are much better.
- rguzman 10y ago
  i agree with all that -- again, the i don't think that whiteboard interviews are perfect. but also, keep in mind that the candidate has a limited amount of time, and often a full-time job on top of the job hunt. the homework questions are only good if the candidate has time and desire to them.
  - enraged_camel 10y ago
    So then wouldn't it make the most sense to let the candidate choose?
    - thaumasiotes 10y ago
      That makes it difficult to compare candidates to each other.
      - prodigal_erik 10y ago
        Why do that? If your screening test says they're both qualified, interview them both. Hire them both if you possibly can.
        thaumasiotes 10y ago
deleted 10y ago
[deleted]
dpeck 10y ago
They can absolutely be timebound, unique URL time stamped at the download, with however many number of hours from then to send in whatever you have, completed or not. I've only experienced this once and it was stressful as could be, but was from a person I absolutely respected and still do. No-bs, either right or wrong and plenty to discuss in the followup conversation. Did it from home, pulled the file at around 11pm right when I knew my best time to work was. Would happily do as part of an interview any time.
tyre 10y ago
> asking people to do work for free in order to explore the potential relationship is a big ask Asking to work somewhere without demonstrating that you can do the work is asking a lot. Unfortunately, job history alone is not enough to weed out people that cannot program (let alone well.) > they are very time-consuming for the company. Not if you implement a simple test spec or scoring system. We're engineers! > the point of asking graph-traversals and similar algorithms questions is not that they are the relevant job skill. So don't ask the generic questions. Ask someone to implement something that they would need to do on the job. Obviously don't ask them to implement a feature that takes 12 hours, but a basic screen that is a snapshot of 45 minutes of on-the-job experience isn't crazy. We also do a take-home exercise first to see how you think without us staring at you. Day-to-day at Seneca Systems we aren't staring at you, so it feels more relevant.
- deleted 10y ago
  [deleted]
- thaumaturgy 10y ago
  > Asking to work somewhere without demonstrating that you can do the work is asking a lot. Which other professions have this attitude? I've either worked or supported a pretty wide variety of other jobs, and I can't off the top of my head think of a single one that does. Even if some examples are brought up (and I'm genuinely interested in what they'd be), it's far from the usual approach in most hiring situations.
  - adevine 10y ago
    Huh? Your answer surprises me, as I can think of tons of examples. Designers are often asked to provide a portfolio to show the work they are capable of. Same thing for photographers. In a previous life I was a ballet dancer, and the whole purpose of auditions is to demonstrate that you are a competent dancer. I generally prefer an existing portfolio of code (e.g. github repos) to see the kind of code someone produces, but if you aren't able to show some fairly extensive examples of your work, homework is usually a bunch better assessment of your skills than what can be accomplished in a ~45 min interview.
skookumchuck 10y ago
The trouble with unbounded time exams is you are pretty much forced to spend enormous time on it, because you assume your competitors will.
- viraptor 10y ago
  That's not my experience, being on both sides of it. (devops kind of tests) Given no bounds, you will not be able to produce a 100% solution from the recruiter side of view. Something will always look weird. There's always another approach, generic parts possible to extract, places that can be left available for future expansion, etc. I'd go as far as, if you can finish the test in 2h, then it doesn't matter if you spend 4h or 8h on it, because you'll just do more of the same in most cases. For example you can document one module and say "the rest of the docs are TODO due to time" - it shows you know how to document and you can move to another bit. My favourite submissions did 5% of many things to show they are aware of those areas, rather than super-polished trivial solutions of just the basic question.
- ufmace 10y ago
  On the contrary, I'd say that having unbounded time tells the interviewer a lot about how you go about building things. Do you know how to build something that works and stop at an appropriate level of abstraction, documentation, and testing for the type of problem at hand, or do you go off into the weeds and build FizzBuzzEnterpriseEdition (https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...)?
hackinthebochs 10y ago
>the point of asking graph-traversals and similar algorithms questions is not that they are the relevant job skill. Actually, my codebase has an honest to god graph traversal in it, on the front-end no less. Basic manipulation of graph relationships without much friction should be standard.

by rguzman 10y ago

jplasmeier 10y ago

The issue with tricky algorithms questions is that they often become tests of how well you can keep your cool. I've (and I'm sure others have also) bombed relatively easy questions due to nervousness compounding. I've also done much better than expected on other interviews because I remained calm and kept the interview flowing smoothly. This is the real skill under test, IMO. Of course, your ability to solve problems is still important, and a subset of what's being tested. The issue is just that there's a lot of (possibly inherently random) factors that cloud that signal.

base698 10y ago
My favorite example of this is when an interviewer asked me about an algorithm. In the course of the problem it occurred to me, what I needed was to break down the input integer into two parts--two equal parts that could be multiplied together. I kept referring to these two parts as such. After the interview he said I was clearly weak on math because I didn't know the definition of a square root. Something I did know (10 or so college math courses in linear algebra, calc etc) and was describing, but was at a loss for words.

saghm 10y ago

> for one, because they are NOT time-bound I find this point interesting, as to me, this would be an argument for, not against. Time constraints on homework assignments are a lot closer to the constraints you'd have programming for a job than "whiteboard coding" would be. I'd hazard a guess that doing a good job on an in-depth, open-ended problem given a few days to a week is a better indicator of ability to be a successful software engineer than being able to solve a logic puzzle on a whiteboard in fifteen minutes.

_delirium 10y ago
If the interview time is paid, sure. But if it's not, agreeing to do unbounded unpaid work for a chance of being hired for a paid gig, especially when it's not even clear the chance is particularly strongly correlated to effort put in, gets worse as a proposition the longer you spend on it.
- deleted 10y ago
  [deleted]
- saghm 10y ago
  I was speaking more from the side of the company; from the side of the interviewee, it's less clear-cut; if you're the type that doesn't tend to do well in interviews, it might be preferable, but if you don't feel like you'll do a better job on a homework-like problem, then it's definitely worse.
- deleted 10y ago
  [deleted]
- jpttsn 10y ago
  By design. In the candidate pool, there will be some who see the odds as too steep. For others the cost of completing assignments is lower, making the proposition better. Not good, but comparatively better (less bad) than for the former kind. And that's how we know whom to hire.
  - 10y ago

enraged_camel 10y ago

>>while homework problems can be better than in-person interviews, they can also be a lot worse. for one, because they are NOT time-bound. Um, this is a good thing, not a bad thing. Research shows that stress significantly reduces cognitive ability, and time pressure definitely adds to stress - sometimes substantially. What this means is that when you ask whiteboard coding questions during the interview, you're not measuring the candidate's on-the-job ability. You're simply measuring how well they perform under severe pressure. If that's fine with you, as in it's representative of your work environment where people have to churn out code around the clock, fine. Otherwise, homework questions are much better.

rguzman 10y ago
i agree with all that -- again, the i don't think that whiteboard interviews are perfect. but also, keep in mind that the candidate has a limited amount of time, and often a full-time job on top of the job hunt. the homework questions are only good if the candidate has time and desire to them.
- enraged_camel 10y ago
  So then wouldn't it make the most sense to let the candidate choose?
  - thaumasiotes 10y ago
    That makes it difficult to compare candidates to each other.
    - prodigal_erik 10y ago
      Why do that? If your screening test says they're both qualified, interview them both. Hire them both if you possibly can.
      - thaumasiotes 10y ago

deleted 10y ago

[deleted]

dpeck 10y ago

They can absolutely be timebound, unique URL time stamped at the download, with however many number of hours from then to send in whatever you have, completed or not. I've only experienced this once and it was stressful as could be, but was from a person I absolutely respected and still do. No-bs, either right or wrong and plenty to discuss in the followup conversation. Did it from home, pulled the file at around 11pm right when I knew my best time to work was. Would happily do as part of an interview any time.

tyre 10y ago

> asking people to do work for free in order to explore the potential relationship is a big ask Asking to work somewhere without demonstrating that you can do the work is asking a lot. Unfortunately, job history alone is not enough to weed out people that cannot program (let alone well.) > they are very time-consuming for the company. Not if you implement a simple test spec or scoring system. We're engineers! > the point of asking graph-traversals and similar algorithms questions is not that they are the relevant job skill. So don't ask the generic questions. Ask someone to implement something that they would need to do on the job. Obviously don't ask them to implement a feature that takes 12 hours, but a basic screen that is a snapshot of 45 minutes of on-the-job experience isn't crazy. We also do a take-home exercise first to see how you think without us staring at you. Day-to-day at Seneca Systems we aren't staring at you, so it feels more relevant.

deleted 10y ago
[deleted]
thaumaturgy 10y ago
> Asking to work somewhere without demonstrating that you can do the work is asking a lot. Which other professions have this attitude? I've either worked or supported a pretty wide variety of other jobs, and I can't off the top of my head think of a single one that does. Even if some examples are brought up (and I'm genuinely interested in what they'd be), it's far from the usual approach in most hiring situations.
- adevine 10y ago
  Huh? Your answer surprises me, as I can think of tons of examples. Designers are often asked to provide a portfolio to show the work they are capable of. Same thing for photographers. In a previous life I was a ballet dancer, and the whole purpose of auditions is to demonstrate that you are a competent dancer. I generally prefer an existing portfolio of code (e.g. github repos) to see the kind of code someone produces, but if you aren't able to show some fairly extensive examples of your work, homework is usually a bunch better assessment of your skills than what can be accomplished in a ~45 min interview.

skookumchuck 10y ago

The trouble with unbounded time exams is you are pretty much forced to spend enormous time on it, because you assume your competitors will.

viraptor 10y ago
That's not my experience, being on both sides of it. (devops kind of tests) Given no bounds, you will not be able to produce a 100% solution from the recruiter side of view. Something will always look weird. There's always another approach, generic parts possible to extract, places that can be left available for future expansion, etc. I'd go as far as, if you can finish the test in 2h, then it doesn't matter if you spend 4h or 8h on it, because you'll just do more of the same in most cases. For example you can document one module and say "the rest of the docs are TODO due to time" - it shows you know how to document and you can move to another bit. My favourite submissions did 5% of many things to show they are aware of those areas, rather than super-polished trivial solutions of just the basic question.
ufmace 10y ago
On the contrary, I'd say that having unbounded time tells the interviewer a lot about how you go about building things. Do you know how to build something that works and stop at an appropriate level of abstraction, documentation, and testing for the type of problem at hand, or do you go off into the weeds and build FizzBuzzEnterpriseEdition (https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...)?

hackinthebochs 10y ago

>the point of asking graph-traversals and similar algorithms questions is not that they are the relevant job skill. Actually, my codebase has an honest to god graph traversal in it, on the front-end no less. Basic manipulation of graph relationships without much friction should be standard.