10 ms·

I see wonderful symmetry here. There are TrendyCos (hot startup unicorns), BigCos (established companies like google or amazon or microsoft) and UnknownCos (not

I see wonderful symmetry here. There are TrendyCos (hot startup unicorns), BigCos (established companies like google or amazon or microsoft) and UnknownCos (not in the limelight so there is not much information about them). Likewise there are TrendyDevs (hotshots who produce one heavily github-starred framework after another), BigCoDevs (multiple years of experience at one of the BigCos, probably were responsible for some important part of one of their numerous services) and UnknownDevs (been there done that, hard to say). And here is the rub. There are certainly many undervalued gems among UnknownCos and UnknownDevs but also many abysmally bad workplaces and programmers. So you either have some inside knowledge about them (a referral, an acquaintance working at UnknownCo), have some magic method for separating the gems (like tptacek claims to possess) or it is just too risky to consider them.

tptacek 11y ago
There's no magic to it at all. Have every candidate work on programming problems related to the work you do at your company. Have every candidate work on the same problems, and let them do it from home. Build and iterate on a rubric for grading those challenges. It is amazing to me that almost nobody does this, but: almost nobody does this. They have programmers write code on a whiteboard, or on some whiteboard-coding site; they have them do programming puzzles ("solve Towers of Hanoi non-recursively"), they have them talk about code, or, more likely, computer science trivia. They'll have them "pair off" with one of their own programmers and "fix a bug". They'll have them work as a 1099 for a month to see if they're any good. In reality, most companies are trying to hire on trendiness; they want people from the right schools and/or the right cohort of companies. They're aggressively courting friends and colleagues of their existing team, and the special people get very different interviews than everyone else. The actual technical evaluation is mostly a facade.
- lagadu 11y ago
  > almost nobody does this. Maybe this is an US thing. I changed jobs last year and nearly every single recruitment process involved exactly that, after a remote interview them giving me a small project worth 2-5 hours of work and then going over what I produced one week later. This was for companies in central and northern Europe.
  - rtpg 11y ago
    Were you paid for your time when doing that? Kind of curious (I guess 2 hours isn't a lot of time but there's the principle)
    - collyw 11y ago
      I have had a few of these the last year or two. Never been paid. More than half of them are a waste of time. I have refused quite a few of them. The last "shouldn't take you more than two hours" exercise that I was given involved jumping through hoops to get set up on Instagrams API, only to discover that I was in a sandboxed mode, and needed my account approved before i could get anything more than metadata out). I didn't even get started coding, so I am not sure what exactly the test was meant to achieve.
      - elptacek 11y ago
        Just curious -- why Instagram's API?
        collyw 11y ago
        I think it was related to the business. Was something to do with photos.
    - mseebach 11y ago
      No, there's no principle, there's a ~20 minutes old idea that we engineers are so obviously awesome that companies should actually be paying us for the privilege of interviewing us. There may exists engineers obviously awesome enough for that to be feasible, and great for them (but they are probably also obviously awesome enough to not have to go through coding tests, so it's a moot point) and even if I could perhaps (judging by certain recruiter-emails) scrape by as one of them, I certainly couldn't five years ago. Back then, if I'd had to be good enough on paper alone to warrant not just being put through the recruitment process, but to be paid for it was well, I am not convinced I'd have been considered (and yes, I did a take-home test, and I aced it and it made up for my near-total lack of on-paper qualities). Of course, companies shouldn't waste their applicants time with needlessly extensive tests, but there certainly exists no 'principle' by which you have a claim to be reimbursed for spending a few hours on an application.
- collyw 11y ago
  Its a crap waste of time. Take home exercises have little more in common with the job than a white board exercises. And not all devs are going to do half a days work for free.
  - tptacek 11y ago
    People say this in every thread about hiring, and it never makes any sense to me at all. Companies that don't hire using objective at-home tests hire instead with grueling on-site interviews that knock out an entire business day (most of them last a whole business day, some recent interviews peers have gone on have taken multiple business days, and all of them at the very least kill the day for anything else). The latter is obviously worse than the former; I don't even see how any other argument could be colorable. What I'm beginning to think is that this kind of pushback comes mostly from developers who are well-connected, and so they never experience the grind that less well-connected but equally-capable developers do with interviews. Yes: if you're at a point in your career where you can get a job in any of 6-7 different companies just by raising your hand, saying "I'm available", and having a 30 minute conversation with the VP/Engineering who you worked with 2 jobs ago, everything is a waste of time. I am not here to tell you that you should make things harder for yourself.
    - dood 11y ago
      If used wisely, at-home coding tests should be far better than the average mess of a technical interview. The problem is in the hands of TrendCo, they can be used crudely to find the trendy hire who has the exact same ideas about engineering as the hiring person. Because giving the test costs TrendCo nothing, they are happy to throw people at it until they randomly find someone who is exactly what they are looking for. Anecdotally, I and 2 people I later befriended applied for the same role. We each spent 4 hours on the task, all 3 of us strong coders who made a solution that would be suitable for any startup, all 3 taking different approaches based on our styles. We were all rejected, because none of the 3 of us hit upon the exact approach the company was hoping for, but didn't ask for. There was no opportunity to ask for this feedback either. And it took them nearly a month to bother looking at my code (even after I called). Since then, I won't take these kind of tests unless I have reason to believe they are being given in good faith as a way to determine if a programmer is capable, not as a way to find their perfect ideal of a programmer.
- mikegioia 11y ago
  Have every candidate work on the same problems, and let them do it from home To be fair, I've seen a lot of good arguments against programming assignments. I think at the end of the day, the employer needs to conduct some method of determining if the employee has the technical capabilities needed for the specific job they're being hired for. However, there are MANY other factors too like "how well they get along with the team" that need to be considered. That's what they're trying to determine with the pairings and other stuff you mentioned I think. Whether that works is another story.
  - tptacek 11y ago
    Every team I've talked to starts out with some X-factor they think they need to assess for. It's my belief that if you can have the discipline to stop filtering for X-factors, you'll build better, more effective teams, because those factors are really just vectors for personal biases. Regardless of whether you agree with me about that, I think we can all stipulate that if work-sample technical evaluations work (and: they do), most of what companies try to evaluate in on-site interviews is stupid. No part of working effectively with a team requires timed recall of how to implement a stable quicksort, or reversing a doubly linked list at a whiteboard. At the very least, using work sample tests allows you to build an on-site interview process that honestly engages with "team fit" (or whatever your X-factor is). My guess, though, is that when more teams adopt work samples and then go through the motions of trying to design a pure team-fit interview, they're going to realize --- once they don't have "implement Bellman-Ford on this whiteboard" to fall back on --- how unequipped they always were to evaluate team-fit in the first place.
    - bcg1 11y ago
      The problem could also be that many teams simply don't have the time or possess the knowledge to do an assessment like you're suggesting (which I agree is a good way to assess candidates). It requires someone to actually design an assignment (which is a task that many coders might not be good at), and requires one or more people to evaluate it. The ridiculous whiteboard coding of puzzles probably stems from laziness or inability to implement what you're suggesting. I know that personally, if someone asked me to developed a work sample evaluation for my job, it would take me many hours to come up with something, and frankly I even once complete we would have no way of knowing if it is actually a good predictor of whether or not someone is the right employee.
- golergka 11y ago
  Why are you saying that nobody does that? My last job search wasn't typical (I thought of starting a consultancy, so didn't search for a job per se), but one before that showed to me that it's exactly what people do. A company that was looking for a guy to write mobile SDKs asked me to build an Android app with embedded webview and animate html stuff with JS based on gyro movement. A slot game company asked me to build a minimalistic slot game from scratch (thanks to them, I have a clean minimalistic and complete game on github now). Another game company asked me to build a small endless shooter — and I ended up being the only candidate who moved texture offsets instead of actually moving game objects to infinity. Each of those tasks took from 4 to 8 hours, and 2 of those got me an offer. And of course, I used the same method when I consulted others on hiring decisions or hired people myself. So, based on personal experience, companies do that, and it works.
  - tptacek 11y ago
    I'm glad that I'm starting to get feedback like this. I believe you that more companies are doing work sample tests. For the the gigs that asked you to do these problems: did you also get interviewed with a standard programming interview? If you had to guess, between 1% and 100%, how much of the weight of your technical evaluation do you feel was on the sample challenges you did at home, versus Q&A and whiteboard coding on-site (or on the phone)?
    - golergka 11y ago
      Don't remember details about interviews, but I'd tell that homework was about 75%, because in those two cases they told me that I exceeded their expectations and they learned something new from my projects.
- djb_hackernews 11y ago
  FWIW I presented this option to my senior management and HR at a "household name" software company and was essentially told we can't do this because it looks like a test and we'd have to do the same test for every candidate or else open our selves up to discrimination issues, no matter which team the position is for, and that was a no go.
  - potatolicious 11y ago
    > "we can't do this because it looks like a test and we'd have to do the same test for every candidate" Oh, what a terrible tragedy, to give candidates consistent evaluations that are actually comparable between each other... /s I get the point you're presenting, but at the end of the day it still boils down to "we can't use tests because then we'd have to be held accountable for objectivity, we prefer instead a system that lacks any kind of rigor such that any result may not be effectively challenged, officially or otherwise".
  - rdl 11y ago
    Work-sample tests are clearly protected; it's IQ tests which aren't permitted (Duke Power v.)
    - djb_hackernews 11y ago
      Right but 2 things: 1) HR (presumably taking direction from in house counsel) takes the position that it will be difficult to distinguish between a work sample test and an IQ test 2) We are a large organization with very broad roles. The Performance Team hiring for a Senior Software Engineer will probably want a different work sample than the Analytics Team...
      - tptacek 11y ago
        It is simply not true that the Perf team must deliver the same work-sample test as the Analytics team. Let's not dignify that argument. I absolutely believe you that your company has allowed HR to sabotage your hiring process, and that sucks. But let's not pretend HR is right to do it.
- ArkyBeagle 11y ago
  But this takes far too long.
  - tptacek 11y ago
    It saved us enormous amounts of time at Matasano. When we designed our work-sample rubric, our #1 concern was filtering out people who interviewed well and/or had great resumes but weren't worth a billable hour on a real project, but our #2 concern was interviewing faster and getting results back to candidates quickly. (We didn't discover the best reason to do work-samples --- discovering hidden talent --- until after we'd started doing them). With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.
    - rorykoehler 11y ago
      Did you ever consider the candidate in all of this?
      - tptacek 11y ago
        Yes: like I said, the goal was to get results back to candidates faster. The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us). After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week. Also, see: https://news.ycombinator.com/item?id=11328090 https://news.ycombinator.com/item?id=11328090 The process I'm advocating for is far better for candidates.
        11y ago
- FLUX-YOU 11y ago
  >It is amazing to me that almost nobody does this, but: almost nobody does this. There isn't enough feedback when someone has a bad idea on hiring. People will very quickly tell you if your code sucks though (this is a good thing), which might be the kind of feedback needed to get hiring changed.
- JoeAltmaier 11y ago
  How is working for a month different that "work on programming problems related to the work"? I'd have thought that is even better as a filter.
  - tptacek 11y ago
    I am fine with asking for candidates to work as a 1099 for a month if you pay them the standard 1099 contractor rate, which is almost invariably more than 2x their FTE rate. I am not fine with the industry norm of paying people their eventual FTE salary on a 1099 basis; that's a scam. I think it's pretty dumb to do that, though, because most software developers can't take a month off of work from their current job to see if they're a fit for your company, and none of them will put their health insurance in jeopardy to do that. I also think it's pretty silly to design a process that deliberately demands a month of time to make a decision that could be made effectively in a week.
    - JoeAltmaier 11y ago
      Yeah last time I did it, it was just a week. And I got my full contract rate (which is about double as you suggest). It went well; got an offer. Its not usually that hard for a contractor to take the time off; they're on their own time already.
      - tptacek 11y ago
        That strikes me as a very costly way to build a team. Independent contractors are expensive; they know, better than most other developers, what their time is worth.
        JoeAltmaier 11y ago
        Microsoft, of course, interns college juniors for 90 days as a summer job. Less costly I imagine per hour, but apparently working for them.
- cp9 11y ago
  cause I have a job that pays me. I'm not working on stuff for your company for free
  - tptacek 11y ago
    That's exactly what you're doing when you go to an on-site job interview, so this objection is slightly less clever than you think it is.
    - cp9 11y ago
      except this is on top of that. nobody does a phone screen, checks a sample of code and says "tada, you're hired" we were going to get together in a room anyways.
      - tptacek 11y ago
        It shouldn't be on top of that! It wasn't for our process. Of course: we did in-person interviews. And in-person interviews are disruptive no matter what you do. But: * Our in-person interviews were shorter than typical in-person interviews because they weren't tasked with fully teching candidates out. * Because they didn't try to tech candidates out, they weren't as stressful, and so were less draining and unpleasant. * Because our in-person interviews were entirely scripted (the interviewers had very little latitude with what they could ask and how they could ask it --- which they hated, by the way), they were easy for everyone to deliver. * Before candidates arrived to their interview, they already knew (because we told them!) that they were likely to receive an offer from us based on their performance on the work-sample tests. I understand why people resist the idea of coding challenges, given: * They're not going to get feedback for weeks, months, or maybe ever * The challenge is going to be graded pass-fail, or whimsically, without rigor or consistency * They're going to have to do the exact same grueling bullshit draining dehumanizing programmer interview anyways * They're not going to see the challenges coming, or, for that matter, know exactly what happens next after the challenges are done That process is super common, even at companies that ostensibly do challenges. It sucks. I agree, no company can really get away with this in the long term. But those are also chickenshit problems. Switching from unstructured interviews to work-samples is hard: you have to change your mindset on how to qualify candidates, and there's a leap of faith involved. But getting candidates feedback, telling them what to expect with your process, keeping a schedule, and having processes in place to be consistent aren't hard problems. They are table-stakes common sense business execution, and if your team can't handle that right now, your team is mismanaged.
- gtolle 11y ago
  I love this approach. I'm also wondering what you think about giving well-designed work-sample tests over Google Hangout instead. The candidate shares their screen with you, so you can watch as they solve the problem in their own dev environment. You can understand how they approach problems (quick and dirty, slow and methodical, lots of rewriting, etc), and you can ask questions at the end. You get a good sense for how they work as an engineer even before having to bring them onsite. This seems to resolve the time asymmetry of take-home tests as well -- the interviewer spends as much time watching as the candidate spends working. The only downside I can see is that you have to design your problems to take about an hour instead of the 2-5 hours you could imagine for a take-home test. But, you can break them up into multiple rounds, and give additional exercises to the candidates who do well on the first one, for no more total time cost than a collection of onsite interviews. For what it's worth, I've done this at my startup and hired a great developer, and got very positive feedback about the process from the candidates I didn't end up hiring.
  - crzwdjk 11y ago
    As someone who's had to do a few take home tests as well as coding with someone looking over my shoulder, I am definitely not a fan of the latter. I prefer to sit down and think about a problem, maybe even read it and let it sit in my head for a day or two before I actually start the coding. I also feel like shortening the time and making it synchronous would remove the opportunity to revise and polish the code I've written, which is one of the advantages of the take-home test format. My ideal interview format would involve a work sample test, followed by a code review of said work sample, and also a reverse code review (candidate is given some code to review). As an added bonus, all of this can easily be done over email without wasting too much of anyone's time, especially compared to an all day in-person interview with lots of whiteboard time.
- bcaine 11y ago
  I agree that this is "better" than the alternative, but it can be absolutely exhausting for candidates actively searching for a job. I feel like it's recently become much, much more common (from my small-ish sample of me and some friends). My issue with this approach is fourfold: 1. Most companies have no idea how to structure a problem that is both informative to them and also not abusive to the candidates time. 2. Companies generally do this right after the recruiter phone screen, which most likely doesn't give the candidate enough information to decide if the next steps are worth their time. 3. Most companies still do a whole suite of normal tech screens after you work on a take home problem. 4. If you're actively looking, getting a bunch of these over a short period of time is likely. I know during my full time search, more than 50% of companies had a take home test right after the recruiter screen. Most of these were 4-8 hours of work each, due within the week. A lot of startups structure it more like hazing or a barrier to entry than an evaluation criteria. I have some fun (read: horrifying) anecdotes from my recent search that illustrate the problems above, but I don't think any of my points are surprising. A nice alternative would have been to simply have one or two projects completed that are straightforward to evaluate and walk companies through them, letting them ask me questions.
  - tptacek 11y ago
    Here's a really simple test for whether a work-sample scheme is effective, or just a bullshit trend-chasing afterthought: Does the work sample test offset all or most of the unstructured tech evaluation the company would otherwise do? If it does, that means they have a work-sample test rubric they believe in, and they're hiring confidently. If it doesn't, they don't believe in what they're doing, and the programming challenges they're assigning can thus reasonably be seen as more hoop-jumping. In the latter case, it's up to you to decide whether a prospective job is worth extra hoop-jumping. Some teams are worth it!
    - bcaine 11y ago
      I think that's fair. I've had both the former and the latter, but unfortunately most of my experiences fall into latter case, where it's simply been hoop jumping. Most of my friends (all about to graduate, so a good number of examples) are experiencing the same. For example one company gave a problem with five parts, with the final part being solve longest path on a bipartite weighted graph (which is quite a hard and time consuming problem). After that, the next step was a phone technical screen, then an on-site with 4-5 more interviews, most being white-boarding. It was basically hazing instead of an evaluation criteria. An alternative is my last job, which had a take home test that took about 6 hours, but that was the whole technical part of the process. Being on the other side reviewing them, the problem absolutely gave enough information. I totally get there's a right way to do it, but like most interviewing trends, companies seem to just be adding this as a step instead of revamping their process.
- abossy 11y ago
  How much time do you give candidates to complete their work samples? I'm curious whether work-sample tests filter out, say, parents with young children, in which a company with a 4-6 hour technical interview might be preferred over an unbound take-home assignment (in addition to the possibility of more interviews).

tptacek 11y ago

There's no magic to it at all. Have every candidate work on programming problems related to the work you do at your company. Have every candidate work on the same problems, and let them do it from home. Build and iterate on a rubric for grading those challenges. It is amazing to me that almost nobody does this, but: almost nobody does this. They have programmers write code on a whiteboard, or on some whiteboard-coding site; they have them do programming puzzles ("solve Towers of Hanoi non-recursively"), they have them talk about code, or, more likely, computer science trivia. They'll have them "pair off" with one of their own programmers and "fix a bug". They'll have them work as a 1099 for a month to see if they're any good. In reality, most companies are trying to hire on trendiness; they want people from the right schools and/or the right cohort of companies. They're aggressively courting friends and colleagues of their existing team, and the special people get very different interviews than everyone else. The actual technical evaluation is mostly a facade.

lagadu 11y ago
> almost nobody does this. Maybe this is an US thing. I changed jobs last year and nearly every single recruitment process involved exactly that, after a remote interview them giving me a small project worth 2-5 hours of work and then going over what I produced one week later. This was for companies in central and northern Europe.
- rtpg 11y ago
  Were you paid for your time when doing that? Kind of curious (I guess 2 hours isn't a lot of time but there's the principle)
  - collyw 11y ago
    I have had a few of these the last year or two. Never been paid. More than half of them are a waste of time. I have refused quite a few of them. The last "shouldn't take you more than two hours" exercise that I was given involved jumping through hoops to get set up on Instagrams API, only to discover that I was in a sandboxed mode, and needed my account approved before i could get anything more than metadata out). I didn't even get started coding, so I am not sure what exactly the test was meant to achieve.
    - elptacek 11y ago
      Just curious -- why Instagram's API?
      - collyw 11y ago
        I think it was related to the business. Was something to do with photos.
  - mseebach 11y ago
    No, there's no principle, there's a ~20 minutes old idea that we engineers are so obviously awesome that companies should actually be paying us for the privilege of interviewing us. There may exists engineers obviously awesome enough for that to be feasible, and great for them (but they are probably also obviously awesome enough to not have to go through coding tests, so it's a moot point) and even if I could perhaps (judging by certain recruiter-emails) scrape by as one of them, I certainly couldn't five years ago. Back then, if I'd had to be good enough on paper alone to warrant not just being put through the recruitment process, but to be paid for it was well, I am not convinced I'd have been considered (and yes, I did a take-home test, and I aced it and it made up for my near-total lack of on-paper qualities). Of course, companies shouldn't waste their applicants time with needlessly extensive tests, but there certainly exists no 'principle' by which you have a claim to be reimbursed for spending a few hours on an application.
collyw 11y ago
Its a crap waste of time. Take home exercises have little more in common with the job than a white board exercises. And not all devs are going to do half a days work for free.
- tptacek 11y ago
  People say this in every thread about hiring, and it never makes any sense to me at all. Companies that don't hire using objective at-home tests hire instead with grueling on-site interviews that knock out an entire business day (most of them last a whole business day, some recent interviews peers have gone on have taken multiple business days, and all of them at the very least kill the day for anything else). The latter is obviously worse than the former; I don't even see how any other argument could be colorable. What I'm beginning to think is that this kind of pushback comes mostly from developers who are well-connected, and so they never experience the grind that less well-connected but equally-capable developers do with interviews. Yes: if you're at a point in your career where you can get a job in any of 6-7 different companies just by raising your hand, saying "I'm available", and having a 30 minute conversation with the VP/Engineering who you worked with 2 jobs ago, everything is a waste of time. I am not here to tell you that you should make things harder for yourself.
  - dood 11y ago
    If used wisely, at-home coding tests should be far better than the average mess of a technical interview. The problem is in the hands of TrendCo, they can be used crudely to find the trendy hire who has the exact same ideas about engineering as the hiring person. Because giving the test costs TrendCo nothing, they are happy to throw people at it until they randomly find someone who is exactly what they are looking for. Anecdotally, I and 2 people I later befriended applied for the same role. We each spent 4 hours on the task, all 3 of us strong coders who made a solution that would be suitable for any startup, all 3 taking different approaches based on our styles. We were all rejected, because none of the 3 of us hit upon the exact approach the company was hoping for, but didn't ask for. There was no opportunity to ask for this feedback either. And it took them nearly a month to bother looking at my code (even after I called). Since then, I won't take these kind of tests unless I have reason to believe they are being given in good faith as a way to determine if a programmer is capable, not as a way to find their perfect ideal of a programmer.
mikegioia 11y ago
Have every candidate work on the same problems, and let them do it from home To be fair, I've seen a lot of good arguments against programming assignments. I think at the end of the day, the employer needs to conduct some method of determining if the employee has the technical capabilities needed for the specific job they're being hired for. However, there are MANY other factors too like "how well they get along with the team" that need to be considered. That's what they're trying to determine with the pairings and other stuff you mentioned I think. Whether that works is another story.
- tptacek 11y ago
  Every team I've talked to starts out with some X-factor they think they need to assess for. It's my belief that if you can have the discipline to stop filtering for X-factors, you'll build better, more effective teams, because those factors are really just vectors for personal biases. Regardless of whether you agree with me about that, I think we can all stipulate that if work-sample technical evaluations work (and: they do), most of what companies try to evaluate in on-site interviews is stupid. No part of working effectively with a team requires timed recall of how to implement a stable quicksort, or reversing a doubly linked list at a whiteboard. At the very least, using work sample tests allows you to build an on-site interview process that honestly engages with "team fit" (or whatever your X-factor is). My guess, though, is that when more teams adopt work samples and then go through the motions of trying to design a pure team-fit interview, they're going to realize --- once they don't have "implement Bellman-Ford on this whiteboard" to fall back on --- how unequipped they always were to evaluate team-fit in the first place.
  - bcg1 11y ago
    The problem could also be that many teams simply don't have the time or possess the knowledge to do an assessment like you're suggesting (which I agree is a good way to assess candidates). It requires someone to actually design an assignment (which is a task that many coders might not be good at), and requires one or more people to evaluate it. The ridiculous whiteboard coding of puzzles probably stems from laziness or inability to implement what you're suggesting. I know that personally, if someone asked me to developed a work sample evaluation for my job, it would take me many hours to come up with something, and frankly I even once complete we would have no way of knowing if it is actually a good predictor of whether or not someone is the right employee.
golergka 11y ago
Why are you saying that nobody does that? My last job search wasn't typical (I thought of starting a consultancy, so didn't search for a job per se), but one before that showed to me that it's exactly what people do. A company that was looking for a guy to write mobile SDKs asked me to build an Android app with embedded webview and animate html stuff with JS based on gyro movement. A slot game company asked me to build a minimalistic slot game from scratch (thanks to them, I have a clean minimalistic and complete game on github now). Another game company asked me to build a small endless shooter — and I ended up being the only candidate who moved texture offsets instead of actually moving game objects to infinity. Each of those tasks took from 4 to 8 hours, and 2 of those got me an offer. And of course, I used the same method when I consulted others on hiring decisions or hired people myself. So, based on personal experience, companies do that, and it works.
- tptacek 11y ago
  I'm glad that I'm starting to get feedback like this. I believe you that more companies are doing work sample tests. For the the gigs that asked you to do these problems: did you also get interviewed with a standard programming interview? If you had to guess, between 1% and 100%, how much of the weight of your technical evaluation do you feel was on the sample challenges you did at home, versus Q&A and whiteboard coding on-site (or on the phone)?
  - golergka 11y ago
    Don't remember details about interviews, but I'd tell that homework was about 75%, because in those two cases they told me that I exceeded their expectations and they learned something new from my projects.
djb_hackernews 11y ago
FWIW I presented this option to my senior management and HR at a "household name" software company and was essentially told we can't do this because it looks like a test and we'd have to do the same test for every candidate or else open our selves up to discrimination issues, no matter which team the position is for, and that was a no go.
- potatolicious 11y ago
  > "we can't do this because it looks like a test and we'd have to do the same test for every candidate" Oh, what a terrible tragedy, to give candidates consistent evaluations that are actually comparable between each other... /s I get the point you're presenting, but at the end of the day it still boils down to "we can't use tests because then we'd have to be held accountable for objectivity, we prefer instead a system that lacks any kind of rigor such that any result may not be effectively challenged, officially or otherwise".
- rdl 11y ago
  Work-sample tests are clearly protected; it's IQ tests which aren't permitted (Duke Power v.)
  - djb_hackernews 11y ago
    Right but 2 things: 1) HR (presumably taking direction from in house counsel) takes the position that it will be difficult to distinguish between a work sample test and an IQ test 2) We are a large organization with very broad roles. The Performance Team hiring for a Senior Software Engineer will probably want a different work sample than the Analytics Team...
    - tptacek 11y ago
      It is simply not true that the Perf team must deliver the same work-sample test as the Analytics team. Let's not dignify that argument. I absolutely believe you that your company has allowed HR to sabotage your hiring process, and that sucks. But let's not pretend HR is right to do it.
ArkyBeagle 11y ago
But this takes far too long.
- tptacek 11y ago
  It saved us enormous amounts of time at Matasano. When we designed our work-sample rubric, our #1 concern was filtering out people who interviewed well and/or had great resumes but weren't worth a billable hour on a real project, but our #2 concern was interviewing faster and getting results back to candidates quickly. (We didn't discover the best reason to do work-samples --- discovering hidden talent --- until after we'd started doing them). With standardized interviews and challenges, our whole hiring pipeline became entirely mechanical. We could send our a challenge and its instructions in 2 minutes, and evaluate the response the next day (or week, or whatever) later in less than 5 minutes. 7 minutes, to produce a technical evaluation that crushed a competing battery of tech interviews by several senior staff members that took 4-5 hours to do.
  - rorykoehler 11y ago
    Did you ever consider the candidate in all of this?
    - tptacek 11y ago
      Yes: like I said, the goal was to get results back to candidates faster. The biggest complaint we'd been getting from candidates before work samples was that our process took too long. Our goal became to get the entire process done under two weeks (from its worst point, a month or two earlier, of 1.5 months, when someone finally posted an anonymous complaint to Glassdoor about us). After iterating a couple times, we could reliably go from first contact to hire/no hire, with a motivated candidates (ie: one who would work with us to schedule the interview process aggressively) inside of a single week. Also, see: https://news.ycombinator.com/item?id=11328090 https://news.ycombinator.com/item?id=11328090 The process I'm advocating for is far better for candidates.
      - 11y ago
FLUX-YOU 11y ago
>It is amazing to me that almost nobody does this, but: almost nobody does this. There isn't enough feedback when someone has a bad idea on hiring. People will very quickly tell you if your code sucks though (this is a good thing), which might be the kind of feedback needed to get hiring changed.
JoeAltmaier 11y ago
How is working for a month different that "work on programming problems related to the work"? I'd have thought that is even better as a filter.
- tptacek 11y ago
  I am fine with asking for candidates to work as a 1099 for a month if you pay them the standard 1099 contractor rate, which is almost invariably more than 2x their FTE rate. I am not fine with the industry norm of paying people their eventual FTE salary on a 1099 basis; that's a scam. I think it's pretty dumb to do that, though, because most software developers can't take a month off of work from their current job to see if they're a fit for your company, and none of them will put their health insurance in jeopardy to do that. I also think it's pretty silly to design a process that deliberately demands a month of time to make a decision that could be made effectively in a week.
  - JoeAltmaier 11y ago
    Yeah last time I did it, it was just a week. And I got my full contract rate (which is about double as you suggest). It went well; got an offer. Its not usually that hard for a contractor to take the time off; they're on their own time already.
    - tptacek 11y ago
      That strikes me as a very costly way to build a team. Independent contractors are expensive; they know, better than most other developers, what their time is worth.
      - JoeAltmaier 11y ago
        Microsoft, of course, interns college juniors for 90 days as a summer job. Less costly I imagine per hour, but apparently working for them.
cp9 11y ago
cause I have a job that pays me. I'm not working on stuff for your company for free
- tptacek 11y ago
  That's exactly what you're doing when you go to an on-site job interview, so this objection is slightly less clever than you think it is.
  - cp9 11y ago
    except this is on top of that. nobody does a phone screen, checks a sample of code and says "tada, you're hired" we were going to get together in a room anyways.
    - tptacek 11y ago
      It shouldn't be on top of that! It wasn't for our process. Of course: we did in-person interviews. And in-person interviews are disruptive no matter what you do. But: * Our in-person interviews were shorter than typical in-person interviews because they weren't tasked with fully teching candidates out. * Because they didn't try to tech candidates out, they weren't as stressful, and so were less draining and unpleasant. * Because our in-person interviews were entirely scripted (the interviewers had very little latitude with what they could ask and how they could ask it --- which they hated, by the way), they were easy for everyone to deliver. * Before candidates arrived to their interview, they already knew (because we told them!) that they were likely to receive an offer from us based on their performance on the work-sample tests. I understand why people resist the idea of coding challenges, given: * They're not going to get feedback for weeks, months, or maybe ever * The challenge is going to be graded pass-fail, or whimsically, without rigor or consistency * They're going to have to do the exact same grueling bullshit draining dehumanizing programmer interview anyways * They're not going to see the challenges coming, or, for that matter, know exactly what happens next after the challenges are done That process is super common, even at companies that ostensibly do challenges. It sucks. I agree, no company can really get away with this in the long term. But those are also chickenshit problems. Switching from unstructured interviews to work-samples is hard: you have to change your mindset on how to qualify candidates, and there's a leap of faith involved. But getting candidates feedback, telling them what to expect with your process, keeping a schedule, and having processes in place to be consistent aren't hard problems. They are table-stakes common sense business execution, and if your team can't handle that right now, your team is mismanaged.
gtolle 11y ago
I love this approach. I'm also wondering what you think about giving well-designed work-sample tests over Google Hangout instead. The candidate shares their screen with you, so you can watch as they solve the problem in their own dev environment. You can understand how they approach problems (quick and dirty, slow and methodical, lots of rewriting, etc), and you can ask questions at the end. You get a good sense for how they work as an engineer even before having to bring them onsite. This seems to resolve the time asymmetry of take-home tests as well -- the interviewer spends as much time watching as the candidate spends working. The only downside I can see is that you have to design your problems to take about an hour instead of the 2-5 hours you could imagine for a take-home test. But, you can break them up into multiple rounds, and give additional exercises to the candidates who do well on the first one, for no more total time cost than a collection of onsite interviews. For what it's worth, I've done this at my startup and hired a great developer, and got very positive feedback about the process from the candidates I didn't end up hiring.
- crzwdjk 11y ago
  As someone who's had to do a few take home tests as well as coding with someone looking over my shoulder, I am definitely not a fan of the latter. I prefer to sit down and think about a problem, maybe even read it and let it sit in my head for a day or two before I actually start the coding. I also feel like shortening the time and making it synchronous would remove the opportunity to revise and polish the code I've written, which is one of the advantages of the take-home test format. My ideal interview format would involve a work sample test, followed by a code review of said work sample, and also a reverse code review (candidate is given some code to review). As an added bonus, all of this can easily be done over email without wasting too much of anyone's time, especially compared to an all day in-person interview with lots of whiteboard time.
bcaine 11y ago
I agree that this is "better" than the alternative, but it can be absolutely exhausting for candidates actively searching for a job. I feel like it's recently become much, much more common (from my small-ish sample of me and some friends). My issue with this approach is fourfold: 1. Most companies have no idea how to structure a problem that is both informative to them and also not abusive to the candidates time. 2. Companies generally do this right after the recruiter phone screen, which most likely doesn't give the candidate enough information to decide if the next steps are worth their time. 3. Most companies still do a whole suite of normal tech screens after you work on a take home problem. 4. If you're actively looking, getting a bunch of these over a short period of time is likely. I know during my full time search, more than 50% of companies had a take home test right after the recruiter screen. Most of these were 4-8 hours of work each, due within the week. A lot of startups structure it more like hazing or a barrier to entry than an evaluation criteria. I have some fun (read: horrifying) anecdotes from my recent search that illustrate the problems above, but I don't think any of my points are surprising. A nice alternative would have been to simply have one or two projects completed that are straightforward to evaluate and walk companies through them, letting them ask me questions.
- tptacek 11y ago
  Here's a really simple test for whether a work-sample scheme is effective, or just a bullshit trend-chasing afterthought: Does the work sample test offset all or most of the unstructured tech evaluation the company would otherwise do? If it does, that means they have a work-sample test rubric they believe in, and they're hiring confidently. If it doesn't, they don't believe in what they're doing, and the programming challenges they're assigning can thus reasonably be seen as more hoop-jumping. In the latter case, it's up to you to decide whether a prospective job is worth extra hoop-jumping. Some teams are worth it!
  - bcaine 11y ago
    I think that's fair. I've had both the former and the latter, but unfortunately most of my experiences fall into latter case, where it's simply been hoop jumping. Most of my friends (all about to graduate, so a good number of examples) are experiencing the same. For example one company gave a problem with five parts, with the final part being solve longest path on a bipartite weighted graph (which is quite a hard and time consuming problem). After that, the next step was a phone technical screen, then an on-site with 4-5 more interviews, most being white-boarding. It was basically hazing instead of an evaluation criteria. An alternative is my last job, which had a take home test that took about 6 hours, but that was the whole technical part of the process. Being on the other side reviewing them, the problem absolutely gave enough information. I totally get there's a right way to do it, but like most interviewing trends, companies seem to just be adding this as a step instead of revamping their process.
abossy 11y ago
How much time do you give candidates to complete their work samples? I'm curious whether work-sample tests filter out, say, parents with young children, in which a company with a 4-6 hour technical interview might be preferred over an unbound take-home assignment (in addition to the possibility of more interviews).