9 ms·

> we use quantifiable work-sample tests as the most important weighted component in selecting candidates Can you speak to that a little? I'm reading it as eith

by dmunoz 13y ago

> we use quantifiable work-sample tests as the most important weighted component in selecting candidates Can you speak to that a little? I'm reading it as either programming challenges in a real environment, or analysis of previous code snippets submitted. In either case, I'm happy to see more companies doing this. I'm always looking for ways to improve my interview skills, but I want to do it honestly. Studying to the test isn't fun for me, I would rather hack on something.

tptacek 13y ago
Most programming puzzles aren't real work-sample tests. A work-sample test has to be representative of the actual stuff you'd do on the job. The trick is, to work in a recruiting context, you also want those tests to be standardized and repeatable. A lot of companies fall down on this. They have candidates do "real work", often in a pair-programming context. There a bunch of problems with this: (1) "Real work" usually isn't standardizable, so you can't compare candidates (2) Signal quality from the test is intensely dependent on who is doing the work with the candidate (3) Two different candidates might end up getting "tests" that are wildly different in terms of predictive power I have a bunch of ideas for pure software development work-sample tests, but I'm not ready to share them. The idea is simple, though: * It's a realistic exercise that approximates actual day-to-day work as much as possible * Every candidate gets the same exercises * The exercises have objective (preferably gradable) outcomes
- balls187 13y ago
  Who asks programming puzzles anymore? When was the last time you applied for a software developer job?
  - tptacek 13y ago
    Before Matasano, if you don't count clients interviewing us. Who asks programming puzzles? Read interview postmortems. The answer appears to be "everyone". We are probably just talking past each other: asking someone how to sanitize an input or parse an HTML document during a face-to-face interview is a "programming puzzle".
    - balls187 13y ago
      It's hard to imagine that Facebook, Twitter, Dropbox, Google, Amazon, Apple, and all the rest of the companies that hire software developers, who have more or less the same type of interview, have gotten things wrong, yet have been so successful in hiring good people and building good products. Perhaps the real problem is that those companies have taken all the people who perform well at whiteboard interviews, and what's left are people who struggle. Still, I wouldn't say the system is flawed because it selects a certain type of people. Clearly those people do well.
      - tptacek 13y ago
        It's not at all hard to imagine that the best known company in the industry, the one that spends the most on direct outreach to clients, whose compensation system more or less sets the standard for the industry, a company that is essentially rolling in money, is able to make a conventional recruiting process viable for itself. Having said that, if you talk to Google people who are close to hiring, especially about the process that connects "interviewers" to "recruiters", I don't think you're going to hear consistently good things about how the system works in practice.
      - benched 13y ago
        The point you are grazing is that this approach results in many false negatives, and few false positives. This is workable for companies that are consistently able to attract large numbers of candidates, such as the ones you listed. (And yes, there are false positives. I personally witnessed someone at a large company who was very adept at whiteboard coding, and a very lazy employee.)
        balls187 13y ago
        You've said it more eloquently than I can, but I want to bring up that: "results in many false negatives..." is not some universal truth. Is there evidence that shows current software interviews turn down way too many great candidates? There should exist a large enough group of developers who have been rejected a number of times[0], who have gone on to be very successful[1] in their careers. Does this data exist? I agree that the current process isn't perfect, and it does produce some false negatives and some false positives. The amount which, has yet to deter me from beliving that the system works, as intended. Certainly there are some fringe candidates, who are turned away, but I don't think it's as onerous as people are making it out to be. Also, most companies didn't start out with unlimited piles of cash, nor could they have their pick of employees. They had to work at it, and I don't believe the culture of their hiring has changed significantly. For the record, I struggle with traditional coding puzzle interviews. But I also know coding on the whiteboard is a skill, which can be practiced, and with time, improved upon, such that if you are a really good developer, but struggle with passing interviews, you can deal with that problem by becoming better at interviewing. Or you know, create something awesome, and never have to worry about interviewing again. [0] Failing one or two interviews isn't really indicative of anything. So this group would require consistently failing multiple standard coding interviews. [1] How to measure success? I say gone to be a major contributor to the success of a company, or built their own company that is considered "successful."
- falsestprophet 13y ago
  This approach will be difficult for firms that use contingent recruiters, because the recruiters will be incentivised to coach candidates.
  - tptacek 13y ago
    Does anyone have any luck with outsourced recruiters? From where I stand, they seem like a bane of the industry.
    - balls187 13y ago
      I have had good enough luck that I won't dismiss a recruiter outright. I treat recruiters like widening the funnel, and haven't found any correlation that suggests they produce better results. I have found the best results is hitting people up through linkedin.
    - rhizome 13y ago
      Perhaps coincidentally, companies say there is an engineer shortage while at least 80% of the Rails jobs on LinkedIn & Craigslist are posted by recruiters. Recruiters may simply be ruining the employment market, probably as a gold rush.
    - baddox 13y ago
      Anecdotally, I know of an experienced and competent engineer who actively sought out a recruiter, hoping to find a job in Silicon Valley (which he did). Some people just can't be bothered with the interview grind, and if you know you're good, why not have a recruiter do a bit of the grunt work for you? But I suppose this could be an edge case, where a legitimately good candidate actually works with a recruiter.
      - vonmoltke 13y ago
        I tried that route to no success. My problem, back when I actually wanted to work in SV, was I found it to be an impenetrable bubble. There seems to be a serious delusion that any engineer worth a damn will move to SV of their own volition, so the only reason to look outside is to try to snatch the top talent from the top schools elsewhere.
- theboss 13y ago
  So accurate. I'm a grad student who studied security deeply during my time in school. I want to keep my hands on a keyboard and I don't like airplanes (which rules out consulting) so I've applied at a lot of companies to be a ``security engineer''. I can't tell you how many well-respected companies ask me to write min-heaps, depth-first searches, etc. I don't understand what they are asking me this for...It isn't even close to a realistic representation of what my day-to-day responsibilities would be. It is also an immediate turn-off....
  - jw2013 13y ago
    Similar story here. Now not only these "big" companies as Facebook, Google, Microsoft focus mainly on algorithms and data structure during interview, to my surprise some of the early startups ask heavily for whiteboard coding... It's no surprise these companies rejects many really good hackers (at least from people I know... yes, I admit the small sample size) and accepts these just optimize for algorithm preparations by doing a set of online coding problems.
    - rhizome 13y ago
      The startup industry saw the same kind of interviewing voodoo in the 90s, copycatting how Microsoft conducted them.
    - deleted 13y ago
      [deleted]
  - GFK_of_xmaspast 13y ago
    They're asking you that because there are lots and lots of candidates who look great on paper but don't actually know anything. I have personally met several. I wouldn't ask a candidate to implement a min-heap (mostly because it's kind of obscure and I'd have to think about it) but a depth (or breadth) first search is something I'd expect any competent candidate to rattle off in a heartbeat, just because it's such a fundamental cs principle that gets used in lots of places and, unlike things like sorts, doesn't really have a good library to fall back on in practice.
- mason55 13y ago
  We actually took a few things that we wrote in the infancy of the company and turned them into work-sample tests. We use the same test for everyone so that we can grade effectively while still ensuring that it's a good sample of what they'd be working on (because it's something we actually did!). We have a front-end and back-end test. I'm sure that at some point we'll grow large enough that solutions to our test will be posted online but I'm not going to worry about it right now.
  - urethra 13y ago
    They won't be posted online because my team will have put you out of business by then.
delinquentme 13y ago
I was about to simply repost this sentence as its the HEAVY substantial bit of the post. What does it mean? Throw engineering problems your CURRENT team has solved in the past at new candidates. I have seen only ONE startup ( even among the nastiest of hard problem seekers ) GET how substantial this is. So many of these companies are too busy to think about smart sourcing, at their own peril. Test new engineers off past solved problems. End of story.
jcurbo 13y ago
I like this guy's stuff: http://codingforinterviews.com/ http://codingforinterviews.com/

by dmunoz 13y ago

tptacek 13y ago

Most programming puzzles aren't real work-sample tests. A work-sample test has to be representative of the actual stuff you'd do on the job. The trick is, to work in a recruiting context, you also want those tests to be standardized and repeatable. A lot of companies fall down on this. They have candidates do "real work", often in a pair-programming context. There a bunch of problems with this: (1) "Real work" usually isn't standardizable, so you can't compare candidates (2) Signal quality from the test is intensely dependent on who is doing the work with the candidate (3) Two different candidates might end up getting "tests" that are wildly different in terms of predictive power I have a bunch of ideas for pure software development work-sample tests, but I'm not ready to share them. The idea is simple, though: * It's a realistic exercise that approximates actual day-to-day work as much as possible * Every candidate gets the same exercises * The exercises have objective (preferably gradable) outcomes

balls187 13y ago
Who asks programming puzzles anymore? When was the last time you applied for a software developer job?
- tptacek 13y ago
  Before Matasano, if you don't count clients interviewing us. Who asks programming puzzles? Read interview postmortems. The answer appears to be "everyone". We are probably just talking past each other: asking someone how to sanitize an input or parse an HTML document during a face-to-face interview is a "programming puzzle".
  - balls187 13y ago
    It's hard to imagine that Facebook, Twitter, Dropbox, Google, Amazon, Apple, and all the rest of the companies that hire software developers, who have more or less the same type of interview, have gotten things wrong, yet have been so successful in hiring good people and building good products. Perhaps the real problem is that those companies have taken all the people who perform well at whiteboard interviews, and what's left are people who struggle. Still, I wouldn't say the system is flawed because it selects a certain type of people. Clearly those people do well.
    - tptacek 13y ago
      It's not at all hard to imagine that the best known company in the industry, the one that spends the most on direct outreach to clients, whose compensation system more or less sets the standard for the industry, a company that is essentially rolling in money, is able to make a conventional recruiting process viable for itself. Having said that, if you talk to Google people who are close to hiring, especially about the process that connects "interviewers" to "recruiters", I don't think you're going to hear consistently good things about how the system works in practice.
    - benched 13y ago
      The point you are grazing is that this approach results in many false negatives, and few false positives. This is workable for companies that are consistently able to attract large numbers of candidates, such as the ones you listed. (And yes, there are false positives. I personally witnessed someone at a large company who was very adept at whiteboard coding, and a very lazy employee.)
      - balls187 13y ago
        You've said it more eloquently than I can, but I want to bring up that: "results in many false negatives..." is not some universal truth. Is there evidence that shows current software interviews turn down way too many great candidates? There should exist a large enough group of developers who have been rejected a number of times[0], who have gone on to be very successful[1] in their careers. Does this data exist? I agree that the current process isn't perfect, and it does produce some false negatives and some false positives. The amount which, has yet to deter me from beliving that the system works, as intended. Certainly there are some fringe candidates, who are turned away, but I don't think it's as onerous as people are making it out to be. Also, most companies didn't start out with unlimited piles of cash, nor could they have their pick of employees. They had to work at it, and I don't believe the culture of their hiring has changed significantly. For the record, I struggle with traditional coding puzzle interviews. But I also know coding on the whiteboard is a skill, which can be practiced, and with time, improved upon, such that if you are a really good developer, but struggle with passing interviews, you can deal with that problem by becoming better at interviewing. Or you know, create something awesome, and never have to worry about interviewing again. [0] Failing one or two interviews isn't really indicative of anything. So this group would require consistently failing multiple standard coding interviews. [1] How to measure success? I say gone to be a major contributor to the success of a company, or built their own company that is considered "successful."
falsestprophet 13y ago
This approach will be difficult for firms that use contingent recruiters, because the recruiters will be incentivised to coach candidates.
- tptacek 13y ago
  Does anyone have any luck with outsourced recruiters? From where I stand, they seem like a bane of the industry.
  - balls187 13y ago
    I have had good enough luck that I won't dismiss a recruiter outright. I treat recruiters like widening the funnel, and haven't found any correlation that suggests they produce better results. I have found the best results is hitting people up through linkedin.
  - rhizome 13y ago
    Perhaps coincidentally, companies say there is an engineer shortage while at least 80% of the Rails jobs on LinkedIn & Craigslist are posted by recruiters. Recruiters may simply be ruining the employment market, probably as a gold rush.
  - baddox 13y ago
    Anecdotally, I know of an experienced and competent engineer who actively sought out a recruiter, hoping to find a job in Silicon Valley (which he did). Some people just can't be bothered with the interview grind, and if you know you're good, why not have a recruiter do a bit of the grunt work for you? But I suppose this could be an edge case, where a legitimately good candidate actually works with a recruiter.
    - vonmoltke 13y ago
      I tried that route to no success. My problem, back when I actually wanted to work in SV, was I found it to be an impenetrable bubble. There seems to be a serious delusion that any engineer worth a damn will move to SV of their own volition, so the only reason to look outside is to try to snatch the top talent from the top schools elsewhere.
theboss 13y ago
So accurate. I'm a grad student who studied security deeply during my time in school. I want to keep my hands on a keyboard and I don't like airplanes (which rules out consulting) so I've applied at a lot of companies to be a ``security engineer''. I can't tell you how many well-respected companies ask me to write min-heaps, depth-first searches, etc. I don't understand what they are asking me this for...It isn't even close to a realistic representation of what my day-to-day responsibilities would be. It is also an immediate turn-off....
- jw2013 13y ago
  Similar story here. Now not only these "big" companies as Facebook, Google, Microsoft focus mainly on algorithms and data structure during interview, to my surprise some of the early startups ask heavily for whiteboard coding... It's no surprise these companies rejects many really good hackers (at least from people I know... yes, I admit the small sample size) and accepts these just optimize for algorithm preparations by doing a set of online coding problems.
  - rhizome 13y ago
    The startup industry saw the same kind of interviewing voodoo in the 90s, copycatting how Microsoft conducted them.
  - deleted 13y ago
    [deleted]
- GFK_of_xmaspast 13y ago
  They're asking you that because there are lots and lots of candidates who look great on paper but don't actually know anything. I have personally met several. I wouldn't ask a candidate to implement a min-heap (mostly because it's kind of obscure and I'd have to think about it) but a depth (or breadth) first search is something I'd expect any competent candidate to rattle off in a heartbeat, just because it's such a fundamental cs principle that gets used in lots of places and, unlike things like sorts, doesn't really have a good library to fall back on in practice.
mason55 13y ago
We actually took a few things that we wrote in the infancy of the company and turned them into work-sample tests. We use the same test for everyone so that we can grade effectively while still ensuring that it's a good sample of what they'd be working on (because it's something we actually did!). We have a front-end and back-end test. I'm sure that at some point we'll grow large enough that solutions to our test will be posted online but I'm not going to worry about it right now.
- urethra 13y ago
  They won't be posted online because my team will have put you out of business by then.

delinquentme 13y ago

I was about to simply repost this sentence as its the HEAVY substantial bit of the post. What does it mean? Throw engineering problems your CURRENT team has solved in the past at new candidates. I have seen only ONE startup ( even among the nastiest of hard problem seekers ) GET how substantial this is. So many of these companies are too busy to think about smart sourcing, at their own peril. Test new engineers off past solved problems. End of story.

jcurbo 13y ago

I like this guy's stuff: http://codingforinterviews.com/ http://codingforinterviews.com/