6 ms·

It continues to surprise and frustrate me that as an industry we continue to highly prize proxy signals for engineering skill, when engineering skill is so dire

It continues to surprise and frustrate me that as an industry we continue to highly prize proxy signals for engineering skill, when engineering skill is so directly measurable. Why even bother with measuring things that are N-degrees removed from actual engineering, when you can just get people to engineer things? I know others on HN have been hammering this point home for years, but until something changes it deserves to be repeated ad nauseum: work samples work samples work samples work samples. Stop with the trivia questions. Stop with the contrived algorithms questions. Ask people to design systems, ask people to defend their designs, ask people to write real runnable code that directly relates to the work they will be doing at your job, ask people to review a real piece of code written at your company, ask people to critique real design produced at your company. Anything but what we're doing right now.

forhayley 9y ago
Nobody good has time to do work samples. Not to mention in terms of time invested they are always high asymmetric. Do you ever stop to consider why most top successful companies ask algorithm questions? Are they all really just so stupid?
williamsmj 9y ago
It's not that simple. Many good engineers, especially those who aren't young men, don't have existing work samples they are able to share. Perhaps, by choice or otherwise, they have a personal life that does not allow them to write code for free. And their employers won't allow them to share work code. And many good engineers are not willing to spend an unpaid weekend writing code auditioning for a job they aren't yet sure they want. And, as you know if you've ever worked with a disruptive colleague, or had customers, engineering in a team for a product is much more than just writing code.
- softawre 9y ago
  You are allowed to pay somebody for a work sample.
- angersock 9y ago
  > don't have existing work samples they are able to share. This, at least, should be fixable. Don't work at places that won't let you talk about your work and demonstrate your competence.
  - pm90 9y ago
    I don't think that's always possible. What if you worked for Healthcare or Security or Finance, where there are strict laws for keeping code/design private?
    - angersock 9y ago
      What are examples of such laws? There are laws allowing prosecution if you break agreements, sure, but the agreements themselves are what can be fixed by encouraging the market that they're incorrect.
      - DanBC 9y ago
        In the UK: https://en.wikipedia.org/wiki/Official_Secrets_Act_1989 https://en.wikipedia.org/wiki/Official_Secrets_Act_1989
        angersock 9y ago
        That seems targeted at .gov employees, which is a little different.
- tptacek 9y ago
  That's not what a work sample is. A work sample is a (usually, hopefully) standardized piece of work you request from all candidates that mirrors the actual work they'd be doing on the job. There's no reason a work sample needs to be so onerous that it costs you a weekend. What people forget when this comes up for discussion is that interviews are work. In fact, they are themselves onerous work, since they require you to be on-site and intensely engaged in ways you don't have to be to do the actual work of a job. A work sample, on the other hand, can be done from your home, with a beer next to your laptop if that's how you roll. The notion of audition work has definitely been abused in our industry. People get work samples that aren't standardized. They get work samples that are later ignored. Work samples don't have objective scoring rubrics. Some annoying companies assign new features for their products as "work samples". Done carefully, though, with objective and predictable grading and calibrated to offset in-person interviewing time, they're superior to any process we have.
  - ryandrake 9y ago
    As an applicant, I'd be all for going the work sample route, but only if it's paid for and if it were used in lieu of the typical face-to-face hazing session rather than in addition. I notice more and more companies are asking for work samples (or "homework assignments" or whatever they want to call them), but they just pile it onto their already onerous process. The major pain point in interviewing (at least for me) is the massive time sink for a very small chance of success. I'm spending a few hours on the application, then a few more doing phone screens, then a weekend doing a work sample, then I blow a VACATION day (which has monetary value and of which I have only 10 per year), and at the end of it all who knows what my odds even are? Multiply that by, lets say 10 companies per year, and: I've invested 31 days of my life, have no vacation left, and I still may end up with nothing.
    - tptacek 9y ago
      I think paying for interviews is problematic for a bunch of reasons. On the other hand, I agree wholeheartedly with people who reject "homework assignments" layered on top of a standard interview process. The onus is on people who want to take advantage of work samples to: * Ensure that the time they're asking of candidates is offset by lowered time demands elsewhere * Ensure that they're using the work samples objectively, so that people aren't asked to do coding work as part of a crap-shoot application process. The process we used at Matasano and NCC was less demanding than typical job interview processes. The challenges were simple and self-contained, and when they were completed we could tell you with a decent degree of confidence whether you were likely to be offered a job at your (shortened) on-site interview.
- potatolicious 9y ago
  I don't mean to propose that people spend their off hours writing work samples - that's unreasonable for all the reasons you've given and more. What I mean to propose is that we dramatically alter the current on-site interview process used by most companies. Instead of 5-7 interviews, consisting of 5-7 independent but ultimately equally contrived algorithms questions on a whiteboard, use the several hours you have the candidate to produce real, working code that reflects the work that your company does. Which is to say, this requires no more time commitment than the existing interview processes at typical tech companies. If the work your company does is deeply algorithmic, this will be reflected in the work sample produced via this process. If the work your company does is more heavily UX-oriented, this will be reflected in the work sample produced, also. Instead of "over the course of a full-day on-site round of interviews, candidate produced several short snippets of hand-scrawled code solving various CS textbook problems, which we will now use to infer general engineering ability"... ... you can say "over the course of a full-day on-site round of interviews, candidate designed and produced a module of code that does [X small thing that company does], which can be assessed with the same rigor and methodology we already apply to our existing work and employees"
- sidlls 9y ago
  While I agree that a focus on "work samples" is inappropriate, I must emphatically agree with the parent's broader point. The way companies in this industry conduct interviews and measure performance is like an aerospace engineering company testing candidates for satellite engineering on their understanding of the Standard Model (physics) or group theory or some such. There's too narrow a focus on what is required of an engineer in the software industry.
kenning 9y ago
I am currently reading "hackers: heroes of the computer revolution" and it seems like this culture of optimizing algorithms ("bumming") comes from the extremely early days of programming, when people were laying the groundwork for all the basic functionality of CS.
- kafkaesq 9y ago
  Well it's not just that -- algorithms do matter, and a very small number of programmers do move the needle in this industry through their ability to shine in that area. But the vast majority of the time -- that's not what your company needs at all. You need someone who's smart and reliable, and more to the point, believes in your mission. And who will go in and do all that far from algorithmic, highly unglamorous stuff that keeps your business from going underwater, day in, day out. But of course - those skills are difficult to evaluate in a short amount of time. So instead companies go for a skill that can be "measured". A skill like -- you guessed it -- algorithms.
  - teen 9y ago
    according to who?? like there are people who post stuff like this all the time, but it's just the anecdotes of a small population. companies have tried many different techniques and this is currently working the best. no top notch engineer is going to do a 8 hour take home test for every interview
    - kafkaesq 9y ago
      companies have tried many different techniques and this is currently working the best. Is it, now? The best thing you can say about is probably "it seems to kind of, sort of work" -- at the cost of burning through a whole lot of candidates, and their presumably worthless time. no top notch engineer is going to do a 8 hour take home test for every interview Yup, take-homes suck, also. But that doesn't mean that complementOf(8-hour-take-home) is the right answer, either.
teen 9y ago
I actually like the current interviewing approach... and I feel like we are a silent majority. I think whiteboarding / coding algorithm and design problems is pretty fair and reflective of engineering skill.
- gautamdivgi 9y ago
  The current approach has a huge false negative rate. It keeps out the bad apples but hugely favors marking a lot of good/great apples bad as well. The time invested to study for these interviews is quite onerous in itself. So, I'll say I'm not a fan of the approach. That being said we do use the method where I currently work to interview. However, the problems are not some made up situation or a test of data structures/algorithms but a set we've encountered in real life. From what I've heard the whiteboard method isn't really popular with many companies today either but everyone is sticking to it until some other alternative which avoid false positives presents itself.
- tptacek 9y ago
  You're the overwhelming majority, which is frustrating, because the method you're advocating for empirically and clearly does not work well.
  - teen 9y ago
    according to some random people on the internet?
- stupidcar 9y ago
  How is whiteboard coding reflective of engineer skill? At what time, during any job you've had, have you needed to instantly write code on a whiteboard, in front of peers, in a matter of minutes. Almost by definition, it's not reflective of engineering as actually practiced in a job. I'm a far bigger fan of at-home coding exercises. Yes, I know, some people get annoyed at these because they think you're asking them to do free work in their spare time. But what other way is there to test people's coding under circumstances that most adequately mirror those of an actual job. E.g. they have a (relatively) unlimited amount of time, they have access to Google, their IDE, etc. I'd much rather see what someone can come up with, in response to some novel problem, based on a couple of days programming, than what they can hack out on a whiteboard based on thirty seconds of thinking. The former seems closer to what coders are actually required to do, day on day.
DavidWoof 9y ago
What's ironic to me is that a lot of people are going to see this video and just start using programming contests as a negative proxy signal for engineering skill.
BatFastard 9y ago
Totally agree, and if someone does not have code they can share, they have no passion for programming. Therefore no job with me.
misingnoglic 9y ago
And with a work sample, how do you know: 1) how well they work with others? 2) how long it took them to come up with the solution? 3) if they actually came up with the solution themselves
- kafkaesq 9y ago
  1) how well they work with others? Whiteboard sessions don't tell you that either. Only whether they got lucky, and you picked a problem they already know (so they can breeze through the solution in front of strangers with confidence). Or if they can at least suck it up, and pretend they enjoy this nonsense for 45 minutes (or 4-6 hours, depending how crazy your company is about this stuff) to get a job. 2) how long it took them to come up with the solution? Does it matter? Really now. What matters on the job is did you go the last mile, think about the corner cases and pitfalls, and make sure your solution was cleanly coded and documented for the next guy or gal. Not whether you solved that silly HackerRank problem in 35 minutes as opposed to 45. 3) if they actually came up with the solution themselves This one's pretty simple to sniff out, actually: "So I'd just be curious, why you did X here and not Y?" There's pretty much no way a bullshitter can answer that question.

forhayley 9y ago

Nobody good has time to do work samples. Not to mention in terms of time invested they are always high asymmetric. Do you ever stop to consider why most top successful companies ask algorithm questions? Are they all really just so stupid?

williamsmj 9y ago

It's not that simple. Many good engineers, especially those who aren't young men, don't have existing work samples they are able to share. Perhaps, by choice or otherwise, they have a personal life that does not allow them to write code for free. And their employers won't allow them to share work code. And many good engineers are not willing to spend an unpaid weekend writing code auditioning for a job they aren't yet sure they want. And, as you know if you've ever worked with a disruptive colleague, or had customers, engineering in a team for a product is much more than just writing code.

softawre 9y ago
You are allowed to pay somebody for a work sample.
angersock 9y ago
> don't have existing work samples they are able to share. This, at least, should be fixable. Don't work at places that won't let you talk about your work and demonstrate your competence.
- pm90 9y ago
  I don't think that's always possible. What if you worked for Healthcare or Security or Finance, where there are strict laws for keeping code/design private?
  - angersock 9y ago
    What are examples of such laws? There are laws allowing prosecution if you break agreements, sure, but the agreements themselves are what can be fixed by encouraging the market that they're incorrect.
    - DanBC 9y ago
      In the UK: https://en.wikipedia.org/wiki/Official_Secrets_Act_1989 https://en.wikipedia.org/wiki/Official_Secrets_Act_1989
      - angersock 9y ago
        That seems targeted at .gov employees, which is a little different.
tptacek 9y ago
That's not what a work sample is. A work sample is a (usually, hopefully) standardized piece of work you request from all candidates that mirrors the actual work they'd be doing on the job. There's no reason a work sample needs to be so onerous that it costs you a weekend. What people forget when this comes up for discussion is that interviews are work. In fact, they are themselves onerous work, since they require you to be on-site and intensely engaged in ways you don't have to be to do the actual work of a job. A work sample, on the other hand, can be done from your home, with a beer next to your laptop if that's how you roll. The notion of audition work has definitely been abused in our industry. People get work samples that aren't standardized. They get work samples that are later ignored. Work samples don't have objective scoring rubrics. Some annoying companies assign new features for their products as "work samples". Done carefully, though, with objective and predictable grading and calibrated to offset in-person interviewing time, they're superior to any process we have.
- ryandrake 9y ago
  As an applicant, I'd be all for going the work sample route, but only if it's paid for and if it were used in lieu of the typical face-to-face hazing session rather than in addition. I notice more and more companies are asking for work samples (or "homework assignments" or whatever they want to call them), but they just pile it onto their already onerous process. The major pain point in interviewing (at least for me) is the massive time sink for a very small chance of success. I'm spending a few hours on the application, then a few more doing phone screens, then a weekend doing a work sample, then I blow a VACATION day (which has monetary value and of which I have only 10 per year), and at the end of it all who knows what my odds even are? Multiply that by, lets say 10 companies per year, and: I've invested 31 days of my life, have no vacation left, and I still may end up with nothing.
  - tptacek 9y ago
    I think paying for interviews is problematic for a bunch of reasons. On the other hand, I agree wholeheartedly with people who reject "homework assignments" layered on top of a standard interview process. The onus is on people who want to take advantage of work samples to: * Ensure that the time they're asking of candidates is offset by lowered time demands elsewhere * Ensure that they're using the work samples objectively, so that people aren't asked to do coding work as part of a crap-shoot application process. The process we used at Matasano and NCC was less demanding than typical job interview processes. The challenges were simple and self-contained, and when they were completed we could tell you with a decent degree of confidence whether you were likely to be offered a job at your (shortened) on-site interview.
potatolicious 9y ago
I don't mean to propose that people spend their off hours writing work samples - that's unreasonable for all the reasons you've given and more. What I mean to propose is that we dramatically alter the current on-site interview process used by most companies. Instead of 5-7 interviews, consisting of 5-7 independent but ultimately equally contrived algorithms questions on a whiteboard, use the several hours you have the candidate to produce real, working code that reflects the work that your company does. Which is to say, this requires no more time commitment than the existing interview processes at typical tech companies. If the work your company does is deeply algorithmic, this will be reflected in the work sample produced via this process. If the work your company does is more heavily UX-oriented, this will be reflected in the work sample produced, also. Instead of "over the course of a full-day on-site round of interviews, candidate produced several short snippets of hand-scrawled code solving various CS textbook problems, which we will now use to infer general engineering ability"... ... you can say "over the course of a full-day on-site round of interviews, candidate designed and produced a module of code that does [X small thing that company does], which can be assessed with the same rigor and methodology we already apply to our existing work and employees"
sidlls 9y ago
While I agree that a focus on "work samples" is inappropriate, I must emphatically agree with the parent's broader point. The way companies in this industry conduct interviews and measure performance is like an aerospace engineering company testing candidates for satellite engineering on their understanding of the Standard Model (physics) or group theory or some such. There's too narrow a focus on what is required of an engineer in the software industry.

kenning 9y ago

I am currently reading "hackers: heroes of the computer revolution" and it seems like this culture of optimizing algorithms ("bumming") comes from the extremely early days of programming, when people were laying the groundwork for all the basic functionality of CS.

kafkaesq 9y ago
Well it's not just that -- algorithms do matter, and a very small number of programmers do move the needle in this industry through their ability to shine in that area. But the vast majority of the time -- that's not what your company needs at all. You need someone who's smart and reliable, and more to the point, believes in your mission. And who will go in and do all that far from algorithmic, highly unglamorous stuff that keeps your business from going underwater, day in, day out. But of course - those skills are difficult to evaluate in a short amount of time. So instead companies go for a skill that can be "measured". A skill like -- you guessed it -- algorithms.
- teen 9y ago
  according to who?? like there are people who post stuff like this all the time, but it's just the anecdotes of a small population. companies have tried many different techniques and this is currently working the best. no top notch engineer is going to do a 8 hour take home test for every interview
  - kafkaesq 9y ago
    companies have tried many different techniques and this is currently working the best. Is it, now? The best thing you can say about is probably "it seems to kind of, sort of work" -- at the cost of burning through a whole lot of candidates, and their presumably worthless time. no top notch engineer is going to do a 8 hour take home test for every interview Yup, take-homes suck, also. But that doesn't mean that complementOf(8-hour-take-home) is the right answer, either.

teen 9y ago

I actually like the current interviewing approach... and I feel like we are a silent majority. I think whiteboarding / coding algorithm and design problems is pretty fair and reflective of engineering skill.

gautamdivgi 9y ago
The current approach has a huge false negative rate. It keeps out the bad apples but hugely favors marking a lot of good/great apples bad as well. The time invested to study for these interviews is quite onerous in itself. So, I'll say I'm not a fan of the approach. That being said we do use the method where I currently work to interview. However, the problems are not some made up situation or a test of data structures/algorithms but a set we've encountered in real life. From what I've heard the whiteboard method isn't really popular with many companies today either but everyone is sticking to it until some other alternative which avoid false positives presents itself.
tptacek 9y ago
You're the overwhelming majority, which is frustrating, because the method you're advocating for empirically and clearly does not work well.
- teen 9y ago
  according to some random people on the internet?
stupidcar 9y ago
How is whiteboard coding reflective of engineer skill? At what time, during any job you've had, have you needed to instantly write code on a whiteboard, in front of peers, in a matter of minutes. Almost by definition, it's not reflective of engineering as actually practiced in a job. I'm a far bigger fan of at-home coding exercises. Yes, I know, some people get annoyed at these because they think you're asking them to do free work in their spare time. But what other way is there to test people's coding under circumstances that most adequately mirror those of an actual job. E.g. they have a (relatively) unlimited amount of time, they have access to Google, their IDE, etc. I'd much rather see what someone can come up with, in response to some novel problem, based on a couple of days programming, than what they can hack out on a whiteboard based on thirty seconds of thinking. The former seems closer to what coders are actually required to do, day on day.

DavidWoof 9y ago

What's ironic to me is that a lot of people are going to see this video and just start using programming contests as a negative proxy signal for engineering skill.

BatFastard 9y ago

Totally agree, and if someone does not have code they can share, they have no passion for programming. Therefore no job with me.

misingnoglic 9y ago

And with a work sample, how do you know: 1) how well they work with others? 2) how long it took them to come up with the solution? 3) if they actually came up with the solution themselves

kafkaesq 9y ago
1) how well they work with others? Whiteboard sessions don't tell you that either. Only whether they got lucky, and you picked a problem they already know (so they can breeze through the solution in front of strangers with confidence). Or if they can at least suck it up, and pretend they enjoy this nonsense for 45 minutes (or 4-6 hours, depending how crazy your company is about this stuff) to get a job. 2) how long it took them to come up with the solution? Does it matter? Really now. What matters on the job is did you go the last mile, think about the corner cases and pitfalls, and make sure your solution was cleanly coded and documented for the next guy or gal. Not whether you solved that silly HackerRank problem in 35 minutes as opposed to 45. 3) if they actually came up with the solution themselves This one's pretty simple to sniff out, actually: "So I'd just be curious, why you did X here and not Y?" There's pretty much no way a bullshitter can answer that question.