5 ms·

I completely disagree that the question is a bad question. He does not say there is a right answer, only that they are able to make a competent argument about

I completely disagree that the question is a bad question. He does not say there is a right answer, only that they are able to make a competent argument about what language they like and why. The point of the question is to see if they have actually put much thought into developing software. Enough thought, in fact, to have an opinion on what they like and why. Furthermore, it tests their ability to verbalize convincing arguments for their opinions with logical evidence for those arguments. These are skills that are needed when working in a development team. Now if he is saying, "well they said python and I hate python, so they are wrong!" Then, obviously, that is not a good interview tactic. But I didn't find that to be the case from the way it was presented in the article.

tptacek 12y ago
The problem isn't that there's a wrong answer, it's that it doesn't immediately admit apples-apples comparisons between candidates, it's hard to structure it so it can do that, and even harder (probably requiring several generations of candidates) to tune the apples-apples criteria to find something that actually predicts how candidates will actually perform the job. Those are the big theoretical problems with questions like that. But the pragmatic problems with it are even worse: they are a lens that focuses and applies subjective biases on the part of interviewers. Developers are terrible psychologists, and subjective interviews are the one time that we demand that software developers perform as psychologists. The best job interview questions are strictly and (as closely as possible) objectively predictive, and the easiest predictive questions to ask are work-sample questions: "here is a problem we solve routinely in this job, candidate, please solve it yourself."
- eli 12y ago
  I get what you're saying, but I think that's true of every interview question. There are no truly objective questions and people are unique; there is no way to make an apples-to-apples comparison of two different people. I think work-sample questions are really good, but you're still optimizing for people who can solve problems in the extremely artificial context of a job interview.
  - philh 12y ago
    "in the extremely artificial context of a job interview" is also true of every interview question. So giving it as a criticism of work-sample questions, when comparing those to other types of question, is meaningless.
    - tptacek 12y ago
      One nice benefit of work-sample hiring processes is that you can rig them to counteract the inherent hostility of a job interview, in ways that you can't easily apply to person-to-person free-form interviews.
      - saturdayplace 12y ago
        I just went through a process like this as a candidate. I was emailed a problem description and told, "Please send us the code that solves this problem. Do so at your leisure, and please don't spend more than x hours on it." This meant I got to use my own setup, and didn't feel as though I was required to come up with the "perfect" solution. When you think about it, it's odd that most programmers aren't interviewed this way. As a firm, the most important question I want answered is "Can the candidate do the job?" There are obviously other important considerations, but that's numero uno. And the best way to tell whether the candidate can do the job is to ask them for a small sample to prove it. I'm 100% certain I've been passed over for positions at other places because the hiring process isn't generally designed to answer that question accurately. It's something I want to keep in mind for when I'm on the other side of the table.
    - eli 12y ago
      That was my point. There is no "objective" question.
      - dllthomas 12y ago
        "Biased" isn't the opposite of "objective", unless the bias has to do with the observer.
- justinpaulson 12y ago
  I still completely disagree with you. You are misunderstanding the point of the question, and probably missing the importance of being able to formulate an opinionated argument about development and present it to a team. That is the point of the question. You should not be apples-apples comparing candidates anyway. Everyone is different, and it is a collection of skills that you need to be a developer. Candidate A may be a person who isn't as quick with coding, but they can quickly architect a solution in their mind and formulate an opinionated argument about why it is superior. Candidate B may be an incredible coder, but unable to explain their methodology, or architect a solution that is generic and re-usable. Both of these candidates can be an asset to the team in different ways. I find a lot of developers, the "JUST LET ME CODE" types, really don't understand the communication skills needed to develop an application with a large group of people. This question helps find people that will be better in that group situation, regardless of how strong their coding skills may be...because coding is only a portion of the job, in the real world. Edit: I also pointed out in my reply, that an interviewer who takes a subjective bias when assessing the answer to this question is doing it wrong.
  - tptacek 12y ago
    You believe that you shouldn't try to hire based on apples-to-apples comparisons of developers. I not only don't agree with you, but I run a (relative to startups) high-volume recruiting practice premised on that belief. We're not going to agree, and that's fine. In the 12 years of my career prior to discovering objective, work-sample, standardized, data-driven hiring, I hired exactly the way you're advocating. I sought out questions like the one you're advocating for. My favorite question from that time: "if you had to put together a libyou.a that you take from project to project, what would be in that library?" I found it to be a great way to start a conversation with a candidate to get inside of their head, and also liked the fact that it quickly gave me a signal as to whether the candidate had actually done the kind of work we did in our shop. As it turns out, the ability to project a signal as to whether a candidate has actually done the kind of work we do in our firm is not actually useful. What's useful is: "if the candidate is put in a reasonable position to try, can they actually perform the tasks the role requires". Surprisingly --- in fact, shockingly --- I've found zero correlation between the capacity to signal ability and the capacity to demonstrate ability. ZERO. Our best performers six months in, as often as not, interview terribly on "signalling" terms, and were hired because work-sample testing overrode those concerns.
- illumen 12y ago
  For your type of industry or company perhaps. Not all jobs or companies are the same, and different styles of questions will be more useful depending on those. A lot of development shops work by picking the best technology for the job. They choose the best plugin, or the best language, or the best component. In those types of shops, this question is exactly as you say - a task they will have to do routinely in the job.
  - tptacek 12y ago
    My type of industry is "consulting software development in every mainstream programming language and on every stack". We could have a very long thread arguing the point, but I'm going to go out on a limb and say that I hire for an unusually representative segment of the software development industry.
ASneakyFox 12y ago
The problem is like its asking you what your favorite type of glue is. Favorite hammer. Favorite color. Favorite grade of gasoline. Unless you have some weird emotional attachment to the subject it makes no sense.
ASneakyFox 12y ago
The problem is like its asking you what your favorite type of glue is. Favorite hammer. Favorite color. Favorite grade of gasoline. Unless you have some weird emotional attachment to the subject it makes no sense.

tptacek 12y ago

The problem isn't that there's a wrong answer, it's that it doesn't immediately admit apples-apples comparisons between candidates, it's hard to structure it so it can do that, and even harder (probably requiring several generations of candidates) to tune the apples-apples criteria to find something that actually predicts how candidates will actually perform the job. Those are the big theoretical problems with questions like that. But the pragmatic problems with it are even worse: they are a lens that focuses and applies subjective biases on the part of interviewers. Developers are terrible psychologists, and subjective interviews are the one time that we demand that software developers perform as psychologists. The best job interview questions are strictly and (as closely as possible) objectively predictive, and the easiest predictive questions to ask are work-sample questions: "here is a problem we solve routinely in this job, candidate, please solve it yourself."

eli 12y ago
I get what you're saying, but I think that's true of every interview question. There are no truly objective questions and people are unique; there is no way to make an apples-to-apples comparison of two different people. I think work-sample questions are really good, but you're still optimizing for people who can solve problems in the extremely artificial context of a job interview.
- philh 12y ago
  "in the extremely artificial context of a job interview" is also true of every interview question. So giving it as a criticism of work-sample questions, when comparing those to other types of question, is meaningless.
  - tptacek 12y ago
    One nice benefit of work-sample hiring processes is that you can rig them to counteract the inherent hostility of a job interview, in ways that you can't easily apply to person-to-person free-form interviews.
    - saturdayplace 12y ago
      I just went through a process like this as a candidate. I was emailed a problem description and told, "Please send us the code that solves this problem. Do so at your leisure, and please don't spend more than x hours on it." This meant I got to use my own setup, and didn't feel as though I was required to come up with the "perfect" solution. When you think about it, it's odd that most programmers aren't interviewed this way. As a firm, the most important question I want answered is "Can the candidate do the job?" There are obviously other important considerations, but that's numero uno. And the best way to tell whether the candidate can do the job is to ask them for a small sample to prove it. I'm 100% certain I've been passed over for positions at other places because the hiring process isn't generally designed to answer that question accurately. It's something I want to keep in mind for when I'm on the other side of the table.
  - eli 12y ago
    That was my point. There is no "objective" question.
    - dllthomas 12y ago
      "Biased" isn't the opposite of "objective", unless the bias has to do with the observer.
justinpaulson 12y ago
I still completely disagree with you. You are misunderstanding the point of the question, and probably missing the importance of being able to formulate an opinionated argument about development and present it to a team. That is the point of the question. You should not be apples-apples comparing candidates anyway. Everyone is different, and it is a collection of skills that you need to be a developer. Candidate A may be a person who isn't as quick with coding, but they can quickly architect a solution in their mind and formulate an opinionated argument about why it is superior. Candidate B may be an incredible coder, but unable to explain their methodology, or architect a solution that is generic and re-usable. Both of these candidates can be an asset to the team in different ways. I find a lot of developers, the "JUST LET ME CODE" types, really don't understand the communication skills needed to develop an application with a large group of people. This question helps find people that will be better in that group situation, regardless of how strong their coding skills may be...because coding is only a portion of the job, in the real world. Edit: I also pointed out in my reply, that an interviewer who takes a subjective bias when assessing the answer to this question is doing it wrong.
- tptacek 12y ago
  You believe that you shouldn't try to hire based on apples-to-apples comparisons of developers. I not only don't agree with you, but I run a (relative to startups) high-volume recruiting practice premised on that belief. We're not going to agree, and that's fine. In the 12 years of my career prior to discovering objective, work-sample, standardized, data-driven hiring, I hired exactly the way you're advocating. I sought out questions like the one you're advocating for. My favorite question from that time: "if you had to put together a libyou.a that you take from project to project, what would be in that library?" I found it to be a great way to start a conversation with a candidate to get inside of their head, and also liked the fact that it quickly gave me a signal as to whether the candidate had actually done the kind of work we did in our shop. As it turns out, the ability to project a signal as to whether a candidate has actually done the kind of work we do in our firm is not actually useful. What's useful is: "if the candidate is put in a reasonable position to try, can they actually perform the tasks the role requires". Surprisingly --- in fact, shockingly --- I've found zero correlation between the capacity to signal ability and the capacity to demonstrate ability. ZERO. Our best performers six months in, as often as not, interview terribly on "signalling" terms, and were hired because work-sample testing overrode those concerns.
illumen 12y ago
For your type of industry or company perhaps. Not all jobs or companies are the same, and different styles of questions will be more useful depending on those. A lot of development shops work by picking the best technology for the job. They choose the best plugin, or the best language, or the best component. In those types of shops, this question is exactly as you say - a task they will have to do routinely in the job.
- tptacek 12y ago
  My type of industry is "consulting software development in every mainstream programming language and on every stack". We could have a very long thread arguing the point, but I'm going to go out on a limb and say that I hire for an unusually representative segment of the software development industry.

ASneakyFox 12y ago

The problem is like its asking you what your favorite type of glue is. Favorite hammer. Favorite color. Favorite grade of gasoline. Unless you have some weird emotional attachment to the subject it makes no sense.

ASneakyFox 12y ago