18 ms·
The Hiring Post
- tptacek 12y agoThis is a great writeup on a similar process: https://push.cx/2015/hiring-apprentices https://push.cx/2015/hiring-apprentices
- deleted 12y ago[deleted]
- SamReidHughes 12y agoAs somebody who agrees in theory with tptacek's posts and now this blog post, what I find very funny is how I perceive typical technical hiring, with the interview gauntlet, to still be way better than what some other industries must do. How the hell does anybody hire a teacher?
- lutorm 12y agoBy asking them to teach something and observing them do it.
- adpirz 12y agoTeacher here. Good schools will ask you to teach a sample lesson on a particular subject and prep your own materials. Principals and other teachers will observe and give feedback. It's what I did to work at my current school, and we're one of the top performing schools in our state.
- eru 12y agoSounds like a work sample test to me. Great!
- tomp 12y agoExcept that it's the kind of work-sample test that you can't do in the evening, and is hard to complete if you already have a full-time job (except if only other teachers are the audience).
- yummyfajitas 12y agoHow would principals/teachers know if and how much the sample lesson transfers knowledge to students? It seems like this would filter out disasters, but can it do more than that?
- adpirz 12y agoThat's when you look at other factors like past performance, including (but certainly not limited to) performance on his / her pupils' standardized tests. Like any other hiring process, it's not always perfect, but you also learn along the way which characteristics make for great teachers (responding to feedback, ability to engage with students, timeliness, etc.).
- scottm01 12y agoI also agree with the sentiment of this post and have always been surprised how poorly prepared most engineers are to fairly conduct interviews. I'm reasonably convinced it's contributing seriously to inequality issues and the "tech labor shortage", and I'm absolutely convinced it's the reason I don't job hop more then I do (hrm, maybe that's part of why the system still sucks?) That said, my wife happens to be a teacher. The process from what I can tell in private schools is pretty similar. Usually a recruiter ("placement agency") sends a bunch of resumes (CVs) to a hiring manager (department lead or school head). They pick some based on how they feel that day and conduct phone screens. Successful applicants then come in to do the equivalent of a whiteboard interview -- they are evaluated on their ability to teach a classroom of students they've never met a lesson plan they've probably not seen more then a day or two in advance (although at least they get that much warning!) edit On reflection I hope that teachers are at least a little more capable of objectively evaluating candidates then your average sysadmin/software engineer (how often are you asked to assess someones ability outside a handful of interviews?), but believe me the sample lesson causes as much stress on the candidate as a whiteboard interview. It also stresses the teacher whose class is being "donated" to candidates and depending on the length of the process can derail what's usually a carefully planned out quarter/semester/etc.
- white-flame 12y agoA big difference is that the benefits of quality programmers manifest over long projects. With many other industries, you can say "Do this" in a shorter, observable time frame and have it more aligned with what they'll be facing on the job. Contrast this to dumb interview coding challenges, for instance.
- dsp1234 12y agoA recent interview for a teaching position involved being given the name of a class to teach. At that point, generating sample classroom materials (a simple worksheet, powerpoint, etc), and a sample lesson plan. Then the interview consists of teaching the sample lesson plan to several interviewers. In other words, a work sample test.
- steakejjs 12y agoGreat post. I hate interviewing. I think a lot about it since I'm the type of person who performs horribly in an interview call when some person I've never met is asking me trivia questions on speaker phone... One thing I wonder. Why are people being interviewed for technical positions in a google doc or whatever fake coding environment? Servers are cheap. If you have to go the way of the technical phone interview, doesn't it make more sense to have both (interviewer and interviewee) parties SSH into a server and work together on some type of problem for an hour? The interviewer can even watch the candidate program or work in screen. This type of interview isn't perfect, but closed book "Programming Jeopardy" style interviews are just worse in every way.
- calcsam 12y agoWe use HackerRank, which has an in-browser IDE that can runs pretty much every language, has highlighting, etc. I can't believe anyone would use Google Docs. Even if you don't care about your interviewees' experience, huge waste of dev time.
- sdevlin 12y agoI once had a phone interview wherein I was required to write some code and then read it to the interviewer (character by character!) over the phone. I asked him whether it would make more sense for us to use etherpad (this was maybe five years ago), but he said no.
- baddox 12y ago> The majority of people who can code can’t do it well in an interview. Major citation needed. I don't have any real data either way, but I find this extremely hard to believe.
- dropit_sphere 12y agoI don't at all. Yeah, sure, I can buzz your fizz with my eyes closed in brainfuck. But that's just measuring language facility. The real tests are more topical, and more complex: is there a way to make this code run faster? There are two types of good candidates that will get that question: 1) the kind who have run into the problem before and know the answer stone cold, and tell you instantly 2) the kind that have no experience in that particular area (say, search algorithms) but are google-proficient, such that if you give them a day or two will talk your ear off about them. In a work situation, the two are really equivalent. After a day, 2) is indistinguishable from 1). But when interviewing?
- MichaelGG 12y agoWhat about kind 3, the ones that can derive the solution from scratch, in their heads? I thought that was the point of most of those questions. Like "given two linked lists, what's the fastest way to determine if they share any nodes (not values, but actual nodes)"? MS gave me that one in a phone screen and I'd never written a linked list before. But if you actually know what a linked list is, this should be simple for you to derive relatively quickly (not Googling for a day). If a candidate needs a day to learn up about something trivial they should be able to derive in minutes, then that kind of thing might compound and end up producing a huge difference in actual productivity ("all other things being equal").
- dropit_sphere 12y ago> But if you actually know what a linked list is, this should be simple for you to derive relatively quickly (not Googling for a day). Actually I think this is the worst kind of question to ask, because it measures: can the candidate code under pressure? Not "we need to get this done by launch/before client meeting" pressure, but right now pressure. If a company wants literal coding ninjas who can reason about computation while in the middle of lightsaber battles, sure. But in the much more likely event that they're hiring people to sit for days on end and work on a large project, then this is an unnecessary obstacle.
- joshdance 12y agoI have a friend who is a good programmer, and likes learning, and is easy to work with, but he is terrible at technical interviews. He stresses about them to no end. This sounds like a much better process.
- BinaryIdiot 12y agoI know no end of people in the exact same bucket. I used to think I wasn't in the bucket at all as I've been through many difficult and time consuming interviews but I always got an offer until I interviewed at Microsoft and Google (though I still maintain my failure at those two companies were not entirely my fault but I digress). After going through the march of those two big companies I found the process incredibly annoying and not useful at all; nothing of substance was really ever discussed it was almost always academic or some sort of trick question where I get to hear the interviewer talk for half an hour how they came up with a superior technique. I've interviewed a lot of people. Interviewing software developers is such a pain in the ass.
- apaprocki 12y agoI found it a funny coincidence that the current top two links (this and a comment on the Holman post[1]) take opposing viewpoints on programmers as soldiers: "Infantry are what most people are." and some are "Commandos" vs Tom's "Engineering teams are not infantry squads. They aren’t selected for their ability to perform under unnatural stress." [1]: https://news.ycombinator.com/reply?id=9159398 https://news.ycombinator.com/reply?id=9159398
- boothead 12y agoThis is great! As a former commando turned programmer I've long been wondering how best to express the commando mindset in programmers terms. The 4 elements of the command spirit are Courage Determination Unselfishness Cheerfulness in the face of adversity These sure come in handy in some programming environemnts :-)
- jowiar 12y agoThis article seems to put forward a good theory on interviewing someone who you want to lock in room with a problem and have code come out the other end. Unfortunately, most of us work in the real world, where that's a very limited portion of the engineering job. When the author says selecting for: "people who have the social skills to actively listen to someone else’s technical points, to guide a discussion with questions of their own, or to spot opportunities to redirect a tough question back to familiar territory." is a mistake, because some of them can't code. I know how to turn someone who can't code into someone who can. I don't know how to turn someone who fails at dealing with humans into someone who succeeds. And, at least in my job, engineers spend a whole lot more time dealing with people than with code.
- tptacek 12y agoI know how to turn someone who can't code into someone who can. If you can do that reliably, you should be making zillions of dollars. My experience is the opposite: if someone is very impressive in an interview but a zero on the team, they're an intractable management problem. In any case: there is a difference between being cripplingly antisocial and being able to deftly handle an interview. The social skills required to flip the script on an interview are distinctive, and they aren't the "team player" skills. In fact, as I re-edited that paragraph and came up with "controlling the conversation", it occurred to me that they might signal a candidate who has problems on teams.
- calcsam 12y agoI'd suspect technical skills are more highly valued in crypto than in other programming domains. That may account for your differing perceptions.
- tptacek 12y agoI'm not sure I follow. But in any case, Matasano isn't a cryptography firm; it's a software consultancy. The set of things we did to boot up a specialty practice in cryptography are a different blog post.
- 12y ago
- Sevzi 12y agowe just need to make narcissism more socially acceptable, so people are always boasting. then an interview doesn't seem so different.
- white-flame 12y ago"Years from now, we’ll look back at the 2015 developer interview as an anachronism..." People were saying similar things 20 years ago, too. Others will likely corroborate that from earlier.
- tptacek 12y agoAuthor. I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions. And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".
- aristus 12y agoThis is a good post, Thomas. From this I think I could make an efficient test for the kind of person who'd make a good performance engineer. Developing a more general work-sample for developers is a harder problem, but still feels doable. This may not be an issue for you yet, but what about plagiarism? And can you expand on your hints about how you changed your pipeline at the front of the process? You don't interview every resume that comes your way, right?
- tptacek 12y agoEvery serious application, regardless of their resume, got a 30-45 minute phone call. You post hiring posts all over the place and you get a lot of unserious submissions, but an easy way to screen them out is to respond with a question, like, "Thanks for writing! We'd be happy to talk to you. Can we ask what has you interested in doing software security?" The answer doesn't matter, as long as there is an answer. I did a lot of calls where I knew a couple minutes in that we were unlikely to hire the person, but (a) I got surprised by outcomes enough not to shirk on those calls, and (b) those calls are a very small price to pay for finding buried talent.
- kasey_junk 12y agoI also advocate for a work sample based hiring pipeline and the plagiarism question often comes up. My response is always: 1) People generally don't cheat. If you have a high percentage of cheaters its how you are filling your pipeline you need to address, not the filter. 2) What most people think of as "plagiarism" is actually very common in the real work of software developers. Very frequently you see/mimic other peoples work to solve problems. Instead of being freaked out about a skill that is central to the job, why not use it to evaluate the candidate. Did they "plagiarize" the right thing? Did they do it effectively. Did they do something backwards that a simple google search would have found a thing to copy? 3) If you are big enough for plagiarism to be a real problem and have addressed points 1 & 2, it is relatively easy to detect mechanically (and there is a surprising amount of research in the field as CS professors invariably write both an automated grader and then an automated cheater detector).
- drewcrawford 12y agoAs a vote of confidence, this article moved me from "I would never ever interview at Matasano" to "I don't want a full-time job at a security firm, but if I did, I would probably start looking here."
- tptacek 12y agoThanks! I hope that carries over to the new firm we started, too. :P
- drewcrawford 12y agoWoah! Don't be afraid to plug your new firm in the blogpost :-) I doubt I'm the only one to mistakenly ascribe your new hiring policy to your old company.
- tptacek 12y agoNo, you're not mistaken! That's how Matasano has hired for something like 2.5 years. I left the company to do hiring-related things, after feeling like I'd more or less disposed of the challenge completely at Matasano/NCC. There's a pretty big plug coming next week, though not from me. What we're doing meshes well with the worldview in my post, but it isn't a productization of it. This is just what I really think about hiring.
- x0x0 12y agoYou gave the game away completely when you named your company Starfighter. Some of us lived through the 80s too =P
- maxbrown 12y agoCan't find a lick of information about Starfighter online - is that on purpose right now then? (Pre-"plug"/launch?) Definitely curious, give us the 2-sentence version if you can?
- 12y ago
- moron4hire 12y agoAll of the good developers I meet already have jobs and they don't want to leave them. So I'm experimenting with "taking what I can get", i.e. subcontracting out a few hours a week. The hypothesis is that X great developers for Y hours a week will provide higher quality results than Y mediocre developers for X hours a week. It seems to be working out so far. I don't have a lot of data to go on, but I haven't had to completely throw out code yet.
- tibbon 12y agoI've been interviewing for the past month or so and its been terrible from many perspectives; I'm fine with the interviews content-wise (silly questions excluded), but the entire process is such a time-sink. I have a current position (full-time job) and really just don't have time for (with each company) 30 minutes of emailing, an hour phone call, a short coffee meeting, an offline coding challenge and then a 3 to 6 hour on-site interview. Additionally the weather in Boston has been dreadful, and getting to places just takes longer than normal. Again, I already have a full-time position, and finding ways to schedule these into my work day just doesn't really work well. The interviews I've had have gone well, but I'm counting down the days until I find the perfect company that makes me an offer that I like, because the entire process is just eating into my life.
- eps 12y agoWe went through the phase when we gave candidates a problem and let them work on it remotely. It was in an embedded C shop that did a lot of kernel work. Basically, we'd give a short programming task (say, to write an intrusive AVL tree container in C) and 24 hours. Guess what? HALF of candidates cheated. Meaning that when the got called for an in-person interview, they stumbled to explain how "their" code worked. To say that I was shocked is an understatement. It's bloody obvious that such blatant cheating would surface within the first week after hiring and yet they cheated :-| In the end we had to switch to the in-person interviews only and implement few other things to discourage cheating. In other words - YMMV, what works for others may misfire miserably for you.
- deleted 12y ago[deleted]
- meowface 12y agoYeah, that's the only real problem with remote work sample tests. Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).
- tptacek 12y agoActually, I think the opposite is true; in appsec, the final work product is a list of vulns, so if you want to cheat, it only takes 4 minutes talking to anyone else who has ever succeeded in the process before. For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on. For my part: we ran this process for over two years and never discovered any plagiarism. Meanwhile, we drastically increased the size of our team and had total retention; from the time I took over recruiting to the time I left the firm, we lost a total of zero of those hires. All of them worked out.
- 12y ago
- xacaxulu 12y agoOverall a really good article. I feel however that if I'm interviewing for a job, I've already lost. It probably means that I haven't been 1) working my network and nurturing those important professional connections, 2) contributing in meaningful ways to Open Source projects (in order to generate interest) or 3) impressing people with real-world solutions. I've never enjoyed a job that I had to seek out or interview for, your mileage may vary however.
- fitzwatermellow 12y agoSo, who feels like building a cloud based "work-sample tests" platform with me?
- spacemaus 12y agoFunny, last week I wrote some notes about doing just this :). It would be hard to build a general purpose platform, but I think if it focused on just replacing the initial technical phone screen, it could be both reliable and useful. There's a common set of skills that most any software engineer should have. I think those skills can probably be checked via a web tool, and in higher fidelity than you can get over the phone. As a candidate applying to N jobs, I'd much rather take a web screen once instead of doing N technical phone interviews. And as an interviewer, I'd much rather not have to ask FizzBuzz again.
- fitzwatermellow 12y agoI envision something akin to the http pipeline in a web app the is composed of several "microservices". One stage of the pipeline is left intentionally out. The candidate then is required to build an actual service. They can use any language they want. Even any cloud host they choose. All that's provided is the entry points, exit points, wire protocol, data encoding, and the spec to be implemented. After passing that, advanced topics like logging, telemetry and performance provide grounds for further discussion to assess development "philosophy". But the real genius would be selling this work-sample platform-as-a-service to enterprise customers. Who could be prompted into building a catalog of microservices based upon their APIs and real data. And out of that hopefully some innovative, hackathon-esque mini-products could arise ;) I definitely never thought I would become an "HR guy". But I really like this idea. Will include a contact email in my profile if anyone is interested in discussing it further off HN as well...
- lordnacho 12y agoI've done a few online tests recently, and I was thinking of how to do a better one. 1) The main idea was to have the results of unit tests visible live. I've done a few tests where I found out I didn't get through even though the test rig says it works. It compiles, it gets the examples right, but there's some hidden test that failed. You'll never know why, even though it's probably not something surprising. So just make it explicit. A bunch of lights for each test, a description of what the test is, and there you go. 2) That solves little problems like "given some set of numbers, how do I find some ridiculous property of those numbers?" type questions. What you really need is something that tells you how people deal with complexity. I have a large problem, how do I solve it? There's always more than one right answer. Here you might be better off doing some kind of subjective voting. People who are looking for work might find it comforting that other people in their situation are judging them. Or gratifying to be able to judge other people's skills. Perhaps there's some incentive structure that cleverly aligns this.
- sytelus 12y agoWork-sample questions is not free from its own evils. They are expensive to device. At big companies, significant portion of candidates dump out their interview questions on Internet so that's a big challenge. Second, work-sample type of questions are also expensive to answer for candidates so you can probably just ask one question as part of whole interview and that's about it. So any conclusion is drawn from sample of 1 as opposed to sample of 5. Third, work-sample type of questions requires knowledge of engineering skills that candidate may or may not have developed well at point in time. At many companies, emphasis is how can we solve problem X from algorithms perspective and engineering is just something you learn on the way if you haven't already. Key is ability to figure out computer science part rather than language and tooling part. Overall, I think many companies blindly try to follow the model of Google, Facebook etc and fall on their face. If you expect your developer to write CRUD apps or do mostly plumbing work, there is no point in asking questions like find common ancestor in binary tree. At other companies, developers are expected to solve computational problems and engineering/plumbing/CRUD is small part of the solution pie. There it makes a LOT of sense to ask candidate to solve problems that requires deep computer science. Also asking as many of as possible so that your sample space is large and outcome has more confidence.
- tptacek 12y agoWe had three work-sample exercises at Matasano. All told, they took candidates mid-single-digit hours to finish (many geeked out, or golfed on them; we did our best to keep this from happening --- you can read about this on our hiring page: http://www.matasano.com/careers http://www.matasano.com/careers This sounds onerous, but it is less onerous than the normal dev hiring process, which involves an onsite interview that eats the whole day. We did on-site interviews, but they took just 2.5 hours to complete; candidates would be out at lunch. Essentially, we were shifting some of the time candidates would spend sweating in a conference room to their couch instead. There are bad work-sample tests. "Bad" comes in a variety of flavors. I think bad work sample tests will out-predict the median interview. But there's a lot of room to optimize and improve this process, sure.
- elptacek 12y ago
- smenko 12y agoGreat, if you want to hire people who "can code", but have no social skills. I am personally done working in environments where people think they "can code" but have not enough of a social skill to at least empathise with your point of view on naming conventions. Don't get me wrong, I can code, I'll write you anything you want, AND IT WILL WORK! and I'll still know there may be a better way but this is the way that works and I still think I don't have enough years to live to argue with social incompetents who think that fighting over tabs or spaces is a good idea and a pocket protector makes you sexy.
- korethr 12y agoReading this, I am filled with both hope and dread. The hope that as more companies figure this out, there will be more and more companies with interesting, challenging and rewarding work as opposed to companies that are only a paycheck, and as soul-sucking one at that. The dread is that I would never make the grade at the former kind of company.
- elptacek 12y agoEvery time I sit down to solve a problem with code, I feel dread. Every time I even think about strapping on my skates, I feel so much dread that my stomach hurts. I think that dread cannot be decoupled from hope. And that this is especially true for folks with higher developmental potentials.
- beefman 12y agoThere are a lot of 'hiring is broken' posts and this is a decent one, but I don't think I'm alone in feeling that none of them convincingly identify the problem, let alone solve it.[1] So I'll do that here. Hiring isn't broken because people use the wrong interview questions. It's broken because firms are engaging in a zero-sum battle for talent. But if what you're doing is worth a damn, it should be worth a damn when ordinary people do it. If what distinguishes your company is that you've managed to hire smarter people, maybe your company isn't so great.[2][3] The proper hiring criterion is pretty simple: do the people who will be working with the candidate like her? In Blink, Gladwell presented good evidence that formal interview process isn't any more objective than this. And when you think about it, what would "objective" even mean? There's also an issue with the labor market -- in particular, the difficulty of firing. You're much better off hiring liberally and firing candidates that don't work out. If all firms did this, the labor market would be more efficient and firing wouldn't be so bad. Apple is a good example. Their profit per employee is second only to Netflix among large tech companies, and this includes a significant fraction of retail personnel. Apple's profit per engineer is probably higher than Netflix's, and that's remarkable considering they're over twenty times bigger. Does Apple achieve this with some optimized interview format? Higher salaries? Onsite massage? No, they do it with better products (and a command and control governance model...). It means that the software industry isn't nearly as innovative as it thinks it is. This is actually so obvious it's the subject of popular jokes ('like airbnb for pet food' etc). It's also obvious when you consider the ratio of effort spent building tools to products. All the latest frameworks but... Where are the apps? Where are the apps that are even half as feature-rich as a typical Windows application from the '90s? Google Plus still can't keep which things I've +1'd straight. [1] It reminds me of the situation with essays claiming to explain why privacy is important even if you 'have nothing to hide'. See https://gist.github.com/clumma/414f0c9a8fedf9ba9db5 https://gist.github.com/clumma/414f0c9a8fedf9ba9db5 [2] https://twitter.com/clumma/status/139082596587012096 https://twitter.com/clumma/status/139082596587012096 [3] Or worse, that you've managed to deprive your competition of smart people. Google has done this -- hired people without having work for them to do, on the reasoning that something might come up and at least they won't be working for the competition.
- Iftheshoefits 12y ago
- spacemaus 12y agoRelevant: a startup called Interview Club[1] presented at the Launch Festival recently. Another startup called PreHire[2] won the hackathon (although their product is more for hiring customer service reps). [1] http://www.launchfestival.com/theinterviewclub http://www.launchfestival.com/theinterviewclub [2] http://www.launchfestival.com/prehire http://www.launchfestival.com/prehire
- bliti 12y agoI really hope companies read and learn from this. It is absolutely maddening how bad hiring currently is. As someone who has been on both sides of the hiring process it's just frustrating. If you are being interviewed you are basically put in front of a firing squad. If you are interviewing you are the one pulling the trigger. Maddening. I'm taking this and showing it to the ceo of the company I work for currently. Now, tptacek: What can I do to make this better? How can I contribute?
- egonschiele 12y agoI have had my share of bad interview questions too. The worst part is when the interviewer starts working on their laptop, like they are so bored of this question they can't think about it any longer. Yet it is now a part of my life for 45 minutes. At the end of every trivia question I'm tempted to ask: so is this typical of what I will be doing here? Will I be implementing rand(7) given rand(5) once a week? I've also had a bad experience with code challenges. These are given before the first phone interview. I'm happy to do a challenge taking < 1 hour. I once did a challenge that took me 3 hours. Got rejected because they didn't like my visual design (I was interviewing for a backend position). I've realized that code challenges require no investment on the company's part, so I don't want to do one that requires significant investment on my part.
- skybrian 12y agoPerhaps they were taking notes about the interview on their laptop?
- egonschiele 12y agoI've had people do that too, but that's noticeably different because they pay attention to you. It's obvious when people are working because you can see them context switch back to you.
- alxndr 12y ago> "At the end of every trivia question I'm tempted to ask: so is this typical of what I will be doing here? Will I be implementing rand(7) given rand(5) once a week?" That sounds like a great question to ask!
- stingraycharles 12y agoBecause being passive-agressive is a great way to get the job?
- ryandrake 12y ago
- avalaunch 12y agoThe same kind of process works for pure dev jobs. So, you’re a Rails shop? Take a Rails application you’ve actually built and deployed. Carve out some functional areas from the application: remove the search feature, or the customer order updater. Bundle up the app with all its assets, so that a single “vagrant up” gives a candidate an app that runs. Have them add back the feature you removed. How would you compensate for candidates that are great developers but are new to Rails, or whatever specific language/framework your company uses?
- MichaelGG 12y agoIt sounds like the candidate gets to do this at their own pace, from home. If so, a good dev shouldn't have trouble learning enough Ruby/Rails to figure this out.
- vehementi 12y agoHow then do you control for time pressure?
- kasey_junk 12y agoAre you asking how do you judge if someone performs under time pressure? If so, my first answer is don't. If that is a serious part of your real world work, you are doing it wrong. The less flippant answer is, you give them a deadline but it is both A) flexible in the face of their "real life" (ie if they tell you they are going on vacation next week deal with it) and B) it deals with realistic time horizons (ie a week, not 4 hours).
- vehementi 12y agoSorry my post was not complete enough - what I meant was that if you have two equally good candidates, but one happens to have experience in Rails, that person is going to perform the coding test more quickly than the other candidate i.e. get a more complete / correct solution in a given amount of time. So I hope in this situation there are no time limits?
- bsdpython 12y agoMy biggest issue with the work sample tests is the companies that waste your time with them. I've had 3 cases over the years where I was asked to work on something, I did it quickly, with quality and exceeded the specs and I was still turned down at that point (I confirmed that the code was good in at least 2 of the cases). As a senior technical person at this point in my career I simply have very little stomach for wasting time. Have at it young'ins.
- deleted 12y ago[deleted]
- geebee 12y agoSame for me. I did a 6-7 hour homework project. I have no idea how I did, the company didn't get back to me for a month, and then it was just a call from a recruiter. Never again.
- bsdpython 12y agoSorry to hear that hang in there
- kasey_junk 12y agoI completely agree with this, and would love to say that as the hiring side you should always give back the feedback to the candidates. I have actively pushed for that every time I've implemented work sample workflows. But it gets trumped by legal/HR. I think they are being overly cautious, but if you talk to HR folks much you will find they all have a crazy candidate lawsuit story.
- joesmo 12y agoYup. Just had this happen. I don't think I will be doing anymore coding tests.
- lsc 12y agoBut the all-day interviews have the same problem. I interviewed at some startup in SF... I mean, this was me looking for my "job with low standards" so I was going through recruiters, right? That's how you get a job with low standards. You put your resume on dice and include a phone number. Answer the phone. Wade through all the bullshit jobs that have nothing to do with your skill-set. Find jobs you could reasonably do, and say yes to the interviews. In my experience, this means that I'm competing, mostly, with people that don't have other options... people that can't get jobs through people they know, because they just aren't that good. For a lot of complex reasons, well, that's exactly what I needed at the time. I negotiate with the recruiter and $150K is within their range. I mean, I'm... not 100%, right? lowered expectations. So I should be looking for less. But I've never had any problem asking for too much; at worst, they laugh and offer me something lower, right? Anyhow, so I show up at this place and they grill me for the usual 8 hours. I mean, it's not too unpleasant; there's beer and food and everything you expect, and they seem to be really into me. The guy who is going to be my boss gives me his email and tells me to send him an email so he can get things started over the weekend. The business guy is, well, he's a business guy, but everyone else seems like people I could work with; people I'd get along with, and the skill level was right in there; even in my reduced capacity, I wouldn't be holding the team back. At the end of the interview, they ask what I'm looking for, money-wise. Now, like I said, I negotiated this with the recruiter, so I said $150K, but I noted I could probably be talked down some, and I mentioned a lower sum, one a friend who has worked for me for most of his career recently got. The mood immediately changed. We said our goodbyes, and I left. I did email the boss, but total radio silence. I never heard from these people again. I eventually got the recruiter to tell me they didn't want me, but... Personally, I'd be less pissed I spent 20 minutes on a coding problem for them at home and then didn't hear about it.
- yeukhon 12y agoI posted this before but someone decided to downvote, whatever, I think this is my version of hiring! 1) start with basic textbook questions like what is authenticity and authentication or XSS 2) catch what the interviewee said and build questions (e.g. I said something about private key so interviewer asked me about pro and cons of asymmetric and symmetric encryption). Oh yeah - know your shit because they are going to catch you! It's okay to say "I don't know." Being straightforward earns respect. My interviewers didn't penalize me much (well I just graduated from college...). 3) the next couple interviews again starts with introduction, then deep dive into what the team does, what the team is building at a high level, then proceed to ask me my interest. Here i would talk about my ideal projects, show them high level how I would go about implementing my idea, challenges I face (and also why I have to build one; are there any existing solution and are they not adequate). Take caution of your words - know the things you say aloud. Somewhere in those 4-6 interviews, add a programming sessions if you haven't done so (for me I skip that and went to onsite because of internal referral). I didn't get an offer probably because I didn't quite know what I really want to build. My idea was too generic and probably too "child play." It was a really intense and yet fun interview. This interview process allows interviewer and interviewee to see if they are a match or not quickly and pleasantly. I always look back at this interview and believe that the rejection is just and great for me and for the team.
- ExploitsforFun 12y agoI started drooling over the part about writing exploit code that doesn't involve doing anything illegal or taking data that doesn't belong to me. Pesky morals :)
- tptacek 12y agoSee: microcorruption.com.
- nerdy 12y agoAlmost makes me want to apply in an effort to abandon my current web development life entirely. Almost
- whiteboarder 12y agoOh man, everything about this post rings true. I myself am a pretty mediocre developer. But fortunately for me, I realized that tech interviews are complete and total BS. So what I did was spend a whole bunch of time studying whiteboard algo questions, and I became really good at interviewing. I've gotten a couple awesome jobs that I had no business getting (and usually get fired after about 6 month, because I'm a shitty dev, but whatever, on to the next job).
- patrickyeon 12y agoHi, I see you're a new user here and I hope you come back to check on this and it's not just a throwaway account. You saw the flaw in the system and studied so that you could exploit it improve your own situation. Some people may be mad about that, but I'm honestly not. I'm sad that you put effort into practicing interviewing and sound like you're not putting effort into practicing your actual job. You say "I'm a shitty dev", which is a fixed mindset type of thing to say. You identify yourself as your current capabilities, as opposed to a growth mindset where you may say "I'm working on improving my dev skills". It sounds like touchy-feely pop-psych, but think about it: you saw that getting better at interviews would land you better jobs, so you worked on interview skills, and you got better jobs. Now if you took your current job (and your next one) as a chance to work on your dev skills, maybe you won't get fired after 6 months. Maybe you'll grow in to absolutely deserving the awesome job, and then grow beyond that. Please think about it. Paraphrasing an Adventure Time character, being a shitty dev is the first step to being a kind of good dev. And we need more good devs and fewer firings.
- jiffyjeff 12y agoTerrific post, thanks! I am somewhat familiar with this hiring process thanks to tptacek's HN contributions (which incidentally led me to recommending a friend to Matasano). I would like to have a better feel for the context of the hiring problems involved here. I am wondering if the company's particular niche makes the hiring process unusually difficult. Or perhaps the company is in a growth phase with a corresponding shortage of employees? With such low turnover rates, how many additional employees are needed? Is it possible that the company is in a position to profitably hire every qualified applicant it receives? If so, the company is essentially losing money on every hire it doesn't make. In addition to a radically improved hiring process, what about offering remote work and/or extremely high starting salaries? Well-advertised high salaries and location flexibility might allow the company to out-compete many alternatives (e.g., Google, FB, Bootstrapped startups).
- elptacek 12y agoWhen I was at Matasano, there was always enough inbound work (sometimes more than enough) to keep the consultants utilized. So yes, a company with a full pipeline will lose money on every qualified hire it doesn't make. Perhaps this is not the case for other industries, but for infosec consulting, there seems to be more work than there are qualified workers. Matasano ran under its own steam and did not take funding. Offering extremely high starting salaries would have killed it in its early years. It was more cost effective to create a process for finding people who could actually do the work rather than rely on the 4-page CV candidates. The expensive ones. The net result was a win-win: you get bargain candidates and they get 4-page CVs.
- dpweb 12y agoSoftware development (please, not "coding").. It's a skill like other skills. It's not magic. Being able to practice medicine or law is a skill. Being able to architect a building is a skill. Being able to effectively develop software is a skill. You judge someone based on their skill level. You have to know what require('fs') is in Node, but you also have to understand truths like the concept of technical debt and so forth. Not everything is broken or "needs a new paradigm". Know what programming is. What high skill means. If you're any kind of leader - TEACH someone younger or less experienced. When you need to judge, for hiring or otherwise, measure against those truths about what good software development is.
- georgemcbay 12y agowat
- NHQ 12y agoheh we just made a day labor site for devs, in part cuz we are trying to circumvent bogus contracting and hiring practices. very beta. http://daylabor.modulha.us http://daylabor.modulha.us
- rrherr 12y agoAny examples of other companies with an awesome hiring process like this?
- logfromblammo 12y agoThank you, tptacek, for writing this post. You have identified several important problems with interviewing and laid out the fundamentals for a reasonable alternative to the standard software interview. It makes me sad that I will probably never see a single one of your suggestions implemented by any potential employer that might want to hire me, specifically. This is because there is one more major problem that you can't fix from where you sit. The people creating the interview protocols do not solve their problems like software professionals. They don't collect and analyze data rationally. They don't control the variables. It never occurs to them that there are a dozen flaws in their system that they just haven't discovered yet. They don't read HN. They don't realize that software developers can write the kind of software that runs on people, too.
- kasey_junk 12y ago[edit] This seemed inappropriate so I changed it The only reason I ever designed a hiring pipeline is because I was asked to be one of the people in the interview pool. Me (and others in the pool) decided we were doing a bad job and started iterating on the problem (much like we would for software). Management was thrilled with both the initiative and the results. That was years ago and working on hiring pipelines has become something I just do. I've never had an employer push back or make that hard. Quite the opposite, they are usually happy to have help. Try fixing the problem in your current job and see. You might be surprised.
- logfromblammo 12y agoI have never been hired by any employer that uses a hiring process that would reject me as an applicant. There's a bias embedded in there, I think--one that may also occur in other people. I also encounter the problems endemic in the hiring process far more often at other companies than I do in my own. The reason why I start interviewing elsewhere is often because my current company has stopped hiring (or started firing). I have occasionally tried giving those other companies feedback, but the response has always been, without a single exception, "We know what we're doing; don't tell me how to do my job." I have neither lever nor fulcrum for this problem. My frustration is not without reasonable cause.
- medell 12y ago"Despite training them not to, they routinely factored “confidence” into their result. Biases like that are pernicious!" pernicious - Having a harmful effect, especially in a gradual or subtle way. New word for me. :)
- joesmo 12y agoI think this method is good in theory but fails in practice. Recently, I applied to three different companies that asked me to do a code sample that took from about two to eight hours each. I generally don't work for free, but I had some spare time and decided why not? I did all three and turned them in. Out of three, one company liked my sample but in the interview that followed, decided that what I was asking was too much (it's what I'm currently making actually). One company never even looked at the code and after two weeks, emailed back saying they had forgotten and decided to hire someone else. The third company was pretty responsive and I'll be going in for an in-person. A seven hour in-person interview that will be more coding and such (at least on a laptop so it's better than most places). So one out of three. I think this is a huge problem with this approach and basically, most companies are not responsible enough to follow this approach. They drop the ball constantly and wonder why they can't find good engineers when they're literally everywhere. Put into the hands of a respectable company, I would definitely prefer this in the future rather than the typical white board bullshit and trivia slinging.
- mrkurt 12y agoWhat you've just described sounds like the exact opposite of a rigorous, objective, work sample based hiring approach. Crappy companies have crappy hiring processes. It sucks to deal with that. When we started doing work sample hiring, we put a process in place to handle many applicants, make sure we were responsive, and work people through the funnel as efficiently as we could. I don't think it's possible to take advantage of it any other way. We got complimented several times on how reasonable the process was (by people we turned down!). This both made me feel good, and caused me existential "the world is broken" sadness. It's not a high bar to be decent to people.
- joesmo 12y agoNo, it's not a high bar to be decent to people, yet many companies (and people) still aren't. The problem then is that prospective employees have no way of knowing that a company is decent versus another company that is not. Will I do another work sample for another company? I'm thinking likely not, but I haven't ruled out the possibility. Note though, that the company must offer something of great value for me to even consider it. In my case, that's remote work. I would never consider it for a non-remote company. Period. Still, a company could have a great, responsive process in place. But my experience, and that of others, is that it's not worth my time and effort to do work for free. Let's be clear: this is what is being asked of the candidates. It therefore starts to undermine any company using such a process even though I don't think it's an inherently bad process.
- ahomescu1 12y ago> Because here is the thing about interviews: they are incredibly hostile experiences for candidates. Minor nitpick from a grad student here: interviews are nowhere near as hostile as academic peer reviews for papers. In an interview, the interviewers aren't actively trying to find any holes in your logic (no matter how small). To me, technical interviews are a walk in the park compared to conference submissions.
- geobmx540 12y agoEr, I sure as hell am looking for flaws in your logic / code. I feel like I'm pretty nice about it. I don't use it to put people down. It allows me to ask follow up and leading questions to see if they too might spot it after a bit more thought.
- michaelochurch 12y agoI realize that I "should" like code samples, on the going ideology and considering the real problem of non-Fizzbuzzing programmers, but I don't find them useful and only do them when I'm unemployed (which hasn't been an issue for a few years). My problem with them, based on experience, is that they only provide downside. Used a language that the team doesn't like? Too bad. (Once I was told that I shouldn't have used Scala for a code sample, because the company-- a shitty Java Shop-- had recently fired someone who was a Scala advocate.) Didn't test your code using the style that the company implicitly expects? Points off. On the other hand, I've had the experience, in the past, of submitting samples that I was told were among the top 3 the company had ever seen... and still been offered a junior or mid-level role (this was in my mid-20s). So I don't see the upside. Why spend 12 hours of my own time on a code sample, only to get the same junior-level position I could have gotten anywhere else without homework? If a kick-ass code sample could have turned a junior-level engineer position at 0.01% into a Director-equivalent (I like being a full-time coder and don't need reports, but I generally prefer that management see me as an equal, so I can get my job done, because a 10x engineer disempowered is a -3x engineer) with above-market compensation and legitimate equity, then I'd say... yeah, these things can spot early talent that nothing else does. But, as far as my experience has led me to think, code samples just provide another reason to reject someone. ("This guy used the TestEase framework instead of EasyTest. What, does he think it's 2013??") I'm a good technical interviewer. I'll say it: I'm very good at it. The first thing I do is that I explain my methodology, rather than hiding what I'm after, so the candidate is at ease and not surprised by difficult questions. I tend to prefer a "rapid fire" approach to collect as much data as possible, so as soon as she's demonstrated that she knows what she's talking about, I'll move on to another question or topic... and I disclose that this isn't to be rude but so I can get the best read possible. I tell the candidate that, unless she is a 0.1% outlier, I will ask questions where she doesn't know the answers, and that's OK. It's not a percentage-based test where you need to get 80% right or it's a no-go, but it's a binary search on the competence spectrum. Generally I'll take something on her CV that I'm familiar with, research it a little bit, and start with a mid-level question that I'd expect 75% of the people I'd want to hire to get. (Statistically, you get your best results by adaptively asking questions at a level of difficulty where there's a 50% chance of success. But I aim for ~75% because most candidates aren't used to difficult interviews, and I don't want anyone to freak out.) That means that there's a 25% false-negative among good candidates, so if she doesn't know the answer (and it's OK to say "I don't know"; I'm in the top couple percent of my field and still have numerous blind spots) then I'll give her another question, also of moderate-high difficulty. (I don't start giving easy questions unless I've decided on a "No", and that's usually to put the candidate at ease, because I might still be wrong, and I want the next interviewer to get a clean read.) A good interviewing process provides multiple paths to success, not multiple ways to trip up and fail. I prefer to ask hard questions rather than easy ones because, as far as I'm concerned, 5 hard questions is multiple paths to success (if you get 1 or 2, you're solid; 3-4 means you're excellent) whereas 5 easy questions is multiple paths to failure. I know I come off as an asshole on Hacker News (because it's turning into this festering pit of pro-corporate Wrongness on the Internet, and someone has to do something about it) but I try to be as nice as possible during interviews. I want to be tough, intellectually, but I want the person at ease as much as possible (which is why I explain, ahead of time, that I've been a notoriously difficult interviewer at every company where I've worked) so I don't get a false negative. I generally give a series of hard questions (not all related to each other) that seem relevant to her experience. I might give four, and a candidate who can competently answer one of them, I would generally say is worth considering. If she gets 2-3 out of 4 hard questions right, then she's probably a good hire and I'll recommend her. I don't think hiring engineers is as hard as people make it out to be. I do like to see code, but I can usually tell the socially adept non-coders and charlatans from people who can actually program, even without looking at a line of code. I tend to prefer to hire for general intelligence than a specialized "hole" that someone is trying to fill, and the main question in my mind isn't, "Did she get all of my questions right?" but "Would I want to work with this person?" and "Would she add something to the team that's not already there?" I won't remember, 90 minutes later, whether she answered every single question right, and if she screwed up and said "Lasso" when she meant "elastic net", that's not terribly important... probably a mistake, and knowing the ideas is more important than the vocabulary. Besides, I care more about hiring the person who has the curiosity and drive to learn new things than hiring the one who already knows what we're "looking for", because the latter changes in most companies by the quarter, while a candidate's general ability doesn't. Bad hires tend to come from two sources. One is when non-technical people override the technical interviewers, either formally or informally. It's best to have an environment where people can speak freely and, preferably, independently (i.e. each interviewer is on the spot to give feedback before discussing the candidate with others, but also feels safe giving an honest read). The worst thing that can happen is when an executive says, "This person is amazing", before any feedback has been given. In many companies, that means that the decision is already made and the engineers are just there to ratify it. That produces a lot of underqualified hires. The second is stack-ranking, which encourages teams to keep an "insurance incompetent" on hand, so that the middling and top players can safely focus on work, knowing that they won't end up in the bottom pool. In companies with stack-ranking, however, I would generally avoid interviewing as much as possible. (In fact, this may be one reason why companies that use stack-ranking tend to decline.) If stack-ranking is in play, you need to focus solely on (1) your main project, (2) getting credit for what you do (the politics of performance, which is more important than performance itself), and (3) being perceived as a top performer without actually trying to outrun the bear (outrun the other guy, and he gets eaten; outrunning the bear is impossible). If you're in a stack-rank company, then interviewing is that class of work that's good for the company but won't improve your Perf score, so you should avoid doing it as much as possible.
- Ilovecroissants 12y agoI think of hiring someone as a process akin to testing experiments in my startup. There are so many variables when people try to work together. Hiring processes that mimic that strategy should be the best bet IMO. This article is a little low on practical advice although I realize it's focused on the issue.
- deleted 12y ago[deleted]
- picardo 12y agoAre you familiar with the psychological research about bias in decisionmaking?[0] It sounds like you have come to the same conclusion psychologists are coming to. Under stress, the human mind uses mental shortcuts, or heuristics, while making important decisions, and act irrationally. I don't often come across a post that takes on these biases so systematically, and translates them into one of the most important decisions a startup has to make, i.e. the hiring decision, and you even invent useful ways to mitigate their effects. Kudos for that! Every company should be doing this. Overcoming bias should be a top priority at all HR departments. ------- [0] http://en.wikipedia.org/wiki/Heuristics_in_judgment_and_decision-making http://en.wikipedia.org/wiki/Heuristics_in_judgment_and_deci...
- stcredzero 12y agoThe software developer job interview doesn’t work. Companies should stop relying on them. The savviest teams will outcompete their peers by devising alternative hiring schemes. I remember talking with a manager after an interview, and listening to him go on about how his company was empirical and data driven. Later, he mentions how this one interview activity was valuable because it resembles something in the job, and I asked him what his empirical data was, and he had to admit that he was caught out.
- rfrey 12y agoLovely read, thanks for the work. Do you think you've learned lessons that could be applied by individual contributors in large orgs who are called on to interview? If so, it'd be great if you added that to your blog backlog. :)