33 ms·

How to conduct a good programming interview

251 points by lihaoyi 9y ago

sillysaurus3 9y ago
While there are a variety of ways you can conduct a programming interview, the goals typically are the same: Will the candidate be able to write working code if they join the team? Can the candidate discuss code and problems with the people they'll be working with? Can the candidate reason about arbitrary problems and constraints? Is the candidate someone we would enjoy working with? All of these can be solved directly by asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. I'm doing it right now as a candidate and it's by far the best interview experience I've had. The others were all artificial. A lot of people have a problem with this model for one reason or another, but in terms of effectiveness it's probably at the top 1%.
- rckclmbr 9y ago
  This is 100% the "best" way to interview/hire a person. A small company who screens someone for basic competence, then does this is great. It does has its drawbacks though 1. Requires a significant time investment on your part 2. Isn't the most efficient -- if you have someone work a week on a project, and decide they're not a good fit, you just lost a week. 3. You can't really have multiple candidates for the same position coding at the same time. They would feel like they would be competing (and really, they would be) 4. For the reasons above, this would never work for a bigger company. By the time they are doing their week-long "trial", you should be pretty confident through screens anyway. Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?
  - morgante 9y ago
    > Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site? I have. I've done plenty of trial weeks where the candidate didn't work out. They got paid for their time, but we didn't extend them further offers.
- wjossey 9y ago
  I also agree. As someone who has recently co-founded a startup, this is something my co-founder and I have spent a lot of discussing and exploring. My personal experience with this style of interview was that it really helped me to personally demonstrate my abilities. I was interviewing at a startup in San Francisco in 2012 who wanted me to do a quick version of "battleship" as part of their hiring exercise. I proposed, mid interview, "how about I help you solve a performance problem with your current system. What's your biggest issue right now?" My interviewer actually showed me, and we ended up solving the problem together and then deploying out the change. The funny thing about that experience was that when we were done improving their platform, the VP of engineering was told what I did, and then had the interviewer go back and do the battleship exercise anyway. They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". I did far more poorly with the battleship exercise than I did helping them on their platform.
  - mousa 9y ago
    Can't really blame him, it might seem kind of arrogant to refuse to do the task they give and propose one yourself, although it is pretty impressive that you could do something useful on a codebase you haven't seen during the code interview.
  - sillysaurus3 9y ago
    It's unfortunate, isn't it? We spend all this time constructing artificial tests when the real thing is all that matters.
  - ThrowawayR2 9y ago
    > They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". If your employer were ever sued by a candidate for discrimination and it came to light that you had received different treatment than other candidates, some very uncomfortable questions regarding the reason would be asked in the courtroom. An unfortunate bit of fallout from the litigious environment we live in today, but it is what it is.
    - 9y ago
- morgante 9y ago
  I used to think that, but honestly after recently going through a job search I think it's untenable for most scenarios. As a candidate, I want to be able to interview with multiple companies simultaneously—if I had to do a trial week with each company, I'd never be able to get multiple parallel offers. There's also significant onboarding complexity which I hadn't factored in. One company had me do a trial period and it took me half a day to get productive—time which is a sunk cost for them (since they had to pay for it) and made me feel bad about my lack of productivity. In the same time period it took to onboard with their codebase/build tools, I could have easily completed a full interview loop with another company.
  - sillysaurus3 9y ago
    True enough, but it sounds like the above model isn't for you. It's for everyone who is talented but doesn't have the option to go interview with multiple companies simultaneously. Most devs are in that position. I'd much rather commit to a single company that I want to work at and do this process to put me ahead of the other candidates. But that's not a decision that everyone feels the same about.
    - deleted 9y ago
      [deleted]
    - readittwice 9y ago
      I get your sentiment. But this is quite problematic if you already have a job. It's not so easy to take a week or month off on short notice. What happens if you don't get your dream job? You can't just quit your current job.
  - TomK32 9y ago
    pair programming should cut the startup time.
    - morgante 9y ago
      It would also substantially increase the cost of this interviewing method.
- baddox 9y ago
  You can still simulate that within the more standard interview process. Pick a real bug or small task in your codebase and give that to every candidate. Obviously you'll want to complete the task for real in your codebase, so just keep an interview branch in your source control.
- gaius 9y ago
  They're compensated for their time, of course. Taking on paid work for another company, possibly a competitor, would put nearly anyone who did so in breach of their contract with their current employer, as well as complicating their tax situation. Not to mention the vacation time they would be burning on it. You will only be able to hire freelancers or people currently unemployed by this method. Do candidates set their rate, or is it just a token amount?
  - sillysaurus3 9y ago
    In my case, I offered to do it for free. (It wasn't their idea to give me a task on the actual codebase.) Almost nobody would be fine with that, but I want to work there. The other option was to hope that none of the other candidates seemed like a better choice. It's good to lock in an opportunity. Perhaps it's unworkable in the general case. That would be unfortunate; it's the most effective type of interview.
  - alkonaut 9y ago
    I think it would be prudent for the interviewer to just say that "of course we will compensate for your time because this takes a couple of hours". If you happen to have a contract that forbids you from doing that, just say "Can't take any comp, I'm happy to do it for free". I don't think the key is compensation, but at least not assuming the subject will work for free. If you are the interviewer in that situation, give them the t-shirt and mug.
    - ng12 9y ago
      Unless your company was exceptionally unique in some way I'd probably just pass instead. It's awfully one-sided: you're just sending me some code and I'm supposed to invest hours on it. At least with an onsite interview our priorities are aligned: neither of us want to waste more time than we have to.
      - alkonaut 9y ago
        Yeah we all have different taste in interviews, I'd much rather spend 2h working for free than writing som tree traversal or fib function on a whiteboard for ten minutes. Interviewing is hard.
- pcmaffey 9y ago
  Rather than working on a real issue, I'd recommend a made-up task/feature, modeled after something they might actually do. Just put it in a repo and share to all candidates. It shouldn't take more than half a day's work. Then the next round is all about the candidates presenting their work and explaining their thought process. It's simulated, but exemplary of real work. And easily repeatable. (Also no need for NDA, etc.) Just requires a thoughtful engineer to develop the task.
- vitus 9y ago
  > asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. At Google, we call these internships. Well, more or less -- there are restrictions regarding enrollment in a degree program, the work is for a fixed duration rather than for a fixed scope, etc. But the goal from the company's perspective is largely the same. I'd be surprised if internships at other companies in the sector don't serve the same purpose. (obligatory disclaimer about how these are my opinion's, not the company's, etc etc)
  - walshemj 9y ago
    doesn't work for those not still at university and there are strict rules in the USA about the role of an "internship" its meant to educate the intern not be a source of cheap labour
    - vitus 9y ago
      I never said anything about internships being a source of cheap labor. Please don't put words in my mouth. While yes, an intern typically makes less than a full-time engineer, you have to realize that the intern reduces the productivity of the host engineer, who has to spend time training the intern, looking over the intern's work, and so forth. The real value in the internship program is that it essentially creates a extended interview process after which we can evaluate the individual (given firsthand experience of how they operate in a work environment), and the individual can evaluate the company.
- Jabbles 9y ago
  This is an interesting candidate-first approach, but I have some concerns: This requires that candidates are experienced at the language or framework you use. Are you not prepared to take someone with different experience? What about graduates? What if their ability with one framework does not transfer to another that you end up assigning them to? Not all bugfixes are equal. If you haven't solved it yourself before, how do you know how hard it will be? If you give candidates questions that vary in difficulty, how do you compensate for that? Is the ability to dive into a codebase and fix (presumably) trivial bugs really what you should be optimizing for? Is it actually better than abstract algorithmic questions for determining if the candidate [can] reason about arbitrary problems and constraints? How would you overcome your unconscious biases when answering the question Is the candidate someone we would enjoy working with?
  - sillysaurus3 9y ago
    After I proposed the idea, they said they liked it and were willing to give it a try. They offered 1.) slight changes to the UX, 2.) add a payments feature, or 3.) some back-end bugs that need to be resolved. They asked which of them sounds most appealing as a test. This was perfect from my perspective, and after I signed the NDA I told them all of those sounded fine. I mentioned that fixing a bug is one of the best ways to get familiar with a new codebase, so I'd probably focus on those, but that I'd like to hear about the rest too. I can probably throw in some of the feature requests or UI tweaks. That's a bit more candidate-driven than might be expected, but it's really wonderful so far. I feel like I have autonomy, while at the same time they'll get the data about whether I'm effective in the field. The bias question is important, and I'm not quite sure the best way for us to collectively deal with it. All I know is that there seems to be far more bias in a traditional interview pipeline. One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?" Obviously the point of that is to see how you think through a problem. But such problems are useless. They're abstracted from what we as engineers do in the field, and the candidate knows that their entire fate rests on how well they answer that question. Yet those are the kind of metrics real companies are currently using.
    - 9y ago
- tptacek 9y ago
  Almost. Instead of doing this, take an old set of bugfixes from the tree, 1-3 of them, give the same bugfixes to every candidate, and come up with a standard rubric for judging the fixes. (If it's an open source project, synthesize 1-3 bugs, using the history of bugs on the project as a guidepost). This is a "work sample test", and it's the gold standard for judging technical qualification. Firms should rely on them to the exclusion of subjective interviews. Within reason, to a first approximation, every candidate gets the same problems (you may iterate over time). You can judge candidates apples-apples. You don't need to pay candidates for their time, because they're not doing work for you; you already fixed these bugs.
  - sillysaurus3 9y ago
    Yeah, I'm a big fan of your approach. Unfortunately it seems quite difficult to get anyone to do it. Most companies have interview pipelines that are well-established and immutable, so only new companies are inclined to try it. And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria. It was a neat hack as a candidate to propose to the interviewing company that they do essentially the same thing as a work-sample test and hear them say yes. That only works if you propose they let you try your hand at doing their actual work. I wonder if it can be generalized?
    - tptacek 9y ago
      I think asking candidates to do "real" work is bad for a bunch of reasons: * It's intrinsically exploitative, even when compensated, because real work done under similar terms is paid far more highly than prorated salary (it also does weird things to salary negotiation, by extracting an up-front concession on rate). * It requires different candidates to work on different problems, which means you're not getting a reliable record of how candidates do on the interview so you can tune it over time. * Because the best candidates will generally refuse to work under these terms, most firms that do paid up-front work have a fast-path for elite candidates, which is an obvious process failure.
      - deleted 9y ago
- johnvega 9y ago
  I wonder where companies or hiring managers can post their job openings that supports this hiring process sometimes called Trial Employment or Job Audition for Software Engineers. Looks like a great way to get the best hire and also prevent a bad hire, although it can only work for a very limited number of companies and candidates.
thiscatis2 9y ago
So reading over some of the questions posted on technical programming interviews I must say that I would definitely fail such an interview without studying for it for a month first. Having recently discussed job interviews of ex-colleagues that went to EBay, Facebook, Google, etc.. I wouldn't even pass the whiteboard test. However I'm shipping scaleable API's used by web and native apps, using Continuous Integration and Deployment on an AWS infrastructure that I set up myself. Furthermore I'm responsible for hiring developers that I work with on agency projects and most important of all getting projects delivered within time, scope and budget AND getting paid for it. Yet I would fail most technical interviews. I've been doing this for about 10 years and never ran into any major problems. I've worked both as a freelancer and in-house permanent dev and never had any complaints about my work. (Feedback and improvements yes, big failures never). Yet I would fail most technical interviews. I talk directly to clients and vendors, I sell projects, I take over projects, I end projects and I've created 2 startups, one of which I sold but I would definitely fail a purely technical interview. I completely understand the need for these technical interviews and weed out obvious bad programmers and get experts on board but sometimes from a business point of view it just doesn't make sense. At a previous company I worked for the team almost crashed & burned because of the TL's focus purely on technical skills. No code was being shipped but hours of meetings and Slack conversations and discussions on PR's for no good reason about 0.01% improvement gains. I know I'm not the best developer out there and never will be. Often I'm in awe of the technical skills of people out there but I rather have a less skilled dev with great "people & shipping" skills as I call it than some of the people that would pass all these tests but that I couldn't put in front of a client or even let them email a vendor.
- perfmode 9y ago
  Many large organizations want their engineers to have the algorithmic ability and their technical project managers and engineering managers to have the strengths you describe. Technical leads who possess both bridge the gap.
  - deleted 9y ago
    [deleted]
- 9y ago
kevmo314 9y ago
> If you ask the candidate the define the requirements for the task you're going to give them, you can't be surprised if the candidate imagines a use case with entirely different requirements from what you expected! This section really resonated with me. I've had a few interviews where I came up with a different answer than what the interviewer expected and this resulted in a lot of tension, as the interviewer asserted they were correct with zero exception. This often led to a lot of dead-end scenarios and the interviewer trying to get me to see an alternate solution without telling me explicitly. I once had an interviewer tell me that O(k) was constant if k was a config parameter (I disagreed because in practice, k would need to be tuned as a function of the size of the data, and it would be like saying O(n) is constant if you just fix n = 1M), and an interviewer tell me that you could solve an optimization problem without an objective function (in actuality, he had an objective function, he just thought it was obvious and didn't want to specify it). In both cases, I understood what they were trying to get at after they explained their solutions, but in the process it was incredibly frustrating as they seemed to just disregard my answer all together.
- baddox 9y ago
  I love interview horror stories! In my most recent job search, which was early this year, I debated with one interviewer whether checking for key existence in a hash was a constant time operation. I insisted it was, and the interviewer insisted otherwise with the same amount of conviction. That was just a small portion of an all-day onsite interview. After leaving, I was told they were looking for a more senior candidate. I was not upset.
  - jerryr 9y ago
    Wouldn't that argument be easily settled by discussing the implementation of a hash table?
    - baddox 9y ago
      I discussed it briefly. My interviewer was unpersuaded, and the other interviewer in the room (who wasn't performing this interview) suggested we just move on. I believe the unconvinced interviewer was simply a very junior developer and an even more junior interviewer. He seemed quite nervous.
sidlls 9y ago
Some points are quite good, and in particular the author calling out the rubbish of asking "neat little algorithmic" questions (e.g. asking things like "implement mergesort"). However one thing I'd point out is that this guide still falls far short of being a good guide for finding engineers. Particularly noteworthy, in my opinion, is that the measurement criteria are still applicable only to the coding question itself. For example, "how well the candidate manages time" still applies only to this limited 40-60 minute question. It doesn't really help evaluate how the candidate might consider prioritization of tasks (implementation or otherwise) in a broader, bigger sense.
bostik 9y ago
A year ago I wrote a (shorter) post on the same topic: https://smarketshq.com/notes-on-interviewing-engineers-a4fa4383968a https://smarketshq.com/notes-on-interviewing-engineers-a4fa4... A reasonable question should never require the candidate to know "just a single trick". Questions that require to combine approaches are far better - they can actually measure if the interviewee can reason about the problem and identify the points for trade-offs. Btw: if you're interviewing more senior candidates, a code review question is often a good idea. It turns the situation around, from "how do you build this?" to "how would you fix this?"
the_mint 9y ago
All his example interview questions are stuff that were actually used at Dropbox!
- kevmo314 9y ago
  His resume (http://www.lihaoyi.com/Resume/ http://www.lihaoyi.com/Resume/) suggests he worked at Dropbox for quite a while. :)
HugoDaniel 9y ago
Here is how to do a good coding interview if you are a recruiter: 1. Do your homework: know what projects drive the interviewee (what kind of things she likes to code, what recent projects has she been involved in), make sure they are a match for your company projects and goals 2. Ask the interviewee for a git repository and take a careful holistic look at it (commit messages and code, timespan, branches, merges etc...) 3. Go for a 2h lunch with the interviewee in a relaxed environment to check speak communication skills (talk about whatever you think is interesting, if you can't have or don't know how to have 2h lunches you shouldn't be interviewing, check your life first and learn how to enjoy it before doing interviews). 4. Stop asking programmers to write code in 40 minutes. Learn how to respect before expecting respect in return, if you didn't take the time to read through their code repositories then don't ask for a 40 minute code problems.
- mwcampbell 9y ago
  I think you missed the part where the OP said not all candidates are willing or able to work on open source projects.
  - HugoDaniel 9y ago
    Well the .2 "Ask the interviewee for a git repository" can be a private repository. It is a very hard problem to solve if they have no visible code to show but it does happen in other areas. Imagine asking for a management candidate for the private e-mails he shared in certain tricky situations of previous jobs. Or even public e-mails. Or even to take 40 minutes to write in a whiteboard a team statement for a certain conflict scenario. Trust is entailed when recruiting in most areas. But in programming not as much.
    - jonathankoren 9y ago
      > Well the .2 "Ask the interviewee for a git repository" can be a private repository. If someone shares a private git repo, they should not be hired, and in fact should be fired from their current job, and possibly sued. (Think Anthony Levandowski sharing Google lidar designs as an example of his work with Uber.) This isn't promoting trust. This is a violation of trust.
Goopplesoft 9y ago
> 5 minutes of resume discussion > 45 minutes of coding As anyone else found that a good resume discussion is often much better than the coding component? I've concluded that implementation difficulties, decisions made (tradeoffs, technologies, etc.) and the collaborative environment around projects, are much better signals than the code part of my interviews. If you've dealt with a broad range of tech and can ask the right questions, it's relatively easy to get a strong signal from a candidate by having a deep discussion on their work on prior projects. Both are needed and very useful.
- majormajor 9y ago
  Yeah. If they can't tell me about what they've built before, and aren't a less-than-one-year-away-from-school new grad, I'm going to be wondering how they deal with real problems regardless of whatever they can do algorithm-wise. If someone can talk about what approach they took, what other approaches they considered, what made it hard, what made it easy, how often the requirements changed, etc, then I'd rather ask them questions about that then spend the whole time in code. Just do some small spot checks to make sure they aren't taking credit for the work of others.
  - Goopplesoft 9y ago
    > Just do some small spot checks to make sure they aren't taking credit for the work of others. Agreed, asking tough technical questions on this front gives a really solid indicator of a 'bullshitter'. For example, I often ask people to diagram a prior project architecture they worked on, and then ask them very fine details of how things work and try to challenge them with alternative approaches to see how they react. I get to experience working with them during the interview through which I learn if they're smart and capable.
- Clubber 9y ago
  Absolutely. If a person can't tell good from bad with that, they shouldn't be interviewing.
Xoros 9y ago
Those white board stuff reminds me when I was a CS student in the beginning of the 90's in France. We had our exams on paper. And believe me, writing LISP on paper, knowing if you forgot one parenthesis you got 0 point ("if I type it on a computer, it fails !"), was pretty and unnecessary stressful. So I relate with candidate facing that. Dude, on your day to day work, you WILL have a computer to check those ! How can you decently check those skill on a white board ? I totally agree with what's stated in this post.
- morgante 9y ago
  No good interviewer is grading whiteboard coding on whether you forgot a parenthesis or semicolon.
- e12e 9y ago
  If you think writing lisp by hand is bad, try implementing a linked list interface in java on paper... shudder. (in fairness, we were sanely graded, minor syntax errors didn't subtract terribly from the grade...).
- metaobject 9y ago
  Writing code for tests, on paper, has been a thing in CS for quite awhile. While writing lisp for tests, I used to sometimes write the code with no parens (using indentation to denote nesting). Then I'd go back and add parens where appropriate. defun add-pos-only v1 v2 let sum 0 if and > v1 0 > v2 0 + v1 v2 nil
XCSme 9y ago
Interviews suck for both parties, a world without interviews would be a better world. Why can't we just hire someone based on his experience and projects he previously worked on? You can have a first month as a "trial" where both parties can cancel the contract. Why interviews suck is because they are very, very biased and subjective and also not a good way to estimate the actual performance of a candidate. Say no to interviews! If you like what someone did, hire him. You can't get to know a person after 1 or even 10 interviews, because the interview environment is totally different from the work environment. PS: I have been interviewed by 14 different people at Google. I didn't get the job. The overall conclusion was that some interviewers loved me, my skill and my work, others didn't like me from the start or simply didn't care about me. Some of them had a good day so they gave a good "review" even though I made mistakes, some of them had a really bad day (one fell of the chair) and gave me a really bad review even though, related to the actual interview questions, I did really well. (14 interviews = applied twice, 2 x (5 on-site + 1 online), and they also wanted 2 extra interviews to take a decision)
- CJefferson 9y ago
  I'm sure most businesses would love that situation, but for employees it would require you quit your current job, but have a good chance of being fired after a month at your new job. I imagine young employees would be happy (I did a bunch of short-term contracts myself), but once you have a mortgage and kids, it's nice to be a little more certain about your new job, before quitting your old one.
  - nolemurs 9y ago
    > I imagine young employees would be happy (I did a bunch of short-term contracts myself), but once you have a mortgage and kids, it's nice to be a little more certain about your new job, before quitting your old one. And just to make this really explicit - this means that a company with this hiring practice would have a very hard time hiring experienced competent engineers. If I were looking for a job, I'd have lots of options, and I would not waste time on a company that demanded this.
- njovin 9y ago
zerr 9y ago
Small take-home paid project.
amriksohata 9y ago
For me I have interviewed many people with a technical test, that was the easiest part i.e determining their competency. The hardest part is figuring out what their work ethic is, can they pull their own weight and will they cause friction in a team, sadly you only find that out often AFTER hiring them.
mwcampbell 9y ago
Here's another disadvantage of whiteboard coding: it's inaccessible to blind people, and impractical (even more than usual) for people with only a little vision (like me). Such candidates will need to do the coding on a computer, so one might as well make that the default, as the OP recommends, and eliminate one variable and a possible source of bias. Of course, once a blind or low-vision programmer stays at a given company long enough, they'll probably be on the other side of the interview at some point. So for consistency among a team of interviewers, I imagine it would be best if all of them have their candidates do coding problems on a computer, not just one of them. I'm curious about why whiteboard coding is still a thing in the first place. Is it a matter of convenience or cost? If so, surely that equation has changed now that we have ubiquitous, cheap, portable computers. Or do some interviewers really think that it's good to deny a candidate access to a real computer while solving a coding problem during the interview?
- mahyarm 9y ago
  These places use the computer for interviews already for the visually impaired. It's just a name for a type of interview. I even give people the option to use the computer if they are more comfortable with that or a whiteboard. It can be 50:50 in who choses what.
spraak 9y ago
What are some good resources for developing good algorithm skills for a non CS grad? Examples in JS, Python or Go preferred.
- SmirkingRevenge 9y ago
  There's lots of stuff online. Check out hackerrank.com for example. There are other sites like that out there. Its becoming a real cottage industry, this whole IT interview prep thing.
tptacek 9y ago
It's good to think carefully about how you interview candidates. Most of the ideas in this post are good. You should definitely let candidates type answers on computers. You should definitely work to put them at ease, in an environment that approximates their actual working environment. But really, this is window dressing. The problem is that technical interviews just don't work reliably. We have better ways to qualify candidates. Rather than trying to read subjective, noisy signals from in-person interviews, give candidates a small battery of work-sample tests. These can be bug-fixes, small features, diagnostic challenges, or even specification writing assignments. A properly designed work-sample challenge has a rubric: a set of instructions for scoring a result. Ideally, the rubric is straightforward enough for someone who hasn't met the candidate to quickly score their result. "Runs correctly". "Passes a battery of unit tests". "Includes appropriate documentation". Whatever is important for your team. Work to give every candidate the same set of challenges, so you can track results over time. You'll find that after refining and iterating for a few months, you'll rapidly build confidence in the tests, to the point where in-person interviews become superfluous for technical qualification. Working on technical qualifiers is stressful. Most candidates will perform more poorly on them than they would on normal work. You should build that into your rubric (though, really, data from candidates will ultimately determine how strict the rubric becomes). Understanding how stressful qualifying work is, you should consider not requiring candidates to work on problems on-site. Instead, administer them online. Suggest the amount of time you expect the problems to take to finish, and offset that amount of time from the amount you demand on-site. When you must interview, you should work hard to standardize the interview. Interviewers hate this. Everyone has their favorite questions. People want interviews to be conversational and adaptive. Interviewing isn't supposed to be fun for the interviewer; it's supposed to be reliable.
- michaelchisari 9y ago
  This is my approach for evaluating new candidates as a first step. For front-end, for instance, I start with a codepen that has the basic skeleton that includes the libraries we use internally, has a few html elements styled and placed, and the required javascript boilerplate. Then I provide 10 tasks that are written much in the way the company would put tasks in asana or git or jira. Each task builds on the previous and gets more complex. In the boilerplate they start with, there is one obvious error and one subtle one. Each step, I encourage them to write notes on how they solved it, and any questions or concerns they might have. I make myself available via chat or email if they have any clarifying questions. They then can do it in their own time, at home. They can google, hit up documentation, stackoverflow, etc., because that's exactly how developers work. Nobody's over their shoulder, there's no arbitrary limitations, just build us some good UI code. It's true that I've been amazed at how many candidates were stopped in their tracks at the simplest things that are well within the domain of their resume. But when I get a candidate that gets through them, hits the requirements and has clean code as a result, I've hardly ever seen them fail at a later step where we might evaluate more complex problems like working around GC slowdowns, obscure browser compatibility, first meaningful paint performance, etc. And most of all, I feel confident that I haven't tried to "trick" anyone by having them do long-solved algorithms offline on a white board.
cbanek 9y ago
> Hopefully in your realistic setting, the challenge comes from the difficult tasks you are given to solve, not some jerk that's sniping at you from the sidelines. I wish I could say that's true. If you're getting sniped from the sidelines in your interview, it's probably a part of the culture where you are interviewing. So it's quite likely to be a non trivial part of the problem you're solving.
jonathankoren 9y ago
I have to disagree with this trend of having people write code on a computer instead of a whiteboard. Sure, it mimics the real world, but in practice you spend a bunch of time dealing with crap about syntax errors, and argument order mismatch instead of dealing with the actual underlying algorithm. Even worse, you may end up with a candidate that realizes that there's probably a library that would be really useful for this, but then has to spend time googling for the name of the library, then installing it, and then desperately skimming for a quick start in documentation because the question isn't, "Can you work google and run pip install?", but rather some other task. So now you've probably lost 15 minutes just dealing with that shit. If you're going to do this, it simply doesn't work in a 40 minute setup without some sort of provided scaffolding and test cases.
- jpatokal 9y ago
  Is doing interviews in an IDE and expecting fully functional code actually a thing? My company uses a simple text editor: perfection is not expected, and it's still way better than a whiteboard.
  - jonathankoren 9y ago
    I can confirm that this is a thing at multiple companies. Instead of of testing the actual algorithm, you'll encounter such blasts from the past like: "Network Socket Debugging", "How The Fuck Does Java Read From STDIN?", and everyone's favorite, "Why The Fuck Did You Give Me Text Data In CSV Format?" I have seen this go well exactly once. And they provided an environment with a working (albeit do nothing) scaffold along with test data. Also it was like a three hours in a room.
pnw_hazor 9y ago
There is no reason to haze potential employees. Just fire them if they don't work out. You set expections by telling them up front they are on probation and that they will be fired if they fail to live up to their resume. Some of the worst programmers I ever worked with could pass hazing interviews like a champ. Whereas some of the best would refuse to step foot in companies with a reputation for hazing their prospects.
- Clubber 9y ago
  That's a funny term to apply to this type of interview process, but it fits. Kudos. Agree on the invalidity of technical interviews. We hired a guy who correctly answered every question I asked, but he could never ship a product. He always got distracted by silly things. He was great at solving tough little problems, but never could grasp the big picture.
Jemaclus 9y ago
I have a lot of thoughts about this particular topic, probably ideas that are unpopular and definitely go against the grain, at least in the way the Bay Area operates. In my opinion, if you bring a candidate on-site for a 4+ hour interview and you ask them to write code, you've failed to properly vet the top of your recruitment funnel. As a software engineer, writing code is the BARE MINIMUM of your job. Imagine hiring a construction worker by merely asking them to hammer a nail for 4 hours. You should have a thorough understanding, through various means such as phone screens, code samples, portfolio work, and simple tech challenge, to determine whether they can write code before you even bring them on-site. Once they're on-site, you should quickly verify that they can actually do what you previously learned that they can do. I'm talking 15-20 minutes worth of time. The remaining time should be 100% on the "soft skills" and other things mentioned, including but not limited to my big criteria: "Can you quickly get up to speed and learn the things you don't already know?" and "Do I want to sit next to you for the next 6 months of my work-life?" If your on-site interview includes 45 minutes of coding (much less multiple sets of 45 minutes of coding!), you're doing it wrong. Very wrong. Keep in mind you've asked your candidate to effectively take a day off work from their current job. They're calling in sick, taking a vacation day, or somehow lying to their boss and taking a risk of not being in their office when they're in yours for the interview. If they're smart, they've lined up multiple interviews with multiple companies, because putting all their eggs in your basket is just a terrible idea. Imagine having to take four days off to interview with four companies. And they probably aren't four days in a row, so you can't just pretend to have the bubonic plague or something. You're asking them for a huge commitment of their time. And if you're willing to kick out a candidate because they didn't know how to invert a binary tree, then you should have found that out before they even walked in the door and saved everyone a whole bunch of time. IMO, an interview should be a max of 4 hours. Any more than that and the candidate is simply repeating themselves over and over to different people, potentially solving the same tech problems (that should have been evaluated beforehand), and generally making it a huge waste of time. My two cents. Happy to debate these ideas with anyone interested.
balls187 9y ago
Being a better interviewer is important. As stated, some types of questions aren't good judge of skills, like implement mergesort. If you can implement mergesort, you probably studied up for your interview. You're not testing anything, other than did the candidate care enough to study. Interviewing should test the candidates ability to write code that is: 1. Correct (solves the problem) 2. Efficient (solves the problem without unnecessarily wasting resources) 3. Understandable (goes hand in hand with maintainable) 4. Maintainable (can be modified by another developer, or even yourself after a few months) Coding on a computer can lend itself to all of these (no brainer there), but you lose a key aspect of whiteboarding that I think is really important--how a candidate approaches solving a problem. Knowing how to implement merge sort, or memorizing every possible linked-list question isn't all that valuable--but having a concrete approach to tackling problems is, and I want to understand that process from my candidates. I want to know the design tradeoffs you make, how you think about optimization, how do you reason about constraints as you learn about them. It seems that explaining your process while coding on a computer would be difficult to do without practice--and if you're going to practice, why not instead just practice whiteboard coding, which is how most tech companies interview.
- rimliu 9y ago
  So do you want someone who is good at explaining how they are looking for the solutions, or someone who is good at finding the solution? For me personally approaching a problem never involves a whiteboard.
  - balls187 9y ago
    I don't think I've ever been at an engineering company that didn't have whiteboards, at the very least in conference rooms--I infer from that that some portion of employees find them valuable. Clearly someone who is good at solving problems is preferred. However, between someone who is good at solving problems, and someone who is good and can explain/demonstrate their process is more desirable.
ilaksh 9y ago
Mostly makes sense but your ability to write a parser is still going to depend on how many parsers you wrote or when you wrote them or whether you are able to use a framework/tool to do it.

251 points by lihaoyi 9y ago

While there are a variety of ways you can conduct a programming interview, the goals typically are the same: Will the candidate be able to write working code if they join the team? Can the candidate discuss code and problems with the people they'll be working with? Can the candidate reason about arbitrary problems and constraints? Is the candidate someone we would enjoy working with? All of these can be solved directly by asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. I'm doing it right now as a candidate and it's by far the best interview experience I've had. The others were all artificial. A lot of people have a problem with this model for one reason or another, but in terms of effectiveness it's probably at the top 1%.

rckclmbr 9y ago
This is 100% the "best" way to interview/hire a person. A small company who screens someone for basic competence, then does this is great. It does has its drawbacks though 1. Requires a significant time investment on your part 2. Isn't the most efficient -- if you have someone work a week on a project, and decide they're not a good fit, you just lost a week. 3. You can't really have multiple candidates for the same position coding at the same time. They would feel like they would be competing (and really, they would be) 4. For the reasons above, this would never work for a bigger company. By the time they are doing their week-long "trial", you should be pretty confident through screens anyway. Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?
- morgante 9y ago
  > Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site? I have. I've done plenty of trial weeks where the candidate didn't work out. They got paid for their time, but we didn't extend them further offers.
wjossey 9y ago
I also agree. As someone who has recently co-founded a startup, this is something my co-founder and I have spent a lot of discussing and exploring. My personal experience with this style of interview was that it really helped me to personally demonstrate my abilities. I was interviewing at a startup in San Francisco in 2012 who wanted me to do a quick version of "battleship" as part of their hiring exercise. I proposed, mid interview, "how about I help you solve a performance problem with your current system. What's your biggest issue right now?" My interviewer actually showed me, and we ended up solving the problem together and then deploying out the change. The funny thing about that experience was that when we were done improving their platform, the VP of engineering was told what I did, and then had the interviewer go back and do the battleship exercise anyway. They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". I did far more poorly with the battleship exercise than I did helping them on their platform.
- mousa 9y ago
  Can't really blame him, it might seem kind of arrogant to refuse to do the task they give and propose one yourself, although it is pretty impressive that you could do something useful on a codebase you haven't seen during the code interview.
- sillysaurus3 9y ago
  It's unfortunate, isn't it? We spend all this time constructing artificial tests when the real thing is all that matters.
- ThrowawayR2 9y ago
  > They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". If your employer were ever sued by a candidate for discrimination and it came to light that you had received different treatment than other candidates, some very uncomfortable questions regarding the reason would be asked in the courtroom. An unfortunate bit of fallout from the litigious environment we live in today, but it is what it is.
  - 9y ago
morgante 9y ago
I used to think that, but honestly after recently going through a job search I think it's untenable for most scenarios. As a candidate, I want to be able to interview with multiple companies simultaneously—if I had to do a trial week with each company, I'd never be able to get multiple parallel offers. There's also significant onboarding complexity which I hadn't factored in. One company had me do a trial period and it took me half a day to get productive—time which is a sunk cost for them (since they had to pay for it) and made me feel bad about my lack of productivity. In the same time period it took to onboard with their codebase/build tools, I could have easily completed a full interview loop with another company.
- sillysaurus3 9y ago
  True enough, but it sounds like the above model isn't for you. It's for everyone who is talented but doesn't have the option to go interview with multiple companies simultaneously. Most devs are in that position. I'd much rather commit to a single company that I want to work at and do this process to put me ahead of the other candidates. But that's not a decision that everyone feels the same about.
  - deleted 9y ago
    [deleted]
  - readittwice 9y ago
    I get your sentiment. But this is quite problematic if you already have a job. It's not so easy to take a week or month off on short notice. What happens if you don't get your dream job? You can't just quit your current job.
- TomK32 9y ago
  pair programming should cut the startup time.
  - morgante 9y ago
    It would also substantially increase the cost of this interviewing method.
baddox 9y ago
You can still simulate that within the more standard interview process. Pick a real bug or small task in your codebase and give that to every candidate. Obviously you'll want to complete the task for real in your codebase, so just keep an interview branch in your source control.
gaius 9y ago
They're compensated for their time, of course. Taking on paid work for another company, possibly a competitor, would put nearly anyone who did so in breach of their contract with their current employer, as well as complicating their tax situation. Not to mention the vacation time they would be burning on it. You will only be able to hire freelancers or people currently unemployed by this method. Do candidates set their rate, or is it just a token amount?
- sillysaurus3 9y ago
  In my case, I offered to do it for free. (It wasn't their idea to give me a task on the actual codebase.) Almost nobody would be fine with that, but I want to work there. The other option was to hope that none of the other candidates seemed like a better choice. It's good to lock in an opportunity. Perhaps it's unworkable in the general case. That would be unfortunate; it's the most effective type of interview.
- alkonaut 9y ago
  I think it would be prudent for the interviewer to just say that "of course we will compensate for your time because this takes a couple of hours". If you happen to have a contract that forbids you from doing that, just say "Can't take any comp, I'm happy to do it for free". I don't think the key is compensation, but at least not assuming the subject will work for free. If you are the interviewer in that situation, give them the t-shirt and mug.
  - ng12 9y ago
    Unless your company was exceptionally unique in some way I'd probably just pass instead. It's awfully one-sided: you're just sending me some code and I'm supposed to invest hours on it. At least with an onsite interview our priorities are aligned: neither of us want to waste more time than we have to.
    - alkonaut 9y ago
      Yeah we all have different taste in interviews, I'd much rather spend 2h working for free than writing som tree traversal or fib function on a whiteboard for ten minutes. Interviewing is hard.
pcmaffey 9y ago
Rather than working on a real issue, I'd recommend a made-up task/feature, modeled after something they might actually do. Just put it in a repo and share to all candidates. It shouldn't take more than half a day's work. Then the next round is all about the candidates presenting their work and explaining their thought process. It's simulated, but exemplary of real work. And easily repeatable. (Also no need for NDA, etc.) Just requires a thoughtful engineer to develop the task.
vitus 9y ago
> asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course. At Google, we call these internships. Well, more or less -- there are restrictions regarding enrollment in a degree program, the work is for a fixed duration rather than for a fixed scope, etc. But the goal from the company's perspective is largely the same. I'd be surprised if internships at other companies in the sector don't serve the same purpose. (obligatory disclaimer about how these are my opinion's, not the company's, etc etc)
- walshemj 9y ago
  doesn't work for those not still at university and there are strict rules in the USA about the role of an "internship" its meant to educate the intern not be a source of cheap labour
  - vitus 9y ago
    I never said anything about internships being a source of cheap labor. Please don't put words in my mouth. While yes, an intern typically makes less than a full-time engineer, you have to realize that the intern reduces the productivity of the host engineer, who has to spend time training the intern, looking over the intern's work, and so forth. The real value in the internship program is that it essentially creates a extended interview process after which we can evaluate the individual (given firsthand experience of how they operate in a work environment), and the individual can evaluate the company.
Jabbles 9y ago
This is an interesting candidate-first approach, but I have some concerns: This requires that candidates are experienced at the language or framework you use. Are you not prepared to take someone with different experience? What about graduates? What if their ability with one framework does not transfer to another that you end up assigning them to? Not all bugfixes are equal. If you haven't solved it yourself before, how do you know how hard it will be? If you give candidates questions that vary in difficulty, how do you compensate for that? Is the ability to dive into a codebase and fix (presumably) trivial bugs really what you should be optimizing for? Is it actually better than abstract algorithmic questions for determining if the candidate [can] reason about arbitrary problems and constraints? How would you overcome your unconscious biases when answering the question Is the candidate someone we would enjoy working with?
- sillysaurus3 9y ago
  After I proposed the idea, they said they liked it and were willing to give it a try. They offered 1.) slight changes to the UX, 2.) add a payments feature, or 3.) some back-end bugs that need to be resolved. They asked which of them sounds most appealing as a test. This was perfect from my perspective, and after I signed the NDA I told them all of those sounded fine. I mentioned that fixing a bug is one of the best ways to get familiar with a new codebase, so I'd probably focus on those, but that I'd like to hear about the rest too. I can probably throw in some of the feature requests or UI tweaks. That's a bit more candidate-driven than might be expected, but it's really wonderful so far. I feel like I have autonomy, while at the same time they'll get the data about whether I'm effective in the field. The bias question is important, and I'm not quite sure the best way for us to collectively deal with it. All I know is that there seems to be far more bias in a traditional interview pipeline. One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?" Obviously the point of that is to see how you think through a problem. But such problems are useless. They're abstracted from what we as engineers do in the field, and the candidate knows that their entire fate rests on how well they answer that question. Yet those are the kind of metrics real companies are currently using.
  - 9y ago
tptacek 9y ago
Almost. Instead of doing this, take an old set of bugfixes from the tree, 1-3 of them, give the same bugfixes to every candidate, and come up with a standard rubric for judging the fixes. (If it's an open source project, synthesize 1-3 bugs, using the history of bugs on the project as a guidepost). This is a "work sample test", and it's the gold standard for judging technical qualification. Firms should rely on them to the exclusion of subjective interviews. Within reason, to a first approximation, every candidate gets the same problems (you may iterate over time). You can judge candidates apples-apples. You don't need to pay candidates for their time, because they're not doing work for you; you already fixed these bugs.
- sillysaurus3 9y ago
  Yeah, I'm a big fan of your approach. Unfortunately it seems quite difficult to get anyone to do it. Most companies have interview pipelines that are well-established and immutable, so only new companies are inclined to try it. And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria. It was a neat hack as a candidate to propose to the interviewing company that they do essentially the same thing as a work-sample test and hear them say yes. That only works if you propose they let you try your hand at doing their actual work. I wonder if it can be generalized?
  - tptacek 9y ago
    I think asking candidates to do "real" work is bad for a bunch of reasons: * It's intrinsically exploitative, even when compensated, because real work done under similar terms is paid far more highly than prorated salary (it also does weird things to salary negotiation, by extracting an up-front concession on rate). * It requires different candidates to work on different problems, which means you're not getting a reliable record of how candidates do on the interview so you can tune it over time. * Because the best candidates will generally refuse to work under these terms, most firms that do paid up-front work have a fast-path for elite candidates, which is an obvious process failure.
    - deleted 9y ago
johnvega 9y ago
I wonder where companies or hiring managers can post their job openings that supports this hiring process sometimes called Trial Employment or Job Audition for Software Engineers. Looks like a great way to get the best hire and also prevent a bad hire, although it can only work for a very limited number of companies and candidates.

thiscatis2 9y ago

So reading over some of the questions posted on technical programming interviews I must say that I would definitely fail such an interview without studying for it for a month first. Having recently discussed job interviews of ex-colleagues that went to EBay, Facebook, Google, etc.. I wouldn't even pass the whiteboard test. However I'm shipping scaleable API's used by web and native apps, using Continuous Integration and Deployment on an AWS infrastructure that I set up myself. Furthermore I'm responsible for hiring developers that I work with on agency projects and most important of all getting projects delivered within time, scope and budget AND getting paid for it. Yet I would fail most technical interviews. I've been doing this for about 10 years and never ran into any major problems. I've worked both as a freelancer and in-house permanent dev and never had any complaints about my work. (Feedback and improvements yes, big failures never). Yet I would fail most technical interviews. I talk directly to clients and vendors, I sell projects, I take over projects, I end projects and I've created 2 startups, one of which I sold but I would definitely fail a purely technical interview. I completely understand the need for these technical interviews and weed out obvious bad programmers and get experts on board but sometimes from a business point of view it just doesn't make sense. At a previous company I worked for the team almost crashed & burned because of the TL's focus purely on technical skills. No code was being shipped but hours of meetings and Slack conversations and discussions on PR's for no good reason about 0.01% improvement gains. I know I'm not the best developer out there and never will be. Often I'm in awe of the technical skills of people out there but I rather have a less skilled dev with great "people & shipping" skills as I call it than some of the people that would pass all these tests but that I couldn't put in front of a client or even let them email a vendor.

perfmode 9y ago
Many large organizations want their engineers to have the algorithmic ability and their technical project managers and engineering managers to have the strengths you describe. Technical leads who possess both bridge the gap.
- deleted 9y ago
  [deleted]
9y ago

kevmo314 9y ago

> If you ask the candidate the define the requirements for the task you're going to give them, you can't be surprised if the candidate imagines a use case with entirely different requirements from what you expected! This section really resonated with me. I've had a few interviews where I came up with a different answer than what the interviewer expected and this resulted in a lot of tension, as the interviewer asserted they were correct with zero exception. This often led to a lot of dead-end scenarios and the interviewer trying to get me to see an alternate solution without telling me explicitly. I once had an interviewer tell me that O(k) was constant if k was a config parameter (I disagreed because in practice, k would need to be tuned as a function of the size of the data, and it would be like saying O(n) is constant if you just fix n = 1M), and an interviewer tell me that you could solve an optimization problem without an objective function (in actuality, he had an objective function, he just thought it was obvious and didn't want to specify it). In both cases, I understood what they were trying to get at after they explained their solutions, but in the process it was incredibly frustrating as they seemed to just disregard my answer all together.

baddox 9y ago
I love interview horror stories! In my most recent job search, which was early this year, I debated with one interviewer whether checking for key existence in a hash was a constant time operation. I insisted it was, and the interviewer insisted otherwise with the same amount of conviction. That was just a small portion of an all-day onsite interview. After leaving, I was told they were looking for a more senior candidate. I was not upset.
- jerryr 9y ago
  Wouldn't that argument be easily settled by discussing the implementation of a hash table?
  - baddox 9y ago
    I discussed it briefly. My interviewer was unpersuaded, and the other interviewer in the room (who wasn't performing this interview) suggested we just move on. I believe the unconvinced interviewer was simply a very junior developer and an even more junior interviewer. He seemed quite nervous.

sidlls 9y ago

Some points are quite good, and in particular the author calling out the rubbish of asking "neat little algorithmic" questions (e.g. asking things like "implement mergesort"). However one thing I'd point out is that this guide still falls far short of being a good guide for finding engineers. Particularly noteworthy, in my opinion, is that the measurement criteria are still applicable only to the coding question itself. For example, "how well the candidate manages time" still applies only to this limited 40-60 minute question. It doesn't really help evaluate how the candidate might consider prioritization of tasks (implementation or otherwise) in a broader, bigger sense.

bostik 9y ago

A year ago I wrote a (shorter) post on the same topic: https://smarketshq.com/notes-on-interviewing-engineers-a4fa4383968a https://smarketshq.com/notes-on-interviewing-engineers-a4fa4... A reasonable question should never require the candidate to know "just a single trick". Questions that require to combine approaches are far better - they can actually measure if the interviewee can reason about the problem and identify the points for trade-offs. Btw: if you're interviewing more senior candidates, a code review question is often a good idea. It turns the situation around, from "how do you build this?" to "how would you fix this?"

the_mint 9y ago

All his example interview questions are stuff that were actually used at Dropbox!

kevmo314 9y ago
His resume (http://www.lihaoyi.com/Resume/ http://www.lihaoyi.com/Resume/) suggests he worked at Dropbox for quite a while. :)

HugoDaniel 9y ago

Here is how to do a good coding interview if you are a recruiter: 1. Do your homework: know what projects drive the interviewee (what kind of things she likes to code, what recent projects has she been involved in), make sure they are a match for your company projects and goals 2. Ask the interviewee for a git repository and take a careful holistic look at it (commit messages and code, timespan, branches, merges etc...) 3. Go for a 2h lunch with the interviewee in a relaxed environment to check speak communication skills (talk about whatever you think is interesting, if you can't have or don't know how to have 2h lunches you shouldn't be interviewing, check your life first and learn how to enjoy it before doing interviews). 4. Stop asking programmers to write code in 40 minutes. Learn how to respect before expecting respect in return, if you didn't take the time to read through their code repositories then don't ask for a 40 minute code problems.

mwcampbell 9y ago
I think you missed the part where the OP said not all candidates are willing or able to work on open source projects.
- HugoDaniel 9y ago
  Well the .2 "Ask the interviewee for a git repository" can be a private repository. It is a very hard problem to solve if they have no visible code to show but it does happen in other areas. Imagine asking for a management candidate for the private e-mails he shared in certain tricky situations of previous jobs. Or even public e-mails. Or even to take 40 minutes to write in a whiteboard a team statement for a certain conflict scenario. Trust is entailed when recruiting in most areas. But in programming not as much.
  - jonathankoren 9y ago
    > Well the .2 "Ask the interviewee for a git repository" can be a private repository. If someone shares a private git repo, they should not be hired, and in fact should be fired from their current job, and possibly sued. (Think Anthony Levandowski sharing Google lidar designs as an example of his work with Uber.) This isn't promoting trust. This is a violation of trust.

Goopplesoft 9y ago

> 5 minutes of resume discussion > 45 minutes of coding As anyone else found that a good resume discussion is often much better than the coding component? I've concluded that implementation difficulties, decisions made (tradeoffs, technologies, etc.) and the collaborative environment around projects, are much better signals than the code part of my interviews. If you've dealt with a broad range of tech and can ask the right questions, it's relatively easy to get a strong signal from a candidate by having a deep discussion on their work on prior projects. Both are needed and very useful.

majormajor 9y ago
Yeah. If they can't tell me about what they've built before, and aren't a less-than-one-year-away-from-school new grad, I'm going to be wondering how they deal with real problems regardless of whatever they can do algorithm-wise. If someone can talk about what approach they took, what other approaches they considered, what made it hard, what made it easy, how often the requirements changed, etc, then I'd rather ask them questions about that then spend the whole time in code. Just do some small spot checks to make sure they aren't taking credit for the work of others.
- Goopplesoft 9y ago
  > Just do some small spot checks to make sure they aren't taking credit for the work of others. Agreed, asking tough technical questions on this front gives a really solid indicator of a 'bullshitter'. For example, I often ask people to diagram a prior project architecture they worked on, and then ask them very fine details of how things work and try to challenge them with alternative approaches to see how they react. I get to experience working with them during the interview through which I learn if they're smart and capable.
Clubber 9y ago
Absolutely. If a person can't tell good from bad with that, they shouldn't be interviewing.

Xoros 9y ago

Those white board stuff reminds me when I was a CS student in the beginning of the 90's in France. We had our exams on paper. And believe me, writing LISP on paper, knowing if you forgot one parenthesis you got 0 point ("if I type it on a computer, it fails !"), was pretty and unnecessary stressful. So I relate with candidate facing that. Dude, on your day to day work, you WILL have a computer to check those ! How can you decently check those skill on a white board ? I totally agree with what's stated in this post.

morgante 9y ago
No good interviewer is grading whiteboard coding on whether you forgot a parenthesis or semicolon.
e12e 9y ago
If you think writing lisp by hand is bad, try implementing a linked list interface in java on paper... shudder. (in fairness, we were sanely graded, minor syntax errors didn't subtract terribly from the grade...).
metaobject 9y ago
Writing code for tests, on paper, has been a thing in CS for quite awhile. While writing lisp for tests, I used to sometimes write the code with no parens (using indentation to denote nesting). Then I'd go back and add parens where appropriate. defun add-pos-only v1 v2 let sum 0 if and > v1 0 > v2 0 + v1 v2 nil

XCSme 9y ago

Interviews suck for both parties, a world without interviews would be a better world. Why can't we just hire someone based on his experience and projects he previously worked on? You can have a first month as a "trial" where both parties can cancel the contract. Why interviews suck is because they are very, very biased and subjective and also not a good way to estimate the actual performance of a candidate. Say no to interviews! If you like what someone did, hire him. You can't get to know a person after 1 or even 10 interviews, because the interview environment is totally different from the work environment. PS: I have been interviewed by 14 different people at Google. I didn't get the job. The overall conclusion was that some interviewers loved me, my skill and my work, others didn't like me from the start or simply didn't care about me. Some of them had a good day so they gave a good "review" even though I made mistakes, some of them had a really bad day (one fell of the chair) and gave me a really bad review even though, related to the actual interview questions, I did really well. (14 interviews = applied twice, 2 x (5 on-site + 1 online), and they also wanted 2 extra interviews to take a decision)

CJefferson 9y ago
I'm sure most businesses would love that situation, but for employees it would require you quit your current job, but have a good chance of being fired after a month at your new job. I imagine young employees would be happy (I did a bunch of short-term contracts myself), but once you have a mortgage and kids, it's nice to be a little more certain about your new job, before quitting your old one.
- nolemurs 9y ago
  > I imagine young employees would be happy (I did a bunch of short-term contracts myself), but once you have a mortgage and kids, it's nice to be a little more certain about your new job, before quitting your old one. And just to make this really explicit - this means that a company with this hiring practice would have a very hard time hiring experienced competent engineers. If I were looking for a job, I'd have lots of options, and I would not waste time on a company that demanded this.
njovin 9y ago

zerr 9y ago

Small take-home paid project.

amriksohata 9y ago

For me I have interviewed many people with a technical test, that was the easiest part i.e determining their competency. The hardest part is figuring out what their work ethic is, can they pull their own weight and will they cause friction in a team, sadly you only find that out often AFTER hiring them.

mwcampbell 9y ago

Here's another disadvantage of whiteboard coding: it's inaccessible to blind people, and impractical (even more than usual) for people with only a little vision (like me). Such candidates will need to do the coding on a computer, so one might as well make that the default, as the OP recommends, and eliminate one variable and a possible source of bias. Of course, once a blind or low-vision programmer stays at a given company long enough, they'll probably be on the other side of the interview at some point. So for consistency among a team of interviewers, I imagine it would be best if all of them have their candidates do coding problems on a computer, not just one of them. I'm curious about why whiteboard coding is still a thing in the first place. Is it a matter of convenience or cost? If so, surely that equation has changed now that we have ubiquitous, cheap, portable computers. Or do some interviewers really think that it's good to deny a candidate access to a real computer while solving a coding problem during the interview?

mahyarm 9y ago
These places use the computer for interviews already for the visually impaired. It's just a name for a type of interview. I even give people the option to use the computer if they are more comfortable with that or a whiteboard. It can be 50:50 in who choses what.

spraak 9y ago

What are some good resources for developing good algorithm skills for a non CS grad? Examples in JS, Python or Go preferred.

SmirkingRevenge 9y ago
There's lots of stuff online. Check out hackerrank.com for example. There are other sites like that out there. Its becoming a real cottage industry, this whole IT interview prep thing.

tptacek 9y ago

It's good to think carefully about how you interview candidates. Most of the ideas in this post are good. You should definitely let candidates type answers on computers. You should definitely work to put them at ease, in an environment that approximates their actual working environment. But really, this is window dressing. The problem is that technical interviews just don't work reliably. We have better ways to qualify candidates. Rather than trying to read subjective, noisy signals from in-person interviews, give candidates a small battery of work-sample tests. These can be bug-fixes, small features, diagnostic challenges, or even specification writing assignments. A properly designed work-sample challenge has a rubric: a set of instructions for scoring a result. Ideally, the rubric is straightforward enough for someone who hasn't met the candidate to quickly score their result. "Runs correctly". "Passes a battery of unit tests". "Includes appropriate documentation". Whatever is important for your team. Work to give every candidate the same set of challenges, so you can track results over time. You'll find that after refining and iterating for a few months, you'll rapidly build confidence in the tests, to the point where in-person interviews become superfluous for technical qualification. Working on technical qualifiers is stressful. Most candidates will perform more poorly on them than they would on normal work. You should build that into your rubric (though, really, data from candidates will ultimately determine how strict the rubric becomes). Understanding how stressful qualifying work is, you should consider not requiring candidates to work on problems on-site. Instead, administer them online. Suggest the amount of time you expect the problems to take to finish, and offset that amount of time from the amount you demand on-site. When you must interview, you should work hard to standardize the interview. Interviewers hate this. Everyone has their favorite questions. People want interviews to be conversational and adaptive. Interviewing isn't supposed to be fun for the interviewer; it's supposed to be reliable.

michaelchisari 9y ago
This is my approach for evaluating new candidates as a first step. For front-end, for instance, I start with a codepen that has the basic skeleton that includes the libraries we use internally, has a few html elements styled and placed, and the required javascript boilerplate. Then I provide 10 tasks that are written much in the way the company would put tasks in asana or git or jira. Each task builds on the previous and gets more complex. In the boilerplate they start with, there is one obvious error and one subtle one. Each step, I encourage them to write notes on how they solved it, and any questions or concerns they might have. I make myself available via chat or email if they have any clarifying questions. They then can do it in their own time, at home. They can google, hit up documentation, stackoverflow, etc., because that's exactly how developers work. Nobody's over their shoulder, there's no arbitrary limitations, just build us some good UI code. It's true that I've been amazed at how many candidates were stopped in their tracks at the simplest things that are well within the domain of their resume. But when I get a candidate that gets through them, hits the requirements and has clean code as a result, I've hardly ever seen them fail at a later step where we might evaluate more complex problems like working around GC slowdowns, obscure browser compatibility, first meaningful paint performance, etc. And most of all, I feel confident that I haven't tried to "trick" anyone by having them do long-solved algorithms offline on a white board.

cbanek 9y ago

> Hopefully in your realistic setting, the challenge comes from the difficult tasks you are given to solve, not some jerk that's sniping at you from the sidelines. I wish I could say that's true. If you're getting sniped from the sidelines in your interview, it's probably a part of the culture where you are interviewing. So it's quite likely to be a non trivial part of the problem you're solving.

jonathankoren 9y ago

I have to disagree with this trend of having people write code on a computer instead of a whiteboard. Sure, it mimics the real world, but in practice you spend a bunch of time dealing with crap about syntax errors, and argument order mismatch instead of dealing with the actual underlying algorithm. Even worse, you may end up with a candidate that realizes that there's probably a library that would be really useful for this, but then has to spend time googling for the name of the library, then installing it, and then desperately skimming for a quick start in documentation because the question isn't, "Can you work google and run pip install?", but rather some other task. So now you've probably lost 15 minutes just dealing with that shit. If you're going to do this, it simply doesn't work in a 40 minute setup without some sort of provided scaffolding and test cases.

jpatokal 9y ago
Is doing interviews in an IDE and expecting fully functional code actually a thing? My company uses a simple text editor: perfection is not expected, and it's still way better than a whiteboard.
- jonathankoren 9y ago
  I can confirm that this is a thing at multiple companies. Instead of of testing the actual algorithm, you'll encounter such blasts from the past like: "Network Socket Debugging", "How The Fuck Does Java Read From STDIN?", and everyone's favorite, "Why The Fuck Did You Give Me Text Data In CSV Format?" I have seen this go well exactly once. And they provided an environment with a working (albeit do nothing) scaffold along with test data. Also it was like a three hours in a room.

pnw_hazor 9y ago

There is no reason to haze potential employees. Just fire them if they don't work out. You set expections by telling them up front they are on probation and that they will be fired if they fail to live up to their resume. Some of the worst programmers I ever worked with could pass hazing interviews like a champ. Whereas some of the best would refuse to step foot in companies with a reputation for hazing their prospects.

Clubber 9y ago
That's a funny term to apply to this type of interview process, but it fits. Kudos. Agree on the invalidity of technical interviews. We hired a guy who correctly answered every question I asked, but he could never ship a product. He always got distracted by silly things. He was great at solving tough little problems, but never could grasp the big picture.

Jemaclus 9y ago

I have a lot of thoughts about this particular topic, probably ideas that are unpopular and definitely go against the grain, at least in the way the Bay Area operates. In my opinion, if you bring a candidate on-site for a 4+ hour interview and you ask them to write code, you've failed to properly vet the top of your recruitment funnel. As a software engineer, writing code is the BARE MINIMUM of your job. Imagine hiring a construction worker by merely asking them to hammer a nail for 4 hours. You should have a thorough understanding, through various means such as phone screens, code samples, portfolio work, and simple tech challenge, to determine whether they can write code before you even bring them on-site. Once they're on-site, you should quickly verify that they can actually do what you previously learned that they can do. I'm talking 15-20 minutes worth of time. The remaining time should be 100% on the "soft skills" and other things mentioned, including but not limited to my big criteria: "Can you quickly get up to speed and learn the things you don't already know?" and "Do I want to sit next to you for the next 6 months of my work-life?" If your on-site interview includes 45 minutes of coding (much less multiple sets of 45 minutes of coding!), you're doing it wrong. Very wrong. Keep in mind you've asked your candidate to effectively take a day off work from their current job. They're calling in sick, taking a vacation day, or somehow lying to their boss and taking a risk of not being in their office when they're in yours for the interview. If they're smart, they've lined up multiple interviews with multiple companies, because putting all their eggs in your basket is just a terrible idea. Imagine having to take four days off to interview with four companies. And they probably aren't four days in a row, so you can't just pretend to have the bubonic plague or something. You're asking them for a huge commitment of their time. And if you're willing to kick out a candidate because they didn't know how to invert a binary tree, then you should have found that out before they even walked in the door and saved everyone a whole bunch of time. IMO, an interview should be a max of 4 hours. Any more than that and the candidate is simply repeating themselves over and over to different people, potentially solving the same tech problems (that should have been evaluated beforehand), and generally making it a huge waste of time. My two cents. Happy to debate these ideas with anyone interested.

balls187 9y ago

Being a better interviewer is important. As stated, some types of questions aren't good judge of skills, like implement mergesort. If you can implement mergesort, you probably studied up for your interview. You're not testing anything, other than did the candidate care enough to study. Interviewing should test the candidates ability to write code that is: 1. Correct (solves the problem) 2. Efficient (solves the problem without unnecessarily wasting resources) 3. Understandable (goes hand in hand with maintainable) 4. Maintainable (can be modified by another developer, or even yourself after a few months) Coding on a computer can lend itself to all of these (no brainer there), but you lose a key aspect of whiteboarding that I think is really important--how a candidate approaches solving a problem. Knowing how to implement merge sort, or memorizing every possible linked-list question isn't all that valuable--but having a concrete approach to tackling problems is, and I want to understand that process from my candidates. I want to know the design tradeoffs you make, how you think about optimization, how do you reason about constraints as you learn about them. It seems that explaining your process while coding on a computer would be difficult to do without practice--and if you're going to practice, why not instead just practice whiteboard coding, which is how most tech companies interview.

rimliu 9y ago
So do you want someone who is good at explaining how they are looking for the solutions, or someone who is good at finding the solution? For me personally approaching a problem never involves a whiteboard.
- balls187 9y ago
  I don't think I've ever been at an engineering company that didn't have whiteboards, at the very least in conference rooms--I infer from that that some portion of employees find them valuable. Clearly someone who is good at solving problems is preferred. However, between someone who is good at solving problems, and someone who is good and can explain/demonstrate their process is more desirable.

ilaksh 9y ago

Mostly makes sense but your ability to write a parser is still going to depend on how many parsers you wrote or when you wrote them or whether you are able to use a framework/tool to do it.