16 ms·
"Then, acting as though this was the first time I'd seen this problem, I would ask if it was ok if I thought aloud as I worked my way through the problem on the
by timr 15y ago
"Then, acting as though this was the first time I'd seen this problem, I would ask if it was ok if I thought aloud as I worked my way through the problem on the board. I'd mumble to myself about moving-this-piece-over-here and-now-we're-going-to-get-this, and lo-and-behold, I accidentally solved it in constant memory space, in C - a language I didn't even claim to be particularly good at! Only someone with amazing problem solving skills would be able to do that their first time thinking through this problem so earnestly!"
All this tells me is that tech interviewing is horribly broken. What we have here is a story of a guy with a little bit of experience who has (if his story is true) successfully learned how to game the interview systems at dozens of startups. If you interview people, this article should terrify you. You can argue that it isn't "gaming" if the guy turns out to be a good employee, but that's wrong. If he can trivially game the system, so can anyone else with the ability to go to dozens of interviews and remember questions.
There's a fix for this problem: make the interview about more than the answer to the stupid question. If the candidate answers your question a little too quickly, ask something harder. Chase down a detail. Pick an experience on the resume (and not an obvious one), and dig as deeply as you can. Good candidates know the details. Bad candidates don't have details you can chase.
If you're interviewing correctly, no candidate should be able to come in and snow you with some regurgitated C code from previous interview experience, because you'll be able to out-flank them at every attempt to cough up a line. Unfortunately, there's a corollary: you need to know the subject better than the person that you're interviewing. If you don't, you have no hope of screening for good people.
- kakuri 15y agoIt sounds like in this case, this guy might be a good candidate - look at all the effort he put in and his ability to remember things, think critically, and learn. On the other hand, there are plenty of competent programmers not willing to put up with this nonsense, and the companies are going to miss out on them.
- sgrove 15y agoYeah, that was more or less the point of the post. For applicants, don't feel nervous, it's a silly game just to see if you know some of the puzzles. For people hiring, go ahead and use this as a first-pass filter, but realize that it's easy to game (by someone who's trying hard and learning, so there is a positive signal in that). It's going to be important to go through and work with them for a period of time in a let's-date-before-we-marry relationship. As an employee, I always insisted on a contracting relationship before an employment relationship. Like I said in the footer, Heroku really seems to do this the right way, and it shows. Kudos to them. And everyone else needs to step up their game.
- groby_b 15y agoIn all my history of interviewing people, I've always found there's almost no info to be had in correct answers to any kind of question. You learn about a candidate once you guide them into territory where the fail. Both where that line is, and how they handle the failure tells you a lot. (Caveat: Make sure you're keeping it on the fail/no-fail edge, and you treat the candidate with respect. The point is not to show them that they're wrong, but to explore the boundaries of their skill set. Without being mean is a major goal here) And if you can't push the candidate there: HIRE THEM. Right away. If they know that much more than you do that you can't get them beyond their limits, they are a fantastic addition to your team. (Provided they match personality-wise, too)
- lancewiggs 15y agoI've done the same - keeping on pushing until the candidate is on uncertain ground. (I interview for consulting/ analytic /business/ senior management roles.) As most of a job is about doing things that haven't been done before, getting the candidate into unknown territory is when you really find out how effective they will be. Some people just stop, others switch seamlessly into problem solving mode and come up with an answer that may or may not work, and also with a plan to test whether it does.
- sgrove 15y agoThis sounds like a great technique. It also gives both sides the chance to talk a problem through, bounce ideas off of one another, and see if there are any interesting insights to be had on the other side of the table. A candidate and interviewer both have to be fairly relaxed and comfortable to do it properly, but that's more of a problem for people new to the interviewing process (probably not senior management roles)
- rhizome 15y agoIt also gives both sides the chance to talk a problem through, bounce ideas off of one another, and see if there are any interesting insights to be had on the other side of the table. Some companies do this intentionally without hiring anybody in order to double-check their plans. It's basically free consulting.
- kenjackson 15y agoI have some simple baseline questions I ask everyone. Getting it right just means we continue the rest of the interview, but doesn't get you a job. I almost always have a programming question I just made up 30 minutes before the interview. I make sure that I can easily do it in 10 minutes and then ask it. It really isn't important to me if they get they answer, but I want to see what questions they have, and how they think. Good candidates will usually get it coded, but some may get hung up on something early on, but still make good progress. If they come in with a precanned response to my question then they're either a mindreader or can time travel. In either case, I still went them on my team.
- bialecki 15y ago>> I almost always have a programming question I just made up 30 minutes before the interview. I make sure that I can easily do it in 10 minutes and then ask it. This is a lot easier said than done. Do you really do this and, if so, any tips on how? (I'm guessing variations on a core set of questions.) I usually don't mind asking a questions someone may have already heard because there's not really a right answer, but I just want to discuss a problem to see how you think through it.
- kenjackson 15y agoI actually don't have an explicit set of core questions, but I'm sure there are clear themes that run through the problems. Probably my only real tip I can think of is not to have them code something related to stuff you're currently working on. You'll be far more likely to underestimate how hard the problem is. But the problems aren't that hard to think of. They're not like ACM programming contests. They're more like: You have two lists. One is a list of IP addresses. The other list contains IP addresses or IP addresses with wildcards. E.g., 192.168.$.1 or $.255.255.255. Return the set of IP addresses from list 1 that are matched in list 2. Additionally return the index from list 2 that matched -- I want the index that is most specific (as defined by having the fewest wildcards, or wildcard furthest to the right when there is a tie).
- 15y ago
- prostoalex 15y agoUnless the end result is a completely incompetent applicant getting an offer, I wouldn't call the system "horribly broken". Very few companies are in the business of reversing linked lists, and hence make binary decisions depending on the outcome of just that one question. I don't think gaming the system is that trivial - an interview is a conversation, so things like computational complexity, runtime restrictions, data structures, implementation differences in various programming languages, and other things are bound to come into conversation. Now, if one can maintain that conversation as it moves along, and seem like a pleasant person to work with, it seems that they're no longer purely "faking".
- tedunangst 15y agoExactly. I've had lots of candidates basically pass the initial question and then sink their own battleship by volunteering some running commentary that happened to be totally wrong. Memorizing enough crap to impress an interviewer about even a basic reverse the linked list question without understanding it is actually quite hard. It's not enough to remember to drop the terms stack and heap, you have to remember which one is which.
- bigbird 15y agoI also don't think that what the article describes is "gaming" the system, except maybe for the part about pretending to have not heard the questions before. Generally, the reason I ask such questions of candidates is to make sure I'm dealing with someone who can actually analyze a problem and get a reasonable result. So if they nail the questions because they took time to study and get good at interview-style brain teasers, that's actually fine with me - it proves the person can execute a plan to get correct results. I'm not sure I'd call anyone who successfully "games" the system incompetent.
- skybrian 15y agoAll that is true, but there's a simpler fix. If they're not reusing each others' questions then it should be significantly harder. Nobody should be using "reverse a linked list" anymore - that's a really old one.
- steve8918 15y agoThat's not a simple fix. If you don't use "reverse a linked list" anymore, then you start asking increasingly harder and harder questions, and the bar gets ridiculously high, to the point where everyone needs to know how to solve NP-class problems within 45 mins. And even then the system still can be gamed by simply memorizing more and more questions. It's a lose-lose situation.
- eru 15y agoYou just have to make the problems so diverse (not necessarily hard), that genuinely understanding the solutions is the easier strategy than rote learning.
- steve8918 15y agoCan you give me an example of a question that isn't located in glassdoor.com and is "diverse but not hard". As far as I'm concerned, the bar is already ridiculously high to the point where I feel like the only people that get jobs are the ones who randomly get lucky enough to get 3 or 4 interviewees that ask easy questions. There are people I know personally that have gone through every single one of the 1000+ Google interview writeups in glassdoor in order to prepare for interviews. One of my former co-workers showed me 10 pages of questions he distilled from glassdoor of various questions that were reported, and he memorized the answers to all of them, ranging from "find the common descendant of two nodes of a directed graph", to programming an AVL tree on the spot, to pure dynamic programming questions. If they asked these types of questions 15+ years ago when I started, there's no way I could have gotten into programming. Even now, the caliber of those questions are simply too hard for me unless I sit down days before the interview to get the answer. There's no way I would be able to figure those out within 45 mins let alone 2 hrs, and I'm not dumb. It's ridiculous the level of expectations people have these days.
- Evgeny 15y agoIt's not really gaming the system. He solved the problems, himself. He asked questions, learned and then came up with more and more effective solutions. Aren't these exactly the traits an employer should be looking for? Didn't he put a lot of effort and time? Here's my own example. I've been recording some of the problems I have to deal with at work, and some of the things I come across in my spare time in the form of a blog, for the last 3 years (with some gaps here and there). I recently started looking for a better place to work. I included the link to my blog in my resume. I got an offer recently, it will be a substantial increase in salary. I only interviewed with ~5 companies to get an offer, that's my easiest offer and shortest job search ever. Now, would you say that I 'gamed the system' because I had a blog, and most don't? I guess you can say so, because when I started my blog, I had that in mind - it may be useful next time I look for a job. But I did not mindlessly type stuff in - I made sure I understood it as much as I can, and quite often writing the solution down helped my understanding. Same with this guy - he did not just memorize how to do it. I had one YC company ask me to create a linked-list structure in ruby, and then reverse it. I did it recursively, mentioned that it would blow the call stack, and then re-implemented it iteratively. I also remember there was some problem where I used symbols rather than strings and that would cause some bad memory leaks over time in a long-running process. - From this passage, he might not be the "top 1 percent" everyone tries to hire, but at least top 5%, that I'm sure of. Or maybe he gamed my impression too.
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- kamaal 15y agoThe definition of gaming the system is different here. These days unless you are implementing a library yourself. You are probably never ever going to even reverse a linked list ever in your life. Or deal sit down with a math text book when you code. Those days are probably gone in the 80's. That means asking these questions is akin to you taking an approach to judge an candidate on things which have nothing to do his job. That's why so many programmers don't about all this. The reason is for 99% of the work they are likely to do they don't need to know it. Under such circumstances when you ask these sort of questions the only way candidate can deal with them is not by actually learning how to solve them. But learn them just for the interview sake. So you end up hiring some one on a criteria doesn't matter. And the candidate manages to beat you by system of work which defeated your original means of judging him. I think this is what gaming the system means.
- huhtenberg 15y agoGoing through the details is the way to estimate candidate's true knowledge of a subject. Ask an easy question, then ask a very hard one, repeat depending on answers, i.e. basically do a binary search: - Do you know what SSL/TLS is? general - What's the maximum size of SSL protocol record? specific - What's a cipher suite? general - How is DH-anon suite is set up? specific ... - What is a linked list? - What is offsetof (and how is it relevant)? ... - What is TCP? general - What is TCP/Vegas? specific ... This is the quickest way to converge to an actual level of expertise, and it really helps with filtering out people with bullshit resumes and those who cheated their way through an offline problem solving. We would say something like "You are not expected to answer all questions, we are just trying to understand what you know and what you don't know."
- arghnoname 15y agoThere's always a downside to every approach though. Some people get flustered if they have to say "I don't know" too often, so instead of bouncing from tough to easy, I'd probably prefer to progress from easy to tough. People will feel confident until they hit their limits. Their limits are what you are looking for (presumably), not their ability to think clearly when they are scared they just blew the interview on a question you didn't really expect them to answer.
- spiffytech 15y agoI recently interviewed for a position with this sort of technical interview. I felt it worked very well. The interviewer was able to cover a wide range of topics in varying depth, and could see what I could easily recall and what I struggled with. With the right set of questions, and some hints when the candidate struggles, you can very effectively gauge a candidate's limits on a subject, and see how they try to puzzle things out when they don't know an answer but know enough to take a good guess. The key distinguishing factor of this method, IMO, is a candidate cannot memorize enough answers to bluff their way through a good questions list. If a candidate really is an N out of 10 with technology $FOO like they claim, they should have no problem answering most/all of your 50+ N/10-level questions for $FOO (my technical interview consisted of ~2 hours of this sort of interrogation for a straightforward junior web dev position. Interview length, in this interviewing style, is important for ensuring you've exhausted a candidate's book knowledge).
- nadam 15y ago"you need to know the subject better than the person that you're interviewing. If you don't, you have no hope of screening for good people." I think it is not necessary and not possible that the interviewer know more about every subtopic than every candidate. It is enough that they have comparable knowledge in most topics. Good candidates can very easily know more about some parts of the subject (and maybe less about other parts). Being a good interviewer is a really tricky business, because good interviewers should see the strength of candidates and should see if the candidate know even more than them about some subtopics. I've seen some bad interviewers who asked only the very narrow topics they knew well, and basically wanted exact copies of themselves.
- ajays 15y agoTell me about it. I have nearly 20 years of experience in CS, and hold an advanced degree. I have 5 years of experience in a very niche field which is relatively hot. I was approached by Facebook for this specific purpose. They invited me in for just a "chat", asked me if I'd be interested in working with them. Obviously I said yeah, I'd be willing to talk. They set up an in-person interview. Just 45 minutes they said. So I show up, fully expecting a discussion about that area, my background, etc. Instead, I get a random coder who wasn't even making eye contact when asking questions. And the questions? Elementary CS stuff that any undergrad would know. Now, obviously I know them too; heck, I taught Algorithms courses in grad school. But why waste my time with this crap? You can hire almost any fresh graduate from a top-40 school who would do a halfway decent job at answering those questions. What does asking a PhD "how to print the levels of a binary tree, one per line" tell you about his work? (That wasn't the question, but that was the level of questioning). The sad part is: I can waste a couple of weeks of my life, go to Glassdoor etc. and basically memorize these questions. I'll probably stand a good chance of blowing the interviewer away by pretending to solve them on the spot. But what does that tell you about how I am at real-life problems?
- jrockway 15y agoThe cost of hiring someone who can't do this stuff makes it worth the risk of putting off someone like you. Sometimes you have to ask rather than be faithful if you haven't personally worked with the person you want to hire. A shitty programmer will kill an entire team.
- sgrove 15y agoThere's a line here - I think the interview should cover a lot of bases, but by no means should a hire be a concrete thing. The first two weeks should be an intensive, fully-paid interview/trial-by-fire. I suppose, I just don't see why a shitty programmer couldn't be filtered out almost immediately through this technique. Maybe it's a company size/age difference - what size company do you have in mind?
- jrockway 15y ago
- zem 15y agoas someone who successfully built a great dev team at a startup, i can say that this is a sign of sheer laziness on the interviewers' parts. which is sad, because in a small company, hiring is one of the most important things you can do. there is no real excuse for having an interview so generic that someone can game it in this manner unless you're so large that you can afford the odd charlatan.
- paganel 15y ago> there is no real excuse for having an interview so generic that someone can game it in this manner unless you're so large that you can afford the odd charlatan. I'm also part of a small start-up (as an employee), and I can certainly vouch for that. I for one put a very high emphasis on the personal/professional projects the guy/dudette in front of me has been involved in, on how best s/he can explain them to me, on how passionate the person seems when explaining all this (and no, I'm not talking about the enthusiastic approach that can be easily faked, I just think that when you're a programmer you can feel what the other programmer in front of you is very interested in). Also, this mania a lot of interviewers have with the interviewees "always ticking the right boxes and giving the correct answers" sort of scares me. It's probably because I work in a start-up, but I can assure everyone that at least half of the time we ourselves don't know the answers (assuming that we even know the correct questions), and there are lots of times when we seem to be knowing in what direction we're going but truly speaking we don't.
- rmc 15y agoIf you interview people, this article should terrify you. If you're interviewing for programming jobs, the (low) calibre of existing applicants has probably already terrified you.
- dunk010 15y agoTrue dat.
- lelele 15y agoLousy employers who employ lousy interviewers deserve lousy employees.