24 ms·

Harder programming questions do a worse job of predicting outcomes

• aantix 8y ago
  If they want something that mimics the common, everyday.. Give the candidate a project with 300,000 loc, tell them to make the most local change possible that fixes the reported bug. Update the tests to reflect the new logic. Bonus: discuss architectural changes that would have resolved the bug and/or improved performance.
  • auston 8y ago
    They do something like this with a couple thousand lines of code and failing tests.
• charliewrites 8y ago
  Another clarification: this data only applies to general programming exercises and not domain-specific knowledge. For example, if your role requires bio-informatics knowledge (or ML or AI etc.), then by all means ask about bio-informatics—even if it means asking harder questions. (We do, however, recommend keeping the content of general programming exercises as vanilla as possible, so that you don't filter out someone because they lack mastery of a subject that you don't actually intend to measure.)
• xiphias2 8y ago
  ,,Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives'' The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google. The other part (whether it's more important to work on long easier questions to see how the candidate works on a large code base) is orthogonal reasoning, and that part may be true, depending on what type of engineers somebody is looking for.
  • maxerickson 8y ago
    It's at least good that they call it a philosophy.
    • xiphias2 8y ago
      For me not being afraid to get kicked out of my workplace was a good thing. Also most of my colleagues were amazing, I miss talking to them, but at the same time I don't miss what the top management has become.
      • bdavisx 8y ago
        What has it become?
  • drugme 8y ago
    The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. Corollary: "If I follow (what I've heard to be) Google's hiring practices (despite not having their brand recognition or candidate pool or a comparable engineering environment to offer) -- then my company is on the way to becoming another Google!"
    • xiphias2 8y ago
      Sure, it's something not to be copied by a small company. But it's important to understand for good developers that they can be great and still fail the process (they just have to prepare again adn reapply)
  • bitL 8y ago
    > The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. Google's problem nowadays is that they have such strict standards only for grunt developers, but not for management. In fact, it looks like that it's more like opposite, i.e. it's better letting in/promoting 3 bad managers/directors than 1 good one.
• drugme 8y ago
  Because they're designed that way. It all comes down to: * Rejecting far more candidates than you need to -- so you can feel like you're hiring "the top 1 percent" * Giving yourself the feeling that you have an objective hiring process (when really you don't) * Making your own team members feel like they're super brilliant and special when really they're not That's what the modern hiring process is designed to do. And in fact it works quite well, to serve this purpose.
  • harryf 8y ago
    Think it’s more a by-product than by-design. At the risk of making a blanket statement, think it’s fair to say most software engineers have average-to-poor people skills, leading them to the fallacy that “logic tests” are a good way to evaluate people. That in turn leads to a bias you describe “did you see how badly the candidate failed the test!?!”
  • tomp 8y ago
    > Rejecting far more candidates than you need to What do you mean by "than you need to"? I'm guessing for many companies (especially the ones with this kinds of interviews), the limit is the number of hires they can do, not the number of candidates that apply... So by definition, you need to reject all but n (the number of open jobs)... Why wouldn't you reject them based on performance on interviews (as opposed to, by their CV or luck or something)?
    • arnvald 8y ago
      In many cases companies don't interview all the candidates before making a decision, it's more on a rolling basis. You consider each candidate separately - you interview a candidate and then decide whether to hire him/her or not. In plenty of cases companies reject candidates who later perform successfully at similar roles, and this is the point of the parent comment. However this is kind of a desired effect, because not hiring a right candidate has lower cost than hiring a wrong candidate: - if you skip good candidate, because you're not sure whether they'll perform well, you just wait for more candidates to apply, it just slows down the process - if you hire wrong candidate, the candidate joins the team, underperforms and eventually is let go, but during the time that person works for you, you don't look for the right candidate for that role, which costs you more money and time
• cshg 8y ago
  Specific examples of what classifies as a “hard” or “easy” interview question would be very helpful to have reference points and assess one’s own interview process.
  • akhilcacharya 8y ago
    Easy - reverse a string, determine if a string is a palindrome, reverse the digits of an integer, determine if one string is an anagram of another. Hard - implement a subset of regex match in optimal time+space, find the operations required to turn 1 word into another word given a list of transitory words, find the median of 2 sorted arrays in optimal time, find the next permuted value.
    • pps43 8y ago
      That's more like small/large than easy/hard.
      • pmiller2 8y ago
        The key word here is “optimal.” These aren’t difficult tasks if you can just write down the straightforward code to do them.
      • akhilcacharya 8y ago
        I'd argue finding the next permutation is actually very difficult if you don't know the standard algorithm.
    • kbp 8y ago
      I think those "easy" examples are too easy to get any meaningful signal from. If they struggle, they have no idea what they're doing, and if they don't, you don't really learn anything about how they work because there's not much to them. Other people have mentioned Triplebyte using console tic tac toe as a question; that seems like a better sort of "easy" question that still lets the interviewee have a chance to show off their problem solving and factoring skills.
  • deleted 8y ago
• wglb 8y ago
  Instead of doing this, read, then follow https://sockpuppet.org/blog/2015/03/06/the-hiring-post/ https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
• dcplogic 8y ago
  A while ago we had a job applicant who had travelled very far and long to reach us. (literally from the other side of the world.) So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-minute whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack overflow/etc access was allowed and encouraged. "Just act as like you would normally do when solving a problem." We spent like 2x4 hours devising this problem, based on our codebase (cutting out something somewhat easily digestible and making it able to run standalone). It took like one hour to get productive. Explaining the problem, setting up editors, compilers, etc. We took turns, but most of the time someone in the interview team (of two) sat next to the guy. We did give him some alone time. This is probably nothing new in terms of interviewing techniques, but to us it was such a revelation. We learned so much more about the applicant. Perhaps it worked well with this guy because he happened to be a bit more outgoing than our typical successful applicant. We'd never felt so confident about giving someone an offer before. I'm really looking forward towards testing out this approach with local candidates to see if we can replicate this "data gathering success".
  • peteri 8y ago
    This can be problematic, you waste a lot of time setting up the PC / explaining the problem. Last time I was involved with this interview style it always seemed to take an hour to get setup which meant a long interview of 3-4 hours particularly if a candidate went down the wrong path. In the end we optimised for SOLID principles with a blackbox dll that had a function that slept for 2 seconds and a calling class that had mixed responsibilities (logging and calling the dll). We started folks off with a test or two and hoped they'd inject a mock to get rid of the delay and split logging off into a separate class. I'm not saying it was a great test but you could do something within an hour or so then maybe spend half an hour talking through what techniques they'd use for a more complicated scenarios. If you can afford the time then more realistic testing is great and I do think you should try.
• systematical 8y ago
  I've never been giving given a practical programming test. Nearly all are useless algorithms. Create a function that takes a integer greater than 0. Build an array equal in size to the argument. Fill the array with numbers that when summed together equal zero. It was worded far worse than that. What exactly is that telling you about the engineer? On the flip side I'm asked to code full fledged applications but not to spend too much time on them... okay... Another time I was asked to code a luhn algorithm. Oh and do it while a room of people watches you on giant screen cause that's what your day to day job will look like... I failed miserably and still got the job. What?!?!?
  • deleted 8y ago
    [deleted]
  • kyralis 8y ago
    FWIW, I regularly pass interview candidates who "fail" at certain questions. The point of those questions isn't to complete it perfectly (and sometimes we get less information from those, honestly), it's to see how you go about getting there, how you fail, and how you deal with that failure. For instance, one of the problems I frequently ask has a structure that really encourages people to try inventing heuristics to solve the problem, even though ultimately all of those heuristics fail. Seeing how people react to "but what if your input looks like this?" questions is often very enlightening- can they rethink their approach? Do they just keep glomming on more special cases? Can they deal with someone pointing out that sort of flaw?
• pps43 8y ago
  So they saw a correlation among people they hired and made a conclusion about people who applied, with no reject inference?
• komali2 8y ago
  >how clean their code is, I got bit in the ass by this one as triplebyte itself. They asked me to make a tic tac toe game, and gave me iirc 30 minutes (less?) to do it. Except, it wasn't "build a tic tac to" game, first it was "draw a board to the console," "take user input from the console," etc a bunch of instructions in a convoluted path that perhaps another engineer would do when knowing from the outset that the goal was to build a tic tac toe game in 30 minutes, but not me. So we'd get to a portion where I'd be writing a quick test on user input, or extrapolating something to a function, and the interviewer would say "don't worry about that, just worry about {getting the grid to print to console or whatever}." Later on I got my feedback and they said they were disappointed with my user input tests and repeated, extractable code in the tic tac toe portion. Triplebyte is trying to do good things in the interview space but I think they're still learning. All in all my interview with them was about as positive an experience as a harried and bad interview could be, from my perspective.
  • closeparen 8y ago
    Telling people not to worry about parts of the problem is often a time management strategy. They want you to move on so you have a chance to show strength in other parts of the problem, or at least come away with the positive experience of finishing something, even though it may count against you.
  • Benjammer 8y ago
    I took shortcuts in my triplebyte coding porting of the interview, assuming we would iterate and expand on the solution. Instead we just moved on to algorithms and I got heavily dinged on my feedback for the shortcuts I took (things like hard-coded strings in some places to get the UI working before wiring up to real data models).
  • localcrisis 8y ago
    Same experience with the Tic Tac Toe, then again with the rest of the interview. There were a lot of Googlable boilerplate questions (e.g. "what does malloc return?", "what's a bloom filter?") that, as a product engineer, never come up. Then there were the classic Big-O notation queries that for most use cases don't come up until much later stage. It felt like the founders were classically trained in CS and over-optimizing for things that aren't practically relevant for the large majority of early/mid-stage startups. Am I familiar with these concepts—e.g. can I go back and refresh myself when they come up?—absolutely. But often times the skills you'd want in an engineer are: 1. Knowing when to optimize 2. Knowing how to profile and identify bottlenecks 3. Familiarity with the available solutions 4. Ability to dig in and evaluate which is the right tool for the jon
• zinclozenge 8y ago
  The less talked about absurdity is that successfully passing programming interviews is a skill itself. It's especially absurd because the time I spend developing that skill is less time spent developing skills and knowledge more directly relevant to my job. Yet, programming interview skill is more relevant to progressing my career. edit: now if you'll excuse me, I need to do some dynamic programming problems.
  • griffinkelly 8y ago
    I've worked with some folks who were decent engineers who originally didn't get offers from FANG companies because they blew the interview; they later studied their ass off on leetcode and got offers. This did not make them better engineers at all. Ultimately, its just studying for the test, very much like the ACT/SAT in high school. You can be great at taking tests but ultimately a terrible student or vice versa.
    • fossuser 8y ago
      It feels very much like that to me which is why it’s strange there isn’t a Kaplan equivalent. There’s interview cake and leetcode, but I think people would pay $2k for a class that focuses on the questions and in person whiteboard practice. They could collect information about the interviews at the major companies and then use those to create the program. For payment could also help candidates negotiate and then take a cut of the signing bonus. If this isn’t part of lambda school already I think it should be. I already work at a competitive tech company, but I’d sign up for this in a second to help me stay competitive for interviews.
      • austenallred 8y ago
        > If this isn’t part of lambda school already I think it should be It’s part of classes for our existing students, but it isn’t a single offering. One day.
        • fossuser 8y ago
• gammateam 8y ago
  I don't give questions like this primarily because: - Workers are going to be around for 15 months or less and they have domain expertise on 1 stack already and I don't need to screen for how they would hypothetically function across all stacks - Worker's process and resource finding skills are more indicative of the time they will spend on a task - Worker's process includes collaborate use of version control and code reviews, if they pass the screening but can't really integrate on these things then thats what will get them booted from the team It isn't always more expensive to have a not great developer. Look in your organization and see if what I experience is true for you, and you'll save everyone a lot of time.
• deleted 8y ago
  [deleted]
• davidhyde 8y ago
  I think that asking a candidate to perform a code review can be an effective method of evaluating quite a few desirable qualities. Can they understand someone else's code? Can they engage in constructive critical discussion? Are they able to effectively refactor something to make it better? Can they spot mistakes and do they have an opinion about how to avoid such mistakes?
  • djhworld 8y ago
    Yeah that's great but what if they can't reverse a binary tree?
    • almost_usual 8y ago
      You spend a lot of work days reversing binary trees?
      • reddit_clone 8y ago
        I think he was being facetious. :-)
    • WilliamEdward 8y ago
      I know you're joking but... If i had never heard of a binary tree, saw it for the first time in my interview, and didn't at least have an idea of how to attempt solving it, I'm probably a lousy hire. The quality of attempt is what these hard questions are really about.
    • abhishekjha 8y ago
      Looks like I am gonna have to look at the solution as I don't fully get how to achieve that structure. Recursion is good for a few standard problems but those are just standard problems. We shouldn't try doing every possible thing using recursion, even if it is possible to do so. Also, debugging an iterative solutions comes naturally to me compared to recursion. I guess I don't have the right mental model for how to track recursion/recursive calls.
  • avip 8y ago
• antibland 8y ago
  A few jobs back I was tasked with hiring new developers to bolster a thin front-end team. The job was very CSS/JavaScript heavy, so I asked questions that were pertinent to what the candidate would be doing if hired. Of the five candidates, only one answered all the questions perfectly, and he turned out to be the biggest bust for us. The other candidates, after answering some of the harder questions incorrectly, seemed very upset with themselves. They knew they were cracking a bit under pressure, but actually showed that they knew the answers when we chatted further. I hired 3/4 of those people because of how well I felt they'd do given the opportunity. All three became leads within a year and a half. I think personality has a lot to do with outcomes. If you are someone who shows they are hungry to learn and knows how to improve their skills, I will never dismiss you for screwing up a few coding questions.
  • TheOtherHobbes 8y ago
    Personality has a huge amount to do with outcomes. Especially in teams, where the best teams have a mix of complementary talents. If you're hiring for Generic Developer Skills you're going to get generic developers - and much less development than a more flexible approach would give you.
• paul7986 8y ago
  One interview of mine asked something i didn't know yet I said there's no doubt I figured it out via Google. The interview pretty much ended there and I'm glad it did! Any place or interviewer that says you shouldn't use Google of OverFlow to get your work done is no place I want to work for.
  • darpa_escapee 8y ago
    What, you couldn't derive a solution in a vacuum from first principles? I bet you think referring to API documentation isn't cheating, too!
    • paul7986 8y ago
      Hmmm hard decipher your comment. But, that interviewer is the type of developer I don't want to work with. A know it all who only acts like he knows it all and uses Google secretly. Who puts others down to make themselves look good. AKA an insecure P word I want nothing to do with.
      • darpa_escapee 8y ago
        > Hmmm hard decipher your comment. I'm being facetious and agree with your sentiment.
      • chrshawkes 8y ago
        Basically, your typical Brogrammer
• taylodl 8y ago
  Another solution to this problem is contract to hire. I realize this is kicking the can down the road to the contracting firm but hear me out: that's the business the contracting firm is in. They can get really good at their hiring practice since that's their core business. That's not our core business. We've been doing this for the past two years and it's worked out great. Now you can see how well people do the actual job and if you're not satisfied, which happens from time to time, just get someone else.
  • tokyodude 8y ago
    How does this help? You still have to find people to contract and so you still have to screen the contractors. Every one that fails you was a huge waste of your time. You've got to spend time telling them what you want them to do, explaining to them your systems, your APIs, training them to your way of doing things or whatever and if they fail than all your time is in the toilet. The only advantage I can see is you can get rid of them easily by not renewing their contract but that in itself is not a solution to finding good people in the first place. Plus it's likely to be hugely limiting. I'm not going to give up my current job and move to your city for a contract you might cancel in 3 months.
  • pmiller2 8y ago
    Why would I leave my W2 job for your contract to hire position? All I can see here is a company that doesn’t want to fire people who aren’t working out, possibly to save on benefits and unemployment insurance premiums. And why do you have so many people not working out that you have to do this? None of this inspires any confidence.
    • extr 8y ago
      Yeah, in my experience asking the basic question of why I would even entertain moving from my W2 job to a contract position where I assume 100% of the risk is met with dead air on the other end, followed by a hasty assurance that we don't really have to go the contract route. It gives me the feeling that I will be working with people who either don't know their own value and so are willing to agree to this kind of deal, or are so poor performing they don't have the leverage to insist on a regular employment contract.
• minimaxir 8y ago
  This is an interesting post given that up until recently TripleByte's facebook/Twitter ads were asking very nonrealworld programming questions: https://twitter.com/minimaxir/status/1054596563585052673 https://twitter.com/minimaxir/status/1054596563585052673 Now, the ads ask simpler things like floating point precision and function variable scoping (https://www.facebook.com/triplebyte/ads/?ref=page_internal https://www.facebook.com/triplebyte/ads/?ref=page_internal ); legit problems, but not sure if they are an indicator of how good a developer they'd be in the real world working on a CRUD app.
  • ummonk 8y ago
    How is that nonrealworld? Dealing with things like endianness is quite normal for a low level programmer.
    • pmiller2 8y ago
      You nailed it. For someone who needs to determine the endianness of a machine on the fly, the question is trivial to solve; for everyone else, it’s trivial in the sense of “unimportant.”
      • ummonk 8y ago
        Their pre-screening quiz uses a bunch of questions on various technologies to ensure that some subset of the questions tests you on technologies you're familiar with. Which is good, since not everyone is a web engineer.
    • marcinzm 8y ago
      Sure but 99% of the companies they're hiring for probably don't need low level programmers.
    • minimaxir 8y ago
      Nonrealworld for the positions TripleByte typically hires for, anyways. It proxies the reader having a CS degree, which is incidentally what TripleByte's testing is supposed to deemphasize.
• weird 8y ago
  On the interview its build a tic tac toe but when you get to the real job they give you a week to fix a line of code. This industry is fucked up.
• curiousDog 8y ago
  The bigger problem is the inconsistency amongst interviewers when judging candidates. All these articles from TripleByte and Gayle (who've built business on the flaws of interviewing) focus on the questions instead. Doesn't matter how hard the question is if the interviewer knows what they're looking for, are experienced enough, show no nepotism and have good communication skills. My worst interview ever was with Facebook when a non-native, new college grad gave me a Leetcode hard problem in half-broken english and went back to his work without even looking up or walking with me through the problem.
• avatarbl 8y ago
  What alternative would you suggest?
• dudul 8y ago
  This article makes a lot of good points. After going through the "implement a red-black tree on this whiteboard" experience as a more junior dev, I always promised myself I would never use this kind of stupid questions to hire. Now, 13 years later, I mostly rely on "homework" type exercises. I think they address most of the issues. They are more "real world", no time pressure, etc. However, even those now are being heavily criticized. What's left to be used?
  • RomanPushkin 8y ago
    There is a problem with homework, candidate spends couple of hours and you spend couple of minutes to assess candidates. It's not fair. Google doesn't do that, Netflix doesn't do that. Why anyone would believe random startup and spend their weekend unless you're desperate looking for any job?
    • dudul 8y ago
      It is both true and false. Yes, the candidate may spend 1 hour (I don't give difficult homework exercises) while it takes me 15 to 20 minutes to review the submission, but I have to do it for maybe 5 candidates. I totally understand the main criticism for homework, it takes time, the company may never call you back after you poured 2 hours into their stupid exercise. But it is an attempt at fixing all other alternatives: * the onsite whiteboarding is BS * using open source contribution is totally unfair to candidates who don't participate * the "contract for 1 month and then maybe we'll hire you" is also total BS in my book. Who would leave a FT job with benefits for a contract that may end? I'm not sure I can think of any alternative that has 0 drawbacks.
      • RomanPushkin 8y ago
        > the candidate may spend 1 hour > after you poured 2 hours and at the end it turns out to be 5-10 hours. Companies say "you should be able to finish it in 1-2 hours", but it's almost never true. I think alternative is to have 1 hour coding session on a good problem. Most problems are complicated, but there could be something else. I was once asked by Uber to implement timer in JavaScript that will update DOM, also create APIs to stop/start/pause. This kind of a challenge doesn't involve any algorithm, but it shows your ability to code. It should be enough to bring you onsite. But I open source should work. If you contributed to Linux Kernel and it was accepted - onsite no questions asked. If you have github repo of 1000+ stars - onsite no questions asked. If you have already passed Google tech screen and was invited onsite - invite onsite no questions asked.
• skybrian 8y ago
  It seems like this article would be stronger with some examples. What are some examples of good and overly-difficult questions, according to TripleByte?
• kylestlb 8y ago
  I've seen this borne out in practice after administering dozens of phone screens and in-person interviews over the years. I started off with questions that were a little too hard and couldn't get a good read of the candidate unless they happened to have already done the problem on some interview prep site. Switching to more practical, simpler problems allowed me to really observe how they work and solve a coding problem. As the article said, I was also able to add requirements or features to the problem which let me see how the candidate adapts to changing requirements, or refactors their own solution to handle a new edge case. Simpler is generally better if you are timeboxed to 45 minutes.
• echevil 8y ago
  I wonder if hard questions, especially hard algorithm questions, has high false positive rate as well. Being good at solving those questions simply requires you to do tons of practice on leetcode, and those who are willing to spend tons of time are probably those who find it difficult to get jobs (or new grads). I don't think the skills for solving hard algorithm questions correlate well with the actual performance at work. Especially if someone is a new grad, they could be good at thise questions but don't have a clue about how real software systems are built.
• qazqwert 8y ago
  too hard?! please... just becuse it may have been to hard for you dosen't mean all programers are idiots/... at least an example of what is considerd hard should'v been given.... you wanna know how to tell if a programer is good or bad in thir job... give them questions from the eular project... and start from page 6 or so... from questions 200-300-to start with.. after he answerd it all correctly and... he should be able to solve at least 1 problem from page 13... only then hire the man.... just because you may be stupid/... find somthing too hard dosen't mean it is too hard! man...
• innocentoldguy 8y ago
  Most programming interviews are a waste of time and energy for everyone involved and the results are, for the most part, a complete facade. Asking puzzle questions gives interviewers the feeling that they're really testing hard for top talent, when in reality they're just demonstrating their lack of interviewing skills. I recently interviewed managers at two different technology companies, both nationally known, for a report I am working on. Most managers admitted their technical interviews were flawed, but didn't know any other way to assess skills. They also admitted that a significant number of people they recruit refuse to even take the technical challenge and end up working elsewhere. In interviewing a couple dozen engineers, I found most just don't want to waste their evenings and weekends on a technical puzzle for a job, especially when there are a lot of companies out there who don't bother with them, so they end up searching for companies that don't waste their time with technical challenges. Another funny thing I discovered during my research is that just under half of the employees at both companies I've interviewed so far were not able to successfully complete their own technical challenges. Another problem with technical challenges is that often times the interviewer knows less about the topic than the interviewee. I recently went through the interview process with a local technology company who uses Elixir and Go (both of which I know). During the onsite interview, the interviewer kept saying things like, "Don't forget to..." or "You forgot..." I kept explaining that I didn't need to do as he was suggesting. In the end, my code worked, my tests worked, and I passed the interview. In spite of this, I was rejected because the interviewer, "Wasn't feeling it." I still have a lot of research to do, but I haven't found anything, so far, that suggests that technical interviews predictably result in top-talent getting hired. It seems to be the same crapshoot interviewing people without using technical challenges is, because in the end, most people decide within the first couple of minutes if they like someone and hire based on that, regardless of the rest of the interview process.
• ummonk 8y ago
  Isn't this just saying that easier questions are more correlated with overall interview performance? That might be a good thing (a sign of consistency), but it could just as easily be a bad thing (i.e. it doesn't provide unique signal).
• SkyPuncher 8y ago
  The best interviews I've been a part of have been a short cultural style interview followed by a (roughly) 2 hour paired-programming session on a relatively simply to-do application. 1. The problem is pretty well understood (but does offer room for interpretation). 2. Provides time to cover all key aspects (Frontend, Backend, Database, Networking, Debugging, Testing, Caching, etc) in at least some capacity. In particular, it shows you what areas the developers focus on. 3. Provides a more relaxed/realist environment. It's also more accommodating to developers switching stacks - familiar with good programming patterns but not the specifics of stack (e.g. "Here's how I'd do [some specific task] in [other stack]. How do I do it here"). 4. It's clearly a throw away task so there's no concern about "interview labor". It can also be pre-prepped so you don't have to worry about jumping too far in. 5. You can cut short with bad candidates and expand the problem for more complex candidates.
• jamestimmins 8y ago
  Heads up I think there's a typo here: "questions that carrying". Interesting piece!
• crimsonalucard 8y ago
  An example and actual numbers would be a good addition to this article.
• Zelphyr 8y ago
  I see one aspect of this trend of asking programming questions that require a lot of memorization: We have had for the past ~10-15 years people in the workforce (and thus acting as interviewers) who went through a public education system where heavy emphasis was placed on passing tests that required a lot of memorization. I'd be interested to know how many of these interviewers actually think they're able to identify a solid candidate this way? Not to mention, are they even factoring in how many people don't test well but are otherwise superb software engineers? Ultimately it seems like there is a soft element to interviewing that is being tossed out now, which is: do I think we can work with this guy/gal? Are they someone that can become part of our team on a personal level? Can they get good work done? Fizz Buzz can't tell you that. What can tell you that is experience. It's a hard-to-put-your-finger-on-it X-factor that I think companies think they can ignore.
  • paulddraper 8y ago
    > It's a hard-to-put-your-finger-on-it X-factor that I think companies think they can ignore. Devil's advocate: It's also the thing that is very hard measure without bias, unless you do something reductionist like total years of experience.
  • alchemism 8y ago
    > Ultimately it seems like there is a soft element to interviewing that is being tossed out now, which is: do I think we can work with this guy/gal? A colleague offered this simple heuristic for the soft side of [the] interview: "If they are going to be equal or junior teammates, I ask myself, 'Would I feel comfortable sitting in a conference room with this person and hashing out a design or troubleshooting a problem for two hours?' If I'm evaluating someone who is going to be senior to me, or my direct boss, I ask myself, 'Would I feel comfortable following this person's technical instructions if they handed them to me in a document?'" In my personal professional opinion, this measure is more useful to an engineering org. than any "Stump the Chump" style technical-trivia screening.
  • nybblesio 8y ago
    So much this. I've been programming for 30+ years. My brain only has so much cache space and it dumps frequently. Asking me questions that clearly are testing my ability to hold large amounts of data in my meat computer isn't testing my ability to design and write computer programs. Sure, I'd love to have a "mind palace" like Sherlock. Alas, I do not. I often admit this as early in the interview process as possible to avoid wasting time.
• RomanPushkin 8y ago
  Let it stay here https://twitter.com/dhh/status/834146806594433025?lang=en https://twitter.com/dhh/status/834146806594433025?lang=en "Hello, my name is David. I would fail to write bubble sort on a whiteboard"
• draw_down 8y ago
  The question I usually give to interviewees (I don't have latitude in choosing) starts out by reducing a list to one of the items in the list. Subsequent steps of the problem build from that very simple premise, but not by a whole lot. A certain portion of interviewees spend a lot of time making that first part way more complicated than it is, which rushes the rest of what they need to get through. (I'm talking about spending over 20 minutes hemming and hawing over the problem of reducing a list to one item in the list.) It's interesting giving the same problem dozens of times to people working in a variety of languages; you get to see the various aspects that people tend to get hung up on. Picking a sentinel value to compare against, how to get a value out of a dictionary and default to some value if it's not there, that sort of stuff. Devs working in Java, C#, C++, and Go seem to really stumble with these things, I don't know why. A big issue there is that the candidates don't know the shape of the problem before they pick their language. I try to be extremely clear about what the problem is, what the expected output is, and say these things multiple times in very clear language. There are test cases in code and examples in the problem description. But all of that doesn't always help. I guess my point is that as many issues as I have with tech hiring, it is also possible to way overthink the problem, and fail to listen to and understand the person giving the interview. I really want people to succeed and try to give them as much as I can toward that end.
• joker3 8y ago
  There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance among candidates who received an offer. Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other. There's also a question about how to mix question difficulty. Should you ask nothing but easy questions, or is it good to throw in a harder question or two to see how the candidate reacts to something they can't answer? I can see a good interviewer getting a lot of signal out of that, but in the hands of a bad interviewer it would not work well.
  • avip 8y ago
    Yep. Lot's of "data analysis" without even defining what would be a "false positive" (someone who did well in an interview, got an offer but turned out to be a bad hire? how is TB exposed to that critical data exactly?)
    • walamaking 8y ago
      It wasn't clear at all in the article what it even meant when it referred to "final outcome".
    • caspar 8y ago
      IIRC, TripleByte only gets their finders fee once (if) the candidate makes it through 3 months of employment. (Of course, you can argue that 3 months of employment isn't enough - but it's a lot more performance data than a bunch of interviews gives.)
    • ggggtez 8y ago
      If a "bad hire" means fired before 3 months... Then I really have to question exactly how they did this analysis. I can't imagine most companies would have the same objective function.
  • Zelphyr 8y ago
    You touch on a very important part of this equation: there are good interviewers and bad interviewers. A good interviewer, in my opinion and experience, will try to get to know the person first. They'll put the candidate at ease. They want to try as much as possible to be talking to the person they're possibly going to be working with, not the anxious candidate that just walked into the room.
• PopeDotNinja 8y ago
  > However, whether or not a candidate answers a question correctly is not the only source of signal during an interview. You can also evaluate their process by, for example, observing how long it takes them to finish, how clean their code is, and how much they struggle while finding a solution. Our analysis shows that this second source of signal (process) is almost as predictive as the first (correctness). I seem to do OK when an interviewer: - asks me a question that sounds like a real problem, not a contrived one (although on occasion I'll have fun with a contrived puzzle if the interviewer has a sense of humor & makes the process light hearted) - doesn't push me down a path that requires me to implement a "simpler" solution I'd never consider (e.g. asking me a question that clearly wants an O(N) solution & then pushes me to try the O(2^n solution first) - talks like a person with a problem, and not as someone who clearly knows what they want & simply won't say it - doesn't try to "see how I think", because I code as much in my head as I do on a screen, meaning most of the code I throw into a text editor is the latest thought in a stream of random ideas until I get to one that works - doesn't constantly interrupt me - states their actual expectations, such as "I don't expect you to finish, what I'm really looking for is X"
  • kyralis 8y ago
    As a frequent interviewer I will say that you are not alone in that: any interviewer that isn't following the above guidelines is unlikely to get a good signal from anyone. To your final point about expectations: One question that we try to ask ourselves about any particular interview question is "how may bits of information are we getting out of this?" Trivia questions are usually less than 1 - you'll find out if they know some trivia, but you don't honestly really even care. The best questions are those that get you several bits over the course of the interview period, rather than a single yes/no answer at the end. Questions where there is no final end point or where we explicitly do not expect candidates to finish, are often the most effective; we can explore the pathways that are working, dive into side channels that seem interesting, and get data along the way rather than just a checkmark on some individually-useless problem.
    • PopeDotNinja 8y ago
      I like the idea of extracting as many useful bits of information as possible!
• je42 8y ago
  I usually ask what's your strongest language; then ask questions about that programming language. f.e. if it is python: > how would you explain the with statement to a junior developer ? then increasingly difficult questions that go into the language runtime/concepts. one other favourite question of mine is: > Imagine, you got a standard website the serves data from a database. When a customer types in the url into the browser bar what needs to happen until the customer see the website. > Go as deep as you can in the answering the question. When you got an answer, you'll see frontend engineers explain more about the browser, while backend engineers talk more about the backend. There was one very senior engineer, that actually talked about the ethernet layer, he talked for more than 15min. Most medior engineers are done in 5mins. ;)
  • retzkek 8y ago
    > Imagine, you got a standard website the serves data from a database. When a customer types in the url into the browser bar what needs to happen until the customer see the website. That's a favorite of mine too. It sets up a broad range of discussion without needing to spend much time setting up hypotheticals and lets the interviewee really delve into topics for which they have expertise and/or interest. It's also easily adapted or sets up follow-up questions for different positions or levels, e.g. "with TLS," or "how would you debug a problem with symptom x?"
    • je42 8y ago
      exactly that you can have very interesting conversations. in a setting that the interviewee is most comfortable with, since the assumptions/details are determined by them. it put the pressure a bit on you as interviewer, since you need to adapt your questions. but in general i found it quite easy when you have a rough plan on the topics to ask.
  • Consultant32452 8y ago
    I'm like the senior engineer you mentioned. I'll spend as much time as possible on them to see if I can wear out the attention of the interviewer. They usually give up by the time I get to how SSL handshakes work.
• zinckiwi 8y ago
  I like some coding in interviews. I try to make the tasks fairly real-world though. I am a UI engineer so my tasks typically fall into one of two buckets: - A take home (2-3 hours) task for retrieving tabular data from an API and displaying it. Here I'm looking for general framework chops, readability, some design sense. - An in-person (~45m) not-quite-pair programming task, with a real computer, tools, editor etc., for doing a typical UI operation, e.g. truncating text. Starts simple and gets more complex as time allows: make a function to truncate text to x chars; now add an ellipsis only if truncation occurred; now make sure not to truncate in the middle of a word, etc.
• bunderbunder 8y ago
  Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying to leverage that correlation for hiring purposes probably also made it go away. Now you've got a whole lot of people practicing hard on these sorts of problems, spending huge chunks of their free time grinding away on Project Euler and Advent of Code and HackerRank. That muddies the quality of this stuff as a proxy for what it was originally trying to detect: natural aptitude. I'm guessing having time to level grind like that also correlates inversely with other traits that are desirable in a programmer.
  • dfilppi 8y ago
    The tests have become a pure filter, screen out those who dont study heavily prior to taking them. They are uncorrelated from the candidates ability at every day tasks.
    • chillacy 8y ago
      It’s like an industry hazing ritual. How bad do you want it?
      • bunderbunder 8y ago
        Hmm. Y'know, that's a great point. I could build a whole conspiracy theory off of that idea: If I'm a FAANG, I'm simply not using my normal interview process to hire for the really interesting jobs. I reserve those ones for people who got the job by virtue of their publication history in the academic literature, or because they built some well-known cool thing, or because they got promoted into the position. Those people get shunted over into the "you didn't come to us, we came to you" interview process. The seats I'm looking to fill with the more public interview process are mostly seats for the grunt coders who work under those people. My ideal candidate for that position isn't some rock star creative genius; it's a workaholic who is resistant to boredom. And what's something a workaholic who's resistant to boredom would be really good at? Grinding away on programming interview questions, of course.
• dfilppi 8y ago
  I've worked for several companies who made up tests that nobody on staff could pass, including the authors. But the test did imply that the existing staff was superhuman, since no applicant could pass.
• spullara 8y ago
  I totally agree with this. You shouldn't ask a question where some amount of luck of figuring out the problem is necessary. Instead ask something practical and relevant to your domain. Also, leave room for exploration if they sail through it and recovery for someone that goes down a bad path. https://github.com/spullara/interviewcode https://github.com/spullara/interviewcode
• cbau 8y ago
  Interesting post! I appreciate this being shared. Would Triplebyte mind sharing the data? I'd love if the numbers could speak for themselves, rather than having to rely on an interpretation of the numbers.
• rajeshmr 8y ago
  You realize that this approach is flawed the moment there are blogposts / books ( e.g. Cracking the coding interview ) on how to crack it. The how to crack 'x' becomes a field altogether ( coaching / youtube videos / blogs / books etc ). Also, platforms like hackerrank are adding fuel to fire. I read the CEO write somewhere that he wished the below "were taught in schools : 1) Communicating complex ideas with clarity 2) Systems thinking 3) Grunt work tasks 4) Boundaryless thinking 5) Self-awareness / EQ" Please note almost all of these are not evaluated on their platform (they profit from coding tests) or during interviews and almost all are soft / intangible skills (skills which are not immediately obvious about the candidate during a typical programming interview). [ Side note : some could say that coding tests are the problem they have chosen to solve - in which case, why are they worried about these skills ? Are the companies seeing coders crack the tests on their platform, while not performing well on the above skills post hiring ? We could only speculate. ] All good work is done by teams, and to be effective in a team requires a lot of intangibles which aren't even assessed in a typical interview. A better approach could be from this article: https://leerob.io/blog/technical-recruiting-is-broken/ https://leerob.io/blog/technical-recruiting-is-broken/ Or : A couple of weeks of work with a task being assigned and the mentor or interviewer looking at how the candidate is approaching the problem and whether he is able to solve the problem within the time constraints (an easy task shouldn't take long, and a hard problem shouldn't be short circuited to give a sub-optimal solution.) and other such observable traits can be evaluated. My two cents! EDIT : Poor wording above (i.e., couple of weeks). A task should be assigned and evaluated post a time (which is ideal for task completion as per the interviewer). No constant interaction with the candidate and spending loads of time with that candidate - that isn't scalable when the demand-supply equation is imbalanced already. EDIT 2 : The idea above is not about spending weeks for recruitment. The idea was about being practical about the kind of questions / tasks that are given during interview (example : code a feature or fix an issue we have, as another user has suggested well in the comments). Took me a while to realize we have missed the point I tried to convey for the logistics of how it should be done.
  • thoman23 8y ago
    > "A couple of weeks of work with a task being assigned" lol, OK
• gumby 8y ago
  I really agree. At companies I've worked at for the last 20 years (at least) the approach has been to ask pretty simple questions; people who struggle on those we don't want; people who don't struggle, even when they make silly mistakes (due to feeling interview pressure) are good. Questions about the standard library of the programming language in question are good. Questions about the dusty corners of said library: bad. And don't ask about floating point: most likely the candidate won't really know more than the usual things; anybody who really does understand them will probably give answers over the interviewer's head :-).
• deleted 8y ago
  [deleted]
• high_derivative 8y ago
  Having recently gone through some interviewing (for machine learning research), a very cynical but overlooked aspect (in this thread) is the following: After receiving an offer from a big tech company, the interviewing process has already completely turned me off from the idea of working there. Now despite this being a dream position for many and me having no alternative but to take it currently, the smug interviewers have already gotten me in the corporate mind-set: No matter, the reputation and salary, treat it like any other job, do not bother being loyal - they will not be. So the terrible interview process has at least the advantage of reminding future employees what they are signing up for. I wonder what kind of psychological filtering is at play. Do employees feel loyalty after an interview process that is best described as hazing? Are they projecting the humiliation they experienced when interviewing future employees? That's always been my impression.
  • seanmcdirmid 8y ago
    Many interview processes generally paint the company in a grimmer picture than it actually is. So I would do additional research than turning down an offer because of a bad interviewing experience. Of course, you are completely in your right to do so, but it might not be the most optimal behavior. Usually, interviewers are just bad at interviewing and aren't really aware of what proper behavior is. It actually isn't easy, nor is it a desirable activity nor do you really get points for it in your performance review.
    • high_derivative 8y ago
      Oh I won't turn down the offer, as said, I have no choice and it would be an irrational career move to do so. The ultra-negative interviewing experience just completely changed the mindset with which I will go into that job. This then got me thinking about the hidden cost of a negative interviewing process.
      • seanmcdirmid 8y ago
        Sure, but in general all of the big company interviewing processes will convey the similar amounts of dread these days. They are so worried about false positives and have so little concern about false negatives that they push many candidates away simply by not focusing enough on the "why should I work here" part of the interview.
• wiremine 8y ago
  At my consultancy we recently streamlined our interview process: 1. Phone screen which takes 15 or 20 minutes. 2. The candidate fills out an essay, including showing us some code they're proud of. 3. If the essay ticks the boxes we conduct a 1 hour on site interview. We use the same a set of questions for every candidate, so the investment is easy to manage, and our team has a shared set of expectations on what is good or bad. 4. If the interview goes well, we give them a take home assignment. Takes between 2 and 6 hours, depending on how experienced the candidate is. Problem is in C and/or Python (or both) 5. We wrap up with a 2 to 3 hour onsite interview. We walk through the assignment and have a deeper conversation about culture and fit. The results have been positive for us: we've made some great hires and weeded out some candidates who weren't a good fit. We've also been able to scale it down to the process we use for interns. The 1 hour interview has some typical programming interview questions, but we wrap them into a real-world example. The goal isn't to prove they know how to program, but more about allowing them to show us how they think/work out a problem.
  • StreamBright 8y ago
    Standard set of questions is a must, some companies do not realise that and surprised when totally random people show up on day 1.
  • gaustin 8y ago
    How do you handle two separate onsites for people that have to travel to get to your office?
  • deleted 8y ago
    [deleted]
  • pmiller2 8y ago
    I’m sure it works out well for you in that the candidates that pass can probably do the job. But, you basically have lost me by the time you’re asking for more than a 20 minute recruiter call, 60 minute phone screen, and a ~5 hour onsite.
  • Zimahl 8y ago
    If you want to do it for junior developers, fine, but as a senior dev, I'm not doing any take home assignment. It's an immediate pass from me. There is very little possibility that you can glean anything from my code other than picking apart it for code review bullshit. I'm not going to waste 2 to 6 hours on this - done it before, never again.
• StreamBright 8y ago
  Is it a surprise to anybody? My favourite example of interviews going bad is a tech company in SF asking to find cross currency arbitrage loops with all known currencies over the phone in the first round hiring data engineers for the ETL team. I am glad I did not pass that round.
• throwawayinter2 8y ago
  Interviews are like taking the SAT or ACT. For better or worse they’re highly standardized and you just need to prepare. There are a bunch of different prep service like InterviewCake.com, Pramp.com, and PracticeCodingInterview.com that are good if you feel like you need more than just leetcode grinding.
• deleted 8y ago
  [deleted]
• Waterluvian 8y ago
  The thing that bothers me about these questions is that there isn't a shortage of practical questions to ask that will test if the candidate can truely contribute to the real problems you're trying to solve. "Right now our roboticists use a hacked together QT based GUI to manage customer robot fleet data. It takes 1-10 minutes to load on a slow network and is hard to add more features to. I know you have far less information than you'd want but walk through your thought process for how you'd replace this system over the next 12 months. We can make assumptions." And then the next 15 mins can be an organic conversation about the problem space. You can direct the conversation into corners most relevant to their potential role: "you mentioned using web tech because we discussed how all usage is across the internet. Can you talk about the merits of Http vs. websocket?" "How would you ensure that we don't accidentally take every single customer offline if we centralised our data store?" "What kinds of UI technology would lend itself to robot mapping? Can we just use Google Maps?" If you really need to dig deeper into technical prowess, find something relevant in your conversation and dig deep into it. "We talked about saving changes to floor layout. Can you whiteboard/laptop how you might implement undo/redo for floor elements?"
• forrestthewoods 8y ago
  I’d love for this article to provide examples of what they consider “hard” and “easy”.
• acroback 8y ago
  What is important to employers? 1. A candidate who can solve puzzles but is not willing to do the dirty work with team, solving production issues, doing debugging, bug fixing with usual stuff. 2. Another Candidate, who is willing to learn, is ready to work with team and do the dirty work. I am on a hiring committee as a Tech Lead, and I always try to weed out 1. Works great, we hire as interns and then assess them. Someone from Google who we hired full time, was detrimental to team's morale, grunting, complaining about code, complaining about food and what not. Another experienced smartass was self centered on his skills and didn't want to teach junior engineers anything or willing to admit he needs to update his skills. The moment he realized his skills have no values, started attending pointless conferences. Now his LinkedIn profile has "aware of block chain technology", "attended machine learning seminars". I said not to interviewing at Google and FB because I don't have cycles to spend months on leetcode. Did I erred? Perhaps. But I am sure neither can provide me same work quality I execute in my current mid size company. I regret nothing :).
  • pmiller2 8y ago
    I didn’t interview with Facebook or Google either, both because I didn’t want to grind leetcode, and because those companies are pure evil.
    • acroback 8y ago
      For me evil is not what I consider. Its pure business, if other party don't need for my skillset, I aint gonna put my time to change my skillset to suit theirs.
• graphememes 8y ago
  I ask what they've built, and how it was done. This shows you way more than what a technical question can answer. There are edge cases to this. Expertise positions specifically.
• dilyevsky 8y ago
  I’d agree but on the condition interviewer actually know ehat they are doing. As I gained more experience (300 interviews and counting, baby) I realized that I pick up more on candidates skills that are not directly related to their performance on particular question. At this point the question itself is just a conversation starter and so it’s better if it’s simpler and more broad because it leaves lots more avenues for the conversation to go.
• otabdeveloper2 8y ago
  That's because professional outcomes for programmers aren't related to aptitude. Nobody ever got promoted for writing good code. (But the reverse is very common.)
• duxup 8y ago
  I changed careers from another technical non programming career to web development. I got a handful of interviews but I never felt like I was showing them what I could do... learn. Until one finally gave me a fairly straightforward homework style project. It was probably more apt for a noob and I threw myself into it and submitted my work and explained what it over the phone. I was in the office the next day and we talked about it and I had an offer. The homework style interviews are understandably controversial, but at least I got to show my work, me doing my work, my thought process, outside of a few moments at a keyboard.
• jorblumesea 8y ago
  One issue I've noticed is that the elite companies are trend setters, and their interview methods are being copied by non elite players. Perhaps Facebook needs and wants engineers that can bust out A* on the spot, but I doubt Nordstrom or Starbucks needs that level of talent. This has changed the field where now to even get an average job at an average company you need to study at the new normal, whose interview ideas were designed to look for the top 1% of the field.
• angarg12 8y ago
  I got my yearly review recently and I got very good feedback. At the same time I have been doing Leetcode at home for fun, starting with easy problems, and I get my ass handed to me. I find it hard to reconcile these two experiences. How can I thrive at a top tech company while failing to solve an 'easy' coding challenge. It makes me concerned about what would be of me if I had to look for a new job now.
  • jhasbro 8y ago
    Thats exactly what your top tech company wants. For you not to look for a another job. For you not to try to get a higher salary.
• deleted 8y ago
  [deleted]
• rdiddly 8y ago
  Given the 5th & 6th paragraphs about "correctness signal" and "process signal" it seems like the obvious solution would be to do both, i.e. ask questions that are easy alongside those that are hard. The easy ones are "process questions" and the hard ones are "correctness questions." Or maybe you have a range of difficulty from easy to hard, and each question has a number attached to it from 0 to 1 that reflects its intended use.
• deleted 8y ago
  [deleted]
• dahart 8y ago
  My group likes starting with easy questions and ramping up the difficulty, not to eliminate people with wrong answers, but for two other reasons: first, to see whether people understand the questions and whether they try to make up answers, or ask questions, or say “I don’t know”. Second, to see what the limits & boundaries of their experience is. We know that people don’t know everything, and we measure more for potential than for knowledge, but it’s still useful to understand someone’s experience and exposure level. More important than question difficulty to me is attitude, and I’d love to see whether attitude is measurable and how it compares to later performance, but curiosity and optimism and communication really do go further than right or wrong on math and engineer questions for me. That point might even be tired already, I know people say it all the time, but I’m going to keep saying it because we still have blog posts on question difficulty, when easy vs hard engineering questions are pretty low on my list of what matters when I’m hiring.
  • ehsankia 8y ago
    At the end of the day, the question itself is a tool, not the goal. When interviewing, I look at how the person approaches a problem and works through it. I don't really care if they get to the end or not. People who complain about memorization and difficulty are kinda missing the point. Just like learning math at school isn't really about knowing how to do trig, but being able to think logically and do problem solving.
• esoterica 8y ago
  Why do developers complain so much about hard interview questions that can be supposedly be gamed by studying to the test? Every high paying industry heavily engages in gatekeeping, because the number of people who want to make 400k/year is far larger than the number of 400k/year jobs available. The traditional forms of gatekeeping involve requiring that people have the right personal/familial connections, or have an elite school pedigree (finance, consulting, law), or that they spend several years and half a million dollars in post-college schooling (medicine). The tech industry's preferred form of gatekeeping is asking people to do algorithmic puzzles, which is far tamer and less exclusionary than what other industries do. If you believe that the only thing standing between you and 400k/year is a few dozen hours of practicing leetcode, why are you whining about it instead of taking advantage of the situation to wildly enrich yourself with a fairly modest amount of effort?
  • Gimpei 8y ago
    I think because some people can't train themselves up through the gate. This is not because they can't figure out the leetcode problems, but because they have more difficulty dealing with the kind of pressure you see in interviewing situations than others.
    • ehsankia 8y ago
      If you have a hard time working under pressure, there's sadly not many viable alternatives. Any process that has a very low % success rate will inherently be stressful. Also, being able to perform under stress could be seen as a desirable skill too.
      • erik_seaberg 8y ago
        If you can't handle stress, you can't join an oncall rotation, which would have ruled out eleven years of my career so far.
      • pseudalopex 8y ago
        The pressures of this kind of interview have very little in common with the pressures of the job.
• wolfgke 8y ago
  "Easier interview questions are also less stressful, which is an important advantage. Stress causes candidates to under-perform. But, on the other hand, when candidates are more comfortable, they perform their true best, which actually makes interviews more predictive." I cannot believe that any interview situation is comfortable.
• drblast 8y ago
  Google recruiters call me a lot. I think I'd do a good if not stellar job working there. I've passed multiple FAANG interviews and been very successful as a senior developer. In my email I have an "interview prep packet" from them that essentially tells me to brush up on algorithms and read Cracking the Coding Interview to prepare for their interview process. I'm fairly happy in my job. If they offered more money or a really interesting project I'd consider working for them. But I'm pretty lazy about redo-ing college algorithms class during my free time at home to go work there, so I probably won't. There's an opportunity cost with interviews like this where an M.S. and long career of getting shit done counts for very little and memorization of undergrad level topics that you can look up in two minutes in Knuth if you have a problem that requires it can make or break an interview. I've made a career fixing a ton of horribly shitty, inefficient code that's been produced exclusively by people who pass these interviews.
  • trishume 8y ago
    I've interviewed with Google and the interviews I got didn't really require having memorized algorithms. Maybe on the level of breadth first and binary search which are pretty basic. But I don't think most interviews used any algorithms from a book at all. I think their interview prep packet overstates the amount of knowledge expected.
    • Ensorceled 8y ago
      We ... had different experiences. It was a day of implementing algorithms on a whiteboard. I was supposed to be interviewing for a management position so I didn’t get the study packet either.
      • koube 8y ago
        I also interviewed at google, in the in-person all of the algos I got were very fair in my opinion, in that it took a little bit of thought to figure out what I needed to do, and then after that, most of the work was just turning that idea into an algorithm. It was all on the whiteboard but I thought it was fair. They were fine with little errors and we talked through them. On the phone interview I was asked how to something I had no idea how to do. I have no idea how I passed the phone interview. Seems like a mixed bag, the question as far as I know is completely up to the interviewer, which makes the interview very subjective when they don't use a normal question.
• crispytx 8y ago
  I took a technical interview today over the phone, and you know what, they were pretty fair with their questions. I don't know what kind of competition I'm up against for the position, but very fair questions.
• dmolony 8y ago
  This whole rock star/ninja interview process is ridiculous. They need to paint a house but they want to hire Michelangelo.
• insie5 8y ago
  I'm like 0/15 on interviews in the last 2 months. more than half of them were with foreigners who i could barley understand.
• g9yuayon 8y ago
  I just finished reading Bill Kilday's Never Lost Again. It was amazing that four engineers produced the ground-breaking Google Maps in 16 months. Algorithm/Math questions used to be really effective at finding engineerings like those four engineers. There was a reason that Microsoft resorted to algorithm puzzles in its hay days as well. It's just unfortunate that there's so much prepping materials online nowadays that the programming puzzles have become ineffective. It gets worse as many interviewers were not good enough to ask follow-up questions. For instance, addition with big integers is a pretty easy interview question, right? But if a candidate can go as deep as this article: https://bearssl.org/bigint.html https://bearssl.org/bigint.html, I can be pretty confident that the candidate is really really good. That said, I personally don't find it necessary to join the rat race. Instead, I'd suggest engineers just take time to thoroughly study just one book on algorithm designs. In fact, an introductory book, such as Kleinberg's Algorithm Design or Udi Manber's Introduction to Algorithms, will be good enough. It may not get you into Google, but it will likely get you into another damn good company. The best part of this approach is that passing interview is really just the byproduct of you trying to become a better engineer.
• alboy 8y ago
  I always thought that the primary quality measured by this popular interviewing style is "being-like-your-interviewer" which leads naturally to the avalanche effect due to which the practice becomes even more popular within the company and simply common sense after a while. The interesting part, to me, is how exactly similar the methodologies should be for companies different by size and domain. Even if you assume that some flavour of technical interview worked for Microfaceboogle, it might as well kill a copycat company (or may be completely irrelevant to its success or failure).
• harry8 8y ago
  Where I earn my money is thinking of a solution to a hard problem while not actually at work. It probably only comes up a few times in a year. Questions you can answer immediately simply don't test if you can do this. I have no idea how you would test to see if you can do this.
• kappi 8y ago
  Another use of these kind of interviews is age discrimination to filter out of older candidates who don't have time or motivation to prepare for this BS. This is just another fad.. In late 90s there was fad for MSFT style puzzle interviews..then it went out of fashion.. this will also go out of trend.. just wait for couple of more years.
• sytelus 8y ago
  If you are asking interview questions that has one beautiful precise answer, you are doing it wrong. Good interview question should start with something very simple that even very beginner can think and answer, then gradually add complexity and constraints little by little. Example: 1. Write function that multiplies two integers. 2. What if these numbers were real numbers but computer can only operate on integers? How do we use same number of bytes as ints to hold a real number? 3. What if I wanted infinite precision? What would be run time of your algorithm and storage complexity? (don't insist that candidate must hit the known optimal). 4. Can I have complex numbers as well? 5. Imagine complex numbers not only has "i" but also "j" and "k". How do we handle this? It is astonishing how many candidates won't be able to move past #2. The key is to look at how candidate approaches handling complexity, create representations and use it to craft clean solutions. Whether they eventually arrive at known optimal/great answers is unimportant.
  • Osiris 8y ago
    I couldn't get past #2. Does that mean I'm a bad programmer or just missing the experience for the particular field? My point is that questions should be tailored to the work the programmer is expected to perform.
    • sytelus 8y ago
      Above example is not a test of your programming skills. May be you are great NodeJS ninja who doesn't need to care about any of this. I think there is two types of hiring that often happens (1) for a specific project, specific task (2) long term member of the mission that company has. For #2, I prefer to hire people who are generalist problem solvers. They need very strong coding skills but more than that they need to be able to operate in new domains easily and adapt. If I was running a startup, I would need them to work on MySQL database one day and react-native stuff other day and perhaps also pick up some of deep learning practitioner skills two months down the line. Above example question doesn't expect candidate to be familiar with floating point representation but it is interesting to see what they might have thought if they were the ones doing it. Assuming everyone works with numbers all the time, people are hopefully familiar with basic issues of precision, rounding etc.
• austincheney 8y ago
  Hold on, who is qualifying the outcome? At one of my jobs I nailed it as the top candidate, by far, of the 79 people they interviewed in person. The interviewers were looking for competence, freedom from frameworks, experience and so forth. I have in this line of work more than 20 years and do it as a hobby so I nailed the interview. At the job though I worked with a bunch of fresh juniors who only know how to write code the one way they learned in school. According to them I am shitty developer because I didn’t write code in the one way they understand. Who qualifies the outcome?
• anon84827820 8y ago
  I've never seen so much salt and bitching, all to justify not studying for a couple weeks for an interview.
• ggggtez 8y ago
  >Harder questions ultimately filter out too many qualified candidates to be optimal. This was the key sentence if anyone missed it. "Optimal" for whom, exactly? Not for FAANG certainly. They don't need to worry about filtering too many candidates out, because they have a nearly infinite pool of applicants, and infinite money to conduct a search. They can ask as difficult questions as they want, because they can pass hundreds and thousands of qualified candidates, and still have plenty more where that came from. Edit: If you consider "optimal" to be the expected cost of a hire compared to the expected profit, it is fully plausible that if your margins are big enough, asking hard questions is the most effective way to ensure low false-positives. But as everyone knows, comments on articles about interviews are never about the economics, it's only about human ego of feeling rejected.
• james_s_tayler 8y ago
  Why do we talk about this every week on HN? I'm pretty sure it's either once a week or once every two weeks.