12 ms·
The tech industry has unfortunately adopted the methodology of centralized hackable tests as the canonical gatekeeping method in the form of programming intervi
by pavlov 7y ago
The tech industry has unfortunately adopted the methodology of centralized hackable tests as the canonical gatekeeping method in the form of programming interviews.
Most big tech companies don't care about how good you have been at delivering some value through creating software: they want to see you deliver a very specific type of performance at a whiteboard. Interviewers are given specific math puzzle questions to ask. Interviewees are explicitly told by the same companies' hiring departments that they should aim to hack the system by studying books like "Cracking the Code Interview".
This is an industry that prides itself on supposedly making data-driven decisions through A/B testing. When it comes to hiring people to make those decisions, everybody just plays along to a decades-old script.
- burlesona 7y agoBeing a hiring manager at a big company, I can tell you this is just as frustrating for me as it is for candidates. I hate “leet code” and frankly find algorithmic interviews to be very low signal compared to more practical, open-ended, domain-specific problems. I will say though, the problem is one of “standardization” across an organization where it’s too big for everyone to fit in a room. Suppose you give each team high autonomy to hire whoever they like using whatever “good” process they come up with. 90% of the time this results in good hires. But as you grow, that ten percent of underperforming people becomes large in absolute numbers, and is very painful to deal with. It becomes a real problem when relatively lower performing people end up concentrated on a team, and then start being the hiring gatekeepers for that team, thus multiplying the number of lower performing hires. Later you start having institutional problems when everyone starts to perceive that the engineers in Department A are generally better than the engineers in Department B. Engineers in Department A are more likely to leave if they perceive the company is getting worse at engineering - it becomes a self-fulfilling prophecy. Then you get enormous pressure to come up with standardized testing - aka algorithms on the whiteboard, or some other academic inspired exercise - imposed by higher level leadership that wants to address a genuine problem (skill disparity across the org) but does not know any better way to do it. I think, as PG points out, there may be a real opportunity to innovate here, and probably a big financial opportunity if anyone can figure out how to productize and scale a solution. I struggle to see an easy answer, though. In a utopian universe (for a hiring manager) I’d do something like pay candidates to come on site and work for a week, then make a hire/no-hire decision based on that. But I think that is far too onerous for candidates (and a big company) to have legs.
- lifeisstillgood 7y agoMy solution is smaller companies - that way the underperforming ones can be eliminated more easily.
- TeMPOraL 7y agoA federation. To be honest, if a large company has multiple departments that essentially govern themselves, perhaps they could just give them proper names, as if they were subsidiary companies? Because it seems to me that the mechanism behind the "best performers leave when they feel the average performance of workers in the company goes down" is really bounded by the company name. I.e. people in department of frobnification leave because of the people in department of foobing bars. But people in Frobnificator perhaps wouldn't leave because of people in FooBars.io, even if legally/administratively the two situations were equivalent.
- scarejunba 7y agoNo, it's not that. It's that often, to make a new product you have to have multiple parts of the business work together. Then you will find yourself working with the other department and they'll frustrate you. Some places attempt to solve this by saying that you can use whatever vendors you need to solve the problem but often the best place to solve it is the other department since the solution is infeasible without them. The problem is that often the solution is infeasible with them because the guys who made them good have left and now you only have the lameos. As to why they can't operate as independent business unit? Well, there's the Theory of the Firm to tell you why that doesn't work often.
- alien_ 7y agoAt Amazon, perhaps one of the largest and most successful employers, the solution to this is that when interviewing, everyone's main goal is to hire people better than the current average of the team. To ensure this, someone experienced with interviewing from outside the team/organization, so called bar raiser, is always added to the interviewing team. This person is there to ensure high quality hires and has hiring veto power. The rest of the interviewers are from within the team, likely under pressure to compromise on quality when they're desperate enough to hire someone, but they can't do it unless the bar raiser agreed.
- manca 7y agoThis is exactly what I said in my retweet to PG's essay. Having hackable bad tests in the very tech industry proves the point that artificial tests are still the way of thinking for many and something we collectively need to unlearn. The question is how.
- loh 7y agoWhen I see a company conducting these kinds of tests during interviews, it's a signal to me that they are probably not a very forward-thinking company, and probably not a good fit for someone like me who tends to think outside the box. If it's another cog in their machine that they want, these kinds of interview tests are probably pretty good. But if they're an innovative company trying to change the world, they'd be much better off with a different, more pragmatic approach designed to attract the independent thinkers. Beyond that, in my opinion, nearly anyone with average programming experience can hack a typical programming/whiteboard interview, so these kinds of interviews are definitely not the best way to find the best fit for particular roles. It's a good way to filter out the inexperienced, for sure, but it's a pretty low bar to set, especially if it's a company looking for top tier engineers, people with the ability to see the bigger picture. It's also likely that a great engineer would pass the test with ease, but because their skillset and abilities have been inaccurately assessed due to the poor interview process, the company could have initially assigned them a much more effective role where both the company and the employee could have benefited to a much greater extent, but didn't. Overall, relatively speaking, the "hackable test" approach wastes time and hurts everyone in the long run, both employers and employees. A little investment up front with more personalized, specific interviews can go a long way. I think we should start refusing these impractical interview processes, or at the very least, from an interviewee standpoint, turn down jobs (if you can) and let the potential employer know that their interview process is the reason.
- manca 7y agoI agree with all you said. However, 99% of major tech companies employ exactly the same process of hiring software engineers. I've been on hundreds of interview loops and unfortunately just one bad interview in a loop of 5-6 could make the candidate look like a "bad" fit and thus receive no offer. That particular candidate might be in fact great fit, but the way he was tested could not reveal that. I don't know the solution to this problem. Eliminating tests whatsoever and just talking with the candidate about his experience and probing his knowledge on different topics is not efficient either. There are a lot of talking heads around who when given a simple task fail miserably. We as a tech community need to come up with better ways to assess other people's competency while also making sure those people fit within our company's culture, work efficiently with others and after all create value. This is a hard problem. So, we try to simplify the problem by imposing the "proven" way of finding such people -- give them arbitrary tests and hope they pass them.
- acangiano 7y agoI work at a very large company (IBM) and I refuse to interview people that way. My track record for great hires (both full-timers and interns) is arguably impeccable. No whiteboard or brain teaser quizzes involved. I wrote a little about my interviewing approach in my latest "hiring" post: https://programmingzen.com/new-ibm-internship-positions-in-my-team/ https://programmingzen.com/new-ibm-internship-positions-in-m...
- lacker 7y agoThe tech industry has unfortunately adopted the methodology of centralized hackable tests I think the opposite is the case. Every industry has some sort of test to get into it. But the other industries are the most centralized and hackable ones. The tech industry is doing the best job of any of the top industries of having an application process that is open to skilled people who don't match the "centralized" standard. Your grades in college, the experience on your resume, those are "centralized" tests. You have one resume and you send it out to everyone. When companies give programming tests to applicants, each company is free to measure the applicants' skill in any way they see fit. One might ask math puzzle questions, but another company might give take-home Rails projects. And companies are free to just do a "soft" interview and look at your resume if they want. Facebook hires far more software engineers with no software engineering education than Chevron hires chemical engineers with no chemical engineering education. Plus, starting a company is open to anyone with a credit card to open an AWS account. No interview required. Just make something people want.
- 29athrowaway 7y agoThere was a time where computers weren't as fast and libraries were not as high-level/user friendly, and you required to know these things in order to get things done. Today, unless you are doing things at scale (tiny fraction of startups), you don't need to know how to make things run in the most optimized way possible.
- suyash 7y agoSoftware Engineering interviews are the worst offenders of this “test for grades approach to hire candidates”. On that other hand, programmers are spending huge amount of time, money and energy into hacking these coding interviews just to get a job. Situation is horrible and needs immediate disruption.
- dominotw 7y agoHaha here is the email i have from my facebook interview thats coming up. "Your initial Facebook interview is coming up and we want you to ace it! Here are some tips and resources so you know what to expect and can prepare adequately. Preparing is key. Our initial interview determines whether to continue with a full series of onsite interviews. This initial interview is primarily a coding interview that will take place between you and a Facebook engineer. It's important for any engineer to brush up on their interview skills, coding skills and algorithms. Practice coding on a whiteboard or with pen and paper, and time yourself. Preparing increases your odds significantly! Below are two helpful links to a Facebook Interview Prep Course led by Gayle Laakmann McDowell, author of “Cracking the Coding Interview”. Use password FB_IPS to access the videos. Cracking the Facebook Coding Interview - The Approach Cracking the Facebook Coding Interview - Problem Walk-Through This article provides much more advice about how to prepare: Preparing for your Software Engineering Interview at Facebook Here's a sample problem to get started. Sample Problem Write a function to return if two words are exactly "one edit" away, where an edit is: Inserting one character anywhere in the word (including at the beginning and end) Removing one character Replacing exactly one character Most importantly, you can view this and other Facebook sample interview questions and solutions here. Want more sample questions? Try HackerRank, LeetCode, and CodeLab. Be sure to practice questions in a variety of subjects and difficulty levels."
- JimboOmega 7y agoThis is an initial phone screen and in this context this a valuable approach. This is one level past fizzbuzz. 'Can you code something past "hello world"?' is what they're looking for. There's nothing worse than an onsite with somebody who just cannot code at all.
- larnmar 7y agoThe particular example is bad, though, because there’s room for a massive algorithmic insight. There’s an obvious-enough but terribly inefficient way to do it, and a fancy efficient algorithm that either you know or you don’t. If given this question I would middle through the inefficient way, aware that there must be a better algorithm out there that’s eluding me, and I’d probably get a “vaguely pass” mark, but the kid fresh out of college who has spent the last three months doing leetcode study will regurgitate the correct solution and pass with flying colours. (Once in a while maybe a complete genius takes the test and reinvents the good algorithm on the spot, but only gets the same grade as the diligent student, for his effort.) Too many basic coding screens seem to rely on some algorithmic insight. I would rather give a candidate a complicated-but-straightforward problem (like a super-fizzbuzz that takes half a page to describe,with a bunch of complicated rules and exceptions to those rules) because that’s closer to what a real world programming task looks like. No room for a clever insight, just translate the problem statement, carefully, into code, and test whether it runs correctly.
- johnrob 7y agoAs PG implies, the work environment at big companies consists of more hackable tests - so (ironically) that type of interview might be a proper test.
- kccqzy 7y agoI think testing algorithmic questions is certainly evil, but it's a necessary evil because all other methods are either too time-consuming or unrealistic. You wouldn't believe in a big company how many clearly unqualified applicants there are, and there has to be some quick and dirty rule to run a filter through them. Perhaps it's the aura of FANG companies that make unqualified applicants try because they have nothing to lose. My experience is that just about 1% of interviewees can actually confidently write a depth first search and explain its time complexity. If they can't even do that (which is basically among the first things schools teach in a CS curriculum), how do you trust they that they have the skills to design and implement even more complicated systems?
- eggsmediumrare 7y agoI've never done that (self-taught dev), but I've set up SSO, terraformed production systems, wrote ETLs for system critical data... I'm not an amazing developer, but I get things done. That test would filter me out simply because I've never tried it before.
- kccqzy 7y agoI totally understand. Different job expectations. Whenever I tell people I actually need to write dynamic programming in my job people are always incredulous. To many, dynamic programming is a technique they learn only to pass interviews and not use in real life. It's really that if you don't know certain algorithmic stuff, you can't actually work on the lowest levels of the stack, but not all companies expect their developer to do that.
- Izkata 7y ago> I think testing algorithmic questions is certainly evil, but it's a necessary evil because all other methods are either too time-consuming or unrealistic. > write a depth first search and explain its time complexity. Part of our test goes "here's a function, in plain English what does it do?", and then "there's a few bugs in this implementation, what are they and how can they be fixed?" I think there's a lot of alternatives to traditional algorithmic questions that just don't usually get considered. This one I liked because it involves things you'd actually do day-to-day: Reading other people's code, figuring out intent, and finding/fixing bugs.
- hkmurakami 7y agoI've been curious for a while about the inflection point between the more informal startup programming interview style and the bigco interviewing style, and while there's obviously variability between orgs, whether it comes with headcount, customer profile, fund raising milestones, etc
- paul7986 7y agoI loathe all types of coding or design challenges when interviewing! I once designed and coded a five page website.. it took awhile, yet never heard back from the company. Thus, Im going to seek out all opportunities that do not force me to waste my time, especially design as it's subject. Overall you liked my portfolio well enough to consider me then lets chat and see if we jive/im a good fit for you/your team/company! So far I've been very fortunate that recruiters reach out on LinkedIn and I rarely have to do such time wasting activities as they vouch for me. Recently dealt with a company who pays parity (everyone makes the same .. no woman, man, etc can negotiate their worth), had five to 6 interviews and 2 to 3 coding/design challenges. WoW I guess they are looking only for a subset of talent who would do all that. People might do that for a FAANg company or when jobs in the field are scarce, but this was no FAANg company and thankfully there is still a good amount of demand!
- areyes 7y agoWhen I started preparing for interviewing, I spent most of my times studying general data structures & algorithms and practicing applying them to different problems. I don't think this is so bad and can lead to a lot of learning you wouldn't do otherwise. The frustrating thing, is how easy it is to hack the test. After one particular interview, I remember talking to a friend about how hard the technical interview was. He told me that he had already seen the problem and knew it would come up because he bought leetcode premium for the interview. Kind of frustrating to spend hours and hours learning data structures and algorithms when the real key to success is getting lucky and memorizing the problem before hand
- quantdev 7y agoLeetcode premium doesn't mean you'll see the problem. I suspect the probability of that is low, given these problems are constantly changing (there's a banned problem list at Google) and the interviewer pool is as well.
- dunkelheit 7y agoThere are alternatives but they are not necessarily better. E.g. academia seems to rely on references and publications for hiring. This seems closer to your wish of caring about "how good you have been". But consider the downsides - it is inherently more insular (to work for a famous professor you need a reference from someone, preferably also a famous professor) and also your thesis advisor can easily ruin your whole career. Contrast that with our industry where a bright kid out of nowhere can study for the (imperfect, hackable, dehumanizing - I agree!) interview and have a realistic shot at that job at FAANG. At least I wish it still works this way, although of course it helps to be a Stanford graduate!
- ericsoderstrom 7y ago> Most big tech companies don't care about how good you have been at delivering some value through creating software: they want to see you deliver a very specific type of performance at a whiteboard PG specifically points out that success within a large tech company is predicated on bogus hackable tests. So it makes sense that their selection criteria should also be a hackable test. The better you do on that artificial selection criterion for admission, the better you're likely to do on the artificial selection criteria of internal career progression. So in this particular case, the hackable admission test is a good proxy.
- 7402 7y agoI think I blame Google, because it was started by a couple of graduate students. I don't know how it was at Stanford, but at Berkeley in Physics there was a series of big honking tests ("Qualifying Exams") that dominated our attention the first two years. You were expected to know all of undergraduate Physics. Fail it, and you're out. Grad students don't know any better. "How do we hire the best people? I know - they'll have to pass a big hard test, just like we did." Then everyone else goes, "How do we hire the best people? I know - we'll do it like Google does, they seem pretty successful." Ugh. No, I don't know that it really went like that. I just suspect it.
- knzhou 7y agoStanford physics PhD student here. We actually don't have qualifying exams, but my unpopular opinion is that we should. Physics is a lot of fun to study. The undergraduate and core graduate curriculum show off all its greatest hits. Why not learn that by heart? I spent two years doing that, it pays off every day in research, and I'd have been happy even if it didn't.
- Mauricio_ 7y agoSure, but what's the alternative? I know big companies do a lot of AB testing with interviews and change their approach over time (Google stooped asking why are manholes round and looking at AST scores because data showed them those things had almost zero correlation with performance).