6 ms·
I've conducted hundreds of FAANG-level coding interviews, and recently tried to run my interview questions through ChatGPT. The model was able to spit out a cor
by angarg12 4y ago
I've conducted hundreds of FAANG-level coding interviews, and recently tried to run my interview questions through ChatGPT. The model was able to spit out a correct, optimal, clean Python solution effortlessly.
Interestingly when I asked a follow up, a harder version of the same problem, ChatGPT spits out code that sort-of looks correct, but is actually nonsense.
So, would it pass the interview? That's difficult to say. Interviews don't happen in a vacuum, you also consider the candidate's thought process, explanations, alternatives, tradeoffs...
Still, I see this is a game changer for this kind of interviews. So long as candidates understand and explain the output code, there is a good chance they would clear the interview. Even if the code is incorrect, it might given them some hints towards the right solution.
So where do we go from here? I always loathed this interview format, and these languages model reinforce it even further. Interview cheating has always been there, but it is generally so rare that it isn't a real concern. However these tools are too effective and easy to use. I can see a real divide between people who use them and people who doesn't. This type of interview might become even more useless at telling good coders apart.
My take is we have 3 choices:
a) Ignore it.
b) Try to fight it.
c) Embrace it.
a) is not an option. b) would make an already pretty dreadful process even more intolerable. My money is on c)
I can envision an interview format where we allow, or even encourage people to use ChatGPT and AlphaCode during the interview, much like you would use your IDE or a search engine. In fact seeing how a candidate understands and uses those code snippets can be a very interesting data point.
Either that or scrap leetcode-style interviews altogether.
P.S.: I was thinking about writing a blog post about this, if people think it'd be interesting.
- red_admiral 4y agoBy the time someone gets an on-premises interview, you'd notice if they're using a bot or not? Or are all the interview rounds remote these days?
- exceptione 4y agoI would expect A, because it is an option. Like in mathematics or other tests students are not allowed to use the internet or an advanced calculator in order to test wether they truly comprehend the stuff. If you think that your FAANG-interviews are any good, then just keep them in the format you already have, by making sure applicants cannot use AI during the test. I would have on-sites with pen+paper, whiteboard or a prepped/supervised machine, whatever. Of course, applicants could use AI to train for the interview, but that is not a problem, as long as you test their comprehension.
- AnimalMuppet 4y agoWould training with such an AI help their comprehension? Or hurt it? My money is on "hurt".
- serjester 4y agoGiven the trends towards remote tech jobs, it could be difficult to convince your companies engineers to show up to the office for what’s usually just a weed out leet code interview.
- mywittyname 4y agoProctored tests are common for certification exams. Perhaps there's a future where candidates go to one of these exam centers for one of their technical interviews.
- antipotoad 4y agoOr just a live (remote) programming interview, with no use of Codex or Copilot.
- angarg12 4y agoI'm not a fan of making the process more dreadful than it already is. Forcing people to go to an office and use pen and paper or a whiteboard would do exactly that. I hope companies (including mine!) see the writing on the wall and stop trying to fight the future. This isn't accounting for even just pretending we are testing candidates on anything remotely indicative of on-the-job performance.
- woeirua 4y agoWe're just going to move to on-site interviews only. ChatGPT can't enter the equation then.
- zamalek 4y ago> c) Embrace it. [...] My money is on c) Agreed. I think developers who don't will be rare in a few years time. Just like developers who primarily rely on assemblers (versus compilers) have basically become extinct. Like we sometimes, very rarely, need to inline some assembly, we will sometimes need to use the ol' gray matter to figure out a novel algorithm or something. I believe that avoiding learning these tools could be a existential issue for your present-day job. You don't have to come to depend on them, or use them daily, but you do need to understand how best to use (and not use) them.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- bcrosby95 4y agoIf a company allows ChatGPT to be used on the job, anything but c) seems foolish. If the company doesn't allow it, c) seems like a bad idea.
- time_to_smile 4y agoThe real issue with any of these AI innovations is they really point out places where humans have already started to behave like an AI in the first place. Ever since the emergence of leetcode style interviews I've been shocked at how many people can reproduce leetcode examples, but still fundamentally have no sense of algorithm design outside of the context of a job interview. Programmers with their sights set on acing a FAANG interview will just keep repeating leetcode problems until they start to memorize the common patterns (not the problems themselves of course, but the structure of these type of problems). What's disturbing to me is that I recall far more interesting discussion about algorithms in the era before leetcode dominated everything. The common solution isn't to understand algorithms better, but to become a leetcode solving robot. So it's no surprise to me that AI can pretty easily replicate humans that have tried to turn themselves into robots. We see similar patterns in the art that AI can create. It's very good at replicating a kind of art style of designers trying to turn themselves into design robots.
- codekilla 4y agoAbsolutely spot on. I actually do algorithm design, usually over a period of weeks (at least), and leet code is a joke for the serious algorist (I’m sure I’d fail an interview based on it). Nothing has so clearly illustrated the robotic nature of the leet code expert quite like this result has.
- savingsPossible 4y agoDo tell! What sort of job leads you do design algorithms?
- codekilla 4y agoMathematical Biology/Bioinformatics. We have to think very carefully about every step in the process of extracting information from large, diverse datasets--often writing things from scratch, combining/transforming things in novel ways, and implementing new mathematical ideas efficiently enough to be computable on large datasets.
- 4y ago
- deegles 4y ago> c) Embrace it asking "here's a chatgpt solution to the problem... what's wrong with it?" would be a solid process imo.
- NegativeLatency 4y agoThis would be way more fun
- kristiandupont 4y agoWell, if they still have access to ChatGPT, they can ask that of it as well. It may or may not give a valid answer, just as it does with code.
- mike_hearn 4y agoA big if. A lot of ChatGPT discussions seem to take for granted that it'll always be available/free/priced low enough that ~everyone has access to it. Seems more likely that at some point OpenAI will close it up and put it back behind an API.
- ilaksh 4y agoYou can use text-davinci-003 from the API now and it works better for many things.
- joe_the_user 4y agoThe company itself could have access to the AI for the interview, since idea is lowering the playing field. But even more, it seems likely someone is going to be willing to supply LLM access at not that much more than the cost of computation. I know OpenAI's business is selling access to their stuff at a premium but since current AI is not much more than brute-forcing of massive public data, it doesn't seem like this premium will be justifiable for long.
- Kinrany 4y agoAnd you can give every candidate a new version of it.
- philjohn 4y agoThat seems like it's likely been trained on various examples of FAANG questions that are posted to the likes of leetcode, with solutions often presented. The push for harder version was clever, and that it fell over is no real surprise.
- jgilias 4y agoYour option d) that you don’t even enumerate (scrap leetcode) would be best.
- malandrew 4y agoThis is why I ask questions in a business domain. It requires the programmer to think not only about solving a straightforward clear problem and only worry about Big-O. Instead they need to figure out the problem by asking thoughtful questions about possible business concerns and think about which ones to optimize for. Ambiguity that requires follow up questions for successful isn't going to be addressed by something focused on solving a problem that "thinks" it has all the information to solve the problem.
- neilv 4y ago> I've conducted hundreds of FAANG-level coding interviews, [...] I always loathed this interview format,* That's impressive perseverance. Did that wear on you?
- nsxwolf 4y agoMyself, I've only conducted dozens not hundreds of FAANG-style coding interviews. I also loathe the format, and it does indeed wear on me. I hate every second.
- hsuduebc2 4y agoDamn. What is your success rate? A lot of times it seems to mee that luck and your ability to withstand stress are one of the main factors. Is it something you can teach yourself?
- angarg12 4y agoIf by success rate you mean inclined rate, it varies a lot by level. I'd say it's around 30% for interns (which is by far more lenient) and closer to 5% for Senior level. However it's also common to downlevel senior candidates rather than downright reject. From my side of the table I can confirm that of course luck plays a role. Nevermind the variability of the coding problems themselves, but the variability on the interviewers. Even in my company, that has a formal process and pretense of objectivity, interviewers quality and expectations vary greatly. In short, I'd say what people on the hiring side hate to hear, but interviewing is ultimately a numbers game. I wouldn't take rejections too seriously. The best you can do is to improve your chances with proper prep (leetcode, system design) and interview broadly.
- nsxwolf 4y agoThe illusion of objectivity is strong. We have an official interviewing process. It has a big survey that turns into "metrics". We have a debrief where we talk about the "metrics" as though it was an "evidence based" process. Then you'll hear someone say "Well, he found a solution for the balanced tree problem, but he just didn't seem all that confident doing it, you know?"
- eulers_secret 4y ago> Interview cheating has always been there, but it is generally so rare that it isn't a real concern. You're the person they're trying to fool, so if they cheat successfully you would never know. You've never seen overt displays of bad cheating, which is different. Think you haven't seen a stick insect in years? Likely you have, but just didn't notice it... Cheaters will always be more motivated than those trying to detect them - because everything is on the line for them.
- sebzim4500 4y ago>You're the person they're trying to fool, so if they cheat successfully you would never know. You've never seen overt displays of bad cheating, which is different. I don't think it's widespread at least, since in my experience people that do well in technical zoom interviews do not drastically decrease in apparent competence when we move to in person rounds. People definitely get told interview questions by recruiters though, if you count that as cheating then it is everywhere.
- ilaksh 4y agoBy the way ChatGPT is not even the best model OpenAI has for writing code.
- busyant 4y agoi think you're correct about option c, primarily because we could never force everyone to adopt the other 2 options. option c makes me nervous, but right now I can't see an ai correctly dealing with the ambiguity of the data sets i typically look at.i do a lot of "asking for clarification."
- LawTalkingGuy 4y agoI get the same results from asking for code. Near 100% success with an initial demo, and then pretty tragic performance when making a series of changes to actually fit the concept. I got it to write 3d rotation code, and to switch to using rotors instead of quaternions but when I asked it to rewrite the combine_rotors() method it rewrote it to combine Enigma rotors. I explained its error and it apologized but just wasn't able to go back to the original code and work with it anymore. Interviewing is going to be shaken up but I think some methods are more timeless. I've been giving the subject code instead of asking them to write it for a while now. Largely because I wanted to talk more and watch them type less. We start by discussing the coding challenge guidelines and I leave them vague. They need to understand the goal and what I've left out and suggest those guidelines themselves. Once we agree though, I give them the code. "Here's what's running now." Then I update the goal and we discuss changes. I get them to "whiteboard" certain things, like what a query looks like with their proposed changes or whatever, and we discuss big-O, etc. This, imho, is how whiteboarding is actually used - not to write whole programs but to provide examples and pick them apart. I feel that this would work even if they were using an AI in another window. We're trying to select for developers with common sense and domain knowledge, and who can clearly discuss engineering tradeoffs. Actually making the changes (the coding itself) was a big part of the job and that's decreasing, but imho all the other requirements remain. They'll still need to know how to handle the issues I talked about above, of trying to get the model to write the right code!
- ren_engineer 4y agoAlgorithm interview and competitive programming type questions are probably some of the easiest for GPT to solve because there are a massive number of problems and solutions publicly available for training. The real benefit of AI is somewhat shown in this paper, it effectively solved the problems through brute force generating millions of possible solutions. For real world problems it would be interesting to let GPT generate a bunch of different solutions and push them into a test environment and see which works best. The biggest problem I see is black swan events where AI coded systems work great until something goes wrong and no human truly knows how all the pieces fit together.
- gautamcgoel 4y agoI expect that the reason it gave the correct answer to your first question is simply that it already saw the problem and memorized the solution - there is some empirical evidence that deep neural networks are able to memorize much of their training data.
- angarg12 4y agoPossibly. The problem itself is not in leetcode, but it almost certainly has been leaked somewhere. However the program was able to make a few changes to the code with some prompting, which hints a little bit more smarts than just regurgitating an answer. Nevertheless the point is moot. I've invented completely novel questions (promise!), and saw them leaked online after asking them twice. The process is fundamentally flawed and large language models are just making that glaringly obvious.
- spuz 4y agoOur coding exercise cannot be solved by ChatGPT and I believe is much more effective at evaluating coding ability than leetcode-style questions. We ask candidates to design an object-oriented booking system which requires 3 or 4 different classes to implement. ChatGPT cannot easily do this without heavy prompting from the user at which point, they'd be better off just writing their design down themselves. We want to evaluate candidates based on the kind of work they will actually be doing - not brainteasing O(n) solutions to contrived algorithm questions and so far it's worked very well.
- MetaWhirledPeas 4y ago> cannot be solved by ChatGPT That's great! For now. But tomorrow's coming fast.
- angarg12 4y agoI agree. Our interview has 3 coding rounds, and one of them I call "clean code". For that I ask a straightforward question that requires candidate to define some APIs and write some classes / functions. Then I ask several follow ups, adding or changing requirements. This is by far my favourite question, the one closer to on-the-job coding. It also lends itself well to deep conversations with candidates. But alas I don't own the process and still have to work within the parameters of the company. Whenever possible I ask this kind of questions, but other interviewers will default to leetcode-style rounds.