34 ms·

Testing how hard it is to cheat with ChatGPT in interviews

msaqibdani 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- frfl 3y ago
  A human did not write this comment, right?
  - howon92 3y ago
    My exact thoughts
    - onlyrealcuzzo 3y ago
      User joined 11 months ago, and that's the first (and so far only) comment. Maybe we'll never know for sure.
  - Beijinger 3y ago
    I can't definitively determine whether a specific piece of text was written by a human or generated by a machine. However, the textprovided seems to be coherent and well-written, so it's plausible that a human could have authored it. If you have any specific concerns or reasons to suspect otherwise, please provide more context.
    - Beijinger 3y ago
      Downvoted. You guys have absolutely no sense of humor. Do you? :-)
      - Jensson 3y ago
        Bot comments are usually not allowed, if you post bot content then it should be clearly marked, like in your post write some blurb below showing it is just a joke rather than a bot comment.
        Beijinger 3y ago
        This would destroy the joke, the ambiguity, wouldn't it?
        Jensson 3y ago
        Yes, it totally does!!!! /s No, not if you do it afterwards. That you have to write these jokes like this is common knowledge, this kind of humor doesn't work otherwise on anonymous forums, you need to write below it that it was a joke and what you did, and since sarcasm is so common you have the "/s" end sarcasm tag.
        dullcrisp 3y ago
        I know humor is subjective, but I think this comment demonstrates the opposite of the point you’re making.
      - nickthegreek 3y ago
        We have a sense of humor, but it’s a low effort joke that’s been done a million times. It wasn’t funny, nor original.
- moffkalast 3y ago
  Touche
- nickthegreek 3y ago
  You forgot to copy & paste the final punctuation.
babyshake 3y ago
IMO asking people to not use available tools in interviews is a bad idea, unless you are trying to do a very basic check that someone knows the fundamentals. Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer. Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.
- leeny 3y ago
  I'm not the author (perhaps he'll chime in as well), but I'm the CEO of interviewing.io (we're the ones who ran the experiment). I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints. In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
  - michael_mroczka 3y ago
    I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.
    - pierat 3y ago
      Well that's the rub. There's no way, even for a senior engineer, to know everything. In fact, one of the required skills is "how to ask the question as to elicit the answer in a reasonable amount of time". The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses. But hiding it? Yeah, probably not a good fit.
      - vinni2 3y ago
bluedino 3y ago
The last couple interviews I have had, warned about using ChatGPT. So it must be happening.
m1el 3y ago
I've had a displeasure of interviewing someone who used ChatGPT in a live setting. It was pretty obvious: I ask a short question, and I say that I expect a short answer on which I will expand further. The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question. Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer. Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?
- ptmcc 3y ago
  Yes of course! I'd be happy to answer your short question with a short answer. I look forward to expanding further on the answer, as you previously stated that you expect me to. Jokes aside, something about LLM responses is very uncanny valley and obvious.
  - chewxy 3y ago
    The peppy, upbeat, ultra-American tone that the LLMs produce can be somewhat toned down with good prompting but ultimately, it does stink of the refinement test set.
    - throwaway2037 3y ago
      True. We need an Aussie bogan mode for ChatGPT. Or, Guy Ritchie villian.
- renewiltord 3y ago
  That sounds great, doesn't it? You got powerful negative signal.
  - lcnPylGDnU4H9OF 3y ago
    It sounds like the problem is really that this is the most obvious cheater. Someone better at manipulation and deception might do a better job cheating the interviewer such that they're hired but then be entirely inadequate in their new position.
michael_mroczka 3y ago
I'm the author of this post. Happy to answer questions if you have any. This was such a fascinating experiment!
- dmazzoni 3y ago
  One of the implicit assumptions in your post is that if you ask a custom question, ChatGPT won't do as well. Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose. It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.
  - michael_mroczka 3y ago
    It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions. Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging. As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are. As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.
nostromo 3y ago
I'm glad ChatGPT could be the end of leetcode interviews. I worry though that it'll just be the end of online leetcode interviews and employers will bring people back into the office to interview.
- zeta0134 3y ago
  Would this necessarily be a bad thing? "In the office" could substitute for a video call. I always got the impression that a coding challenge during an interview was much less "did you memorize this solution in advance" and much more indirect, like "what is your general problem solving methodology, do you ask good questions, etc." Maybe I haven't been on the receiving end of enough bad interviews?
- nottorp 3y ago
  I wouldn't mind doing an interview face to face for a fully remote job :) The reverse, yes I would mind.
- doctorpangloss 3y ago
  They'll change the questions. No chance in hell leetcoding is going away. It will be even more important, with even greater ceremony. > employers will bring people back into the office to interview. Nothing stops people from getting the questions they'll be asked ahead of time from insiders, like their friends or a recruiter, which is really how people have been cheating. This is how it is possible to be Google and have identical standards for years but nonetheless observe overall quality of hires go down.
  - timeagain 3y ago
    Based on my experience receiving the questions beforehand is not super likely. Getting other interviewers’ questions beforehand /as another interviewer in the loop/ is already like pulling teeth.
  - nostromo 3y ago
    > They'll change the questions. No chance in hell leetcoding is going away. I'd take that bet. Leetcoding is something that GPT 4 is very good at doing -- and it does it faster than any engineer can type, let alone think.
hijinks 3y ago
as someone that has been remove for 10 years now and interviewed a lot of people. You can 100% tell when someone is reading off a screen and not looking at you during an interview via webcam
- michael_mroczka 3y ago
  I'm not sure if you read the post, but with some of the new cheating tools that exist, they overlay the GPT responses in front of your screen with concise bullet points. You wouldn't even need to look away from your screen or interviewer to cheat. The bullet points are also small enough to where it is incredibly difficult to tell that someone is reading anything - even if they have a webcam enabled and are looking right at you. This, coupled with some interviewers that don't care a lot about the process, it is getting easier for cheaters to slip into places for sure!
nine_zeros 3y ago
I've seen people use chrome extensions like leetbuddy.
qweqwe14 3y ago
The core problem with interviews is that it's basically impossible to tell how well someone's going to perform on the job. It's always possible to grind leetcode or whatever and make it look like you know what you're talking about, by using the model people can just skip that part entirely. Not to mention the fact that some interviewers feel obliged to ask useless cliche questions like "why do you think you are a good fit for this position" yada yada. Not going to be surprised if picking people based on random chance (if they meet basic requirements) is going to actually be better statistically than bombarding them with questions trying to determine if they are good enough. Really feels like we are trying to find a pattern in randomness at that point. Bottom line is that if ChatGPT is actually a problem for the interview process, then the process is just broken.
- svachalek 3y ago
  I think the difference is effort. If someone actually bothers to go grind LeetCode for a couple weeks before the interview, then they have demonstrated some form of persistence and work ethic at a minimum. Someone slow rolling questions with ChatGPT is demonstrating pretty much the opposite.
- adamredwoods 3y ago
  Without a dedicated bar exam, we have little to vet hires against. Everyone is a senior engineer, until they're not. I think the next evolution of technical interviews will be hands-off, talking through problems where the criteria changes on the fly, to prevent typing while talking.
  - maxvt 3y ago
    Or we'll go back to doing more onsite interviews at a whiteboard.
    - michael_mroczka 3y ago
      I can't imagine many companies going back to this. The savings are too shiny to resist
OnionBlender 3y ago
I've wondered about cheating with a friend who can (out of ear shot but can hear the call) type in the question and display the result on a screen the interviewee can see. I often get stuck on leetcode problems and simple hints like "O(n), prefix sum" can make a huge difference. Especially if I haven't seen the problem before or is having a brain fart. I would still need to get good at leetcode, just not _as_ good.
- blharr 3y ago
  Yea, it's always been possible to cheat. Also through searching Google. Its just now, you can use ChatGPT, and it's a lot easier for someone to do so.
locallost 3y ago
Personally I am waiting for deep faked video chats with chatgpt generating the answers. And maybe even questions.
- jacques_chester 3y ago
  ChatGPT gushes apologies if you contradict its answers, that would probably be a reliable tell for now.
  - Nition 3y ago
    I suppose you could catch those like so: "What is one plus three?" candidate frantic typing "It's four" "Are you sure it isn't five?" frantic typing "I apologise, one plus three is five."
tetha 3y ago
Hm, interesting. To me, team fit, curiosity and, depending on the level of seniority I'm looking for, an impression of experience are the most important things in an interview. The latter might look like you could fake it with ChatGPT, but it'd be hard. For example, some time ago I was interviewing an admin with a bit of a monitoring focus and.. it's hard to replicate the amount of trust I gained to the guy when he was like "Oh yeah, munin was ugly AF but it worked.. well. Now we have better tech". I guess that's consistent with the article?
- doubled112 3y ago
  Real world experience sometimes comes across better like that than in technical Q&A. One time in an interview they asked how I felt about systemd. At first I thought it was a technical question, but quickly realized he was just probing to see if we'd get along. I got a job offer that night.
  - ekimekim 3y ago
    I think asking "controversial" tech questions like that can be a great signal, because it steers the conversation towards features of the system and discussion of tradeoffs. If the question is good, then it shouldn't matter what the answer is - the fact they HAVE an opinionated answer and arguments to back it up is the point.
    - tetha 3y ago
      Yeah, and systemd is an excellent example there. I can totally understand the issues of unification there, and very much understand issues with poetterings perfectionist attitude to some issues. But do you know how much time I've spent on shitty, arcane, hand-crafted init-scripts? Containers as a whole would be another great question there. I have a certain class of applications I wouldn't want to run without a container orchestration anymore after a certain scale. But on the other hand, I do have a bunch of systems I'd almost never want to run as containers for serious data.
  - dash488 3y ago
nottorp 3y ago
Well, i've started to use ChatGPT instead of google when looking for quickie examples for something. Mainly because of how bad google has become. It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up". Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT. So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up. Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.
- huytersd 3y ago
  There’s your problem. GPT4 is an order of magnitude better than the free version, there’s no comparison.
- blharr 3y ago
  The problem is its difficult to track what ChatGPT can and can't do. One day it'll give you junk and then an update or different prompt might fix that problem.
  - ilc 3y ago
    That's why you look at what it says with a critical eye. I think of ChatGPT like a pretty smart co-worker. Just because they are smart doesn't mean they are always right.
    - nottorp 3y ago
      I'm using it mostly instead of API documentation. And for stuff I already have an idea of. Don't trust it further than that.
      - ilc 3y ago
        There's times when there's no good docs.... ChatGPT can give you a spot to start from, even if it is wrong. Trusting it blindly is stupid. But, so is trusting any sources without verification.
lkdfjlkdfjlg 3y ago
I've tried a couple of time using ChatGPT on a coding assignment (because.... if I can NOT do it, better right?) and both times I got garbage and ended up doing the coding assignment myself.
rvz 3y ago
This conclusively tells us that the Leetcode grind has been (without any dispute) been gamed to the ground and is no longer an accurate measure of exceptional performance in the role. Even the interviewers would struggle with the questions themselves. Why waste each other's time in the interview when I (if I was the interviewer) can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool. I don't need to test you if you have already made significant contributions in the open. Easy assumptions can be made with the very least: * Has knowledge of Git. * Knows how to code in X language in a large project. * Has done code reviews on other people's code. * Is able to maintain a sophisticated project with external contributors. Everything else beyond that is secondary or optional and it's a very efficient evaluation and hard to fake. When there are too many candidates in the pipeline, Leetcoding them all is a waste of everyone's time. Overall leetcode optimizes to be gamed and is now a solved problem by ChatGPT.
- randmeerkat 3y ago
  > …can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool. Get your hiring done now while you can, when the economy rebounds you won’t be able to hire anyone. Also give your team a raise, because they’ll probably be the first to go once new options open up.
- bpye 3y ago
  Not everyone works in the open. I do have open source side projects and contributions I’ve made on my own time - but almost everything I’ve done at work is closed source.
- leononame 3y ago
  > can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool Not everyone spends their free time contributing (to major nonetheless) to open source projects. There are a lot of great engineers that have enough work on their desks with their day job and there are also plenty of idiots in open source. Asking for relevant projects or asking for GitHub profiles to gauge relevant projects yourself is what people were already doing years ago and it wasn't a great hiring strategy. Turns out judging a software engineers skills is extremely hard.
andrewstuart 3y ago
- Using ChatGPT is not cheating. - Using an IDE is not cheating. - Using StackOverflow is not cheating. - Reading the documentation is not cheating. I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills. I would also expect them to be skilled in using their choice of IDE. I would expect them to know how to use Google and StackOverflow for problem solving. I would expect programmers applying for jobs to use every tool at their disposal to get the job done. If you come to an interview without any AI coding skills you would certainly be marked down. And if I gave you some sort of skills test, then I would expect you to use all of your strongest tools to get the best result you can. When someone is interviewed for a job, the idea is to work out how they would go doing the job, and doing the job of programming means using AI copilots, IDEs, StackOverflow, Google, github, documentation, with the goal being to write code that builds stuff. Its ridiculous to demonise certain tools for what reason - prejudice? Fear? Lack of understanding? There's this idea that when you assess programmers in a job interview they should be assessed whilst stripped of their knowledge tools - absolute bunk. If your recruiting process trips candidates of knowledge tools then you're holding it wrong.
- jddj 3y ago
  The "would" suggests the latter, but are you in this position or is this hypothetical?
  - andrewstuart 3y ago
    I don't understand what you are asking. Are you asking if I am qualified to comment on this topic? I think so yes I have relevant experience in recruiting and programming and job hunting.
    - azemetre 3y ago
      They're asking if you're a hiring manager at a company that does a lot of interviews. We all see people commenting how much leetcode sucks and how it's not realistic, but companies that pay good money still asks leetcode regardless of what the general SWE public thinks. The only public companies I know that give hiring managers a lot of leeway in deciding their subordinates are Netflix and Apple.
p0w3n3d 3y ago
Two cameras interview. One from the laptop, another from the back of the interviewee head showing the whole screen
- hamandcheese 3y ago
  At this point, just do an on-site interview.
p0w3n3d 3y ago
Also AI will make us dumb. Those of us who decide to use AI extensively will get lazy, and brai removes the lazy parts of knowledge as they are no longer needed. Meanwhile AI will learn from internet only based on AI generated text, which as we know, causes AI models to deteriorate. nobody will write anything. The society collapses. We admire and worship big computer and a man who can fix it. Basically a Wizard Of OZ scenario
- Charlescurt 3y ago
  Has the Internet made us dumber?
  - outworlder 3y ago
    Some of us, definitely.
  - smithcoin 3y ago
    100% - it's shortened our attention spans and ability to focus on a single thing.
  - p0w3n3d 3y ago
    Internet made us dumber by overwhelming with information. If we allowed it of course. I consider that a man's brain originally is like a little empty attic, and you have to stock it with such furniture as you choose (Sherlock Holmes) However what I mean is, once we start relying on AI, we will never know things for sure, always will need to have the AI asked and no knowledge or skills will we have on our own
jawr 3y ago
Is it really cheating if they’re allowed to use online tools for their day to day?
ngneer 3y ago
I am fortunate to be in a field that AI has not caught up with. I interview security researchers. Would ChatGPT spot a vulnerability in a function it has never seen before?
- outworlder 3y ago
  You can try to answer that yourself, if your company's policy allows you to feed that function to it.
  - ngneer 3y ago
    I know. It was a hypothetical question. I cannot imagine an LLM reasoning well enough to perform security analysis. I made up the function, so no concern over intellectual property.
willcipriano 3y ago
If I pass the interview with ChatGPT I don't want the job.
justin_oaks 3y ago
> We ended up conducting 37 interviews overall, 32 of which we were able to use (we had to remove 5 because participants didn’t follow directions) Removing interviewees because they don't follow directions seems like a good strategy. And I mean removing them as job candidates, not removing them from the study. It's good to have something in the interview process used explicitly for weeding out people who don't follow directions. Something like "Email us your application, and put the word 'eggplant' somewhere in the subject line. We use it to filter out spam." And then literally delete any subjects that don't have "eggplant" in them.
- michael_mroczka 3y ago
  So true!
SheinhardtWigCo 3y ago
There's an app for that: https://github.com/leetcode-mafia/cheetah https://github.com/leetcode-mafia/cheetah > Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and live coding platform integration.
User23 3y ago
Why not just let the candidate use ChatGPT or autopilot or whatever if it’s permitted at your workplace? It’s not hard to detect if they actually understand what it’s doing. In fact competence at LLM assisted coding is hire signal as far as I’m concerned.
rbut 3y ago
We recently interviewed some candidates and got them to complete an initial DevSkiller test. In the interview I asked each what tools they used to complete the DevSkiller test. All admitted to using ChatGPT or Copilot to complete the test. So now DevSkiller is really just a litmus test to determine whether they can be bothered completing the test in order to get interviewed.
- neilv 3y ago
  Sounds like it's not only whether they can be bothered to do DevSkiller, but whether they're not turned off by your company asking them to use DevSkiller. So it's a hiring filter that only passes cheaters who have low standards.
bspates 3y ago
The most interesting part is that the control group (no cheating) has a ~50% pass rate on random leetcode questions. Tech hiring is so arbitrary.
- esafak 3y ago
  That's not arbitrary. The half who passes are probably better programmers, on average.
jhawk28 3y ago
I interviewed a number of people for a few positions and I never told them that I detected them using ChatGPT. We structured our interviews in 2 parts. The first one was finding a bug. First clue if they were using AI was that they would solve it instantly. Second part was to write something related to our work that had definitive start/end. If they were using AI, they often were able to get something out, but they had no foundation to reason about it and modify it. They would quickly become lost. We always said that they could use whatever "helps" as long as they showed what they were doing on screen. For some reason, only one person openly showed that they were using AI, but that was only because they couldn't figure out how to turn it off in the UI. We didn't disqualify anyone for using AI, we disqualified them because of their dishonesty. If you can't trust someone in an interview, how can you trust them in a remote environment?
- yieldcrv 3y ago
  I just did an interview where the collaborative coding session had an ai assistant in it, just a wrapper around chatgpt that was interesting, upvoting that employer for honesty and pragmatism
- consp 3y ago
  This sounds like the "coding for engineers" course I was a TA of. Everybody copied everybodies code and depending on their effort they either modified the variable names, flow or nothing (including the original authors name). Long story short: asking them to make small changes and then tell us what would happen was a shurefire way to detect the true cheaters and not the lazy people. I also fondly remember triggering float errors in loops so you'd get an extra cycle due to it ending in .999etc instead of 0.
- saiya-jin 3y ago
  Yes basically when interviewing you should be looking for warning signs. CV is as it is, you can't cover any bigger one extensively in that short time, so you poke randomly and go deep. There is no bigger warning sign than outright lying. A normal mature person would ask just before if AI is allowed.
- isaacfrond 3y ago
pcthrowaway 3y ago
For junior level candidates, I'll admit that ChatGPT might make it harder to interview. For senior+ candidates I honestly think the correct approach is to just lean into it though. Encourage them to use ChatGPT at the outset, and select questions that you've already fed to the prompt. When you ask them the question, you can show them ChatGPT's output on a screenshare. The candidate can then talk you through what they like about the answer, as well as where it falls short. A senior-level developer should almost always be capable of improving on any response given by ChatGPT, even when it gives a good solution. And if they're not able to give better output than the current AI tooling, it's a pretty good signal that you'd be better off just using LLMs yourself instead of hiring them.
kmoser 3y ago
Instead of asking the candidate to solve coding challenges, ask them to create a coding challenge. The interviewer can then use ChatGPT to see if the coding challenge the candidate wrote was already known to ChatGPT or if it was indeed created from whole cloth by the candidate. Example: Interviewer: "Create a coding challenge that requires sorting a CSV file by timestamp. Require the timestamps to be in some weird, nonstandard format and describe the format. Provide a few entries in the sample data that contain a timestamp which is ambiguous."
- echelon 3y ago
  I did lots of hiring (500+) in my previous company. The challenge is to develop an interview that gives a wide dynamic range of signal, a lot of opportunities for the candidate to shine, easy escape hatches for different skill sets (or even different problems), and a sequence of achievable milestones. This is not something you can do in an hour or without play-testing your interview problem.
aryehof 3y ago
I’d suggest the wrong questions are being asked. Interview for understanding not just information.
edanm 3y ago
I'm not sure if I missed it, but why do they call using ChatGPT "cheating"? It's only "cheating" if you are explicitly asked not to use ChatGTP. (Also, not sure if it really counts as cheating, wouldn't it be more like fraud?) Some interviewers wouldn't mind, or would even encourage, using all available tools to solve problems.
- notfed 3y ago
  For software engineering interviews (all kinds of interviews/tests?), using any outside resource should be assumed to be cheating by default, unless you've asked for, or been given, permission.
  - planede 3y ago
    No, I don't think that's reasonable. Any outside resource ranges from pen and paper, a basic calculator to all the books you have, the Internet, ChatGPT and the help of other people. I would say that the only thing that should be assumed implicitly is that it's forbidden to use the help of other people. Anything else should be explicitly laid out. Having said that if some rule is not clear or evident then the interviewee should ask. And they should never be dishonest.
    - ramraj07 3y ago
      Resource here means something with external knowledge. Pen and paper is not a resource in that sense. If you keep an algorithms book and look into it that would be considered cheating as well unless the interviewer pre-agreed that it’s open book so to speak.
  - baq 3y ago
    Quite the contrary - this is the exact opposite of how the job looks like. All these resources should be available and the candidate should get a mark for efficiency of their usage. Not using google and/or chatgpt efficiently lowers the grade.
  - tmtvl 3y ago
mirekrusin 3y ago
What's the point of all of this? ChatGPT (or local/hosted LLMs) should be tools available at workplace nowadays. Interview while using LLMs, wikipedia, google, SO, o'reilly or whatnot should be not only allowed but encouraged. Just have conversation/pair programming like session with gpts open and shared - just how you'd work with that person. That's how they'll work for/with you. Mission. Fucking. Accomplished. [0] [0] https://xkcd.com/810 https://xkcd.com/810
svilen_dobrev 3y ago
yeah.. most interview procedures/ways in recent ~10+years do not check if interviewees can think. Only if they can "remember". Which is akin to education's problem of last few decades.. there are plenty of machines for remembering now. But dumb-grinding is still the expected way in schools.. While, wait, thinking... So all this "cheating" is maybe a (bit delayed) response to the above trend..
- gampleman 3y ago
  I'm curious what education system you are talking about. One of the trends of modern (read post 1900) education was a move to emphasise higher-order skills (such as analysis or critical thinking) over lower-order skills (like memorisation). In some systems this has gotten to the point where kids seem unusually happy about going to school but don't actually learn almost anything while there...
  - sibit 3y ago
    It may not be like everywhere but in the US most of my education (k-12) was memorization. All we did was memorize names, places, dates, formulas, or a series of steps (depending on the class) over the course of ~2 weeks then we'd be tested to see what we could remember. After the test, we'd move on to the next thing and could forget all about what we just memorized. The only classes I can recall that required higher-order skills were some woodworking, CAD, and CNC/machining classes I took while at an off-site vocational school alongside my required high school classes.
    - theultdev 3y ago
      This was my experience as well. Down to the metal fabrication / woodworking being the most critical thinking I did in K-12 other than programming and reading on my own time
    - dansalvato 3y ago
      I went to a high school (in the US) that was full of extremely passionate and engaging teachers. After I graduated, I had a conversation with one of them about the curriculum, and she told me that their number one job was effectively to teach whatever will be on the standardized tests used for college admissions. I've had some amazing teachers whose passion shone through and cultivated students' genuine interest in the subject matter. Unfortunately, there is only so much that can be done when, at the end of the day, your curriculum is designed for the purpose of getting a good standardized test score.
jgilias 3y ago
I really hope this ends up killing LeetCode interviews with fire. The negative assessment of one of the interviewers about a candidate how “he hadn’t prepared to solve even the most basic LeetCode problems” is especially telling. Maybe the candidate had really honed their sudoku solving skills instead.
megamix 3y ago
For one's own sake, use the approach that will lead to self-actualization.
andreagrandi 3y ago
If you are asking questions which can be answered by Chat-GPT, maybe you are asking the wrong questions. GPT is a tool which can legitimately be used to do your job. There are so many things that GPT can't do: take decisions, find the best approach to talk with a human being, resolve conflicts between two members of the team, and last but not least explain why of a certain solution. Is it a coding test? Pair with the candidate. See how they think. Ask yourself: would I enjoy working with this person? And make your own decision.
- Miraltar 3y ago
  GPT is actually great at explaining why it provided a specific solution
  - sanderjd 3y ago
    Definitely. But if you ask a candidate for a solution, and watch them ask chatgpt, and then you ask what the pros and cons are of the solution, and they ask chatgpt to explain them, then you have still learned a lot about the candidate. The key is to be able to see the conversation, just like how you would previously want to see what people were searching or looking up on the web if you allowed that.
black_13 3y ago
[dead]
gregors 3y ago
All's fair in love, war and the job market. Good for them, interviewers have been jerking people around for a long long time. Maybe they should come up with some new material or start having actual conversations.
TrackerFF 3y ago
My takeaway from this is: Companies need to start conducting interviews where tools are available. When everyone is cheating, no-one is cheating. Trying to customize your questions is just a race to the bottom, and will always be an arms race against the LLMs. So, instead, let the candidate use whatever tools they want - in the open, and rather probe them on their thought process.
tnel77 3y ago
I think the perfect interview process is: 1) A couple basic coding questions. FizzBuzz and such. Can they actually solve something basic? 2) Do a real code review with this person. Share your screen and let them review code. Observe what questions they ask and the comments they leave for the author. 3) Ask some design questions. Digging in on how they would design the classes for some new product and purposely throwing a twist in there from time to time. How do they handle this new information and adapt their design? Do they take constructive criticism well? 4) Talking to this person. Are they polite and respectful? You can help someone grow as an engineer, but good luck getting them to be a better coworker if they are rude.
- sensanaty 3y ago
  For point 2 and 3, I had a very pleasant experience interviewing for a Frontend role recently where the CTO just screenshared their actual production site, and asked me some high-level questions about how I'd implement some random elements on the screen. Stuff like a dropdown filter list, a searchbar, customizable list layouts etc. it was nice cause it lends itself naturally to casual conversations about different possible implementations and lets you sort of riff on possible solutions, probably the most enjoyable interview I've ever done.
glonq 3y ago
I interviewed two guys last month who were definitely using some form of real-time assistance on a remote (MS Teams) interview. Their English language skills were atrocious, but their answers were still peppered with ten-dollar words that they couldn't pronounce.

[flagged]

deleted 3y ago
[deleted]
frfl 3y ago
A human did not write this comment, right?
- howon92 3y ago
  My exact thoughts
  - onlyrealcuzzo 3y ago
    User joined 11 months ago, and that's the first (and so far only) comment. Maybe we'll never know for sure.
- Beijinger 3y ago
  I can't definitively determine whether a specific piece of text was written by a human or generated by a machine. However, the textprovided seems to be coherent and well-written, so it's plausible that a human could have authored it. If you have any specific concerns or reasons to suspect otherwise, please provide more context.
  - Beijinger 3y ago
    Downvoted. You guys have absolutely no sense of humor. Do you? :-)
    - Jensson 3y ago
      Bot comments are usually not allowed, if you post bot content then it should be clearly marked, like in your post write some blurb below showing it is just a joke rather than a bot comment.
      - Beijinger 3y ago
        This would destroy the joke, the ambiguity, wouldn't it?
        Jensson 3y ago
        Yes, it totally does!!!! /s No, not if you do it afterwards. That you have to write these jokes like this is common knowledge, this kind of humor doesn't work otherwise on anonymous forums, you need to write below it that it was a joke and what you did, and since sarcasm is so common you have the "/s" end sarcasm tag.
        dullcrisp 3y ago
        I know humor is subjective, but I think this comment demonstrates the opposite of the point you’re making.
    - nickthegreek 3y ago
      We have a sense of humor, but it’s a low effort joke that’s been done a million times. It wasn’t funny, nor original.
moffkalast 3y ago
Touche
nickthegreek 3y ago
You forgot to copy & paste the final punctuation.

IMO asking people to not use available tools in interviews is a bad idea, unless you are trying to do a very basic check that someone knows the fundamentals. Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer. Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.

leeny 3y ago
I'm not the author (perhaps he'll chime in as well), but I'm the CEO of interviewing.io (we're the ones who ran the experiment). I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints. In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
- michael_mroczka 3y ago
  I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.
  - pierat 3y ago
    Well that's the rub. There's no way, even for a senior engineer, to know everything. In fact, one of the required skills is "how to ask the question as to elicit the answer in a reasonable amount of time". The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses. But hiding it? Yeah, probably not a good fit.
    - vinni2 3y ago

bluedino 3y ago

The last couple interviews I have had, warned about using ChatGPT. So it must be happening.

m1el 3y ago

I've had a displeasure of interviewing someone who used ChatGPT in a live setting. It was pretty obvious: I ask a short question, and I say that I expect a short answer on which I will expand further. The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question. Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer. Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?

ptmcc 3y ago
Yes of course! I'd be happy to answer your short question with a short answer. I look forward to expanding further on the answer, as you previously stated that you expect me to. Jokes aside, something about LLM responses is very uncanny valley and obvious.
- chewxy 3y ago
  The peppy, upbeat, ultra-American tone that the LLMs produce can be somewhat toned down with good prompting but ultimately, it does stink of the refinement test set.
  - throwaway2037 3y ago
    True. We need an Aussie bogan mode for ChatGPT. Or, Guy Ritchie villian.
renewiltord 3y ago
That sounds great, doesn't it? You got powerful negative signal.
- lcnPylGDnU4H9OF 3y ago
  It sounds like the problem is really that this is the most obvious cheater. Someone better at manipulation and deception might do a better job cheating the interviewer such that they're hired but then be entirely inadequate in their new position.

michael_mroczka 3y ago

I'm the author of this post. Happy to answer questions if you have any. This was such a fascinating experiment!

dmazzoni 3y ago
One of the implicit assumptions in your post is that if you ask a custom question, ChatGPT won't do as well. Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose. It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.
- michael_mroczka 3y ago
  It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions. Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging. As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are. As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.

nostromo 3y ago

I'm glad ChatGPT could be the end of leetcode interviews. I worry though that it'll just be the end of online leetcode interviews and employers will bring people back into the office to interview.

zeta0134 3y ago
Would this necessarily be a bad thing? "In the office" could substitute for a video call. I always got the impression that a coding challenge during an interview was much less "did you memorize this solution in advance" and much more indirect, like "what is your general problem solving methodology, do you ask good questions, etc." Maybe I haven't been on the receiving end of enough bad interviews?
nottorp 3y ago
I wouldn't mind doing an interview face to face for a fully remote job :) The reverse, yes I would mind.
doctorpangloss 3y ago
They'll change the questions. No chance in hell leetcoding is going away. It will be even more important, with even greater ceremony. > employers will bring people back into the office to interview. Nothing stops people from getting the questions they'll be asked ahead of time from insiders, like their friends or a recruiter, which is really how people have been cheating. This is how it is possible to be Google and have identical standards for years but nonetheless observe overall quality of hires go down.
- timeagain 3y ago
  Based on my experience receiving the questions beforehand is not super likely. Getting other interviewers’ questions beforehand /as another interviewer in the loop/ is already like pulling teeth.
- nostromo 3y ago
  > They'll change the questions. No chance in hell leetcoding is going away. I'd take that bet. Leetcoding is something that GPT 4 is very good at doing -- and it does it faster than any engineer can type, let alone think.

hijinks 3y ago

as someone that has been remove for 10 years now and interviewed a lot of people. You can 100% tell when someone is reading off a screen and not looking at you during an interview via webcam

michael_mroczka 3y ago
I'm not sure if you read the post, but with some of the new cheating tools that exist, they overlay the GPT responses in front of your screen with concise bullet points. You wouldn't even need to look away from your screen or interviewer to cheat. The bullet points are also small enough to where it is incredibly difficult to tell that someone is reading anything - even if they have a webcam enabled and are looking right at you. This, coupled with some interviewers that don't care a lot about the process, it is getting easier for cheaters to slip into places for sure!

nine_zeros 3y ago

I've seen people use chrome extensions like leetbuddy.

qweqwe14 3y ago

The core problem with interviews is that it's basically impossible to tell how well someone's going to perform on the job. It's always possible to grind leetcode or whatever and make it look like you know what you're talking about, by using the model people can just skip that part entirely. Not to mention the fact that some interviewers feel obliged to ask useless cliche questions like "why do you think you are a good fit for this position" yada yada. Not going to be surprised if picking people based on random chance (if they meet basic requirements) is going to actually be better statistically than bombarding them with questions trying to determine if they are good enough. Really feels like we are trying to find a pattern in randomness at that point. Bottom line is that if ChatGPT is actually a problem for the interview process, then the process is just broken.

svachalek 3y ago
I think the difference is effort. If someone actually bothers to go grind LeetCode for a couple weeks before the interview, then they have demonstrated some form of persistence and work ethic at a minimum. Someone slow rolling questions with ChatGPT is demonstrating pretty much the opposite.
adamredwoods 3y ago
Without a dedicated bar exam, we have little to vet hires against. Everyone is a senior engineer, until they're not. I think the next evolution of technical interviews will be hands-off, talking through problems where the criteria changes on the fly, to prevent typing while talking.
- maxvt 3y ago
  Or we'll go back to doing more onsite interviews at a whiteboard.
  - michael_mroczka 3y ago
    I can't imagine many companies going back to this. The savings are too shiny to resist

OnionBlender 3y ago

I've wondered about cheating with a friend who can (out of ear shot but can hear the call) type in the question and display the result on a screen the interviewee can see. I often get stuck on leetcode problems and simple hints like "O(n), prefix sum" can make a huge difference. Especially if I haven't seen the problem before or is having a brain fart. I would still need to get good at leetcode, just not _as_ good.

blharr 3y ago
Yea, it's always been possible to cheat. Also through searching Google. Its just now, you can use ChatGPT, and it's a lot easier for someone to do so.

locallost 3y ago

Personally I am waiting for deep faked video chats with chatgpt generating the answers. And maybe even questions.

jacques_chester 3y ago
ChatGPT gushes apologies if you contradict its answers, that would probably be a reliable tell for now.
- Nition 3y ago
  I suppose you could catch those like so: "What is one plus three?" candidate frantic typing "It's four" "Are you sure it isn't five?" frantic typing "I apologise, one plus three is five."

tetha 3y ago

Hm, interesting. To me, team fit, curiosity and, depending on the level of seniority I'm looking for, an impression of experience are the most important things in an interview. The latter might look like you could fake it with ChatGPT, but it'd be hard. For example, some time ago I was interviewing an admin with a bit of a monitoring focus and.. it's hard to replicate the amount of trust I gained to the guy when he was like "Oh yeah, munin was ugly AF but it worked.. well. Now we have better tech". I guess that's consistent with the article?

doubled112 3y ago
Real world experience sometimes comes across better like that than in technical Q&A. One time in an interview they asked how I felt about systemd. At first I thought it was a technical question, but quickly realized he was just probing to see if we'd get along. I got a job offer that night.
- ekimekim 3y ago
  I think asking "controversial" tech questions like that can be a great signal, because it steers the conversation towards features of the system and discussion of tradeoffs. If the question is good, then it shouldn't matter what the answer is - the fact they HAVE an opinionated answer and arguments to back it up is the point.
  - tetha 3y ago
    Yeah, and systemd is an excellent example there. I can totally understand the issues of unification there, and very much understand issues with poetterings perfectionist attitude to some issues. But do you know how much time I've spent on shitty, arcane, hand-crafted init-scripts? Containers as a whole would be another great question there. I have a certain class of applications I wouldn't want to run without a container orchestration anymore after a certain scale. But on the other hand, I do have a bunch of systems I'd almost never want to run as containers for serious data.
- dash488 3y ago

nottorp 3y ago

Well, i've started to use ChatGPT instead of google when looking for quickie examples for something. Mainly because of how bad google has become. It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up". Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT. So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up. Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.

huytersd 3y ago
There’s your problem. GPT4 is an order of magnitude better than the free version, there’s no comparison.
blharr 3y ago
The problem is its difficult to track what ChatGPT can and can't do. One day it'll give you junk and then an update or different prompt might fix that problem.
- ilc 3y ago
  That's why you look at what it says with a critical eye. I think of ChatGPT like a pretty smart co-worker. Just because they are smart doesn't mean they are always right.
  - nottorp 3y ago
    I'm using it mostly instead of API documentation. And for stuff I already have an idea of. Don't trust it further than that.
    - ilc 3y ago
      There's times when there's no good docs.... ChatGPT can give you a spot to start from, even if it is wrong. Trusting it blindly is stupid. But, so is trusting any sources without verification.

lkdfjlkdfjlg 3y ago

I've tried a couple of time using ChatGPT on a coding assignment (because.... if I can NOT do it, better right?) and both times I got garbage and ended up doing the coding assignment myself.

rvz 3y ago

This conclusively tells us that the Leetcode grind has been (without any dispute) been gamed to the ground and is no longer an accurate measure of exceptional performance in the role. Even the interviewers would struggle with the questions themselves. Why waste each other's time in the interview when I (if I was the interviewer) can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool. I don't need to test you if you have already made significant contributions in the open. Easy assumptions can be made with the very least: * Has knowledge of Git. * Knows how to code in X language in a large project. * Has done code reviews on other people's code. * Is able to maintain a sophisticated project with external contributors. Everything else beyond that is secondary or optional and it's a very efficient evaluation and hard to fake. When there are too many candidates in the pipeline, Leetcoding them all is a waste of everyone's time. Overall leetcode optimizes to be gamed and is now a solved problem by ChatGPT.

randmeerkat 3y ago
> …can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool. Get your hiring done now while you can, when the economy rebounds you won’t be able to hire anyone. Also give your team a raise, because they’ll probably be the first to go once new options open up.
bpye 3y ago
Not everyone works in the open. I do have open source side projects and contributions I’ve made on my own time - but almost everything I’ve done at work is closed source.
leononame 3y ago
> can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool Not everyone spends their free time contributing (to major nonetheless) to open source projects. There are a lot of great engineers that have enough work on their desks with their day job and there are also plenty of idiots in open source. Asking for relevant projects or asking for GitHub profiles to gauge relevant projects yourself is what people were already doing years ago and it wasn't a great hiring strategy. Turns out judging a software engineers skills is extremely hard.

andrewstuart 3y ago

- Using ChatGPT is not cheating. - Using an IDE is not cheating. - Using StackOverflow is not cheating. - Reading the documentation is not cheating. I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills. I would also expect them to be skilled in using their choice of IDE. I would expect them to know how to use Google and StackOverflow for problem solving. I would expect programmers applying for jobs to use every tool at their disposal to get the job done. If you come to an interview without any AI coding skills you would certainly be marked down. And if I gave you some sort of skills test, then I would expect you to use all of your strongest tools to get the best result you can. When someone is interviewed for a job, the idea is to work out how they would go doing the job, and doing the job of programming means using AI copilots, IDEs, StackOverflow, Google, github, documentation, with the goal being to write code that builds stuff. Its ridiculous to demonise certain tools for what reason - prejudice? Fear? Lack of understanding? There's this idea that when you assess programmers in a job interview they should be assessed whilst stripped of their knowledge tools - absolute bunk. If your recruiting process trips candidates of knowledge tools then you're holding it wrong.

jddj 3y ago
The "would" suggests the latter, but are you in this position or is this hypothetical?
- andrewstuart 3y ago
  I don't understand what you are asking. Are you asking if I am qualified to comment on this topic? I think so yes I have relevant experience in recruiting and programming and job hunting.
  - azemetre 3y ago
    They're asking if you're a hiring manager at a company that does a lot of interviews. We all see people commenting how much leetcode sucks and how it's not realistic, but companies that pay good money still asks leetcode regardless of what the general SWE public thinks. The only public companies I know that give hiring managers a lot of leeway in deciding their subordinates are Netflix and Apple.

p0w3n3d 3y ago

Two cameras interview. One from the laptop, another from the back of the interviewee head showing the whole screen

hamandcheese 3y ago
At this point, just do an on-site interview.

p0w3n3d 3y ago

Also AI will make us dumb. Those of us who decide to use AI extensively will get lazy, and brai removes the lazy parts of knowledge as they are no longer needed. Meanwhile AI will learn from internet only based on AI generated text, which as we know, causes AI models to deteriorate. nobody will write anything. The society collapses. We admire and worship big computer and a man who can fix it. Basically a Wizard Of OZ scenario

Charlescurt 3y ago
Has the Internet made us dumber?
- outworlder 3y ago
  Some of us, definitely.
- smithcoin 3y ago
  100% - it's shortened our attention spans and ability to focus on a single thing.
- p0w3n3d 3y ago
  Internet made us dumber by overwhelming with information. If we allowed it of course. I consider that a man's brain originally is like a little empty attic, and you have to stock it with such furniture as you choose (Sherlock Holmes) However what I mean is, once we start relying on AI, we will never know things for sure, always will need to have the AI asked and no knowledge or skills will we have on our own

jawr 3y ago

Is it really cheating if they’re allowed to use online tools for their day to day?

ngneer 3y ago

I am fortunate to be in a field that AI has not caught up with. I interview security researchers. Would ChatGPT spot a vulnerability in a function it has never seen before?

outworlder 3y ago
You can try to answer that yourself, if your company's policy allows you to feed that function to it.
- ngneer 3y ago
  I know. It was a hypothetical question. I cannot imagine an LLM reasoning well enough to perform security analysis. I made up the function, so no concern over intellectual property.

willcipriano 3y ago

If I pass the interview with ChatGPT I don't want the job.

justin_oaks 3y ago

> We ended up conducting 37 interviews overall, 32 of which we were able to use (we had to remove 5 because participants didn’t follow directions) Removing interviewees because they don't follow directions seems like a good strategy. And I mean removing them as job candidates, not removing them from the study. It's good to have something in the interview process used explicitly for weeding out people who don't follow directions. Something like "Email us your application, and put the word 'eggplant' somewhere in the subject line. We use it to filter out spam." And then literally delete any subjects that don't have "eggplant" in them.

michael_mroczka 3y ago
So true!

SheinhardtWigCo 3y ago

There's an app for that: https://github.com/leetcode-mafia/cheetah https://github.com/leetcode-mafia/cheetah > Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and live coding platform integration.

User23 3y ago

Why not just let the candidate use ChatGPT or autopilot or whatever if it’s permitted at your workplace? It’s not hard to detect if they actually understand what it’s doing. In fact competence at LLM assisted coding is hire signal as far as I’m concerned.

rbut 3y ago

We recently interviewed some candidates and got them to complete an initial DevSkiller test. In the interview I asked each what tools they used to complete the DevSkiller test. All admitted to using ChatGPT or Copilot to complete the test. So now DevSkiller is really just a litmus test to determine whether they can be bothered completing the test in order to get interviewed.

neilv 3y ago
Sounds like it's not only whether they can be bothered to do DevSkiller, but whether they're not turned off by your company asking them to use DevSkiller. So it's a hiring filter that only passes cheaters who have low standards.

bspates 3y ago

The most interesting part is that the control group (no cheating) has a ~50% pass rate on random leetcode questions. Tech hiring is so arbitrary.

esafak 3y ago
That's not arbitrary. The half who passes are probably better programmers, on average.

jhawk28 3y ago

I interviewed a number of people for a few positions and I never told them that I detected them using ChatGPT. We structured our interviews in 2 parts. The first one was finding a bug. First clue if they were using AI was that they would solve it instantly. Second part was to write something related to our work that had definitive start/end. If they were using AI, they often were able to get something out, but they had no foundation to reason about it and modify it. They would quickly become lost. We always said that they could use whatever "helps" as long as they showed what they were doing on screen. For some reason, only one person openly showed that they were using AI, but that was only because they couldn't figure out how to turn it off in the UI. We didn't disqualify anyone for using AI, we disqualified them because of their dishonesty. If you can't trust someone in an interview, how can you trust them in a remote environment?

yieldcrv 3y ago
I just did an interview where the collaborative coding session had an ai assistant in it, just a wrapper around chatgpt that was interesting, upvoting that employer for honesty and pragmatism
consp 3y ago
This sounds like the "coding for engineers" course I was a TA of. Everybody copied everybodies code and depending on their effort they either modified the variable names, flow or nothing (including the original authors name). Long story short: asking them to make small changes and then tell us what would happen was a shurefire way to detect the true cheaters and not the lazy people. I also fondly remember triggering float errors in loops so you'd get an extra cycle due to it ending in .999etc instead of 0.
saiya-jin 3y ago
Yes basically when interviewing you should be looking for warning signs. CV is as it is, you can't cover any bigger one extensively in that short time, so you poke randomly and go deep. There is no bigger warning sign than outright lying. A normal mature person would ask just before if AI is allowed.
isaacfrond 3y ago

pcthrowaway 3y ago

For junior level candidates, I'll admit that ChatGPT might make it harder to interview. For senior+ candidates I honestly think the correct approach is to just lean into it though. Encourage them to use ChatGPT at the outset, and select questions that you've already fed to the prompt. When you ask them the question, you can show them ChatGPT's output on a screenshare. The candidate can then talk you through what they like about the answer, as well as where it falls short. A senior-level developer should almost always be capable of improving on any response given by ChatGPT, even when it gives a good solution. And if they're not able to give better output than the current AI tooling, it's a pretty good signal that you'd be better off just using LLMs yourself instead of hiring them.

kmoser 3y ago

Instead of asking the candidate to solve coding challenges, ask them to create a coding challenge. The interviewer can then use ChatGPT to see if the coding challenge the candidate wrote was already known to ChatGPT or if it was indeed created from whole cloth by the candidate. Example: Interviewer: "Create a coding challenge that requires sorting a CSV file by timestamp. Require the timestamps to be in some weird, nonstandard format and describe the format. Provide a few entries in the sample data that contain a timestamp which is ambiguous."

echelon 3y ago
I did lots of hiring (500+) in my previous company. The challenge is to develop an interview that gives a wide dynamic range of signal, a lot of opportunities for the candidate to shine, easy escape hatches for different skill sets (or even different problems), and a sequence of achievable milestones. This is not something you can do in an hour or without play-testing your interview problem.

aryehof 3y ago

I’d suggest the wrong questions are being asked. Interview for understanding not just information.

edanm 3y ago

I'm not sure if I missed it, but why do they call using ChatGPT "cheating"? It's only "cheating" if you are explicitly asked not to use ChatGTP. (Also, not sure if it really counts as cheating, wouldn't it be more like fraud?) Some interviewers wouldn't mind, or would even encourage, using all available tools to solve problems.

notfed 3y ago
For software engineering interviews (all kinds of interviews/tests?), using any outside resource should be assumed to be cheating by default, unless you've asked for, or been given, permission.
- planede 3y ago
  No, I don't think that's reasonable. Any outside resource ranges from pen and paper, a basic calculator to all the books you have, the Internet, ChatGPT and the help of other people. I would say that the only thing that should be assumed implicitly is that it's forbidden to use the help of other people. Anything else should be explicitly laid out. Having said that if some rule is not clear or evident then the interviewee should ask. And they should never be dishonest.
  - ramraj07 3y ago
    Resource here means something with external knowledge. Pen and paper is not a resource in that sense. If you keep an algorithms book and look into it that would be considered cheating as well unless the interviewer pre-agreed that it’s open book so to speak.
- baq 3y ago
  Quite the contrary - this is the exact opposite of how the job looks like. All these resources should be available and the candidate should get a mark for efficiency of their usage. Not using google and/or chatgpt efficiently lowers the grade.
- tmtvl 3y ago

mirekrusin 3y ago

What's the point of all of this? ChatGPT (or local/hosted LLMs) should be tools available at workplace nowadays. Interview while using LLMs, wikipedia, google, SO, o'reilly or whatnot should be not only allowed but encouraged. Just have conversation/pair programming like session with gpts open and shared - just how you'd work with that person. That's how they'll work for/with you. Mission. Fucking. Accomplished. [0] [0] https://xkcd.com/810 https://xkcd.com/810

svilen_dobrev 3y ago

yeah.. most interview procedures/ways in recent ~10+years do not check if interviewees can think. Only if they can "remember". Which is akin to education's problem of last few decades.. there are plenty of machines for remembering now. But dumb-grinding is still the expected way in schools.. While, wait, thinking... So all this "cheating" is maybe a (bit delayed) response to the above trend..

gampleman 3y ago
I'm curious what education system you are talking about. One of the trends of modern (read post 1900) education was a move to emphasise higher-order skills (such as analysis or critical thinking) over lower-order skills (like memorisation). In some systems this has gotten to the point where kids seem unusually happy about going to school but don't actually learn almost anything while there...
- sibit 3y ago
  It may not be like everywhere but in the US most of my education (k-12) was memorization. All we did was memorize names, places, dates, formulas, or a series of steps (depending on the class) over the course of ~2 weeks then we'd be tested to see what we could remember. After the test, we'd move on to the next thing and could forget all about what we just memorized. The only classes I can recall that required higher-order skills were some woodworking, CAD, and CNC/machining classes I took while at an off-site vocational school alongside my required high school classes.
  - theultdev 3y ago
    This was my experience as well. Down to the metal fabrication / woodworking being the most critical thinking I did in K-12 other than programming and reading on my own time
  - dansalvato 3y ago
    I went to a high school (in the US) that was full of extremely passionate and engaging teachers. After I graduated, I had a conversation with one of them about the curriculum, and she told me that their number one job was effectively to teach whatever will be on the standardized tests used for college admissions. I've had some amazing teachers whose passion shone through and cultivated students' genuine interest in the subject matter. Unfortunately, there is only so much that can be done when, at the end of the day, your curriculum is designed for the purpose of getting a good standardized test score.

jgilias 3y ago

I really hope this ends up killing LeetCode interviews with fire. The negative assessment of one of the interviewers about a candidate how “he hadn’t prepared to solve even the most basic LeetCode problems” is especially telling. Maybe the candidate had really honed their sudoku solving skills instead.

megamix 3y ago

For one's own sake, use the approach that will lead to self-actualization.

andreagrandi 3y ago

If you are asking questions which can be answered by Chat-GPT, maybe you are asking the wrong questions. GPT is a tool which can legitimately be used to do your job. There are so many things that GPT can't do: take decisions, find the best approach to talk with a human being, resolve conflicts between two members of the team, and last but not least explain why of a certain solution. Is it a coding test? Pair with the candidate. See how they think. Ask yourself: would I enjoy working with this person? And make your own decision.

Miraltar 3y ago
GPT is actually great at explaining why it provided a specific solution
- sanderjd 3y ago
  Definitely. But if you ask a candidate for a solution, and watch them ask chatgpt, and then you ask what the pros and cons are of the solution, and they ask chatgpt to explain them, then you have still learned a lot about the candidate. The key is to be able to see the conversation, just like how you would previously want to see what people were searching or looking up on the web if you allowed that.

black_13 3y ago

[dead]

gregors 3y ago

All's fair in love, war and the job market. Good for them, interviewers have been jerking people around for a long long time. Maybe they should come up with some new material or start having actual conversations.

TrackerFF 3y ago

My takeaway from this is: Companies need to start conducting interviews where tools are available. When everyone is cheating, no-one is cheating. Trying to customize your questions is just a race to the bottom, and will always be an arms race against the LLMs. So, instead, let the candidate use whatever tools they want - in the open, and rather probe them on their thought process.

tnel77 3y ago

I think the perfect interview process is: 1) A couple basic coding questions. FizzBuzz and such. Can they actually solve something basic? 2) Do a real code review with this person. Share your screen and let them review code. Observe what questions they ask and the comments they leave for the author. 3) Ask some design questions. Digging in on how they would design the classes for some new product and purposely throwing a twist in there from time to time. How do they handle this new information and adapt their design? Do they take constructive criticism well? 4) Talking to this person. Are they polite and respectful? You can help someone grow as an engineer, but good luck getting them to be a better coworker if they are rude.

sensanaty 3y ago
For point 2 and 3, I had a very pleasant experience interviewing for a Frontend role recently where the CTO just screenshared their actual production site, and asked me some high-level questions about how I'd implement some random elements on the screen. Stuff like a dropdown filter list, a searchbar, customizable list layouts etc. it was nice cause it lends itself naturally to casual conversations about different possible implementations and lets you sort of riff on possible solutions, probably the most enjoyable interview I've ever done.

glonq 3y ago

I interviewed two guys last month who were definitely using some form of real-time assistance on a remote (MS Teams) interview. Their English language skills were atrocious, but their answers were still peppered with ten-dollar words that they couldn't pronounce.