5 ms·

The impact of AI on the technical interview process

vunderba 2y ago
TLDR: Another article in favor of code reviews vs leetcode challenges. About a decade ago during an interview, I was shown a piece of code (directly from their code base) which was about 50 lines long and we had a back and forth technical conversation in which the interviewer asked me the following questions: - What does this code appear to do (both technically and more importantly in the context of the business domain of the company)? - Are there are any red flags, edge cases, etc. that this code might not cover? - If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently? This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as: - Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code. - Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc) - Are they able to communicate and convey their thoughts in a coherent and logical manner?
- CharlieDigital 2y ago
  Yup, I went through a similar experience with a YC startup no less. First round was a code review with the CTO/co-founder and it was a web API and SQL schema with lots of little bits and varying "areas of interest" from data validation to exception handling (or lack thereof) to indices to type selection and other bits and pieces. Left a very strong and favorable impression on me for how much more enjoyable it was as a candidate and also how much more it felt like he presented a sandbox within which we could discover and discuss.
- username135 2y ago
  I like this.
- inerte 2y ago
  Meanwhile there's a handful of companies (the one I work at included) using AI to automate code reviews. Sure, these tools are not as good as a real human. Yet. Just like 3 years ago companies were starting to ban candidates to use GitHub Copilot, pretty soon candidates WILL find a way to either have ChatGPT open on the side, or listening to the operating system audio and providing real time suggestions during the interview (yeah I know this can be done today). We either go back doing things on site, or we go crazy and let candidates use whatever they want during an interview. Want to use Cursor and Perplexity + R1? Sure, why not. This is what people are using at work anyway. This probably means interview conclusions are going to be even more nuanced, it can't simply be they solved the problem or not (and the interviewer liked the person), but during the actual conversation things moved organically to whatever direction the topic led to - which is what actually happens at work when pairing. As you can see I have no idea how to "solve" this problem moving on, but one thing is for sure, the bigger the gap between tooling involved in actual work vs the interview, the bigger this problem is. It's one thing to ask people to write code on a whiteboard for 30 minutes (because they "write" code in a text editor), and another thing to ask people to write code in a whiteboard and the actual work is people chatting with AI Agents to produce so much software in 30 minutes that the entire thing would look like side projects 5 years ago.
- pugworthy 2y ago
  Per a separate comment I made, I think you could add to this and call it an AI assisted code review. Not just can the user review the code shown, but can they review the code that AI creates? Yes, AI code is sort of like any other code in that it can have errors, but I think taking it to the next step is whether they can actually modify or follow up with the prompt to get it to do the right thing. AI is kind of like having a really smart and confident intern who's sometimes wrong or doesn't quite understand your request. But you're not a manager or a lead. You're just some junior hire who's got AI saying, "Hey I have this figured out and am very confident in my results" and it's your job to check their work.
pityJuke 2y ago
Off-topic: does the font on this site render horribly for anyone else? Windows, Firefox & Edge.
- tomrod 2y ago
  Fedora, Firefox and Chrome
- low_tech_punk 2y ago
  I think the issue is that the font used by the author does not support fraction pixel sizing. I changed the body font size from 1.2em (16.8px) to 16px and the issues are gone. I'm curious too as for why it happened. I'm guessing the font was originally designed for Japanese, not Latin. See: https://fontmeme.com/fonts/m-plus-rounded-1c-font/ https://fontmeme.com/fonts/m-plus-rounded-1c-font/
  - CharlieDigital 2y ago
    Thanks for that insight!
throwup238 2y ago
> The system design ones were somehow even more outrageous given that system design is often the deciding factor between a mid/senior level and a staff/principal level engineer. Yet watching the IGotAnOffer video series successively reveals just how formulaic and predictable the response and banter is designed to be. Oh my god, yes. Every architectural interview I’ve had in the last five years has been a rehashing of the same concepts, pulled straight out of Designing Data Intensive Applications. Anyone who has read that book once or twice can easily pass. Half the interviewers even use the Twitter example presented in the book, with a few arbitrary extra requirements tacked on.
- tthhrrooww_away 2y ago
  To be fair, DDIA is a pretty deep book, so if you read it, took something out of it, and read it the second time (!), you probably deserve an offer.
- CharlieDigital 2y ago
  There's very little room for deviation as well because it is so formulaic. Present a novel and interesting system? It may not be realistically solvable in a 30-60 minute session to a satisfying degree because the candidate doesn't have familiarity. From the candidate side, you can take a risk and make it more interesting by presenting more novel solutions, but the interviewer might be looking for very specific response patterns; "signaling" of a sort. So both sides just stick to a narrow range of scenarios and responses. Once you've done one or two, it's hard to see how you couldn't do every other one after it.
- satvikpendem 2y ago
  > pulled straight out of Designing Data Intensive Applications I mean, because it works. Why would you create a novel system just for the sake of it?
- superfrank 2y ago
  I think the root of the problem is actually an org design problem. Big companies want consistency in their process and their hires. There's a lot of benefits if you can create a Google type hiring pipeline you're just constantly interviewing candidates and you worry about where they go once they've passed the interview, like: - Since all candidates at the same level are passing essentially the same interview you can move people from team to team easier. You reduce the risk of a team with lower standards causing you problems when their engineers are moved to other places in the company. You know that most of your hires have a certain consistency in their skills. This makes things like reorgs a lot easier. - Your recruiters and hiring managers save time because in most cases they don't need to meet to set up unique hiring pipelines per team. You just have one big hiring pipeline and teams with open spots only step in at the last minute. - You reduce the risk of being sued for bias or discrimination. If interviewers are just pulling from a bank of questions that are somewhat equal it's hard for a candidate to make a case that they were singled out and given a tougher question because of their identity. The problem with that is that consistency breeds predictability. The only way to get a big pool of interview questions that are about the same difficulty is by repeating the same patterns between questions. Once candidates know those patterns, candidates can just focus their studying on that and the interview just boils down to pattern matching.
rvz 2y ago
> Rather than entirely shifting away from existing processes, teams should start to consider incorporating code reviews as a key step in the candidate selection process to identify those with a proficiency to effectively evaluating code. They should indeed shift to code reviews to test the reasoning ability of the candidate's written code rather than already solved leetcode puzzles which is very easy for candidates to use LLMs to cheat. On last resort, I'd use leetcode but focused on the relevant language that I am looking for. (If I want to see the leetcode problem solved in Golang, I do not want to see you use Python to solve it.) I would give the candidate a chance to skip the leetcode puzzles if they are able to demonstrate that they have contributed (at least two) to significant open source project(s) in the relevant technology in the job description and show an example code review of those commits. This gives a public example of a candidate that is able to contribute to very high quality repositories that have been used by millions of developers and isn't afraid of public scrutiny or code reviews by the maintainers. Otherwise, I would give a standard leetcode task (hard) in the relevant language and code-review that in the second interview. Eliminating 95% of AI frauds in one technical interview.
drummojg 2y ago
I helped interview a guy recently who was (a) ESL, and (b) Typing in our questions and reading the answers at a breakneck pace to the point where it was almost pathological; we could not get a word in. In unspoken horror and synchronization, we got through the formal part of the interview as quickly as we possibly could.
madeofpalk 2y ago
Even before AI tools, anyone can write code. It's not impressive, and as an interviewer you're not going to gain deep insights on whether they're a developer you want to work with by watching then squirm with some boring leetcode exercise. We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler? Nothing's perfect, but I do think asking people to explain their code to be much more resilliant to those who treat coding as a rote exercise, or just blindly ChatGPT-ed their way through.
- floren 2y ago
  > We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler? I applied this approach recently and it was... depressingly useful. Sent out 3 short tasks which should take an experienced programmer less than 30 minutes to solve in total: one "find, fix, and explain the problem in this code" and two "write the ~50 LOC to implement these exact specifications". These went out only to those whose resumes looked like a potential fit. Something like half of the responses we got back didn't even compile/run. Mostly due to hallucinated library functions. One candidate had decent responses, so we went on to the full interview, asking (as you mentioned) if he could explain his approach. He was having some trouble so I just asked him to explain how a sync.WaitGroup (which he'd used in his code) actually functions, and we got treated to an intensely painful few minutes as he desperately tried to spin some BS about how important concurrency is, and how Go's so powerful at concurrency, etc etc etc. but nothing about what a WaitGroup actually does. I don't really know how to solve this shit going forward.
  - satvikpendem 2y ago
    Just need to keep interviewing more people. Ironically, AI might make (competent) devs more valuable, not less, as lots of them will simply be filtered out because they use AI as a crutch, thus the overall pool of software engineers decreases (and supply is constantly increasing). It's akin to the COBOL situation at banks, they don't have enough competent COBOL devs (as new devs don't learn the language and older ones die off) so they need to pay out the ass to get them to work on their systems.
tthhrrooww_away 2y ago
(throwaway account) Can’t comment on the AI angle of the post, but the system design question moaning is interesting. I did many years of hiring in a FAANG for senior/staff positions and “one simple trick” to break a candidate out of formulaic response like this is a simple “You can put an off-the-shelf product (Postgres, Redis, Cassandra, memcached, whatever) in your diagram here, but you would need to explain to me how it works in good detail. Is that OK?” Like, the author’s point #4 mentions shards and replicas — those are driven by a crazily complex software. Want to really test your candidate’s system design knowledge? Stop them as soon as they draw a “replica” rectangle and ask to explain the replication process. Otherwise they may as well just draw one rectangle saying “Magic” in it, really.
- CharlieDigital 2y ago
  My counter to this is then consider just asking those types of questions to begin with and save time. If you already know the types of questions and answers that would knock your socks off in an interview, just do those! Like, if your favorite part of the trail mix is the pecans, just buy a bag of pecans and get to the good part.
pugworthy 2y ago
The thing is though, we "all" are using AI in the work place to help us with coding. It's a thing. It's the new Stack Overflow. So why shouldn't someone use it in an interview? Of course the main thing is whether they deceptively use it or openly use it. If an applicant is very open about how they use AI in the interview process with full disclosure, and do very good work, maybe they are a good hire? If I asked a question and someone wrote a very good prompt, got results, took the results and cleaned it up and confirmed it wasn't hallucinating, maybe they are a person I want to hire? I think in a lot of ways the new technical interview would revolve around showing the applicant a simulated AI prompt, and a response that included various mistakes and hallucinations. Then let the applicant tell me what's wrong with the code, and how the prompt could be improved or a followup prompt could be used to fix the issues. Getting AI to do what you want is a skill, and one you want in an applicant. Knowing when AI is wrong and being able to correct it is also something you want in an applicant.
- kenschu 2y ago
  In the future we'll be essentially testing how well candidates can steer these models anyway
- typs 2y ago
  True, though the lack of Stack Overflow / internet use during interviews has always been a critique as they've never seemed fully tied to the experience of the job.
- ajcp 2y ago
  One thing I look for during an interview is does the candidate use Google as part of their answer workflow. While having a candidate who just "knows" the answer might be great, eventually they wont, and what then? I'd rather see how a candidate goes about finding an answer to a problem, and part of that is looking it up. Why would LLM be any different? The real test is: do they know the difference between a right answer and a wrong one. That's what I'm interviewing for.
  - klipklop 2y ago
Swizec 2y ago
> reveals just how formulaic and predictable the response and banter is designed to be Accurate. Interviews have to be formulaic and predictable. That's how you combat bias in the system. Kahneman's book on Noise (in decision making) has a whole chapter on interviews. With careful design to avoid bias you can get to about 75% accuracy measured as correlation between Yeses and Job Performance. Without having repeatable interviews judged against a pre-defined metric, your signal becomes meaningless. You're basically flipping a coin so why even bother. Or worse, you get people passing interviews because they went to the same school as the interviewer or whatever. In that case your un-designed un-formulaic interviews become literally worse than random.
- typs 2y ago
  Yeah, I think interviews are to get some signal of competence and ability to prepare / offer some filtering ability even if they're a bit arbitrary or banal.
kenschu 2y ago
*disclaimer that I'm the founder of Ropes AI, & we're building a new way to evaluate engineering talent* Discourse here always tends to be negative - but I think that AI really opens the door positively here. It allows us to effectively vet talent asynchronously for the first time. Our thesis is that live interviews, while imperfect, work. If an engineer sits down with a candidate and watches them work for an hour (really you probably only need 5 minutes), you have a good read on their technical ability. There's all of these subtle hints that come out during an interview (how does the candidate approach a problem? What's their debugging reflex when something goes wrong? etc) - seeing enough of those signals give you confidence in your hiring decision. Well - LLMs can do that too, meaning we can capture these subtle signals asynchronously for the first time. And that's a big deal - if we can do that, then everyone gets the equivalent of a live interview - it doesn't matter your YOE or where you went to school etc - those that are technically gifted open a slot. And that's what we've built - a test that has the same signal as a live interview. If you're able to do that reliably, it doesn't just provide a new interview method for a new system - it might change how the recruiting process itself is structured.
- madeofpalk 2y ago
  If a company I am interviewing at tried to make me interview with some LLM instead of sitting down with an actual person, I would dip from the process. To me, only junk companies would use such a tool, so I guess it does serve the candiates as a massive red flag.
  - kenschu 2y ago
    Agreed! An LLM interviewer is probably almost insulting. The idea here is that the signals are implicit in the user's coding patterns (e.g in a take-home format etc)
    - mystified5016 2y ago
      So the candidate is being interviewed and rejected by an AI without their knowledge or consent. Most people would consider that quite rude, yes.
calflegal 2y ago
Aren't the LLMs also pretty decent code reviewers? Sure, not perfect, but the suggestion that we should use code review as the replacement for the coding challenge 'because AI' seems a bit misplaced...
galkk 2y ago
I don't understand the entire premise of "ohh, they will use ai/llm/whisper to cheat". Companies already succeeded in returning people to the offices. Soon they will return onsite interviews. Be ready to fly to the campus and have your whiteboard there, like it was pre-covid era.
swalsh 2y ago
As i've been interviewing people, i've found that when I used AI to generate questions people gave me near perfect answers. When I wrote my own, I got real answers. That was useful for gauging peoples general knowledge, but frankly I just lowered the value of simple "knowledge" questions. I shifted my interview to focus more on design exercises, and scenarios. How do you think and approach problems rather than what you've memmorized. That was quite a distinguishing factor for me that AI doesn't help you with.

TLDR: Another article in favor of code reviews vs leetcode challenges. About a decade ago during an interview, I was shown a piece of code (directly from their code base) which was about 50 lines long and we had a back and forth technical conversation in which the interviewer asked me the following questions: - What does this code appear to do (both technically and more importantly in the context of the business domain of the company)? - Are there are any red flags, edge cases, etc. that this code might not cover? - If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently? This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as: - Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code. - Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc) - Are they able to communicate and convey their thoughts in a coherent and logical manner?

CharlieDigital 2y ago
Yup, I went through a similar experience with a YC startup no less. First round was a code review with the CTO/co-founder and it was a web API and SQL schema with lots of little bits and varying "areas of interest" from data validation to exception handling (or lack thereof) to indices to type selection and other bits and pieces. Left a very strong and favorable impression on me for how much more enjoyable it was as a candidate and also how much more it felt like he presented a sandbox within which we could discover and discuss.
username135 2y ago
I like this.
inerte 2y ago
Meanwhile there's a handful of companies (the one I work at included) using AI to automate code reviews. Sure, these tools are not as good as a real human. Yet. Just like 3 years ago companies were starting to ban candidates to use GitHub Copilot, pretty soon candidates WILL find a way to either have ChatGPT open on the side, or listening to the operating system audio and providing real time suggestions during the interview (yeah I know this can be done today). We either go back doing things on site, or we go crazy and let candidates use whatever they want during an interview. Want to use Cursor and Perplexity + R1? Sure, why not. This is what people are using at work anyway. This probably means interview conclusions are going to be even more nuanced, it can't simply be they solved the problem or not (and the interviewer liked the person), but during the actual conversation things moved organically to whatever direction the topic led to - which is what actually happens at work when pairing. As you can see I have no idea how to "solve" this problem moving on, but one thing is for sure, the bigger the gap between tooling involved in actual work vs the interview, the bigger this problem is. It's one thing to ask people to write code on a whiteboard for 30 minutes (because they "write" code in a text editor), and another thing to ask people to write code in a whiteboard and the actual work is people chatting with AI Agents to produce so much software in 30 minutes that the entire thing would look like side projects 5 years ago.
pugworthy 2y ago
Per a separate comment I made, I think you could add to this and call it an AI assisted code review. Not just can the user review the code shown, but can they review the code that AI creates? Yes, AI code is sort of like any other code in that it can have errors, but I think taking it to the next step is whether they can actually modify or follow up with the prompt to get it to do the right thing. AI is kind of like having a really smart and confident intern who's sometimes wrong or doesn't quite understand your request. But you're not a manager or a lead. You're just some junior hire who's got AI saying, "Hey I have this figured out and am very confident in my results" and it's your job to check their work.

pityJuke 2y ago

Off-topic: does the font on this site render horribly for anyone else? Windows, Firefox & Edge.

tomrod 2y ago
Fedora, Firefox and Chrome
low_tech_punk 2y ago
I think the issue is that the font used by the author does not support fraction pixel sizing. I changed the body font size from 1.2em (16.8px) to 16px and the issues are gone. I'm curious too as for why it happened. I'm guessing the font was originally designed for Japanese, not Latin. See: https://fontmeme.com/fonts/m-plus-rounded-1c-font/ https://fontmeme.com/fonts/m-plus-rounded-1c-font/
- CharlieDigital 2y ago
  Thanks for that insight!

throwup238 2y ago

> The system design ones were somehow even more outrageous given that system design is often the deciding factor between a mid/senior level and a staff/principal level engineer. Yet watching the IGotAnOffer video series successively reveals just how formulaic and predictable the response and banter is designed to be. Oh my god, yes. Every architectural interview I’ve had in the last five years has been a rehashing of the same concepts, pulled straight out of Designing Data Intensive Applications. Anyone who has read that book once or twice can easily pass. Half the interviewers even use the Twitter example presented in the book, with a few arbitrary extra requirements tacked on.

tthhrrooww_away 2y ago
To be fair, DDIA is a pretty deep book, so if you read it, took something out of it, and read it the second time (!), you probably deserve an offer.
CharlieDigital 2y ago
There's very little room for deviation as well because it is so formulaic. Present a novel and interesting system? It may not be realistically solvable in a 30-60 minute session to a satisfying degree because the candidate doesn't have familiarity. From the candidate side, you can take a risk and make it more interesting by presenting more novel solutions, but the interviewer might be looking for very specific response patterns; "signaling" of a sort. So both sides just stick to a narrow range of scenarios and responses. Once you've done one or two, it's hard to see how you couldn't do every other one after it.
satvikpendem 2y ago
> pulled straight out of Designing Data Intensive Applications I mean, because it works. Why would you create a novel system just for the sake of it?
superfrank 2y ago
I think the root of the problem is actually an org design problem. Big companies want consistency in their process and their hires. There's a lot of benefits if you can create a Google type hiring pipeline you're just constantly interviewing candidates and you worry about where they go once they've passed the interview, like: - Since all candidates at the same level are passing essentially the same interview you can move people from team to team easier. You reduce the risk of a team with lower standards causing you problems when their engineers are moved to other places in the company. You know that most of your hires have a certain consistency in their skills. This makes things like reorgs a lot easier. - Your recruiters and hiring managers save time because in most cases they don't need to meet to set up unique hiring pipelines per team. You just have one big hiring pipeline and teams with open spots only step in at the last minute. - You reduce the risk of being sued for bias or discrimination. If interviewers are just pulling from a bank of questions that are somewhat equal it's hard for a candidate to make a case that they were singled out and given a tougher question because of their identity. The problem with that is that consistency breeds predictability. The only way to get a big pool of interview questions that are about the same difficulty is by repeating the same patterns between questions. Once candidates know those patterns, candidates can just focus their studying on that and the interview just boils down to pattern matching.

rvz 2y ago

> Rather than entirely shifting away from existing processes, teams should start to consider incorporating code reviews as a key step in the candidate selection process to identify those with a proficiency to effectively evaluating code. They should indeed shift to code reviews to test the reasoning ability of the candidate's written code rather than already solved leetcode puzzles which is very easy for candidates to use LLMs to cheat. On last resort, I'd use leetcode but focused on the relevant language that I am looking for. (If I want to see the leetcode problem solved in Golang, I do not want to see you use Python to solve it.) I would give the candidate a chance to skip the leetcode puzzles if they are able to demonstrate that they have contributed (at least two) to significant open source project(s) in the relevant technology in the job description and show an example code review of those commits. This gives a public example of a candidate that is able to contribute to very high quality repositories that have been used by millions of developers and isn't afraid of public scrutiny or code reviews by the maintainers. Otherwise, I would give a standard leetcode task (hard) in the relevant language and code-review that in the second interview. Eliminating 95% of AI frauds in one technical interview.

drummojg 2y ago

I helped interview a guy recently who was (a) ESL, and (b) Typing in our questions and reading the answers at a breakneck pace to the point where it was almost pathological; we could not get a word in. In unspoken horror and synchronization, we got through the formal part of the interview as quickly as we possibly could.

madeofpalk 2y ago

Even before AI tools, anyone can write code. It's not impressive, and as an interviewer you're not going to gain deep insights on whether they're a developer you want to work with by watching then squirm with some boring leetcode exercise. We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler? Nothing's perfect, but I do think asking people to explain their code to be much more resilliant to those who treat coding as a rote exercise, or just blindly ChatGPT-ed their way through.

floren 2y ago
> We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler? I applied this approach recently and it was... depressingly useful. Sent out 3 short tasks which should take an experienced programmer less than 30 minutes to solve in total: one "find, fix, and explain the problem in this code" and two "write the ~50 LOC to implement these exact specifications". These went out only to those whose resumes looked like a potential fit. Something like half of the responses we got back didn't even compile/run. Mostly due to hallucinated library functions. One candidate had decent responses, so we went on to the full interview, asking (as you mentioned) if he could explain his approach. He was having some trouble so I just asked him to explain how a sync.WaitGroup (which he'd used in his code) actually functions, and we got treated to an intensely painful few minutes as he desperately tried to spin some BS about how important concurrency is, and how Go's so powerful at concurrency, etc etc etc. but nothing about what a WaitGroup actually does. I don't really know how to solve this shit going forward.
- satvikpendem 2y ago
  Just need to keep interviewing more people. Ironically, AI might make (competent) devs more valuable, not less, as lots of them will simply be filtered out because they use AI as a crutch, thus the overall pool of software engineers decreases (and supply is constantly increasing). It's akin to the COBOL situation at banks, they don't have enough competent COBOL devs (as new devs don't learn the language and older ones die off) so they need to pay out the ass to get them to work on their systems.

tthhrrooww_away 2y ago

(throwaway account) Can’t comment on the AI angle of the post, but the system design question moaning is interesting. I did many years of hiring in a FAANG for senior/staff positions and “one simple trick” to break a candidate out of formulaic response like this is a simple “You can put an off-the-shelf product (Postgres, Redis, Cassandra, memcached, whatever) in your diagram here, but you would need to explain to me how it works in good detail. Is that OK?” Like, the author’s point #4 mentions shards and replicas — those are driven by a crazily complex software. Want to really test your candidate’s system design knowledge? Stop them as soon as they draw a “replica” rectangle and ask to explain the replication process. Otherwise they may as well just draw one rectangle saying “Magic” in it, really.

CharlieDigital 2y ago
My counter to this is then consider just asking those types of questions to begin with and save time. If you already know the types of questions and answers that would knock your socks off in an interview, just do those! Like, if your favorite part of the trail mix is the pecans, just buy a bag of pecans and get to the good part.

pugworthy 2y ago

The thing is though, we "all" are using AI in the work place to help us with coding. It's a thing. It's the new Stack Overflow. So why shouldn't someone use it in an interview? Of course the main thing is whether they deceptively use it or openly use it. If an applicant is very open about how they use AI in the interview process with full disclosure, and do very good work, maybe they are a good hire? If I asked a question and someone wrote a very good prompt, got results, took the results and cleaned it up and confirmed it wasn't hallucinating, maybe they are a person I want to hire? I think in a lot of ways the new technical interview would revolve around showing the applicant a simulated AI prompt, and a response that included various mistakes and hallucinations. Then let the applicant tell me what's wrong with the code, and how the prompt could be improved or a followup prompt could be used to fix the issues. Getting AI to do what you want is a skill, and one you want in an applicant. Knowing when AI is wrong and being able to correct it is also something you want in an applicant.

kenschu 2y ago
In the future we'll be essentially testing how well candidates can steer these models anyway
typs 2y ago
True, though the lack of Stack Overflow / internet use during interviews has always been a critique as they've never seemed fully tied to the experience of the job.
ajcp 2y ago
One thing I look for during an interview is does the candidate use Google as part of their answer workflow. While having a candidate who just "knows" the answer might be great, eventually they wont, and what then? I'd rather see how a candidate goes about finding an answer to a problem, and part of that is looking it up. Why would LLM be any different? The real test is: do they know the difference between a right answer and a wrong one. That's what I'm interviewing for.
- klipklop 2y ago

Swizec 2y ago

> reveals just how formulaic and predictable the response and banter is designed to be Accurate. Interviews have to be formulaic and predictable. That's how you combat bias in the system. Kahneman's book on Noise (in decision making) has a whole chapter on interviews. With careful design to avoid bias you can get to about 75% accuracy measured as correlation between Yeses and Job Performance. Without having repeatable interviews judged against a pre-defined metric, your signal becomes meaningless. You're basically flipping a coin so why even bother. Or worse, you get people passing interviews because they went to the same school as the interviewer or whatever. In that case your un-designed un-formulaic interviews become literally worse than random.

typs 2y ago
Yeah, I think interviews are to get some signal of competence and ability to prepare / offer some filtering ability even if they're a bit arbitrary or banal.

kenschu 2y ago

*disclaimer that I'm the founder of Ropes AI, & we're building a new way to evaluate engineering talent* Discourse here always tends to be negative - but I think that AI really opens the door positively here. It allows us to effectively vet talent asynchronously for the first time. Our thesis is that live interviews, while imperfect, work. If an engineer sits down with a candidate and watches them work for an hour (really you probably only need 5 minutes), you have a good read on their technical ability. There's all of these subtle hints that come out during an interview (how does the candidate approach a problem? What's their debugging reflex when something goes wrong? etc) - seeing enough of those signals give you confidence in your hiring decision. Well - LLMs can do that too, meaning we can capture these subtle signals asynchronously for the first time. And that's a big deal - if we can do that, then everyone gets the equivalent of a live interview - it doesn't matter your YOE or where you went to school etc - those that are technically gifted open a slot. And that's what we've built - a test that has the same signal as a live interview. If you're able to do that reliably, it doesn't just provide a new interview method for a new system - it might change how the recruiting process itself is structured.

madeofpalk 2y ago
If a company I am interviewing at tried to make me interview with some LLM instead of sitting down with an actual person, I would dip from the process. To me, only junk companies would use such a tool, so I guess it does serve the candiates as a massive red flag.
- kenschu 2y ago
  Agreed! An LLM interviewer is probably almost insulting. The idea here is that the signals are implicit in the user's coding patterns (e.g in a take-home format etc)
  - mystified5016 2y ago
    So the candidate is being interviewed and rejected by an AI without their knowledge or consent. Most people would consider that quite rude, yes.

calflegal 2y ago

Aren't the LLMs also pretty decent code reviewers? Sure, not perfect, but the suggestion that we should use code review as the replacement for the coding challenge 'because AI' seems a bit misplaced...

galkk 2y ago

I don't understand the entire premise of "ohh, they will use ai/llm/whisper to cheat". Companies already succeeded in returning people to the offices. Soon they will return onsite interviews. Be ready to fly to the campus and have your whiteboard there, like it was pre-covid era.

swalsh 2y ago

As i've been interviewing people, i've found that when I used AI to generate questions people gave me near perfect answers. When I wrote my own, I got real answers. That was useful for gauging peoples general knowledge, but frankly I just lowered the value of simple "knowledge" questions. I shifted my interview to focus more on design exercises, and scenarios. How do you think and approach problems rather than what you've memmorized. That was quite a distinguishing factor for me that AI doesn't help you with.