7 ms·

IMO asking people to not use available tools in interviews is a bad idea, unless you are trying to do a very basic check that someone knows the fundamentals. A

IMO asking people to not use available tools in interviews is a bad idea, unless you are trying to do a very basic check that someone knows the fundamentals. Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer. Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.

leeny 3y ago
I'm not the author (perhaps he'll chime in as well), but I'm the CEO of interviewing.io (we're the ones who ran the experiment). I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints. In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
- michael_mroczka 3y ago
  I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.
  - pierat 3y ago
    Well that's the rub. There's no way, even for a senior engineer, to know everything. In fact, one of the required skills is "how to ask the question as to elicit the answer in a reasonable amount of time". The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses. But hiding it? Yeah, probably not a good fit.
    - vinni2 3y ago
      > The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. I work at a university and most of our exams are open book or project based. You probably want to update your image of universities.
      - ziddoap 3y ago
        >You probably want to update your image of universities. I have a close friend who is a prof at a university and most exams remain closed book. You probably want to update your image of universities. Or, perhaps, we can agree that it depends on the university, the subject, etc. and blanket statements based on single anecdotes are silly?
  - randomdata 3y ago
    > an interview that clearly is meant to be "closed book," I am not sure that is clear. It seems the expectation was not "closed book", but "never opened a book before, not even in the past": "It's tough to determine if the candidate breezed through the question because they're actually good or if they've heard this question before." Clearly the interviewers were looking not for knowledge, but for uncanny ability. How well was that communicated to the interviewees? It is not cheating if the rules of the game are not defined.
  - gabrieledarrigo 3y ago
    > but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating. I would argue that is the opposite: it's fair to say that the interview is a cheat.
  - jacques_chester 3y ago
    > I also am in favor of "open book" interviews. I recall reading an interviewing.io blog post[0] in which the dominant considerations interviewers weighed were (my interpretation): (1) Did they solve the problem optimally? (2) How fluid was their coding? With "communication" turning out to be basically worthless for predicting hire/no-hire decisions. Perception of coding fluidity seems like it would be affected by how often the candidate stops and looks up things like library functions or obscure syntax. For that reason I've been investing time in committing a lot of library functions to memory, so they instantly flow from my fingers rather than spending a minute looking it up. It's dumb that I need to do this, but I don't make the rules. I'm just at the bottom of the information cascade that led to how things are done now. [0] https://interviewing.io/blog/does-communication-matter-in-technical-interviewing-we-looked-at-100k-interviews-to-find-out https://interviewing.io/blog/does-communication-matter-in-te...
    - madeofpalk 3y ago
      Companies that optimise for memorising obscure stdlib functions don't seem like great places to work.
      - spongebobstoes 3y ago
        Occasionally looking something up is normal, but if you don't know how to append to a list in Python or iterate over a vector in C++ then you probably are not currently writing much code in those languages. That's a signal by itself, and one that is too often a negative.
        mark38848 3y ago
        Why? Maybe you just use good programming languages instead.
        doix 3y ago
        I'm not saying you're wrong, but that reasoning is why I have to prepare for interviews. I'm _really_ bad at remembering that stuff, I think it really depends on how you think while programming. I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent. It's not a huge deal, but I have to actively sit and memorize that stuff before an interview. Usually by writing it out on paper or writing it in some foreign editor that I'm not used too. It probably wouldn't be _too_ bad today, because I just sit and write Typescript everyday, but when I was switching between perl/ruby/python/tcl/kotlin/javascript/bash/csh/lisp my brain was basically mush. I couldn't tell you how to do any basic operation in any language.
- BriggyDwiggs42 3y ago
  Given your knowledge of the subject, do you think leetcode-type questions are meaningfully able to appraise an employee’s performance in a production environment? I’ve always thought it was basically unrelated beyond testing basic coding experience.
  - michael_mroczka 3y ago
    The short answer is, "Yes." They are very flawed, but one of the most reliable ways to avoid "bad" hires. The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions. While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code. We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.
    - consp 3y ago
      You forget where you lose good hires since leetcode questions are pathetic and demeaning for anyone with experience.
      - trgdr 3y ago
        Keep it civil, dude. This is a pretty reductive take that is not obviously true and adds little to the discussion.
      - michael_mroczka 3y ago
        Let's keep it civil, my friend. I totally get the frustration with leetcode-style questions and acknowledge that they aren't perfect. They can indeed overlook the diverse strengths that experienced engineers bring to the table. My perspective is that understanding the basics, like the efficiency of different data structures, is crucial, not just for interviews but for making informed decisions in our work. I'm not advocating for spot tests on complex algorithms without context. However, I believe a conversation about fundamental concepts like big-O notation reflects on one's approach to problem-solving and software design. It's not about dismissing anyone's experience or capability but ensuring a solid foundation that benefits all aspects of engineering work. I understand senior engineers being out of practice and not being able to derive things like topological sort on the fly, but an inability to talk about the basics of big-O or the simplest of data structures is more commonly a red flag than "rust." If they never learned the material, that's a red flag. If they learned the material and they've been building software for the last decade without taking the basics of the material into account while coding, then that is a red flag as well. Again, happy to admit that this is a flawed approach that will lose some great engineers, but most engineers that have your line of thinking are ones I actively avoid hiring. These concepts are practical, available to learn for free, and easy to understand. If an engineer feels that testing these concepts is beneath them, then they definitely aren't a good fit for any team I'm on. Big-O is to software engineers what Ohm's Law is to electricians. Imagine a world where electricians thought it was demeaning to talk about Ohm's law in an interview. ¯\_(ツ)_/¯
- blitzar 3y ago
  I have been part of a few interview tests that were proudly "open book". However, if you are going to read the context and tasks then build 20 spreadsheets in 20 minutes you have 0 time to google something.
danielvaughn 3y ago
I'm fairly open book, but I wouldn't accept LLM usage in an interview. I don't need someone to have all the facts in their head, so if they have to look up some syntax or whatever, then totally fine. However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.
- michael_mroczka 3y ago
  Exactly. People end up throwing the baby out with the bathwater on this. "Data structure & algorithm interviews aren't perfect, so let's not ask people to code at all." It's an absurd overcorrection, but most people think these interviews are about demanding optimal code and perfection when they mostly just are making sure you're not using arrays when you should be using hashmaps... and that you know what a hashmap is, I suppose.
  - blkhawk 3y ago
    Maybe combine the two - Have the applicant do a small task then talk about it. Maybe give them a slightly dubious direction then see if they followed to the letter or deviated and their reasoning for it. I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds. Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
    - danielvaughn 3y ago
      My favorite style of interview was actually a take-home exercise. Back in 2016 I interviewed with Blue Apron and they asked me to create a custom javascript framework from scratch to render a recipe app or something like that. Then I had an interview where I discussed my solution and we reviewed the code together. It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.
elicksaur 3y ago
> Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another. Consider that this may not be typical.
- ilc 3y ago
  Consider... it might be. Seriously, I work for a company very protective of its IP. And I can still use ChatGPT and similar tools for some of what I do. It is a huge force multiplier.
- CaptainFever 3y ago
  > 70% of all respondents are using or are planning to use AI tools in their development process this year. Those learning to code are more likely than professional developers to be using or use AI tools (82% vs. 70%). Source: https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage To be fair, the number of "Yes" was "just" 43% but that's still a very large amount of developers, not including those who plan to use it.
  - elicksaur 3y ago
    I know a high amount of people have ever tried it, but my primary reason for responding was the “constantly” qualifier, which admittedly I could’ve made clearer. I would’ve answered yes to that survey question, but I wouldn’t say I use AI tools constantly or that it is typically how I solve a problem at work.
- madeofpalk 3y ago
  Do you consider it typical for development to look things up on google, documentation websites, or stack overflow?
- sanderjd 3y ago
  Consider that it definitely is not atypical.
- baq 3y ago
kjkjadksj 3y ago
This is what they get for assigning homework instead of just having a candid conversation where you verbally probe the depth of knowledge like everyone interviewing anyone ever outside of computer science.
- bomewish 3y ago
  This is a really strong point. For the people who cannot code at all apparently — wouldn’t a conversation like this show a lack of understanding across a bunch of relevant topics??
iLoveOncall 3y ago
An interview is aimed at verifying only basics skills and then mostly general intelligence. This narrative of "interviews should be like real work" needs to stop. It doesn't make any sense, this is not the goal of an interview.
- rvba 3y ago
  It does make sense if you want to know if employees can do their jobs.
  - michael_mroczka 3y ago
    The implied assumption here is that if you can figure out the number of anagrams that can be made from a given string in just 30 minutes, you probably are smart enough to set up a web server in a couple of days with documentation and ChatGPT to guide you. While the above sounds sarcastic, honestly, it isn't as ridiculous of a thought as it sounds. I don't think being good at DS&A problems guarantees that you're a good coder, but in general, the people who get good at these problems are also great coders. I can think of less than a handful of people who are good at CS problems but bad at actual coding.
    - rvba 3y ago
      Alternative is that there are people who cannot do anagrams, who probably cannot follow a ChatGPT manual even if their life depended on it. Since IT is paid very well, there are lots of people who try to lie their way to get the job. If you are someone on minimum wage, then why not try your luck - if they fire you after 2 months, you probably earned more in those 2 months than in whole year of your shit job.
notfed 3y ago
LLMs are too powerful. It's basically like having someone next to you telling you what to say in the interview, if not better. The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.
- planede 3y ago
  > "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily. This gives you a plan for designing the interview questions.
  - deleted 3y ago
    [deleted]
- baq 3y ago
  Except when it is, which will only be a bigger part of the job. I'm definitely looking at googling proficiency when interviewing. Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
- blitzar 3y ago
  > "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt. Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
  - sibit 3y ago
    > Unfortunately that requires skill and awareness on behalf of the interviewers; In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.
reidrac 3y ago
> Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another. I don't, and I don't know anyone working with me thar is using any LLM (we pair). Some tried Copilot at some point and concluded it was useless for our use case. Not sure on what context one would constantly use LLMs.
- dgellow 3y ago
  It’s ok at writing comments :)
koonsolo 3y ago
> Interviews should be consistent with what day to day work actually looks like I'm not hiring at the level of day to day work, I'm hiring at the level of when things go complex or bad. To give an example, I'm not going to test a coders ability to write some CRUD application. I'm testing the ability when a junior developers comes with a crazy problem, that they can find the cause and solve it. If I only test day to day work, you get these kind of developers that keep changing faulty code until it magically works. I don't want those in my team. I want developers that can figure out what is going wrong, understand it, and provide a solution. Do such things happen day to day? No. And a person that can reason his way out of things instead of trying random stuff, sure as hell can do the same day to day tasks just as well or even better.
em1sar 3y ago
[dead]

leeny 3y ago

I'm not the author (perhaps he'll chime in as well), but I'm the CEO of interviewing.io (we're the ones who ran the experiment). I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints. In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.

michael_mroczka 3y ago
I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.
- pierat 3y ago
  Well that's the rub. There's no way, even for a senior engineer, to know everything. In fact, one of the required skills is "how to ask the question as to elicit the answer in a reasonable amount of time". The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses. But hiding it? Yeah, probably not a good fit.
  - vinni2 3y ago
    > The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge. I work at a university and most of our exams are open book or project based. You probably want to update your image of universities.
    - ziddoap 3y ago
      >You probably want to update your image of universities. I have a close friend who is a prof at a university and most exams remain closed book. You probably want to update your image of universities. Or, perhaps, we can agree that it depends on the university, the subject, etc. and blanket statements based on single anecdotes are silly?
- randomdata 3y ago
  > an interview that clearly is meant to be "closed book," I am not sure that is clear. It seems the expectation was not "closed book", but "never opened a book before, not even in the past": "It's tough to determine if the candidate breezed through the question because they're actually good or if they've heard this question before." Clearly the interviewers were looking not for knowledge, but for uncanny ability. How well was that communicated to the interviewees? It is not cheating if the rules of the game are not defined.
- gabrieledarrigo 3y ago
  > but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating. I would argue that is the opposite: it's fair to say that the interview is a cheat.
- jacques_chester 3y ago
  > I also am in favor of "open book" interviews. I recall reading an interviewing.io blog post[0] in which the dominant considerations interviewers weighed were (my interpretation): (1) Did they solve the problem optimally? (2) How fluid was their coding? With "communication" turning out to be basically worthless for predicting hire/no-hire decisions. Perception of coding fluidity seems like it would be affected by how often the candidate stops and looks up things like library functions or obscure syntax. For that reason I've been investing time in committing a lot of library functions to memory, so they instantly flow from my fingers rather than spending a minute looking it up. It's dumb that I need to do this, but I don't make the rules. I'm just at the bottom of the information cascade that led to how things are done now. [0] https://interviewing.io/blog/does-communication-matter-in-technical-interviewing-we-looked-at-100k-interviews-to-find-out https://interviewing.io/blog/does-communication-matter-in-te...
  - madeofpalk 3y ago
    Companies that optimise for memorising obscure stdlib functions don't seem like great places to work.
    - spongebobstoes 3y ago
      Occasionally looking something up is normal, but if you don't know how to append to a list in Python or iterate over a vector in C++ then you probably are not currently writing much code in those languages. That's a signal by itself, and one that is too often a negative.
      - mark38848 3y ago
        Why? Maybe you just use good programming languages instead.
      - doix 3y ago
        I'm not saying you're wrong, but that reasoning is why I have to prepare for interviews. I'm _really_ bad at remembering that stuff, I think it really depends on how you think while programming. I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent. It's not a huge deal, but I have to actively sit and memorize that stuff before an interview. Usually by writing it out on paper or writing it in some foreign editor that I'm not used too. It probably wouldn't be _too_ bad today, because I just sit and write Typescript everyday, but when I was switching between perl/ruby/python/tcl/kotlin/javascript/bash/csh/lisp my brain was basically mush. I couldn't tell you how to do any basic operation in any language.
BriggyDwiggs42 3y ago
Given your knowledge of the subject, do you think leetcode-type questions are meaningfully able to appraise an employee’s performance in a production environment? I’ve always thought it was basically unrelated beyond testing basic coding experience.
- michael_mroczka 3y ago
  The short answer is, "Yes." They are very flawed, but one of the most reliable ways to avoid "bad" hires. The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions. While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code. We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.
  - consp 3y ago
    You forget where you lose good hires since leetcode questions are pathetic and demeaning for anyone with experience.
    - trgdr 3y ago
      Keep it civil, dude. This is a pretty reductive take that is not obviously true and adds little to the discussion.
    - michael_mroczka 3y ago
      Let's keep it civil, my friend. I totally get the frustration with leetcode-style questions and acknowledge that they aren't perfect. They can indeed overlook the diverse strengths that experienced engineers bring to the table. My perspective is that understanding the basics, like the efficiency of different data structures, is crucial, not just for interviews but for making informed decisions in our work. I'm not advocating for spot tests on complex algorithms without context. However, I believe a conversation about fundamental concepts like big-O notation reflects on one's approach to problem-solving and software design. It's not about dismissing anyone's experience or capability but ensuring a solid foundation that benefits all aspects of engineering work. I understand senior engineers being out of practice and not being able to derive things like topological sort on the fly, but an inability to talk about the basics of big-O or the simplest of data structures is more commonly a red flag than "rust." If they never learned the material, that's a red flag. If they learned the material and they've been building software for the last decade without taking the basics of the material into account while coding, then that is a red flag as well. Again, happy to admit that this is a flawed approach that will lose some great engineers, but most engineers that have your line of thinking are ones I actively avoid hiring. These concepts are practical, available to learn for free, and easy to understand. If an engineer feels that testing these concepts is beneath them, then they definitely aren't a good fit for any team I'm on. Big-O is to software engineers what Ohm's Law is to electricians. Imagine a world where electricians thought it was demeaning to talk about Ohm's law in an interview. ¯\_(ツ)_/¯
blitzar 3y ago
I have been part of a few interview tests that were proudly "open book". However, if you are going to read the context and tasks then build 20 spreadsheets in 20 minutes you have 0 time to google something.

danielvaughn 3y ago

I'm fairly open book, but I wouldn't accept LLM usage in an interview. I don't need someone to have all the facts in their head, so if they have to look up some syntax or whatever, then totally fine. However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.

michael_mroczka 3y ago
Exactly. People end up throwing the baby out with the bathwater on this. "Data structure & algorithm interviews aren't perfect, so let's not ask people to code at all." It's an absurd overcorrection, but most people think these interviews are about demanding optimal code and perfection when they mostly just are making sure you're not using arrays when you should be using hashmaps... and that you know what a hashmap is, I suppose.
- blkhawk 3y ago
  Maybe combine the two - Have the applicant do a small task then talk about it. Maybe give them a slightly dubious direction then see if they followed to the letter or deviated and their reasoning for it. I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds. Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
  - danielvaughn 3y ago
    My favorite style of interview was actually a take-home exercise. Back in 2016 I interviewed with Blue Apron and they asked me to create a custom javascript framework from scratch to render a recipe app or something like that. Then I had an interview where I discussed my solution and we reviewed the code together. It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.

elicksaur 3y ago

> Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another. Consider that this may not be typical.

ilc 3y ago
Consider... it might be. Seriously, I work for a company very protective of its IP. And I can still use ChatGPT and similar tools for some of what I do. It is a huge force multiplier.
CaptainFever 3y ago
> 70% of all respondents are using or are planning to use AI tools in their development process this year. Those learning to code are more likely than professional developers to be using or use AI tools (82% vs. 70%). Source: https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage To be fair, the number of "Yes" was "just" 43% but that's still a very large amount of developers, not including those who plan to use it.
- elicksaur 3y ago
  I know a high amount of people have ever tried it, but my primary reason for responding was the “constantly” qualifier, which admittedly I could’ve made clearer. I would’ve answered yes to that survey question, but I wouldn’t say I use AI tools constantly or that it is typically how I solve a problem at work.
madeofpalk 3y ago
Do you consider it typical for development to look things up on google, documentation websites, or stack overflow?
sanderjd 3y ago
Consider that it definitely is not atypical.
baq 3y ago

kjkjadksj 3y ago

This is what they get for assigning homework instead of just having a candid conversation where you verbally probe the depth of knowledge like everyone interviewing anyone ever outside of computer science.

bomewish 3y ago
This is a really strong point. For the people who cannot code at all apparently — wouldn’t a conversation like this show a lack of understanding across a bunch of relevant topics??

iLoveOncall 3y ago

An interview is aimed at verifying only basics skills and then mostly general intelligence. This narrative of "interviews should be like real work" needs to stop. It doesn't make any sense, this is not the goal of an interview.

rvba 3y ago
It does make sense if you want to know if employees can do their jobs.
- michael_mroczka 3y ago
  The implied assumption here is that if you can figure out the number of anagrams that can be made from a given string in just 30 minutes, you probably are smart enough to set up a web server in a couple of days with documentation and ChatGPT to guide you. While the above sounds sarcastic, honestly, it isn't as ridiculous of a thought as it sounds. I don't think being good at DS&A problems guarantees that you're a good coder, but in general, the people who get good at these problems are also great coders. I can think of less than a handful of people who are good at CS problems but bad at actual coding.
  - rvba 3y ago
    Alternative is that there are people who cannot do anagrams, who probably cannot follow a ChatGPT manual even if their life depended on it. Since IT is paid very well, there are lots of people who try to lie their way to get the job. If you are someone on minimum wage, then why not try your luck - if they fire you after 2 months, you probably earned more in those 2 months than in whole year of your shit job.

notfed 3y ago

LLMs are too powerful. It's basically like having someone next to you telling you what to say in the interview, if not better. The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.

planede 3y ago
> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily. This gives you a plan for designing the interview questions.
- deleted 3y ago
  [deleted]
baq 3y ago
Except when it is, which will only be a bigger part of the job. I'm definitely looking at googling proficiency when interviewing. Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
blitzar 3y ago
> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt. Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
- sibit 3y ago
  > Unfortunately that requires skill and awareness on behalf of the interviewers; In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.

reidrac 3y ago

> Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another. I don't, and I don't know anyone working with me thar is using any LLM (we pair). Some tried Copilot at some point and concluded it was useless for our use case. Not sure on what context one would constantly use LLMs.

dgellow 3y ago
It’s ok at writing comments :)

koonsolo 3y ago

> Interviews should be consistent with what day to day work actually looks like I'm not hiring at the level of day to day work, I'm hiring at the level of when things go complex or bad. To give an example, I'm not going to test a coders ability to write some CRUD application. I'm testing the ability when a junior developers comes with a crazy problem, that they can find the cause and solve it. If I only test day to day work, you get these kind of developers that keep changing faulty code until it magically works. I don't want those in my team. I want developers that can figure out what is going wrong, understand it, and provide a solution. Do such things happen day to day? No. And a person that can reason his way out of things instead of trying random stuff, sure as hell can do the same day to day tasks just as well or even better.

em1sar 3y ago

[dead]