6 ms·

LLMs are too powerful. It's basically like having someone next to you telling you what to say in the interview, if not better. The point of (software engineeri

by notfed 3y ago

LLMs are too powerful. It's basically like having someone next to you telling you what to say in the interview, if not better. The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.

planede 3y ago
> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily. This gives you a plan for designing the interview questions.
- deleted 3y ago
  [deleted]
baq 3y ago
Except when it is, which will only be a bigger part of the job. I'm definitely looking at googling proficiency when interviewing. Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
blitzar 3y ago
> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt. Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
- sibit 3y ago
  > Unfortunately that requires skill and awareness on behalf of the interviewers; In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.
  - redditor98654 3y ago
    I did this. Thrice, and worked hard on the problems since coming up with new ones and making them relevant to day to day work while still be something reasonable to ask in a 45 min interview is tough. After about three interviews it ended up on Leetcode.
GuB-42 3y ago
Unless you can't use ChatGPT for some reason (ex: security), I think it is a totally valid way of solving problems. The interesting part is what to do when ChatGPT does it wrong, and if the problem is not trivial and an exact solution is not available online, it is usually wrong. Sometimes, not by much, but it takes skill to notice the problem and fix it, either manually or by asking ChatGPT for it. Same idea as for libraries. One could argue that those using the "sort" function don't know how to write a sorting algorithm, but in real life, unless there are particularly good reasons, rewriting a "sort" function would be crazy, and probably not what you want from your employees. If you want your candidate not to use the tools at his disposal, you may frame it as a requirement. "on this test, imagine you are working on a sensitive project, you are not allowed to upload any details to a third party service, you can search the internet for generic information, but do as if your development machine has no internet access, so no copy-pasting". Or, for libraries "on this test, imagine we absolutely want to limit dependencies on third party libraries, even if it means sometimes reinventing the wheel, so only <list of libraries> are allowed".
- sesm 3y ago
  Re: security - I personally never copy company code into ChatGPT prompt and just ask questions like “I have X how do I do Y” replacing any names with placeholders. Is there an environment where such usage would be considered a security issue?
  - GuB-42 3y ago
    I don't know if it is. However, the process of anonymizing the data, but still get a useful answer and then adapting it to working code on your side takes some skills. If you can do that effectively, you are not completely clueless and probably could do well enough without ChatGPT.
    - alexb_ 3y ago
      It 100% is a security issue. If you think you can just trust people to "anonymize the data effectively" you are very mistaken.
TrackerFF 3y ago
Ideally they're interested in the "how" part. Unfortunately though, there are lots and lots of interviewers out there that do not give a shit about "how", but rather if you can provide the right answer or not. Here are three questions, you have 60 minutes. Please provide the optimal solution to at least two of those questions. Now excuse me while I'm listening to a teams/zoom meeting in the background. Good luck, have fun.
Jaygles 3y ago
Yeah, if someone is leaning on an LLM, conceptually you are interviewing the LLM and not the candidate. I think the only signal you can get in such a case is how effective the candidate can use an LLM, and maybe something about the candidate's skills if they are able to catch mistakes the LLM makes. But I don't think that's enough signal to ensure someone would not be a bad hire.

by notfed 3y ago

planede 3y ago

> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily. This gives you a plan for designing the interview questions.

deleted 3y ago
[deleted]

baq 3y ago

Except when it is, which will only be a bigger part of the job. I'm definitely looking at googling proficiency when interviewing. Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.

blitzar 3y ago

> "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for. Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt. Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.

sibit 3y ago
> Unfortunately that requires skill and awareness on behalf of the interviewers; In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.
- redditor98654 3y ago
  I did this. Thrice, and worked hard on the problems since coming up with new ones and making them relevant to day to day work while still be something reasonable to ask in a 45 min interview is tough. After about three interviews it ended up on Leetcode.

GuB-42 3y ago

Unless you can't use ChatGPT for some reason (ex: security), I think it is a totally valid way of solving problems. The interesting part is what to do when ChatGPT does it wrong, and if the problem is not trivial and an exact solution is not available online, it is usually wrong. Sometimes, not by much, but it takes skill to notice the problem and fix it, either manually or by asking ChatGPT for it. Same idea as for libraries. One could argue that those using the "sort" function don't know how to write a sorting algorithm, but in real life, unless there are particularly good reasons, rewriting a "sort" function would be crazy, and probably not what you want from your employees. If you want your candidate not to use the tools at his disposal, you may frame it as a requirement. "on this test, imagine you are working on a sensitive project, you are not allowed to upload any details to a third party service, you can search the internet for generic information, but do as if your development machine has no internet access, so no copy-pasting". Or, for libraries "on this test, imagine we absolutely want to limit dependencies on third party libraries, even if it means sometimes reinventing the wheel, so only <list of libraries> are allowed".

sesm 3y ago
Re: security - I personally never copy company code into ChatGPT prompt and just ask questions like “I have X how do I do Y” replacing any names with placeholders. Is there an environment where such usage would be considered a security issue?
- GuB-42 3y ago
  I don't know if it is. However, the process of anonymizing the data, but still get a useful answer and then adapting it to working code on your side takes some skills. If you can do that effectively, you are not completely clueless and probably could do well enough without ChatGPT.
  - alexb_ 3y ago
    It 100% is a security issue. If you think you can just trust people to "anonymize the data effectively" you are very mistaken.

TrackerFF 3y ago

Ideally they're interested in the "how" part. Unfortunately though, there are lots and lots of interviewers out there that do not give a shit about "how", but rather if you can provide the right answer or not. Here are three questions, you have 60 minutes. Please provide the optimal solution to at least two of those questions. Now excuse me while I'm listening to a teams/zoom meeting in the background. Good luck, have fun.

Jaygles 3y ago

Yeah, if someone is leaning on an LLM, conceptually you are interviewing the LLM and not the candidate. I think the only signal you can get in such a case is how effective the candidate can use an LLM, and maybe something about the candidate's skills if they are able to catch mistakes the LLM makes. But I don't think that's enough signal to ensure someone would not be a bad hire.