5 ms·

I'm fairly open book, but I wouldn't accept LLM usage in an interview. I don't need someone to have all the facts in their head, so if they have to look up some

I'm fairly open book, but I wouldn't accept LLM usage in an interview. I don't need someone to have all the facts in their head, so if they have to look up some syntax or whatever, then totally fine. However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.

michael_mroczka 3y ago
Exactly. People end up throwing the baby out with the bathwater on this. "Data structure & algorithm interviews aren't perfect, so let's not ask people to code at all." It's an absurd overcorrection, but most people think these interviews are about demanding optimal code and perfection when they mostly just are making sure you're not using arrays when you should be using hashmaps... and that you know what a hashmap is, I suppose.
- blkhawk 3y ago
  Maybe combine the two - Have the applicant do a small task then talk about it. Maybe give them a slightly dubious direction then see if they followed to the letter or deviated and their reasoning for it. I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds. Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
  - danielvaughn 3y ago
    My favorite style of interview was actually a take-home exercise. Back in 2016 I interviewed with Blue Apron and they asked me to create a custom javascript framework from scratch to render a recipe app or something like that. Then I had an interview where I discussed my solution and we reviewed the code together. It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.
  - michael_mroczka 3y ago
    Your thought is a good one, and I think it is a valid approach. The barrier that comes up when doing this is still going to be cheating. How do you separate the people who are cheating on these at-home tests? Salesforce had a good interview practice a few years back where they invited you to a meeting. Started a recording, then asked you to keep your microphone and camera on and do several simple programming tasks. It was an "open book," and you could use whatever you wanted, but you just had to show how you got to where you were (and you could only use one monitor so that it was clear what you were looking at at all times). The engineer who met you on the call left after just a couple of minutes, and you could work in peace without having to worry about "entertaining them."
sanderjd 3y ago
This is intriguingly dissonant. If your style of interview is "LLM-proof" then why would you care whether people choose to use whatever tools they're comfortable with - including LLMs - if they so desire? For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews. (But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)
- danielvaughn 3y ago
  Fair enough, this was a drive-by comment that I didn't give much thought to. What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem. It actually is very important that you're able to think through problems on your own, sans-llm.
  - sanderjd 3y ago
    > But LLMs simulate the abstract thinking for you This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal. The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them! But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.

michael_mroczka 3y ago

Exactly. People end up throwing the baby out with the bathwater on this. "Data structure & algorithm interviews aren't perfect, so let's not ask people to code at all." It's an absurd overcorrection, but most people think these interviews are about demanding optimal code and perfection when they mostly just are making sure you're not using arrays when you should be using hashmaps... and that you know what a hashmap is, I suppose.

blkhawk 3y ago
Maybe combine the two - Have the applicant do a small task then talk about it. Maybe give them a slightly dubious direction then see if they followed to the letter or deviated and their reasoning for it. I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds. Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
- danielvaughn 3y ago
  My favorite style of interview was actually a take-home exercise. Back in 2016 I interviewed with Blue Apron and they asked me to create a custom javascript framework from scratch to render a recipe app or something like that. Then I had an interview where I discussed my solution and we reviewed the code together. It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.
- michael_mroczka 3y ago
  Your thought is a good one, and I think it is a valid approach. The barrier that comes up when doing this is still going to be cheating. How do you separate the people who are cheating on these at-home tests? Salesforce had a good interview practice a few years back where they invited you to a meeting. Started a recording, then asked you to keep your microphone and camera on and do several simple programming tasks. It was an "open book," and you could use whatever you wanted, but you just had to show how you got to where you were (and you could only use one monitor so that it was clear what you were looking at at all times). The engineer who met you on the call left after just a couple of minutes, and you could work in peace without having to worry about "entertaining them."

sanderjd 3y ago

This is intriguingly dissonant. If your style of interview is "LLM-proof" then why would you care whether people choose to use whatever tools they're comfortable with - including LLMs - if they so desire? For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews. (But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)

danielvaughn 3y ago
Fair enough, this was a drive-by comment that I didn't give much thought to. What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem. It actually is very important that you're able to think through problems on your own, sans-llm.
- sanderjd 3y ago
  > But LLMs simulate the abstract thinking for you This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal. The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them! But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.