3 ms·

This is intriguingly dissonant. If your style of interview is "LLM-proof" then why would you care whether people choose to use whatever tools they're comfortabl

This is intriguingly dissonant. If your style of interview is "LLM-proof" then why would you care whether people choose to use whatever tools they're comfortable with - including LLMs - if they so desire? For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews. (But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)

danielvaughn 3y ago
Fair enough, this was a drive-by comment that I didn't give much thought to. What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem. It actually is very important that you're able to think through problems on your own, sans-llm.
- sanderjd 3y ago
  > But LLMs simulate the abstract thinking for you This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal. The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them! But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.

danielvaughn 3y ago

Fair enough, this was a drive-by comment that I didn't give much thought to. What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem. It actually is very important that you're able to think through problems on your own, sans-llm.

sanderjd 3y ago
> But LLMs simulate the abstract thinking for you This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal. The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them! But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.