4 ms·

I'm the author of this post. Happy to answer questions if you have any. This was such a fascinating experiment!

dmazzoni 3y ago
One of the implicit assumptions in your post is that if you ask a custom question, ChatGPT won't do as well. Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose. It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.
- michael_mroczka 3y ago
  It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions. Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging. As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are. As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.
  - fragmede 3y ago
    right, it's really good at cheater code. if you had to write a function equivalent to is_odd, it would take the example cases and if those values directly instead of doing modulo. of course if you ask it that directly, it'll do modulo, but that's the kind of code it'll output. doesn't mean it's useless, but you gotta be careful when using code its output. if a dumb person is using ChatGPT to cheat on an interview, it'll be easy to tell from the code given if it's a bad question thats being given.

One of the implicit assumptions in your post is that if you ask a custom question, ChatGPT won't do as well. Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose. It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.

michael_mroczka 3y ago
It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions. Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging. As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are. As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.
- fragmede 3y ago
  right, it's really good at cheater code. if you had to write a function equivalent to is_odd, it would take the example cases and if those values directly instead of doing modulo. of course if you ask it that directly, it'll do modulo, but that's the kind of code it'll output. doesn't mean it's useless, but you gotta be careful when using code its output. if a dumb person is using ChatGPT to cheat on an interview, it'll be easy to tell from the code given if it's a bad question thats being given.