4 ms·

I'm an active interviewer for a FAANG company. I've run some of my interview questions through ChatGPT with mixed results. I'd say that on it's current form it

I'm an active interviewer for a FAANG company. I've run some of my interview questions through ChatGPT with mixed results. I'd say that on it's current form it could easily help a Junior candidate to pass the interview, not so much for a mid or senior (yet). In an internal slack channel I proposed to generate (incorrect) solutions for coding problems with ChatGPT and have candidate do code reviews as an alternative interview format. I was almost chastised for proposing this. Someone proposed that, if I wanted to go down that path, I could manually generate a wrong solution so that it is curated. I think he is missing the point.

8organicbits 4y ago
Wouldn't you want to curate? Maybe I'm missing the point too.
- memalign 4y ago
  OP might be implying that from now on many engineers will use AI tools to write the first draft of their code so it’s smart to hire people skilled at correcting AI-generated code.
  - h0l0cube 4y ago
    Or that the inherent variability in ChatGPT output could make it harder to game examinations based on the same set of prompts
  - angarg12 4y ago
    You get the point.
  - dicknuckle 4y ago
    It's also good to hire people who are skilled at correcting bad code in general. This is also a good litmus(?) test to see the interesting ways people come to solutions to other's faults.
  - shanebellone 4y ago
    "many engineers will use AI tools to write the first draft of their code" I fail to see how ChatGPT would be useful for capable engineers. Is boilerplate code actually a time suck for anyone?
    - williamcotton 4y ago
      Yes, and also having to scan through search engine results and documentation just to find out how to do something like this: const ts = require("typescript"); const options = { target: ts.ScriptTarget.ES2017, module: ts.ModuleKind.CommonJS, }; const someJavaScriptCode = ts.transpileModule(someTypeScriptCode, { compilerOptions: options }); console.log(someJavaScriptCode.outputText); (Actual ChatGPT response when asked to convert TS to JS) That saved me a good five minutes with minimal disruption to my flow. Those little things add up to a significant amount of time and energy. I have two young children, ages 1 and 4, and these tools make it possible for me to program in little 10 minutes bursts between basically endless cooking and cleaning chores. I actually take offense to the notion that you would consider people who find these tools useful to be incapable engineers.
      - shanebellone 4y ago
        "I actually take offense to the notion that you would consider people who find these tools useful to be incapable engineers." I apologize for offending you. I can't imagine parsing code for errors being faster than typing it out. I suppose I assumed most have boilerplate memorized. I'll try to avoid being so presumptuous in the future.
        williamcotton 4y ago
        There's two ways to check if something is correct: run it or read it. In this case I just copy and pasted the results and it worked as expected. YMMV. I will say that you seem to be operating under the assumption that these tools are less reliable at producing the correct code than they really are! Ironically, the latest GPT-3 models trained before OpenAI made changes to their API resulting in basically useless assistance for things like the openai npm library!
- jerrre 4y ago
  He can still curate right?
- michaelt 4y ago
  The eternal loop in job interview questions. Should you ask very similar questions, to keep candidates on a level playing field by always measuring to the same standard? Or should you vary the questions a lot, to keep candidates on a level playing field by not advantaging those who've seen leaked questions on glassdoor?
- angarg12 4y ago
  I must have expressed myself incorrectly. Generating coding problems automatically with ChatGPT doesn't mean you just throw whatever to the candidates. You can both auto generate AND curate. The part I'm against it is manually creating problems. There are several reasons for this. One of the biggest problems with FAANG interviewing is that questions get leaked almost instantly. I spent days creating a brand new original problem, and it got leaked by the second time I asked it. ChatGPT gives you a unique problem for each candidate. There is also the angle of leaning in the tech. I would encourage candidates to use any means to review the code as they would in the workplace, including IDE, web search, and LLMs. If these tools are going to revolutionize the way we work, I want to see how well people can use them. Lastly I'm also intrigued by the dynamics of this kind of interview. As an interviewer I have asked the same question 50 times and I have seen all variations of answers. But what if the problem is relatively new to me? This seems closer to a pair programming exercise with two colleagues working together to solve a problem.
weird-eye-issue 4y ago
Curating it would give a level playing field to candidates and allow you to decide what you are actually testing for, but maybe I'm also missing the point
- angarg12 4y ago
  You can do both. Generate 10 solutions with ChatGPT and pick the ones that work best. Do this for each candidate.
  - weird-eye-issue 4y ago
    That is curating

8organicbits 4y ago

Wouldn't you want to curate? Maybe I'm missing the point too.

memalign 4y ago
OP might be implying that from now on many engineers will use AI tools to write the first draft of their code so it’s smart to hire people skilled at correcting AI-generated code.
- h0l0cube 4y ago
  Or that the inherent variability in ChatGPT output could make it harder to game examinations based on the same set of prompts
- angarg12 4y ago
  You get the point.
- dicknuckle 4y ago
  It's also good to hire people who are skilled at correcting bad code in general. This is also a good litmus(?) test to see the interesting ways people come to solutions to other's faults.
- shanebellone 4y ago
  "many engineers will use AI tools to write the first draft of their code" I fail to see how ChatGPT would be useful for capable engineers. Is boilerplate code actually a time suck for anyone?
  - williamcotton 4y ago
    Yes, and also having to scan through search engine results and documentation just to find out how to do something like this: const ts = require("typescript"); const options = { target: ts.ScriptTarget.ES2017, module: ts.ModuleKind.CommonJS, }; const someJavaScriptCode = ts.transpileModule(someTypeScriptCode, { compilerOptions: options }); console.log(someJavaScriptCode.outputText); (Actual ChatGPT response when asked to convert TS to JS) That saved me a good five minutes with minimal disruption to my flow. Those little things add up to a significant amount of time and energy. I have two young children, ages 1 and 4, and these tools make it possible for me to program in little 10 minutes bursts between basically endless cooking and cleaning chores. I actually take offense to the notion that you would consider people who find these tools useful to be incapable engineers.
    - shanebellone 4y ago
      "I actually take offense to the notion that you would consider people who find these tools useful to be incapable engineers." I apologize for offending you. I can't imagine parsing code for errors being faster than typing it out. I suppose I assumed most have boilerplate memorized. I'll try to avoid being so presumptuous in the future.
      - williamcotton 4y ago
        There's two ways to check if something is correct: run it or read it. In this case I just copy and pasted the results and it worked as expected. YMMV. I will say that you seem to be operating under the assumption that these tools are less reliable at producing the correct code than they really are! Ironically, the latest GPT-3 models trained before OpenAI made changes to their API resulting in basically useless assistance for things like the openai npm library!
jerrre 4y ago
He can still curate right?
michaelt 4y ago
The eternal loop in job interview questions. Should you ask very similar questions, to keep candidates on a level playing field by always measuring to the same standard? Or should you vary the questions a lot, to keep candidates on a level playing field by not advantaging those who've seen leaked questions on glassdoor?
angarg12 4y ago
I must have expressed myself incorrectly. Generating coding problems automatically with ChatGPT doesn't mean you just throw whatever to the candidates. You can both auto generate AND curate. The part I'm against it is manually creating problems. There are several reasons for this. One of the biggest problems with FAANG interviewing is that questions get leaked almost instantly. I spent days creating a brand new original problem, and it got leaked by the second time I asked it. ChatGPT gives you a unique problem for each candidate. There is also the angle of leaning in the tech. I would encourage candidates to use any means to review the code as they would in the workplace, including IDE, web search, and LLMs. If these tools are going to revolutionize the way we work, I want to see how well people can use them. Lastly I'm also intrigued by the dynamics of this kind of interview. As an interviewer I have asked the same question 50 times and I have seen all variations of answers. But what if the problem is relatively new to me? This seems closer to a pair programming exercise with two colleagues working together to solve a problem.

weird-eye-issue 4y ago

Curating it would give a level playing field to candidates and allow you to decide what you are actually testing for, but maybe I'm also missing the point

angarg12 4y ago
You can do both. Generate 10 solutions with ChatGPT and pick the ones that work best. Do this for each candidate.
- weird-eye-issue 4y ago
  That is curating