I get the same results from asking for code. Near 100% success with an initial demo, and then pretty tragic performance when making a series of changes to actua

I get the same results from asking for code. Near 100% success with an initial demo, and then pretty tragic performance when making a series of changes to actually fit the concept. I got it to write 3d rotation code, and to switch to using rotors instead of quaternions but when I asked it to rewrite the combine_rotors() method it rewrote it to combine Enigma rotors. I explained its error and it apologized but just wasn't able to go back to the original code and work with it anymore. Interviewing is going to be shaken up but I think some methods are more timeless. I've been giving the subject code instead of asking them to write it for a while now. Largely because I wanted to talk more and watch them type less. We start by discussing the coding challenge guidelines and I leave them vague. They need to understand the goal and what I've left out and suggest those guidelines themselves. Once we agree though, I give them the code. "Here's what's running now." Then I update the goal and we discuss changes. I get them to "whiteboard" certain things, like what a query looks like with their proposed changes or whatever, and we discuss big-O, etc. This, imho, is how whiteboarding is actually used - not to write whole programs but to provide examples and pick them apart. I feel that this would work even if they were using an AI in another window. We're trying to select for developers with common sense and domain knowledge, and who can clearly discuss engineering tradeoffs. Actually making the changes (the coding itself) was a big part of the job and that's decreasing, but imho all the other requirements remain. They'll still need to know how to handle the issues I talked about above, of trying to get the model to write the right code!