6 ms·

When I saw this, I had a good belly laugh! As someone who does a lot of interviewing who also hates all the leetcode style interviews, I love this. I hope it

When I saw this, I had a good belly laugh! As someone who does a lot of interviewing who also hates all the leetcode style interviews, I love this. I hope it ushers in a new era of non-leetcode style interviews. But I suspect what it will actually do is usher in an era of in-person only interviews or having to use that same crap spyware that schools use to lock down your computer by essentially rooting it.

moose_man 4y ago
They need to come up with something better than leetcode. Build a project and explain it is a much better way to get a sense of skill. It's the sort of skill that won't be made extinct by ChatGPT. Your creativity and ability to architect and build a project and then explain it should be the standard. Leetcode also penalizes good engineers who have anxiety during interviews.
- breckenedge 4y ago
  I’ve been doing interviews like this for a few years now. Leetcode sucks. Bonus points the next time a candidate (correctly) uses ChatGPT or Copilot. Let the machines do what the machines do well (grinding leetcode), good riddance.
  - moose_man 4y ago
    Yeah exactly. Since these tools commoditize repetitive skills, the goal should be to see how creative and awesome the project is within the parameters given. Being creative is what you should be testing for since all these other skills are now commodities. It's the one thing we have that AI can't replicate (yet?).
    - leetcode-mafia 4y ago
      Often one of the parameters given is a timeframe that no sane person would request in a real-world setting. The justification I've heard for that is "we don't wan't people spending too long on it."
  - DANmode 4y ago
    Discussion: what does correctly programming with chatbots look like, specifically? Probably easier for most to list what it doesn't look like. I'm interested in both.
    - breckenedge 4y ago
      Asking a chatbot a question rather than Google. I never minded if candidates said they’d use Google for looking up something in the docs. But now you’ve got something a heckuva lot more efficient. For example: What’s the method to select a random item from a range in Ruby? (ChatGPT used to get this wrong.) I don’t mean to say that I give trivia questions in interview, but if the need to know this came up during an interview or code pairing, having the candidate know when a chatbot response was invalid (and where to look for a correct answer) is a good sign. I’m also open to a candidate stubbing out an idea with a chatbot/copilot and then checking the solution and adapting it to fit a given context.
- LanceH 4y ago
  Worked one place where we received seven (7!) resumes that were almost identical, with the same project of what they could do. Not saying leetcode is the answer, but it does solve for something things. My interviews tend to walk through the framework of a project and they can speak to me in the abstract, or an example of what that part of the project does. Say there is a standard way of connecting to the database like in Rails. They can tell me about database.yml and how it has different entries for each of the databases. Then we have a conversation about checking in passwords to git, env variables, secrets managers, etc... This avoids asking stock questions which might be coached/studied more and aims for what the person doing the work might practically know. It also keeps the discussion in a context that makes understanding the questions (hopefully) easier. My style of interviews are very much non-standardized and there has to be some trust that I have some idea what I'm doing. Leetcode at least has some standardization around the problem. Of course everyone could have looked them up and studied the exact solution, and these solutions don't correspond very well to daily efforts. But, given everyone knows the game, it does demonstrate some horsepower I guess. Or maybe I like these interviews because I'm good at them.
  - moose_man 4y ago
    They should all fail if they are all identical. You look for someone who can build something extraordinary, who takes the parameters and runs with them.
    - yieldcrv 4y ago
      Extraordinary is too subjective in my experience When I’m competing for one position and “we just want to see how you think” is always not true but instead a completely arbitrary set of criteria not presented to the candidate, it should be sanctionable
- itake 4y ago
  > Build a project and explain it is a much better way to get a sense of skill. I HATE project interviews, because their skills aren't transferable to the next interview. Take home projects also tend to "go-over", because you want to put your best foot forward and you're betting that the other people they interview are also going to exceed time.
  - IanCal 4y ago
    I don't know if this is what they meant but I read it as you describe an existing project that you've worked on, not a mini one just for the interview. That's how I've interviewed before.
    - itake 4y ago
      > I read it as you describe an existing project that you've worked on IMHO, there is some risk of violating NDAs. For the most part companies don't care if you share how their tech stack works, but I get nervous about revealing IP.
    - munchbunny 4y ago
      I’ve encountered both, significantly more of the “build a mini project for us” variety. I also dislike those because I’ve already got a full time engineering job and a family and don’t have time to put 4-5 hours each into every 3-hour project the interviewers ask for. But 5 years ago I did. I’ve concluded there really isn’t a single way to interview that checks all the boxes, and if you’re running the interview process you have to pay the costs somewhere, unfortunately.
      - grumpy_coder 4y ago
        This is more of a problem than it seems at first. At one place I worked that did mini-project interview most of the hires were currently unemployed and could spend all week on it. (then claim they did it in 3 hours)
qup 4y ago
Couldn't you just run a second PC right behind your monitor that was listening and providing tips?
- jedberg 4y ago
  Yeah but then you'd have to retype the output instead of just copy/pasting it. :)
  - qup 4y ago
    I'll just have gpt email it to me
runnerup 4y ago
> or having to use that same crap spyware that schools use to lock down your computer by essentially rooting it. The most extreme one I’m aware of is “Lockdown Browser” and the student who lives with me has effectively rendered it useless. Students use MST with DisplayPort to mirror a monitor in a way such that the OS cannot see that two monitors exist. (I don’t think there’s a windows API for LDB to see this? Could be wrong) Anyways, one student faces the other, each looking at a mirrored display, one student out of view of the webcam. Then a microphone with a hardwired switch soldered in which, again, doesn’t alert the OS that a microphone has been disconnected, is switched off. Then student 2 can freely speak at the student taking the test and announce all the answers. The test taker is able to keep their eyes on the monitor at all times (so eye tracking won’t show anything weird). If I were black-hat enough to lead a product like lockdownbrowser, I would beat this technique by looking for electrical hum frequency signatures in the audio feed which can pinpoint what time the audio was recorded due to fluctuations in the grid electricity. https://en.m.wikipedia.org/wiki/Electrical_network_frequency_analysis https://en.m.wikipedia.org/wiki/Electrical_network_frequency... If there is zero audio waveform, that would be a flag for review. If the cheater attempts to loop pre-recorded audio, it will be noticeable that the ~60hz frequency signature is “wrong” for the time that the test was being taken. This would then in turn be defeated by the cheating student running a remote microphone to a quiet room which can be switched to. I’ve put a lot of thought into this and no matter what, the cheater eventually wins given infinite cat-and-mouse iterations. But most would be caught somewhere along the way.
- jackblemming 4y ago
  The solution is to trust the people you hire, treat them with respect and decency. And when they start and don't know what a for-loop is because they obviously gamed the process, you fire them. It's really that simple. If people complain it's impossible to fire anyone, that's the issue. The solution isn't to implement an insane anti-cheat hiring pipeline that will drive away competent people.
  - wodenokoto 4y ago
    A common estimate is that the hiring process costs a year of salary. You kinda want to nail it and not have too many do-overs, even if firing is easy.
leetcode-mafia 4y ago
Unfortunately, I think you're right, at least initially. Not much happens unless the cost of losing access to remote talent becomes too significant to ignore.
Nathanba 4y ago
the example video isn't of leetcode though, it's general questions about database tech and it gives you all the right points to mention
weq 4y ago
My cross platform c# apps are used by millions of students weekly to take assessments. We try and work as close with the OS venders as possible to implement lockdowns via native Kiosk functionality (ive posted bugs with all the majors MS/Google/Chrome/Android/Apple as their updates/move fast attitudes are probably my biggest enermies right now). We sometimes cant though and have to find other means because our clients demand we support xOS because vendors agreesively push hardware onto students. We definitely arnt spywhere, schools are very much against anything that will phone home alot of the time. We have pentests and issues like every other game/app in this cat and mouse race but as soon as you close our app it done, nothing lingers. ChatGTP is a growing problem but we have defenses against it (until its on device maybe?). Things like Frida are probably worse right now.
switch007 4y ago
> But I suspect what it will actually do is usher in an era of in-person only interviews A man can dream! I come across really badly on video. I hate the compressed audio, lack of body language, latency etc
angarg12 4y ago
I also do tons of interview for a Big Tech. I ran my interview questions through chatGPT (3, the one available at the time), and the answers varied from pretty good to total garbage. I brought this to the interviewer community in the company, suggesting that it might be time to put leetcode-style interviews in the bin and start over. They dismissed my concerns and will continue as usual. A part of me wants cheating to be rampant to force companies hand in the matter. Unfortunately I'm with you and I suspect they will just enforce in-person, whiteboard interviews again (a colleague explicitly asked for it), rather than trying to come up with a better system.
theRealMe 4y ago
This “burn it all to the ground” mentality is naive. There is nothing leetcode-specific about this cheating platform. OP just angled it that way because HN readers would lap it up that way. Assuming someone was using this cheating platform, how would you run your interviews? Wouldn’t this screw up the actual legit interviews(whatever that is…) too?
- jedberg 4y ago
  I don't do live coding in my interviews because it's silly. We talk about the person's past projects, their contributions to it, insights the gained, mistakes they made and how they would avoid them again, etc. Sometimes they may share some code they wrote and we walk through it and they explain it to me and why they made the choices they made. Sometimes I'll make suggestions and see how well they receive that feedback. I'm looking for people who can demonstrate that they have faced challenges and overcame them, that they communicate their decisions effectively, and that they can learn new information quickly, can receive and incorporate feedback, and so on. Can't really cheat your way out of that.
  - theRealMe 4y ago
    I’d argue that pieces of that interview could still be gamed using this cheating platform depending on how accurate, fast, and realistic sounding the gpt responses are. And all of those attribute will just get better as new models are released. I realize that this one project isn’t the only way to cheat in interviews, but I still think it’s naive to think that this tech will only harm what you perceive to be “bad interviews” and not affect your own preferred interviews. At the absolute minimum, it adds additional overhead to performing interviews where you have to also be aware and try to figure out if the interviewee is being coached like this.
    - jedberg 4y ago
      If the candidate uses a tool to "cheat", and then keeps using that tool while they work for me, is anything lost? Right now people "cheat" by using an IDE, but no one has trouble with that (and rightly so!). So why should I care if someone is using LLMs to pass the interview and do their job if they are being successful?

moose_man 4y ago

They need to come up with something better than leetcode. Build a project and explain it is a much better way to get a sense of skill. It's the sort of skill that won't be made extinct by ChatGPT. Your creativity and ability to architect and build a project and then explain it should be the standard. Leetcode also penalizes good engineers who have anxiety during interviews.

breckenedge 4y ago
I’ve been doing interviews like this for a few years now. Leetcode sucks. Bonus points the next time a candidate (correctly) uses ChatGPT or Copilot. Let the machines do what the machines do well (grinding leetcode), good riddance.
- moose_man 4y ago
  Yeah exactly. Since these tools commoditize repetitive skills, the goal should be to see how creative and awesome the project is within the parameters given. Being creative is what you should be testing for since all these other skills are now commodities. It's the one thing we have that AI can't replicate (yet?).
  - leetcode-mafia 4y ago
    Often one of the parameters given is a timeframe that no sane person would request in a real-world setting. The justification I've heard for that is "we don't wan't people spending too long on it."
- DANmode 4y ago
  Discussion: what does correctly programming with chatbots look like, specifically? Probably easier for most to list what it doesn't look like. I'm interested in both.
  - breckenedge 4y ago
    Asking a chatbot a question rather than Google. I never minded if candidates said they’d use Google for looking up something in the docs. But now you’ve got something a heckuva lot more efficient. For example: What’s the method to select a random item from a range in Ruby? (ChatGPT used to get this wrong.) I don’t mean to say that I give trivia questions in interview, but if the need to know this came up during an interview or code pairing, having the candidate know when a chatbot response was invalid (and where to look for a correct answer) is a good sign. I’m also open to a candidate stubbing out an idea with a chatbot/copilot and then checking the solution and adapting it to fit a given context.
LanceH 4y ago
Worked one place where we received seven (7!) resumes that were almost identical, with the same project of what they could do. Not saying leetcode is the answer, but it does solve for something things. My interviews tend to walk through the framework of a project and they can speak to me in the abstract, or an example of what that part of the project does. Say there is a standard way of connecting to the database like in Rails. They can tell me about database.yml and how it has different entries for each of the databases. Then we have a conversation about checking in passwords to git, env variables, secrets managers, etc... This avoids asking stock questions which might be coached/studied more and aims for what the person doing the work might practically know. It also keeps the discussion in a context that makes understanding the questions (hopefully) easier. My style of interviews are very much non-standardized and there has to be some trust that I have some idea what I'm doing. Leetcode at least has some standardization around the problem. Of course everyone could have looked them up and studied the exact solution, and these solutions don't correspond very well to daily efforts. But, given everyone knows the game, it does demonstrate some horsepower I guess. Or maybe I like these interviews because I'm good at them.
- moose_man 4y ago
  They should all fail if they are all identical. You look for someone who can build something extraordinary, who takes the parameters and runs with them.
  - yieldcrv 4y ago
    Extraordinary is too subjective in my experience When I’m competing for one position and “we just want to see how you think” is always not true but instead a completely arbitrary set of criteria not presented to the candidate, it should be sanctionable
itake 4y ago
> Build a project and explain it is a much better way to get a sense of skill. I HATE project interviews, because their skills aren't transferable to the next interview. Take home projects also tend to "go-over", because you want to put your best foot forward and you're betting that the other people they interview are also going to exceed time.
- IanCal 4y ago
  I don't know if this is what they meant but I read it as you describe an existing project that you've worked on, not a mini one just for the interview. That's how I've interviewed before.
  - itake 4y ago
    > I read it as you describe an existing project that you've worked on IMHO, there is some risk of violating NDAs. For the most part companies don't care if you share how their tech stack works, but I get nervous about revealing IP.
  - munchbunny 4y ago
    I’ve encountered both, significantly more of the “build a mini project for us” variety. I also dislike those because I’ve already got a full time engineering job and a family and don’t have time to put 4-5 hours each into every 3-hour project the interviewers ask for. But 5 years ago I did. I’ve concluded there really isn’t a single way to interview that checks all the boxes, and if you’re running the interview process you have to pay the costs somewhere, unfortunately.
    - grumpy_coder 4y ago
      This is more of a problem than it seems at first. At one place I worked that did mini-project interview most of the hires were currently unemployed and could spend all week on it. (then claim they did it in 3 hours)

qup 4y ago

Couldn't you just run a second PC right behind your monitor that was listening and providing tips?

jedberg 4y ago
Yeah but then you'd have to retype the output instead of just copy/pasting it. :)
- qup 4y ago
  I'll just have gpt email it to me

runnerup 4y ago

> or having to use that same crap spyware that schools use to lock down your computer by essentially rooting it. The most extreme one I’m aware of is “Lockdown Browser” and the student who lives with me has effectively rendered it useless. Students use MST with DisplayPort to mirror a monitor in a way such that the OS cannot see that two monitors exist. (I don’t think there’s a windows API for LDB to see this? Could be wrong) Anyways, one student faces the other, each looking at a mirrored display, one student out of view of the webcam. Then a microphone with a hardwired switch soldered in which, again, doesn’t alert the OS that a microphone has been disconnected, is switched off. Then student 2 can freely speak at the student taking the test and announce all the answers. The test taker is able to keep their eyes on the monitor at all times (so eye tracking won’t show anything weird). If I were black-hat enough to lead a product like lockdownbrowser, I would beat this technique by looking for electrical hum frequency signatures in the audio feed which can pinpoint what time the audio was recorded due to fluctuations in the grid electricity. https://en.m.wikipedia.org/wiki/Electrical_network_frequency_analysis https://en.m.wikipedia.org/wiki/Electrical_network_frequency... If there is zero audio waveform, that would be a flag for review. If the cheater attempts to loop pre-recorded audio, it will be noticeable that the ~60hz frequency signature is “wrong” for the time that the test was being taken. This would then in turn be defeated by the cheating student running a remote microphone to a quiet room which can be switched to. I’ve put a lot of thought into this and no matter what, the cheater eventually wins given infinite cat-and-mouse iterations. But most would be caught somewhere along the way.

jackblemming 4y ago
The solution is to trust the people you hire, treat them with respect and decency. And when they start and don't know what a for-loop is because they obviously gamed the process, you fire them. It's really that simple. If people complain it's impossible to fire anyone, that's the issue. The solution isn't to implement an insane anti-cheat hiring pipeline that will drive away competent people.
- wodenokoto 4y ago
  A common estimate is that the hiring process costs a year of salary. You kinda want to nail it and not have too many do-overs, even if firing is easy.

leetcode-mafia 4y ago

Unfortunately, I think you're right, at least initially. Not much happens unless the cost of losing access to remote talent becomes too significant to ignore.

Nathanba 4y ago

the example video isn't of leetcode though, it's general questions about database tech and it gives you all the right points to mention

weq 4y ago

My cross platform c# apps are used by millions of students weekly to take assessments. We try and work as close with the OS venders as possible to implement lockdowns via native Kiosk functionality (ive posted bugs with all the majors MS/Google/Chrome/Android/Apple as their updates/move fast attitudes are probably my biggest enermies right now). We sometimes cant though and have to find other means because our clients demand we support xOS because vendors agreesively push hardware onto students. We definitely arnt spywhere, schools are very much against anything that will phone home alot of the time. We have pentests and issues like every other game/app in this cat and mouse race but as soon as you close our app it done, nothing lingers. ChatGTP is a growing problem but we have defenses against it (until its on device maybe?). Things like Frida are probably worse right now.

switch007 4y ago

> But I suspect what it will actually do is usher in an era of in-person only interviews A man can dream! I come across really badly on video. I hate the compressed audio, lack of body language, latency etc

angarg12 4y ago

I also do tons of interview for a Big Tech. I ran my interview questions through chatGPT (3, the one available at the time), and the answers varied from pretty good to total garbage. I brought this to the interviewer community in the company, suggesting that it might be time to put leetcode-style interviews in the bin and start over. They dismissed my concerns and will continue as usual. A part of me wants cheating to be rampant to force companies hand in the matter. Unfortunately I'm with you and I suspect they will just enforce in-person, whiteboard interviews again (a colleague explicitly asked for it), rather than trying to come up with a better system.

theRealMe 4y ago

This “burn it all to the ground” mentality is naive. There is nothing leetcode-specific about this cheating platform. OP just angled it that way because HN readers would lap it up that way. Assuming someone was using this cheating platform, how would you run your interviews? Wouldn’t this screw up the actual legit interviews(whatever that is…) too?

jedberg 4y ago
I don't do live coding in my interviews because it's silly. We talk about the person's past projects, their contributions to it, insights the gained, mistakes they made and how they would avoid them again, etc. Sometimes they may share some code they wrote and we walk through it and they explain it to me and why they made the choices they made. Sometimes I'll make suggestions and see how well they receive that feedback. I'm looking for people who can demonstrate that they have faced challenges and overcame them, that they communicate their decisions effectively, and that they can learn new information quickly, can receive and incorporate feedback, and so on. Can't really cheat your way out of that.
- theRealMe 4y ago
  I’d argue that pieces of that interview could still be gamed using this cheating platform depending on how accurate, fast, and realistic sounding the gpt responses are. And all of those attribute will just get better as new models are released. I realize that this one project isn’t the only way to cheat in interviews, but I still think it’s naive to think that this tech will only harm what you perceive to be “bad interviews” and not affect your own preferred interviews. At the absolute minimum, it adds additional overhead to performing interviews where you have to also be aware and try to figure out if the interviewee is being coached like this.
  - jedberg 4y ago
    If the candidate uses a tool to "cheat", and then keeps using that tool while they work for me, is anything lost? Right now people "cheat" by using an IDE, but no one has trouble with that (and rightly so!). So why should I care if someone is using LLMs to pass the interview and do their job if they are being successful?