4 ms·

Guess we can throw out "look at a job candidate's github repo" as a screening tool.

langitbiru 4y ago
There are two ways forward for testing candidates' skills: - offline testing - asking candidates to create ridiculous and hardcore apps that even ChatGPT is not sufficient. For example, creating a web browser with Rust.
samtheprogram 4y ago
If the candidate can get the work done, does it matter how? Tools like this are going to become more and more ubiquitous, and I’m questioning myself whether it makes sense to avoid it almost entirely as I have been or to embrace using it to accomplish tasks. Unfortunately (IMO) we are through the looking glass, as they say.
- joenot443 4y ago
  Copilot has a free trial, I think it’s worth anyone giving it a try just to see how it feels. I’m using it in my workflow for when I’m writing really repeatable test cases or boilerplate. For anything more complicated, very quickly the limitations will start to appear which has brought me comfort that it’ll be a long time before this is going to replace me entirely. I imagine it’ll be a bit similar to self driving cars, in that the technology behind the base 80% of cases is incredibly impressive, but the human element to solve the other 20% remains out of reach for longer than we hope.
- VectorLock 4y ago
  Using it as a boilerplate generator I've found is pretty handy. That's something I've always wanted in my workflow.
nearmuse 4y ago
If to "look" means to verify the existence of things in it and to skim the code for style etc without even running it, then yes. Many people have these repos with a fork of some opensource project with like 2 commits in it, or something not parting too far from a template project. I am not saying this does not qualify - because those commits might be useful or insightful, or merely sufficient to solve a problem - but they are way too easy to fake.
- VectorLock 4y ago
  They're not even easier to fake. You could spread out these commits over time with each commit just being the next iteration from chatGPT. I'd bet sooner rather than later someone will offer this as a service. Most first line screens/recruiters aren't going to do much more than skim code and aren't going to run it. Best case scenario now your recruiting pipeline is inflated with more poor candidates. Worst case scenario they're not one step closer to bamboozling their way to a job they're not qualified for.
koboll 4y ago
Have you tried asking it your company's tech interview coding challenge questions yet? I have, and I'm pretty convinced coding challenges are going to be a thing of the past soon. Either that or there will need to be extreme anti-cheating measures in place.
- agolio 4y ago
  It's better than 90% of our applicants for junior positions. I doubt we would hire it, but it would definitely pass the first tech screens.
  - koboll 4y ago
    I guess by thing of the past soon I don't mean ChatGPT will literally end coding challenges immediately, but whatever comes next (analogous to DALL-E 2 -> Stable Diffusion -> Midjourney 4 -> ???) probably will. For junior positions first, then mid-level...
- hoosieree 4y ago
  I teach CS/Engineering at a university and it's really making me rethink how we should be teaching and what we should be emphasizing. This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum. I can see this becoming a standard tool, which shifts the burden on the student from writing code to critically reading code, writing tests, and learning how to judge correctness.
  - nerdponx 4y ago
    > This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum. But the counter-argument is the same: you don't want to be reliant on a calculator at all times. And that's assuming the tech can even get to the point of replacing the curriculum. It only works as well as it does because it's been trained on text that is effectively many past curricula. What if you add something new? The model will be useless. I'd be shocked if ChatGPT knew as much about, say, Coq as it did about Python.
rvz 4y ago
Then, I will go a step further and ask the candidate if they have contributed any functioning code with code review in the open to other large existing open-source projects that are certainly not generated by ChatGPT. Examples such as Node,js, TypeScript, Rust, WebKit, Linux, etc. Asking for an AI to generate code is one thing. Explaining the code you have contributed to large open-source projects and why you have done so is another and ChatGPT is quite bad at transparently explaining its own decisions and why it got to that solution in the first place.
ornornor 4y ago
In my experience it was never a thing in the first place. I have quite a few non trivial and complete OSS projects on my GH, and in my 10+ year career as a SDE no one has ever really looked at any of it. But they do frequently ask me to take their code challenge to prove I can program.
- ChrisMarshallNY 4y ago
  Yup. I have a huge GH portfolio[0]. Dozens of complete, soup-to-nuts, documented, tested, and in-use projects. I have complete source for shipping apps, backend frameworks, APIs, SDKs, communication libraries, UI widgets, teaching material (complete course modules), etc. I also have many personal blog entries, and online documentation for stuff; not to mention years of tags and commit history. No one ever looked at them. In fact, when I asked one interviewer, why they didn't bother looking at them, I was told "You probably faked it." Needless to say, that interview did not proceed any farther. [0] https://github.com/ChrisMarshallNY#browse-away https://github.com/ChrisMarshallNY#browse-away (NOTE: I just archived and retired a bunch of stuff, because I maintain what I publish; even if I am the only consumer. I dogfood most of my work).
  - mike_hearn 4y ago
    Sadly they said that because it does happen. I hired someone on the back of their GitHub repo once. He talked me into it by saying he struggled to control nerves during coding interviews etc. Regretted it immediately - the code he wrote once he joined inexplicably bore no resemblance to what was in his repo. He also turned out to have a major attitude problem with the language we were actually working in, suggesting at one point that maybe he should be able to write his code in Common Lisp and then we should hire cheap Indians to rewrite it into our actual language. This sort of thing seems hard to detect unless you actually watch someone code. The stuff on his repo could have come from anywhere or anyone, really.
    - ChrisMarshallNY 4y ago
      > This sort of thing seems hard to detect This was not my experience. I never gave coding tests, and would have killed for a GH repo. I was a manager for 25 years, and we did pretty hairy C++ programming. What a code repo gives me, is a fulcrum for a discussion. I would never hire, based on a code repo, or a test. What I did hire on, was fairly long, casual conversations, where I drew the applicant out, and got them to talk about their work, problems they solved, and design decisions they made. A GH repo would give me an "in" for that. "I see that you wrote the firmware for an AI-powered cheese straightener. Tell me about how it came to be. ... Really? How did you solve that problem?" etc. A lot of the folks that I interviewed were not "people persons." I found that once we got into one of these conversations, they opened right up. Also, and this actually doesn't have much to do with my tech experience, I'm a pretty hard person to BS. One reason, is that I come across as a naive, credulous, person, and bulshitters can't seem to help themselves. They just have to try yanking my chain. It's kind of fun, actually.

There are two ways forward for testing candidates' skills: - offline testing - asking candidates to create ridiculous and hardcore apps that even ChatGPT is not sufficient. For example, creating a web browser with Rust.

samtheprogram 4y ago

If the candidate can get the work done, does it matter how? Tools like this are going to become more and more ubiquitous, and I’m questioning myself whether it makes sense to avoid it almost entirely as I have been or to embrace using it to accomplish tasks. Unfortunately (IMO) we are through the looking glass, as they say.

joenot443 4y ago
Copilot has a free trial, I think it’s worth anyone giving it a try just to see how it feels. I’m using it in my workflow for when I’m writing really repeatable test cases or boilerplate. For anything more complicated, very quickly the limitations will start to appear which has brought me comfort that it’ll be a long time before this is going to replace me entirely. I imagine it’ll be a bit similar to self driving cars, in that the technology behind the base 80% of cases is incredibly impressive, but the human element to solve the other 20% remains out of reach for longer than we hope.
VectorLock 4y ago
Using it as a boilerplate generator I've found is pretty handy. That's something I've always wanted in my workflow.

nearmuse 4y ago

If to "look" means to verify the existence of things in it and to skim the code for style etc without even running it, then yes. Many people have these repos with a fork of some opensource project with like 2 commits in it, or something not parting too far from a template project. I am not saying this does not qualify - because those commits might be useful or insightful, or merely sufficient to solve a problem - but they are way too easy to fake.

VectorLock 4y ago
They're not even easier to fake. You could spread out these commits over time with each commit just being the next iteration from chatGPT. I'd bet sooner rather than later someone will offer this as a service. Most first line screens/recruiters aren't going to do much more than skim code and aren't going to run it. Best case scenario now your recruiting pipeline is inflated with more poor candidates. Worst case scenario they're not one step closer to bamboozling their way to a job they're not qualified for.

koboll 4y ago

Have you tried asking it your company's tech interview coding challenge questions yet? I have, and I'm pretty convinced coding challenges are going to be a thing of the past soon. Either that or there will need to be extreme anti-cheating measures in place.

agolio 4y ago
It's better than 90% of our applicants for junior positions. I doubt we would hire it, but it would definitely pass the first tech screens.
- koboll 4y ago
  I guess by thing of the past soon I don't mean ChatGPT will literally end coding challenges immediately, but whatever comes next (analogous to DALL-E 2 -> Stable Diffusion -> Midjourney 4 -> ???) probably will. For junior positions first, then mid-level...
hoosieree 4y ago
I teach CS/Engineering at a university and it's really making me rethink how we should be teaching and what we should be emphasizing. This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum. I can see this becoming a standard tool, which shifts the burden on the student from writing code to critically reading code, writing tests, and learning how to judge correctness.
- nerdponx 4y ago
  > This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum. But the counter-argument is the same: you don't want to be reliant on a calculator at all times. And that's assuming the tech can even get to the point of replacing the curriculum. It only works as well as it does because it's been trained on text that is effectively many past curricula. What if you add something new? The model will be useless. I'd be shocked if ChatGPT knew as much about, say, Coq as it did about Python.

rvz 4y ago

Then, I will go a step further and ask the candidate if they have contributed any functioning code with code review in the open to other large existing open-source projects that are certainly not generated by ChatGPT. Examples such as Node,js, TypeScript, Rust, WebKit, Linux, etc. Asking for an AI to generate code is one thing. Explaining the code you have contributed to large open-source projects and why you have done so is another and ChatGPT is quite bad at transparently explaining its own decisions and why it got to that solution in the first place.

ornornor 4y ago

In my experience it was never a thing in the first place. I have quite a few non trivial and complete OSS projects on my GH, and in my 10+ year career as a SDE no one has ever really looked at any of it. But they do frequently ask me to take their code challenge to prove I can program.

ChrisMarshallNY 4y ago
Yup. I have a huge GH portfolio[0]. Dozens of complete, soup-to-nuts, documented, tested, and in-use projects. I have complete source for shipping apps, backend frameworks, APIs, SDKs, communication libraries, UI widgets, teaching material (complete course modules), etc. I also have many personal blog entries, and online documentation for stuff; not to mention years of tags and commit history. No one ever looked at them. In fact, when I asked one interviewer, why they didn't bother looking at them, I was told "You probably faked it." Needless to say, that interview did not proceed any farther. [0] https://github.com/ChrisMarshallNY#browse-away https://github.com/ChrisMarshallNY#browse-away (NOTE: I just archived and retired a bunch of stuff, because I maintain what I publish; even if I am the only consumer. I dogfood most of my work).
- mike_hearn 4y ago
  Sadly they said that because it does happen. I hired someone on the back of their GitHub repo once. He talked me into it by saying he struggled to control nerves during coding interviews etc. Regretted it immediately - the code he wrote once he joined inexplicably bore no resemblance to what was in his repo. He also turned out to have a major attitude problem with the language we were actually working in, suggesting at one point that maybe he should be able to write his code in Common Lisp and then we should hire cheap Indians to rewrite it into our actual language. This sort of thing seems hard to detect unless you actually watch someone code. The stuff on his repo could have come from anywhere or anyone, really.
  - ChrisMarshallNY 4y ago
    > This sort of thing seems hard to detect This was not my experience. I never gave coding tests, and would have killed for a GH repo. I was a manager for 25 years, and we did pretty hairy C++ programming. What a code repo gives me, is a fulcrum for a discussion. I would never hire, based on a code repo, or a test. What I did hire on, was fairly long, casual conversations, where I drew the applicant out, and got them to talk about their work, problems they solved, and design decisions they made. A GH repo would give me an "in" for that. "I see that you wrote the firmware for an AI-powered cheese straightener. Tell me about how it came to be. ... Really? How did you solve that problem?" etc. A lot of the folks that I interviewed were not "people persons." I found that once we got into one of these conversations, they opened right up. Also, and this actually doesn't have much to do with my tech experience, I'm a pretty hard person to BS. One reason, is that I come across as a naive, credulous, person, and bulshitters can't seem to help themselves. They just have to try yanking my chain. It's kind of fun, actually.