4 ms·
Over Christmas I met up with a friend who teaches part-time at a local college. He said he’s failed more people this year than the cumulative total of his entir
by Aurornis 2y ago
Over Christmas I met up with a friend who teaches part-time at a local college. He said he’s failed more people this year than the cumulative total of his entire past teaching career due to LLM abuse.
He doesn’t use LLM detection tools, but he says it’s easy to identify papers with warning signs of LLM use. For some reason, using ChatGPT for his specific niche topic overuses a few obscure, rarely-used words that most people wouldn’t even recognize. The ChatGPT abusers some times have these words appearing multiple times through their essays.
He’s also caught people who cited a lot of different works and books in their reports that were outside of the assigned reading, or in some cases books that don’t exist at all. Catching them is as simple as asking them about their sources or where they acquired a copy of the text.
I see a lot of parallels in hiring and talking to junior software engineers right now. We had a take-home problem that was well liked that we used for many years, but now it’s obvious that a majority of young applicants are just using LLMs to get an answer. When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method.
It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved.
- Shocka1 2y agoOverly expressive words like "robust" and words that would appear in the thesaurus for "extremely" seem like tell tale signs in my experience. I've noticed personally that sometimes I have to tell ChatGPT to "sound more human and concise." I'd love to hear if anyone else has had the same experience as me. Regardless, if ChatGPT is tailored enough, or a custom model is created, I can't think of any way to detect if an LLM has authored something generic. The lazy college student will probably get caught, but the cunning one not so much.
- the_snooze 2y ago>It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved. Technology enables scale and reach, which solves some problems but also creates its own set of issues. I think you're right on with the solution: do things that are anti-scale. If you make things a bit more inconvenient, a bit more costly, and a bit more local, you create an environment where there's space for trust and humanity---values that don't scale.
- salomon812 2y agoI know the on-site is time-consuming and expensive, but so is firing people (at least in United States it is.) I've had a few on-site interviews where having them on-site made us realize we could never work with them. Given how much time they will spend with you, it's totally worth it to spend your resources on hiring.
- sarchertech 2y agoFiring people isn’t as expensive as people make it out to be. People vastly overestimate the chance of a lawsuit. And they overestimate the chance of a lawsuit that makes it far enough that it costs significant money even more. Hiring fast and firing fast (for lying or misrepresentation) is almost always a better business decision than being ultra defensive in the hiring process.
- salomon812 2y agoIt's not the lawsuit, it's about the time wasted as a manager and salary to the person as you work out if it's actually time to fire. Performance Improvement Plans, a bunch of back-and-forths. I'm not going to be the kind of person that fires quickly, so there's a bunch of sunk cost we have to take. Plus, fast firing creates a cooling effect among everyone there. And for what? To save money on hiring? Not worth it.
- floating-io 2y agoThe counter argument is that firing too slowly can be a serious drag on morale. Leaving your team to carry dead weight can really suck for the team. Ask me how I know... :)
- salomon812 2y agoOkay, so the original argument is about whether or not it's worth it to fly people out for an on-site. Hotel and airfare: $2000 absolute max. Salary at $100/hr for one month for me to figure out it's not going to work out, then pull the trigger to fire: $24,000. I mean, being a manager is hard, but putting in the time and money to hire and then putting in the time to make sure your team doesn't have a morale drag, it's worth it.
- throwaway2037 2y ago> When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method. Sweet! That sounds like perfect signal for "used ChatGPT" to answer this question. So, you can send take home test, candidate sends reply (many from ChatGPT), then you do quick follow-up phone/video call to discuss the code. When you get the "signal" (should be quick!), then you immediately close the interview and move to the next candidate.