9 ms·
I'm a software manager that has been doing some form of interviewing/hiring for 13 years. I did two rounds of hiring software engineers last year, one in sprin
by salomon812 2y ago
I'm a software manager that has been doing some form of interviewing/hiring for 13 years.
I did two rounds of hiring software engineers last year, one in spring that seemed normal, and one in the fall that was was brutal. The fall hiring had a flood of applicants, and in retrospect, most seemed like AI was used in some way.
For the fall round, I suddenly had a higher percentage of applicants that qualified after resume screening and initial phone screen, but they all collapsed when I did a technical round. And failure rate on the technical was much much worse than usual.
We have a full-time recruiter working with us, and I'm not 100% sure what tools he used, but I switched to manually reviewing each resume, and given that it was 100s, it took a long time, but I still had my problem of great initial screen, terrible technical interview.
Then, I decided to throw out anyone who heavily mentioned AI, LLM, or data science. After all, with almost a thousand applicants, I needed to sort some how. (To be fair, our use case is more esoteric, we're not writing Javascript or parsers, so it's not as much of a time-saver.) Large chunks of applicants dropped and the process felt more normal.
I also switched to only on-site interviews. My initial technical screenings are still done remotely. Before COVID we were 100% on-site interviews, but did hybrid after COVID. Now, I'm back to enforcing on-site for my group.
- deleted 2y ago[deleted]
- pknomad 2y agoI am a Platform Engineer and it feels like your experience mirrors mine. Like you, our challenge is filtering out large volume but also filtering out LLM abusers. We're not opposed to people using LLMs, when appropriate. I find that candidates inappropriately use it to circumvent the process and that is a big deal for me (and our team). We try to do the right thing(TM) by the candidates by creating minimal interview workloads, asking highly relevant questions that aren't "gotchas", and updating their candidacy as soon as possible. It doesn't feel like many candidates are interested in returning the same courtesy. This kind of behavior means we have to lean harder into tapping our existing networks for sourcing "trust-worthy" candidates. That puts us at risk for creating additional blinders and also unfairly filters out "un-networked" candidates. For whatever it's worth, we are remote-first org so all of our interviews are done remotely. One of the things I'm thinking about doing in the future is sharing the screen with diagrams and adding irrelevant annotations to it (while clearly indicating to the candidates that those are irrelevant) as a primitive adversarial AI technique. Perhaps on-site interviews is part of the solution.
- mvanbaak 2y agoFellow Platform Engineer here, and I can relate 100% with your comment. We decided to stop announcing our engineering jobs and go back to mouth to mouth for sourcing candidates for now. It's a move I didn't want to make as, like you said, it means a lot of less networked engineers will not know about it and all. but for now this was the only way we got rid of the constant stream of letters from AI.
- throwaway2037 2y agoWhat is a "Platform Engineer"? I never heard that term before today.
- beardedwizard 2y agoElitism is alive and well in this little nook. Equating platform engineering, SRE and sysadmin to the same thing. Platforms are often large scale distributed systems, dealing with problems like ensuring 100000s of compute nodes are in a deployed and in consistent state. Millions of lines of code are written, peer reviewed and committed to solve this problem. This mirrors an attitude I have frequently encountered from "traditional" or "mainstream" software engineers who devalue any work that isn't features, and don't want to have to work on problems like "make my feature appear on all deployments and work well" - it's just something sysadmins do amirite?
- hipadev23 2y agoLatest rebranding of “sysadmin”, which became “devops engineer” or SRE a decade ago. It’s the people who shove kubernetes, datadog, and CI/CD tools into every corner.
- pknomad 2y agoPlatform Engineers are operationally focused software engineers who focus on enablement of other software engineering groups through building self-service tooling and create unified platform for app deployment. The cultural focus is placed on enablement of teams through self service, whereas DevOps is more about reducing silos and SRE is more about doing infra through the software engineering lens with metrics (SLO/SLA/SLI).
- deleted 2y ago[deleted]
- Aurornis 2y agoOver Christmas I met up with a friend who teaches part-time at a local college. He said he’s failed more people this year than the cumulative total of his entire past teaching career due to LLM abuse. He doesn’t use LLM detection tools, but he says it’s easy to identify papers with warning signs of LLM use. For some reason, using ChatGPT for his specific niche topic overuses a few obscure, rarely-used words that most people wouldn’t even recognize. The ChatGPT abusers some times have these words appearing multiple times through their essays. He’s also caught people who cited a lot of different works and books in their reports that were outside of the assigned reading, or in some cases books that don’t exist at all. Catching them is as simple as asking them about their sources or where they acquired a copy of the text. I see a lot of parallels in hiring and talking to junior software engineers right now. We had a take-home problem that was well liked that we used for many years, but now it’s obvious that a majority of young applicants are just using LLMs to get an answer. When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method. It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved.
- Shocka1 2y agoOverly expressive words like "robust" and words that would appear in the thesaurus for "extremely" seem like tell tale signs in my experience. I've noticed personally that sometimes I have to tell ChatGPT to "sound more human and concise." I'd love to hear if anyone else has had the same experience as me. Regardless, if ChatGPT is tailored enough, or a custom model is created, I can't think of any way to detect if an LLM has authored something generic. The lazy college student will probably get caught, but the cunning one not so much.
- the_snooze 2y ago>It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved. Technology enables scale and reach, which solves some problems but also creates its own set of issues. I think you're right on with the solution: do things that are anti-scale. If you make things a bit more inconvenient, a bit more costly, and a bit more local, you create an environment where there's space for trust and humanity---values that don't scale.
- cleanerbob 2y ago[dead]
- johnnyanmac 2y ago>We have a full-time recruiter working with us, and I'm not 100% sure what tools he used, but I switched to manually reviewing each resume, and given that it was 100s, it took a long time, but I still had my problem of great initial screen, terrible technical interview. I think it's very scary when even manual review is still yielding you results with horrible technical screenings. I wonder at that point if your technical review is very hard or specific (specific makes sense, yo did you you are looking for esoteric), or if it's just truly that polarized a market. Many are laid off and I imagine those qualified with such specialized knowledge and anchoring themselves instead of searching. >I also switched to only on-site interviews Kind of crazy. Not that I mind on-sites, but I haven't even heard a mention of on-site in the interview process since COVID. And I'm basically applying to any relevant position, locally or remotely. Just another curiosity.
- salomon812 2y ago> I think it's very scary when even manual review is still yielding you results with horrible technical screenings. It was bad. It was starting to affect my life outside of work. > I wonder at that point if your technical review is very hard My technical review is very hard, but it is directly applicable to the work I'm doing. And I've seen some candidates just do outstanding based entirely on their natural curiosity to look a bit deeper. I've been using a form of it for five years, so it's well reviewed.
- casenmgreen 2y agoSpeaking as a contractor since 2017, I have given up using recruitment agencies in the UK to find work. I am likely the number one expert, in my field, globally. I apply for roles which specifically ask for an SME in my field. There is no question here of skills, and it is as certain as it can be without actually knowing that I am a light year ahead of all other applicants (because there is practically no one else actually qualified in my field). I'm not flapping my ego, this is how things look to actually be. I find now I never get even contacted by agencies. I think they are not reading my CV/application, and I think this is happening because they are flooded - hundreds of applications in the first hour. They take the first person who looks good enough (and they're not good - there are practically no people in this field who actually have skills and experience, as opposed to just "I've worked with") and run with that, and then turn to filling the next contract. The upshot of this is that it doesn't matter how good you are, because your CV isn't going to be seen, not unless you apply in the first ten minutes or so. You have to play that game, and automate your applications, to be seen. So the question is, if you don't want to play that game, how now do you find companies who need skills?
- iamacyborg 2y agoThe UK job market is pretty screwy. I got made redundant back in March, applied for a bunch of stuff I matched profile for and maybe got 5-6 interviews off the back of it. The worst was the agency that lined me up for a contract role, got me to fill out all the paperwork only for the job to fall through because the client apparently never got budget signed off for the position.
- casenmgreen 2y ago> The worst was the agency that lined me up for a contract role, got me to fill out all the paperwork only for the job to fall through because the client apparently never got budget signed off for the position. I could be wrong, but I think that might have been a lie. I've heard it before, too, and I've come to doubt it; I think it too unlikely to be heard with such frequency as I do hear it. I've also had on three occasions agents call up after a day or two and tell "something about the budget, so the rate is now less than expected". In two cases I came to know the agency was simply lying, and was keeping the difference for itself, and I expect it to be true also in the third.
- nitwit005 2y agoI had a recruiter basically hold onto me after I passed more than one technical screen, even though I clearly did get all the way through the hiring process at either role. They were maintaining a pool of competent people.
- selimthegrim 2y agoDid not get?
- riwsky 2y agoIf only they could crowd source/gpg web of trust such a designation. We solved this for email by aggregating spam/not spam ratings from tons of recipients. It’d be great if we could do that here.
- ttyprintk 2y agoI’d be willing to sign someone’s (possibly incomplete) background check if I worked with them for six months.
- Seb-C 2y agoThat's why I don't answer LinkedIn recruiters anymore, unless they can show me a clear opportunity that matches my profile. Every time I answered their bait and they "just wanted a quick phone call", it was clear that they only were interested in filling their database. Then I get spammed by them with irrelevant automated offers forever, unless I block them completely.
- umvi 2y agoSame experience, we are getting absolutely flooded with hundreds, sometimes thousands of applicants who are presumably using some sort of automation/AI to adapt their resumes to the position yet they are very weak when it comes time for a job challenge or tech screening
- cj 2y ago> I also switched to only on-site interviews. My initial technical screenings are still done remotely. Pro tip for anyone hiring engineers for remote positions: Tell the applicant that there “might be” an in person technical assessment, even if you know the process will be 100% remote. The amount of fake candidates at the moment is insane. The only thing that makes fake candidates self-select out is knowing there’s the possibility that they will be required to be somewhere in person. Another trick I’ve used is saying “Oh, you live in Flint Michigan?? We happen to have an employee 20 minutes away, would you be open to meeting them?” And then suddenly they drop out of the interview process. There are a lot of foreign scammers exploiting the WFH trend in the US to the point where it drowns out real candidates. It’s really bad.
- evantbyrne 2y agoPeople who are serious about doing remote work are going to pass on anything that indicates hybrid. The simplest screening technique is to give instructions within the job post to submit via email rather than the job board form. Even before LLM slop became the norm people were spamming their resumes with Easy Apply.
- apitman 2y agoWhat do you do when a real person says they'd be happy to meet your fake employee?
- cj 2y agowere honest with the candidate about why we asked the question.
- generalizations 2y agoAs someone looking for remote work atm, can confirm this sounds fair to me: if the employer looks legit and would fly me out (like my current employer did), I'd be totally willing to do an onsite interview. Right now my approach has been focused less on proving my skills, and more on proving I'm a real person. Hah.
- bovermyer 2y ago
- la64710 2y agoWhat exactly you can do as a HM to make the life of a job seeker easy? I dare say nothing except to just make the quality of response better. Even after six rounds of interviews candidates who are not selected get not a single positive feedback and is treated like scrap with a soulless rejection. Beleive me as a job applicant I have zero sympathy for the corporates that hire me and I will use every thing at my disposal including AI to be more efficient in any way I deem fit. The job is just a business transaction to me and I don’t care about your high and mighty lecture as a HM. GTFO.
- throwaway2037 2y agoI agree with your sentiment. I am curious how this person will fare when they return to the job hunt. Then, they will see how adversarial the process has become, even for highly qualified candidates. Suddenly, AI looks like a good idea to game some of the process.
- salomon812 2y agoSince I think I'm the person you're referencing, I really do want to give good feedback, but experience has shown it's really perilous (discussion here: https://news.ycombinator.com/item?id=42533899 https://news.ycombinator.com/item?id=42533899) And I know how adversarial the process has become. I have friends looking for jobs plus I try to get to know my candidates. And I have my recent hires and their stories. I want to make it a better process but I'm so burnt out figuring out how to make it better. Some people talk about professional 3rd party recruiters but I've been burned by those as well. When it comes to dating and hiring, both can be pretty brutal.
- twelve40 2y agofwiw the article seems to describe a pretty mild type of automation to deal with tons of job ads and mundane stuff like cover letters that often get completely ignored by all sides, so why not try to automate that in good faith? didn't find anything about fake or cheating or misrepresenting one's skills in there.
- ttyprintk 2y agoThat’s what I see, too. I see someone who is easy to manage, as opposed to someone leaking AI slop into production.
- charlie0 2y agoOn-site interviews. This is not ideal, but is the way. As long as you're willing to shell out on my flight and cover the expenses, I'll fly anywhere for an interview.
- paulluuk 2y ago> Then, I decided to throw out anyone who heavily mentioned AI, LLM, or data science. As someone who graduated in the field of AI (so it's on my resume), and is now working in the Data Science field, often with LLMs, this hurts. Although I'm not sure what role you're hiring for, so perhaps I wouldn't be in the list of candidates anyway.