9 ms·

AI can’t do hiring because it lacks the data

52 points by leeny 3y ago

thomasfromcdnjs 3y ago
I've never been interested in the automated matching aspect of HR but this piece has me inspired to attempt the challenge. To add a point, my favorite reason to use recruiters, is having them do the negotiation process for me. High quality post as usual.
stemlord 3y ago
Neither can people for the same reason
- leeny 3y ago
  Author here. 100%. Most recruiters are terrible, and they're kinda set up to fail because the data isn't there in exactly the same way. The difference is that good human recruiters can make some meaningful warm intros, let 2 engineers get in a room together to see if there's chemistry, and get the hell out of the way. I should make this clearer in the post. "Recruiters can't do hiring... and neither can AI. Both can't because the data's not there."
  - leeny 3y ago
    Post updated. Thanks for the feedback!
  - aigoochamna 3y ago
    As it stands now, modern recruiting is roughly the same as an AI recruiter. Scrape 10,000 emails/basic biographics and email blast everyone with an exciting opportunity at an {X} company doing {Y} thing that raised {Z} money. Insert unlimited leave, work life balance, and some other thing no one cares about but the company.
vonnik 3y ago
A major problem with data in the job market is that there are bad actors on both sides, employer and employee, with incentives to lie about themselves; ie companies with toxic cultures won't reveal that, nor will malicious or incompetent job candidates. Many readers of HN know that companies can be bad, but don't recognize that, say, competent software engineers are not the median candidate applying for most roles. The other side has a filtering problem, too. A second major problem with data is that it's often hard to know why something didn't work out. Yeah, not everybody's a good fit for a role. But where would you get an objective source of truth about an individual's performance. Or ... transpose it to dating: would you trust a group of ex-girlfriends' or ex-boyfriends' to give an accurate assessment of someone as a partner? Might they have incentives to distort the record? Where does good data come from? So it's not just that AI lacks the data. It's that there are structural problems with ever gathering the data. It's not like the data's out there and we just don't have access. If it even exists, it's poisoned at the source.
- ipaddr 3y ago
  "competent software engineers are not the median candidate applying for most roles" Most developers with a few years of experience are pretty competent. Most employers are not toxic waste dumps. Give the worst interview candidates a try and they will do well. Give the best interview candidates a try and they will fail or leave early. Interviewing is a skill, anyone who is extremely good at it most likely practiced more often and that could mean they needed to. The more experienced and poor interview they give the more of an unearthed gem you've discovered
  - zdragnar 3y ago
    I just interviewed someone who I am certain lied on their resume; they could barely write valid code during their interview, despite supposedly being a senior engineer with nearly 10 years of experience. Randomly hiring the first people to apply (or "giving them a shot") is a terrible strategy unless you have tons of money to burn and no deadlines to meet. We have internship positions for people willing to learn, but when we are trying to fill a senior position, we expect people to be at a certain level.
nextworddev 3y ago
I have seen human recruiters and companies blatantly lie about what they want, and what they are willing to offer as well. Bad data problem doesn't go away because there's a human doing the matchmaking manually. If anything, it's adversarial against job seekers if the process isn't done algorithmicly.
- leeny 3y ago
  Author here. I replied to a similar comment, but I'll reply to this one too because it's a really good point, and I think I should have done a better job of calling out in the post that humans suck at hiring as well. Most recruiters are terrible, and they're kinda set up to fail because the data isn't there in exactly the same way. The difference is that good human recruiters can make some meaningful warm intros, let 2 engineers get in a room together to see if there's chemistry, and get the hell out of the way.
  - oplaadpunt 3y ago
    Similarly, I have so many terrible experiences at 'job fairs' for engineering students, where they only sent HR. Things like asking what stack they use, and getting answers in the form of 'Diverse an challenging technology is at the heart of our innovative agenda ...'. Even when I _know_ the insider details through friends, HR always does such an appalling job of selling it to you, even when they're actually good.
  - satisfice 3y ago
    Humans being bad at social decision X is never an argument against humans making those social decisions. This is because of our moral status: we are responsible for our own social systems. We cannot morally evade or nullify that responsibility by asking a Magic Eight Ball to make decisions for us. Science Fiction writers going back to Asimov have already imagined such dystopias as when machines imprison or enslave humans because humans cannot be trusted to know what is “best” for their own lives. And of course that is how despots like Putin justify themselves. Maybe I am biased or suboptimal in my hiring, but I own my business and I will live or die by my OWN decisions, thank you. Does any CEO feel different?
Art9681 3y ago
Venture backed CEO makes desperate attempt to justify the value of her obsolete business. More news at 10. Edit: Corrected gender.
- leeny 3y ago
  Venture backed CEO here. I'm a "her". Also, please see the first footnote in the post. "If I were you, I’d be justifiably skeptical at this point. Wow, a recruiter is writing about how AI can’t do recruiting. Classic Luddite trope, right? Let’s burn down the shoe factory because it can never compete with the artisanal shoes we make in our homes. In this case, though, you’d be wrong. I walked away from a very lucrative recruiting agency that I built to start interviewing.io, precisely because I wanted to be the guy who owned the shoe factory, not the guy setting it on fire out of spite. Recruiting needs to change, it needs disintermediation, and it needs more data. It’s the only way hiring will ever become efficient and fair. I just don’t think AI is going to be that change. If I’m wrong, I’ll be the first in line to pivot interviewing.io to an AI-first solution."
  - esafak 3y ago
    Maybe it's a fad, but one that could subsidize your operation with a VC- and AI-backed offering "that scales" :)
angarg12 3y ago
This is more subtle and profound that it might seem. I work for a Big Tech and do tons of interviews, and not even us we have the data about what works and what doesn't (and if that data does exist, it's a very closely guarded secret). Even if we had hard data, we would still be blindsided by our inherent biases i.e. we know how candidates we accepted did, but not how candidates we rejected could have done.
- oplaadpunt 3y ago
  I would even argue that knowing just how well the accepted candidates did is also extremely difficult. Some assesment might be possible based on some simple metrics, but the ultimate 'how much value does this person add' is a hard one. I wouldn't always trust performance reviews and the like for someone's added value, since that adds another point for added bias.
- bippihippi1 3y ago
  the data doesn't exist because it can't. there are no independent variables. even a team of 1 relies on good management, getting assigned good work that is well specified, and thousands of other variables. maybe a super expensive unsupervised model could figure out how well one person with a ton of data on their working would work on a specific team that also has produced a lot of data on how they work, but that doesn't seem to happen so it's likely not more cost effective than letting people decide.
aaron695 3y ago
[dead]
deleted 3y ago
[deleted]
smeagull 3y ago
Honestly, this sounds like it is on par with the current state of human hiring.
nitwit005 3y ago
If you wanted ideal data, you'd want an employer to hire large numbers of people for a single role, hire them as randomly as possible, and be willing to train them. You could then see what sort of candidate was successful there. There are a handful of militaries that effectively do that, but it's pretty much impossible for a normal employer.
- sxp 3y ago
  When I was at Google, I remember hearing that this experiment was done on interns once. It wasn’t completely random, but they let in people who failed the standard Google interview to validate the effectiveness of that interview. Since interns were hired for a limited time, there was a lower cost to a false positive hire compared to an FTE.
  - testfoobar 3y ago
    What were the results?
- somenameforme 3y ago
  Which militaries? I'm just genuinely curious. To my knowledge, most militaries in the world use IQ tests to determine placement qualification which is more or less completely antithetical to this concept. It'd be interesting to compare and contrast how it works out.
  - swexbe 3y ago
    A high IQ won't help you pilot a fighter jet on it's own.
    - wnkrshm 3y ago
      That's why you need a Bachelor's degree to be eligible for training, at least in the US Air Force. Edit: They need people who have proven they can study and understand material.
    - 3y ago
tippytippytango 3y ago
Totally agree with this. I was tasked with getting some insights from hiring data once and everyone was disappointed we just couldn’t get anything meaningful. There were echos of something. But no signal could rise above the noise into the realm of statistical significance. Seems like we just don’t know how to measure what matters.
- cwalv 3y ago
  > everyone was disappointed we just couldn’t get anything meaningful. Nice .. my cynical side thinks that most of the time this is the expected outcome, in which case it wouldn't lead to disappointment, but just the next step of finding something that could plausibly be meaningful
Sparkyte 3y ago
Largest problem with data is that it can be falsified. So if you look at jobs like on Glassdoor they are well under the going rate. This is what is used to supress the job market.
Sparkyte 3y ago
It doesn't lack data, it lacks cultural. Nuances change rapidly AI would have to have data from these nuances and so AI is more fit as an assistant. You'd be surprised how we have to adapt to cultural changes when hiring. Things last year are no longer acceptable today.
runtime_blues 3y ago
But it will. Because it doesn't matter if it's fundamentally better than a good recruiter if it's orders of magnitude cheaper. If you can have it pursue far more leads, maybe the outcomes are going to be the same or better. And if you used it to replace a bad or a mediocre recruiter? In any case, you might not care: hiring is a crapshoot anyway, and AI is saving you millions of dollars. You want to weed out people who are clearly unqualified, but that's not rocket science. Beyond that, every company has a different hiring bar, a different process... and approximately zero data that their approach works better than anybody else's. Interview performance is a poor predictor of job performance. Whether the bar is high or comparatively relaxed, around 70% of the people you hire will be good, and the rest will underperform, leave after a couple of months, have difficult personalities, and so forth.
- godelski 3y ago
  This is the only "pro-AI" comment that I've seen that makes sense. This isn't too different from the non-CS interviewing strategy. Try to make interviewing cheap and accept that it is a noisy (and biased) process. This also makes it easier to let bad candidates ("false hires") cheap replace. As I see it, you're trying to optimize: p(X|F,C) > T, F=filter, and C=cost, and T=threshold. Treating this as a probabilistic problem seems important. So reducing C is valuable, even if F is not as good.
  - wiseowise 3y ago
    > This also makes it easier to let bad candidates ("false hires") cheap replace. Not in Europe. It’s almost impossible to fire someone in a reasonable time in Europe.
    - olddustytrail 3y ago
      "in Europe" doesn't mean anything because every European country has different employment law. In the UK it's trivially easy to fire someone quickly if they're a recent hire.
godelski 3y ago
There's always a lot of talk about meritocracy, but I want to convince you that this is an unachievable thing. We often hear "it doesn't matter your university's prestige/your credentials, just the quality of your work." As if the two don't strongly correlate. But the question is why they correlate. Social networks matter, a lot. If you're at a top 10 university your university's career fair is going to be filled with top companies. This isn't true for less prestigious universities. If you're a grad student, you also know that the connections your advisor has strongly correlates with your ability to get a job (nepotism playing a significant role here). We also see everyone using LeetCode to filter candidates but there's so much evidence that this is just a noisy filter. Not only do people cheat and succeed[0] but just consider how it is a meme that this doesn't correlate with the actual job. Our whole community has the position "study for the test, then forget it". Honestly, this feels insane to continue doing this, and to even pump more money into lifting this system up. We also know that grades are noisy[1] and does anyone remember those brain teasers that google used to do (also [1])? The ones that people still use? So how do we hire? Well, to do that we need to look at what the job actually requires. But the job will change. We also need to know how well a candidate will work with the team. These are really difficult questions to answer and it would be insane to think that a resume or in person interview could be a non-noisy process (or at least where noise doesn't dominate). Nepotism has succeeded because it is a decent filter. It has a lot of false rejects but it works because you are outsourcing a lot of questions to someone else that has intimate knowledge. It is easy to determine this, logically. You have three candidates who all perform equally well on their resumes, interviews, and whatever. You only have this information for candidate A. Candidate B was recommended by a close friend who you trust. Candidate C also knows a close friend but that friend dislikes them. Who are you going to hire? Why? The nepotism is actually giving you more information. This is not an argument for nepotism but rather a illustration about how the blind interview process is highly noisy and that there is still a lot of missing information. CS seems to be very weird in a lot of these respects. In a standard engineering job (e.g. Aerospace, electrical, mechanical) you generally send in your resume, talk with a few people (2-3 interviews) where a few technical questions will be asked, and that's about it. Resume -> phone screen (~30 minutes) -> In person interview (30min - 90 min). No whiteboard problems/puzzles. No take home tests/projects. In person interviews have several engineers on the team that is hiring and they ask behavioral questions as well as a few relevant technical problems. At most you'd have to do a back of the napkin calculation. Why do these firms do it this way? They've recognized that the system is noisy and that essentially you apply a few filters and then just hire the person. In 3-6 months if they don't work out, you let them go. If the hiring process is easy/cheap then you can also turn over bad hires easily. It also isn't uncommon to get a call 3-6 months down the line from your final interview. So you don't always have to repeat the whole process because there are still often available candidates. I've met plenty of people who work at FAANG jobs and brag about how they work 20hrs a week making $150k/yr. I've never met an engineer like that. Maybe that means something, maybe it doesn't. [0] https://news.ycombinator.com/item?id=35885342 https://news.ycombinator.com/item?id=35885342 [1] https://www.cnet.com/culture/google-gpas-are-worthless/ https://www.cnet.com/culture/google-gpas-are-worthless/
quickthrower2 3y ago
AI doesn’t have the data. Neither do humans. So why are humans better again? I see it answered in another part. Humans are better at deciding who they like working with. Which is fair enough. Although AI might help remove some biases that creates. By the way this is a very hard problem because it is like solving “war” in a sense. People need money to not die. They are very motivated to get a job. Therefore any system anyone comes up with gets gamed. There are 2 types of skill: ones you use in jobs and ones you use in interviews and there is very little overlap. It will be hard to ungame something where people need a job.
- laratied 3y ago
  I would even question how good humans are at deciding who they like working with better. Even more so than war it is like dating and that is exactly why I question the above. Humans can't even tell better than random who is a good marriage candidate with nearly unlimited time to make the decision. There is a game theoretic poker match going on but we pretend as if both sides don't have cards they hold close to the vest and that there isn't this huge random element with how the cards come off the deck. The flush draw hits and the hiring manager pretends that it was some kind of fortune telling skill they have. It is really a great example of where we think of ourselves as this highly evolved society from the Enlightenment but using a process that is practically no different than something the medieval church would come up with.
fbrncci 3y ago
AI can't do X or Y because it lacks Z, _for now_.
Animats 3y ago
Firing, though... There, the data is available.
precompute 3y ago
"AI" (LLM) hype is so great, people are faulting it for not solving issues they overlook. Which means the solution isn't in the data that it's trained on. Most people don't want to hire an employee that is competent. That alone is the truth. They value other non-technical factors highly, even when those factors might negatively affect performance. As I learnt from a senior developer, the best developers are the people who have horrible impostor syndrome and constantly downplay their own abilities. These guys usually turn out to be geniuses, no matter where they've been (or haven't been) employed before. Plus, when selecting for teamwork, instead of following the usual dogma, it's always better to hire someone that's been though shit rather than hiring someone who's always been cushy. People that have been through the ringer know how to cooperate well and even better, know why such a thing is necessary in the first place. The other kind of people usually are horrible backstabbers. Yes, they've had extensive corporate experience, that's usually not a good sign.
fatneckbeard 3y ago
fortunately the employers are now tracking like everything we click on and every time we move a mouse, every key we type, every second we are logged on... also every single IM/teams chat we send and receive, every phone call we make...every email we send, every sentence we typed out and then deleted because it sounded wrong,... so... they will have the data pretty soon.
ly3xqhl8g9 3y ago
AI can't do hiring because it knows the best hire is the one you don't do: why bother with a pesky human when it can do the job itself cheaper/faster/soonish better. Aside: having LinkedIn is a red flag in my book. I understand that one's desperation could lead to try getting a job using that cesspool of anti-patterns glued together with pure spite for the user, nevertheless, a red flag. Or to put it otherwise, if having a LinkedIn is a requirement for you to get hired, your job will disappear in 5 to 7 years (due to AI or other corporate movements).
- lisasays 3y ago
  So I guess that means you wouldn't hire this guy. Walking red flag that he is: https://www.linkedin.com/in/jeff-dean-8b212555/ https://www.linkedin.com/in/jeff-dean-8b212555/ I hate that "service" as much as you do, but people use it for all sorts of reasons. So if you're going to use it as a blocking filter, well, I guess that's the kind of luxury you can treat yourself to.
  - ly3xqhl8g9 3y ago
    Yes, sorry Mr. Dean, no job for you here. Even if he told me he wanted to work for me, I would just call an ambulance, for he must have some kind of stroke. As I typed the initial rant I opened an incognito tab and searched "Andrej Karpathy LinkedIn", had the same curiosity as you, if the top stars use that "service". He has an account. I clicked the link, LinkedIn took me through a quick security check/Verification process through which I had to select which bull was standing in the "right position", then it opened Mr. Karpathy's LinkedIn page and I saw as his last activity/posted article that he hires for Tesla AI, which is weird because I knew he quit, I clicked the article to read more, and LinkedIn opened a modal to Log In. Closed the page, and promised myself I will never again bait myself to open LinkedIn (probably I will break the promise in a year or two, one must check LinkedIn to see the state of the tart† in what not to do in user experience). † wanted to use "f-art" but ChatGPT recommended "t-art" as wordplay: 'This phrase combines the concept of a "tart," which can refer to a promiscuous or morally questionable person, with the original phrase. It adds a touch of humor while implying a negative connotation. However, please note that wordplay involving potentially derogatory terms or stereotypes should be used with caution and sensitivity.'
tokenspender 3y ago
From: https://interviewing.io/blog/why-ai-cant-do-hiring#user-content-fn-4 https://interviewing.io/blog/why-ai-cant-do-hiring#user-cont... "The future of technical assessments, in the age of generative AI is a noble and difficult topic, though, and it’s something we’ll tackle in a future post." This is exactly what we have built (are now applying to YC as well). Can share more details soon (we are integrating some much needed love on our site), but we are: - FAANG + Startup + Fintech founders - super technical and our expertise is in Product Engineering + ML-for-products - keeping everything 3.5-turbo based free, and probably will have to charge for 4 Our early users (Google/Meta SWEs and a few small AI startups) are pretty shocked with the results they are seeing, which they consider to be "as good if not better than human interviews + feedback". I'll post here next week with a link to try!
nologic01 3y ago
The type of ML algorithm that would do "automated hiring" has nothing to do with LLM and the current AI mania. Classification algorithms work well in objective contexts where the data contains fundamental and stable associations between attributes and the property one wants to predict. When applying this stuff to human affairs, whether that is credit scoring or dating apps or employee scoring you must be comfortable with very poor and unstable performance, extreme biases, and gaming behavior. The alternative is ofcourse the HR department so pick your poisson.
aigoochamna 3y ago
> AI can’t do hiring because it lacks the data Challenge accepted.

52 points by leeny 3y ago

I've never been interested in the automated matching aspect of HR but this piece has me inspired to attempt the challenge. To add a point, my favorite reason to use recruiters, is having them do the negotiation process for me. High quality post as usual.

stemlord 3y ago

Neither can people for the same reason

leeny 3y ago
Author here. 100%. Most recruiters are terrible, and they're kinda set up to fail because the data isn't there in exactly the same way. The difference is that good human recruiters can make some meaningful warm intros, let 2 engineers get in a room together to see if there's chemistry, and get the hell out of the way. I should make this clearer in the post. "Recruiters can't do hiring... and neither can AI. Both can't because the data's not there."
- leeny 3y ago
  Post updated. Thanks for the feedback!
- aigoochamna 3y ago
  As it stands now, modern recruiting is roughly the same as an AI recruiter. Scrape 10,000 emails/basic biographics and email blast everyone with an exciting opportunity at an {X} company doing {Y} thing that raised {Z} money. Insert unlimited leave, work life balance, and some other thing no one cares about but the company.

vonnik 3y ago

A major problem with data in the job market is that there are bad actors on both sides, employer and employee, with incentives to lie about themselves; ie companies with toxic cultures won't reveal that, nor will malicious or incompetent job candidates. Many readers of HN know that companies can be bad, but don't recognize that, say, competent software engineers are not the median candidate applying for most roles. The other side has a filtering problem, too. A second major problem with data is that it's often hard to know why something didn't work out. Yeah, not everybody's a good fit for a role. But where would you get an objective source of truth about an individual's performance. Or ... transpose it to dating: would you trust a group of ex-girlfriends' or ex-boyfriends' to give an accurate assessment of someone as a partner? Might they have incentives to distort the record? Where does good data come from? So it's not just that AI lacks the data. It's that there are structural problems with ever gathering the data. It's not like the data's out there and we just don't have access. If it even exists, it's poisoned at the source.

ipaddr 3y ago
"competent software engineers are not the median candidate applying for most roles" Most developers with a few years of experience are pretty competent. Most employers are not toxic waste dumps. Give the worst interview candidates a try and they will do well. Give the best interview candidates a try and they will fail or leave early. Interviewing is a skill, anyone who is extremely good at it most likely practiced more often and that could mean they needed to. The more experienced and poor interview they give the more of an unearthed gem you've discovered
- zdragnar 3y ago
  I just interviewed someone who I am certain lied on their resume; they could barely write valid code during their interview, despite supposedly being a senior engineer with nearly 10 years of experience. Randomly hiring the first people to apply (or "giving them a shot") is a terrible strategy unless you have tons of money to burn and no deadlines to meet. We have internship positions for people willing to learn, but when we are trying to fill a senior position, we expect people to be at a certain level.

nextworddev 3y ago

I have seen human recruiters and companies blatantly lie about what they want, and what they are willing to offer as well. Bad data problem doesn't go away because there's a human doing the matchmaking manually. If anything, it's adversarial against job seekers if the process isn't done algorithmicly.

leeny 3y ago
Author here. I replied to a similar comment, but I'll reply to this one too because it's a really good point, and I think I should have done a better job of calling out in the post that humans suck at hiring as well. Most recruiters are terrible, and they're kinda set up to fail because the data isn't there in exactly the same way. The difference is that good human recruiters can make some meaningful warm intros, let 2 engineers get in a room together to see if there's chemistry, and get the hell out of the way.
- oplaadpunt 3y ago
  Similarly, I have so many terrible experiences at 'job fairs' for engineering students, where they only sent HR. Things like asking what stack they use, and getting answers in the form of 'Diverse an challenging technology is at the heart of our innovative agenda ...'. Even when I _know_ the insider details through friends, HR always does such an appalling job of selling it to you, even when they're actually good.
- satisfice 3y ago
  Humans being bad at social decision X is never an argument against humans making those social decisions. This is because of our moral status: we are responsible for our own social systems. We cannot morally evade or nullify that responsibility by asking a Magic Eight Ball to make decisions for us. Science Fiction writers going back to Asimov have already imagined such dystopias as when machines imprison or enslave humans because humans cannot be trusted to know what is “best” for their own lives. And of course that is how despots like Putin justify themselves. Maybe I am biased or suboptimal in my hiring, but I own my business and I will live or die by my OWN decisions, thank you. Does any CEO feel different?

Art9681 3y ago

Venture backed CEO makes desperate attempt to justify the value of her obsolete business. More news at 10. Edit: Corrected gender.

leeny 3y ago
Venture backed CEO here. I'm a "her". Also, please see the first footnote in the post. "If I were you, I’d be justifiably skeptical at this point. Wow, a recruiter is writing about how AI can’t do recruiting. Classic Luddite trope, right? Let’s burn down the shoe factory because it can never compete with the artisanal shoes we make in our homes. In this case, though, you’d be wrong. I walked away from a very lucrative recruiting agency that I built to start interviewing.io, precisely because I wanted to be the guy who owned the shoe factory, not the guy setting it on fire out of spite. Recruiting needs to change, it needs disintermediation, and it needs more data. It’s the only way hiring will ever become efficient and fair. I just don’t think AI is going to be that change. If I’m wrong, I’ll be the first in line to pivot interviewing.io to an AI-first solution."
- esafak 3y ago
  Maybe it's a fad, but one that could subsidize your operation with a VC- and AI-backed offering "that scales" :)

angarg12 3y ago

This is more subtle and profound that it might seem. I work for a Big Tech and do tons of interviews, and not even us we have the data about what works and what doesn't (and if that data does exist, it's a very closely guarded secret). Even if we had hard data, we would still be blindsided by our inherent biases i.e. we know how candidates we accepted did, but not how candidates we rejected could have done.

oplaadpunt 3y ago
I would even argue that knowing just how well the accepted candidates did is also extremely difficult. Some assesment might be possible based on some simple metrics, but the ultimate 'how much value does this person add' is a hard one. I wouldn't always trust performance reviews and the like for someone's added value, since that adds another point for added bias.
bippihippi1 3y ago
the data doesn't exist because it can't. there are no independent variables. even a team of 1 relies on good management, getting assigned good work that is well specified, and thousands of other variables. maybe a super expensive unsupervised model could figure out how well one person with a ton of data on their working would work on a specific team that also has produced a lot of data on how they work, but that doesn't seem to happen so it's likely not more cost effective than letting people decide.

aaron695 3y ago

[dead]

deleted 3y ago

[deleted]

smeagull 3y ago

Honestly, this sounds like it is on par with the current state of human hiring.

nitwit005 3y ago

If you wanted ideal data, you'd want an employer to hire large numbers of people for a single role, hire them as randomly as possible, and be willing to train them. You could then see what sort of candidate was successful there. There are a handful of militaries that effectively do that, but it's pretty much impossible for a normal employer.

sxp 3y ago
When I was at Google, I remember hearing that this experiment was done on interns once. It wasn’t completely random, but they let in people who failed the standard Google interview to validate the effectiveness of that interview. Since interns were hired for a limited time, there was a lower cost to a false positive hire compared to an FTE.
- testfoobar 3y ago
  What were the results?
somenameforme 3y ago
Which militaries? I'm just genuinely curious. To my knowledge, most militaries in the world use IQ tests to determine placement qualification which is more or less completely antithetical to this concept. It'd be interesting to compare and contrast how it works out.
- swexbe 3y ago
  A high IQ won't help you pilot a fighter jet on it's own.
  - wnkrshm 3y ago
    That's why you need a Bachelor's degree to be eligible for training, at least in the US Air Force. Edit: They need people who have proven they can study and understand material.
  - 3y ago

tippytippytango 3y ago

Totally agree with this. I was tasked with getting some insights from hiring data once and everyone was disappointed we just couldn’t get anything meaningful. There were echos of something. But no signal could rise above the noise into the realm of statistical significance. Seems like we just don’t know how to measure what matters.

cwalv 3y ago
> everyone was disappointed we just couldn’t get anything meaningful. Nice .. my cynical side thinks that most of the time this is the expected outcome, in which case it wouldn't lead to disappointment, but just the next step of finding something that could plausibly be meaningful

Sparkyte 3y ago

Largest problem with data is that it can be falsified. So if you look at jobs like on Glassdoor they are well under the going rate. This is what is used to supress the job market.

Sparkyte 3y ago

It doesn't lack data, it lacks cultural. Nuances change rapidly AI would have to have data from these nuances and so AI is more fit as an assistant. You'd be surprised how we have to adapt to cultural changes when hiring. Things last year are no longer acceptable today.

runtime_blues 3y ago

But it will. Because it doesn't matter if it's fundamentally better than a good recruiter if it's orders of magnitude cheaper. If you can have it pursue far more leads, maybe the outcomes are going to be the same or better. And if you used it to replace a bad or a mediocre recruiter? In any case, you might not care: hiring is a crapshoot anyway, and AI is saving you millions of dollars. You want to weed out people who are clearly unqualified, but that's not rocket science. Beyond that, every company has a different hiring bar, a different process... and approximately zero data that their approach works better than anybody else's. Interview performance is a poor predictor of job performance. Whether the bar is high or comparatively relaxed, around 70% of the people you hire will be good, and the rest will underperform, leave after a couple of months, have difficult personalities, and so forth.

godelski 3y ago
This is the only "pro-AI" comment that I've seen that makes sense. This isn't too different from the non-CS interviewing strategy. Try to make interviewing cheap and accept that it is a noisy (and biased) process. This also makes it easier to let bad candidates ("false hires") cheap replace. As I see it, you're trying to optimize: p(X|F,C) > T, F=filter, and C=cost, and T=threshold. Treating this as a probabilistic problem seems important. So reducing C is valuable, even if F is not as good.
- wiseowise 3y ago
  > This also makes it easier to let bad candidates ("false hires") cheap replace. Not in Europe. It’s almost impossible to fire someone in a reasonable time in Europe.
  - olddustytrail 3y ago
    "in Europe" doesn't mean anything because every European country has different employment law. In the UK it's trivially easy to fire someone quickly if they're a recent hire.

godelski 3y ago

There's always a lot of talk about meritocracy, but I want to convince you that this is an unachievable thing. We often hear "it doesn't matter your university's prestige/your credentials, just the quality of your work." As if the two don't strongly correlate. But the question is why they correlate. Social networks matter, a lot. If you're at a top 10 university your university's career fair is going to be filled with top companies. This isn't true for less prestigious universities. If you're a grad student, you also know that the connections your advisor has strongly correlates with your ability to get a job (nepotism playing a significant role here). We also see everyone using LeetCode to filter candidates but there's so much evidence that this is just a noisy filter. Not only do people cheat and succeed[0] but just consider how it is a meme that this doesn't correlate with the actual job. Our whole community has the position "study for the test, then forget it". Honestly, this feels insane to continue doing this, and to even pump more money into lifting this system up. We also know that grades are noisy[1] and does anyone remember those brain teasers that google used to do (also [1])? The ones that people still use? So how do we hire? Well, to do that we need to look at what the job actually requires. But the job will change. We also need to know how well a candidate will work with the team. These are really difficult questions to answer and it would be insane to think that a resume or in person interview could be a non-noisy process (or at least where noise doesn't dominate). Nepotism has succeeded because it is a decent filter. It has a lot of false rejects but it works because you are outsourcing a lot of questions to someone else that has intimate knowledge. It is easy to determine this, logically. You have three candidates who all perform equally well on their resumes, interviews, and whatever. You only have this information for candidate A. Candidate B was recommended by a close friend who you trust. Candidate C also knows a close friend but that friend dislikes them. Who are you going to hire? Why? The nepotism is actually giving you more information. This is not an argument for nepotism but rather a illustration about how the blind interview process is highly noisy and that there is still a lot of missing information. CS seems to be very weird in a lot of these respects. In a standard engineering job (e.g. Aerospace, electrical, mechanical) you generally send in your resume, talk with a few people (2-3 interviews) where a few technical questions will be asked, and that's about it. Resume -> phone screen (~30 minutes) -> In person interview (30min - 90 min). No whiteboard problems/puzzles. No take home tests/projects. In person interviews have several engineers on the team that is hiring and they ask behavioral questions as well as a few relevant technical problems. At most you'd have to do a back of the napkin calculation. Why do these firms do it this way? They've recognized that the system is noisy and that essentially you apply a few filters and then just hire the person. In 3-6 months if they don't work out, you let them go. If the hiring process is easy/cheap then you can also turn over bad hires easily. It also isn't uncommon to get a call 3-6 months down the line from your final interview. So you don't always have to repeat the whole process because there are still often available candidates. I've met plenty of people who work at FAANG jobs and brag about how they work 20hrs a week making $150k/yr. I've never met an engineer like that. Maybe that means something, maybe it doesn't. [0] https://news.ycombinator.com/item?id=35885342 https://news.ycombinator.com/item?id=35885342 [1] https://www.cnet.com/culture/google-gpas-are-worthless/ https://www.cnet.com/culture/google-gpas-are-worthless/

quickthrower2 3y ago

AI doesn’t have the data. Neither do humans. So why are humans better again? I see it answered in another part. Humans are better at deciding who they like working with. Which is fair enough. Although AI might help remove some biases that creates. By the way this is a very hard problem because it is like solving “war” in a sense. People need money to not die. They are very motivated to get a job. Therefore any system anyone comes up with gets gamed. There are 2 types of skill: ones you use in jobs and ones you use in interviews and there is very little overlap. It will be hard to ungame something where people need a job.

laratied 3y ago
I would even question how good humans are at deciding who they like working with better. Even more so than war it is like dating and that is exactly why I question the above. Humans can't even tell better than random who is a good marriage candidate with nearly unlimited time to make the decision. There is a game theoretic poker match going on but we pretend as if both sides don't have cards they hold close to the vest and that there isn't this huge random element with how the cards come off the deck. The flush draw hits and the hiring manager pretends that it was some kind of fortune telling skill they have. It is really a great example of where we think of ourselves as this highly evolved society from the Enlightenment but using a process that is practically no different than something the medieval church would come up with.

fbrncci 3y ago

AI can't do X or Y because it lacks Z, _for now_.

Animats 3y ago

Firing, though... There, the data is available.

precompute 3y ago

"AI" (LLM) hype is so great, people are faulting it for not solving issues they overlook. Which means the solution isn't in the data that it's trained on. Most people don't want to hire an employee that is competent. That alone is the truth. They value other non-technical factors highly, even when those factors might negatively affect performance. As I learnt from a senior developer, the best developers are the people who have horrible impostor syndrome and constantly downplay their own abilities. These guys usually turn out to be geniuses, no matter where they've been (or haven't been) employed before. Plus, when selecting for teamwork, instead of following the usual dogma, it's always better to hire someone that's been though shit rather than hiring someone who's always been cushy. People that have been through the ringer know how to cooperate well and even better, know why such a thing is necessary in the first place. The other kind of people usually are horrible backstabbers. Yes, they've had extensive corporate experience, that's usually not a good sign.

fatneckbeard 3y ago

fortunately the employers are now tracking like everything we click on and every time we move a mouse, every key we type, every second we are logged on... also every single IM/teams chat we send and receive, every phone call we make...every email we send, every sentence we typed out and then deleted because it sounded wrong,... so... they will have the data pretty soon.

ly3xqhl8g9 3y ago

AI can't do hiring because it knows the best hire is the one you don't do: why bother with a pesky human when it can do the job itself cheaper/faster/soonish better. Aside: having LinkedIn is a red flag in my book. I understand that one's desperation could lead to try getting a job using that cesspool of anti-patterns glued together with pure spite for the user, nevertheless, a red flag. Or to put it otherwise, if having a LinkedIn is a requirement for you to get hired, your job will disappear in 5 to 7 years (due to AI or other corporate movements).

lisasays 3y ago
So I guess that means you wouldn't hire this guy. Walking red flag that he is: https://www.linkedin.com/in/jeff-dean-8b212555/ https://www.linkedin.com/in/jeff-dean-8b212555/ I hate that "service" as much as you do, but people use it for all sorts of reasons. So if you're going to use it as a blocking filter, well, I guess that's the kind of luxury you can treat yourself to.
- ly3xqhl8g9 3y ago
  Yes, sorry Mr. Dean, no job for you here. Even if he told me he wanted to work for me, I would just call an ambulance, for he must have some kind of stroke. As I typed the initial rant I opened an incognito tab and searched "Andrej Karpathy LinkedIn", had the same curiosity as you, if the top stars use that "service". He has an account. I clicked the link, LinkedIn took me through a quick security check/Verification process through which I had to select which bull was standing in the "right position", then it opened Mr. Karpathy's LinkedIn page and I saw as his last activity/posted article that he hires for Tesla AI, which is weird because I knew he quit, I clicked the article to read more, and LinkedIn opened a modal to Log In. Closed the page, and promised myself I will never again bait myself to open LinkedIn (probably I will break the promise in a year or two, one must check LinkedIn to see the state of the tart† in what not to do in user experience). † wanted to use "f-art" but ChatGPT recommended "t-art" as wordplay: 'This phrase combines the concept of a "tart," which can refer to a promiscuous or morally questionable person, with the original phrase. It adds a touch of humor while implying a negative connotation. However, please note that wordplay involving potentially derogatory terms or stereotypes should be used with caution and sensitivity.'

tokenspender 3y ago

From: https://interviewing.io/blog/why-ai-cant-do-hiring#user-content-fn-4 https://interviewing.io/blog/why-ai-cant-do-hiring#user-cont... "The future of technical assessments, in the age of generative AI is a noble and difficult topic, though, and it’s something we’ll tackle in a future post." This is exactly what we have built (are now applying to YC as well). Can share more details soon (we are integrating some much needed love on our site), but we are: - FAANG + Startup + Fintech founders - super technical and our expertise is in Product Engineering + ML-for-products - keeping everything 3.5-turbo based free, and probably will have to charge for 4 Our early users (Google/Meta SWEs and a few small AI startups) are pretty shocked with the results they are seeing, which they consider to be "as good if not better than human interviews + feedback". I'll post here next week with a link to try!

nologic01 3y ago

The type of ML algorithm that would do "automated hiring" has nothing to do with LLM and the current AI mania. Classification algorithms work well in objective contexts where the data contains fundamental and stable associations between attributes and the property one wants to predict. When applying this stuff to human affairs, whether that is credit scoring or dating apps or employee scoring you must be comfortable with very poor and unstable performance, extreme biases, and gaming behavior. The alternative is ofcourse the HR department so pick your poisson.

aigoochamna 3y ago

> AI can’t do hiring because it lacks the data Challenge accepted.