21 ms·
I was part of a very in-depth evaluations of both Pymetrics & HireVue for a large company recently (2 different evals, among others). Some interesting things w
by curiousllama 6y ago
I was part of a very in-depth evaluations of both Pymetrics & HireVue for a large company recently (2 different evals, among others). Some interesting things we observed:
(1) It was not clear to us, with any level of certainty, whether either tool did anything more useful than filter out people who don't want to take the test / don't trust "AI".
(2) It was not clear to us, with any level of certainty, whether (1) was a useful, arbitrary, or counterproductive selection criterion for the roles we were hiring for.
(3) It was very clear to us, with a high degree of certainty, that both impressed a LOT of the higher ups
(4) It seemed somewhat clear that both were unlikely to get us sued (note: this is a high bar). However, HireVue (as of a year ago) could not in ANY meaningful way substantiate this claim "AI is more impartial than a human interviewer, as it has no bias," and it angers me that they make that claim.
After conducting a bunch of reviews like this (including other tools), I concluded that most AI recruiting software is a combo of (a) very useful process automation and (b) hocus-pocus, magicical, pseudo-AI. I had people telling me they used supervised learning because they had a team of supervisors in India. I had people telling me they had "Custom AI" when they were just calling some random unverified third party API. And I OFTEN heard nonsense, unverified claims of "unbiased" AI, as if eg training a hiring model using geographical factors won't tell you "don't hire non-college-age people from the south side of Chicago".
If anyone is looking to use recruiting AI, get a good IO psych person and listen to them.
- finolex1 6y agoThe pymetrics test mentioned in the article involves among other things, clicking on a balloon till it pops and pressing the keyboard as many times as possible in a minute. As much as people love to hate on leetcode-style interview programming questions, they are at least remotely related to performance on the job. Personally, I've found it to be a good filter in terms of deciding where to interview at. Companies that have adopted these pop-psychology tests (management consulting companies and big banks like BCG, JP Morgan) signal that they've let non-technical upper management hold sway over recruiting in comparison to tech-centric companies/hedge funds etc. that seem to have more rigorous recruiting standards.
- mattgreenrocks 6y ago> Personally, I've found it to be a good filter in terms of deciding where to interview at Emphasizing this point for others. This is the best way to approach it IMO. The company should work to impress you as much as you work to impress them.
- naravara 6y ago> The company should work to impress you as much as you work to impress them. This becomes more true the further in your career you go. But for entry level positions it's really just a numbers game.
- treeman79 6y agoGot a call from a recruiter that was looking for an oddly specific combination of skills. Me, so it’s Company X? He was flabbergasted. I gave 3 months notice. I was literally the only person in the world with that combination of skills. So I asked how much, went to bosses, I’ll stay on longer if you pay me that rate. Good money for a few months.
- 2112 6y agoI'm curious about what the combination of skills was ?
- treeman79 6y agoObscure mainframe tech and ruby, VB. About 15 years ago.
- Ruthalas 6y agoI suppose that speaks positively of the recruiter's ability to find the right person for the job!
- o-__-o 6y agoThis. I got headhunted for a very very specific skill in my city. Asked them if it was company X? “How did you know??” “I wrote that rec”. I then proceeded to work directly with the recruiter to find a candidate and got an easy $1000 for about 4 hours of total work.
- aspaceman 6y agoHow do you find good psych people? Showing my bias but it’s a genuine question. I remember in school all the psych and social psych folks being total jackoffs in blowoff classes. How do you find people you can take seriously? Who don’t just quote psychology today and act like it’s insight? Like I know the field is professional and has skilled professionals - but how do you find them?
- Balgair 6y agoWell, first you put out a job ad. Then because the psych people are so numerous, you'll be inundated with resumes. Since you don't know what to really look for, you decide that an AI from some company will help out with that. So you use an AI to hire people to get rid of the AI, except in the case where you hire people to get rid of the AI, because you don't know how to hire those people. ;)
- beckingz 6y agoSo how do you find the right psych person to monitor the psych hiring AI?
- curiousllama 6y agoI actually built an AI to do that for you
- metrics314 6y agoI’m an IO psych at a large tech company that evaluated pymetrics against job performance and there was no relationship, just near 0 correlations so recommended not using them and we don’t. These AI tools are not transparent enough in explaining their outcomes and lack what we call face validity or job relevance. A coding test is at least a good filter at the top for entry level roles with large pools because it has relevance and false negatives aren’t as much of a concern (but we test still for disparate impact) otherwise for most roles a structured interview is the best option.
- kyrieeschaton 6y agoEven fairly strong correlations break down on restricted ranges of inputs. Eg, if you're hiring between 80-90th percentile candidates (top ones get better offers, lower ones get filtered out), yes, the correlation will get swamped by noise; not so much if you are hiring between 0 and 100.
- curiousllama 6y agoThis is why tech companies are good. Our output basically said "this is entirely indistinguishable from random choice," and we got the response "Great! Can't get sued over randomness!"
- stuaxo 6y agoBased on how these things go, it's almost a certainly that this will be found to be discriminating in some way that is unethical/illegal within the next few years.
- Nacdor 6y ago> (1) It was not clear to us, with any level of certainty, whether either tool did anything more useful than filter out people who don't want to take the test / don't trust "AI". There's at least one way this is beneficial for the employer: Desperate, obsequious job-seekers will be more willing to tolerate the degradation of being judged by a computer. It's perfect for finding employees who will ask "How high?" when you tell them to jump. I applied to a few jobs at United Healthcare and they kept sending me HireVue interview invitations. Every time I rejected them and when prompted for feedback I said I don't want to be evaluated by a computer. Never heard anything from them (and didn't expect to).
- 908B64B197 6y ago> I had people telling me they used supervised learning because they had a team of supervisors in India Love it.