3 ms·
If you are not sure if your screening processes work, do an RCT. I have zero idea why the article raises this clearly provocative and correct approach and dismi
by notafraudster 4y ago
If you are not sure if your screening processes work, do an RCT. I have zero idea why the article raises this clearly provocative and correct approach and dismisses it immediately. Allocate a percentage of the positions you're hiring for to candidates that are hired at random. The empirical decision about what percentage depends on your primary concern: if you think your hiring process is completely useless, then you want to get this percentage close to 50% because this is what's optimal, experimentally. If you think your hiring process works OK and want to guard against downside risk on the random hires, get the percentage lower. This will cost you statistical power, but less than you expect.
No one is saying "random" means "no standards at all". If you're a graduate pure math department you don't need to admit a grad student whose background is that they have a GED and worked as a welder for 5 years before getting a 135 on the GRE Math side. Instead, split your screening into two phases: one phase that is targeted to exclude people who absolutely positively do not meet requirements and could never work out, and one phase designed to pick the best feasible candidates among those who meet requirements. Do the RCT in the latter phase.
If you have relatively continuous or regular hiring processes, you'll be well powered to assess your hiring screen in pretty short order. The larger your organization or the faster your overall turnover is the more robust you are to the risk of this going wrong. Not every company can do this -- academic departments that hire one person a year and sometimes have blown searches or people going back on the market within a year or two are clearly going to be a little risk averse. But if we're talking Deloitte hiring a cohort of 150 office admins, they can absolutely, positively make use of this strategy.
The other thing I'd add is that a lot of people are under the mistaken belief that hiring is about determining the best candidates. In fact that is not true and it has never been true. It's about maximizing the probability that a candidate you offer a job to takes it and is good... or else maximizing the frontier of some function of probability the candidate accepts, is good, and how good they are. Hiring nothing but the best candidates will increase the likelihood of a blown search, increase the rate of attrition, etc. This makes almost all of the discussion around mErIt In HiRiNg a distraction from how hiring actually works.
There are more narrowly tailored options. The more local you make the randomization, the less power and the less downside risk. Suppose you rank candidates from 1 (terrible) to 100 (ideal) and tend to hire candidates in the 80-90 range. Even a willingness to allocate some positions to candidates scored at 70-80 is going to give you lots information about whether your ranking system is useful at all.
Do more RCTs!!! Please!
- balsam 4y agoErm, no. Applications are needed (or, more charitably to TFA, useful) precisely because qualifications can be hard to quantify. That’s why they are called Applications and not Forms. And Qualifications not Quantifications. Hence RCTs are not really a solution. Look at clinical trials. They are already heavily quantified compared to Social Science, which hiring is, and its a consensual mess