If you asked a researcher on fairness and discrimination in machine learning, "what's an application where ML could go horribly wrong", they would describe exac

If you asked a researcher on fairness and discrimination in machine learning, "what's an application where ML could go horribly wrong", they would describe exactly this kind of example. I don't know a thing about this company, maybe they're doing great work. But here are the alarm bells. - Application domain is extremely difficult even for humans who get in-person interviews with the subject. Claiming to solve a task with ML that humans cannot reliably solve is a symptom of snake-oil. - Compounding this, feedback is very limited and noisy. It's hard to tell if your algorithm is doing a good or bad job. So it's very hard to train the algorithm to do better. - Data is likely to be low-quality (because humans can't even label this task well). Much worse, data is likely to reflect human biases, conscious and unconscious both. Racist or sexist data lead to racist or sexist algorithms. - Algorithms in the space are highly susceptible to learn stereotypes (even from apparently-unbiased data). We can expect tons of noise in most features of an applicant, while more reliable signal would come from very generic features like what college someone attended or other factors closely tied to race or socioeconomic background. - Feedback loops can be a problem if the algorithm is fed data that is produced by the algorithm. I.e. the algorithm mostly recommends men (let's say) and is fed positive feedback, it will discriminate even more against women and so on. On a more personal note, it's not clear the premise of the task makes sense. A diverse team will have a greater mix of perspectives, which may make it more effective even if its 'culture fit' score is lower.