9 ms·

Job Applicant Resumes Are Effectively Impossible to De-Gender

• enriquto 5y ago
  > The model had learned through historical hiring data that men were more likely to be hired, and therefore rated male resumes higher than female resumes. A lot of machine learning sounds like the computers are just cargo-culting. But this concrete example is spectacularly so.
  • vidarh 5y ago
    Well, it's not the computers faults that the people designing the system failed to (and probably are completely unable to) provide a proper measure of "value once hired". In the absence of a signal to tell you that you're stuck looking for proxies, and that it then ends up doing the same as many humans and assume that "past success at getting hired" is a good proxy for "value once hired" is not very surprising. The machine learning is "cargo-culting" in many instances because humans are cargo-culting and thus are creating models that repeat the same motions rather than try to provide qualitatively better data. But the interesting thing is that in doing so it does reveal a lot of things about human decision making that may help inform improvements.
    • jsmith99 5y ago
      If one gender really did provide more 'value once hired', would it be reasonable for the AI to favour them?
      • lsiebert 5y ago
        the AI is just a product of it's inputs, how it's used, and whether it violates discrimination laws, are totally up to the users.
        • delusional 5y ago
          Sort of, but we have to be careful not the wash the hands of people all too willing to discard responsibility. It's correct to say "it's just statistics on data" but it also provides cover to the processes that use it. "How can you be against hiring the best person for the job" the argument goes, completely ignoring that the "best" isn't scientifically defined, and includes the same sexist biases we were trying to get rid of.
          • yjftsjthsd-h 5y ago
            I read that as specifically saying that the ethical and legal responsibility is entirely on the people. (i.e. I think you're agreeing with the comment you're replying to)
            • lsiebert 5y ago
              Yes, sorry for the lack of clarity. Users shouldn't blame the machine when they control every aspect of it and how it's used. It's on you as a user to understand your tools and data.
      • JaumeGreen 5y ago
        Not necesarily, because it could have given more value for the wrong reasons. Imagine a misoginistic, but horny, boss, that only hires good working ugly men and hot dumb women, specifically looking for these traits in order to have a "playing" field. If the AI got in here it would be constrained to the same parameters, af if it was looking for good workers of any gender it would prefer men. An the thing is that although this boss is part of history (we hope) the influence they had on the workforce at large was enormous, and it'll be a long time before this is reversed. And this without taking into account all the coworkers that could hinder your job because you are not part of their cadre (gender, football club, fortnite clan, whatever). We really should exepriment with equality, or even the opossite extreme, before considering one gender "superior" than another for a given role. And even then there would be exceptions.
      • tialaramex 5y ago
        How much more value? Suppose men were providing $10 in additional value over the time they work for you. That's barely even worth a Starbucks gift card, the sort of thing you probably give away to people who guess the number of marbles in a jar at a company mandatory fun event. You definitely shouldn't hire James rather than Deborah because of an average $10 gender difference unless they are literally interchangeable. If their sole distinguishing feature was somehow that Deborah's smile was nicer, or James' laugh was more annoying, that's probably still enough difference to out-weigh this $10 lifetime value. And the reality is your candidates just aren't that similar. Deborah's six years experience in a similar role while James is applying with no relevant background should not lead to the conclusion that all that matters is the average "value once hired" for James will be higher based on gender averages. Now, suppose women are providing $10M in additional value. Well, I think we'd have noticed something that dramatic by now right? "Gee, my start-up with five women broke even in the first six weeks because of how awesome women are". So we can rule that sort of large figure out as implausible. I think a discussion about whether it's reasonable needs to take the magnitude of the difference into consideration, which would first mean (if you wanted to do this) finding out for sure how large that difference even is. My guess is, if you did this, you get a very small answer and so then the discussion is really easy: No. But before you could do that, you'd need to have a decent objective measurement of this "value once hired", and that part is far harder so it won't get done.
        • newsclues 5y ago
          If you are in a gunfight, would you rather have a slight advantage? You also need to consider the risks and circumstances, sometimes you want every tiny edge possible.
          • tialaramex 5y ago
            Sure, you spend as long as you like figuring out how best to achieve the slight advantage, meanwhile I shoot you in the head. Analysis paralysis, figuring out what to do costs time and time is expensive so you should avoid this unless you've got a good reason to expect it actually matters and you believe you can actually do it correctly.
      • newsclues 5y ago
        Yes, because of the AI is picking my brain surgeon, I want the best person, and I’m absolutely opposed to any equity, just raw talent and merit when it comes to someone operating on my brain again.
      • vidarh 5y ago
        In this instance most societies have decided that discrimination by gender is a negative for society irrespective of whether or not you can come up with a metric that gives one gender a higher value to the company, so no. But it would still be important to understand how the AI would assign value because it would matter to ensure that it does not apply criteria that are not legal. At the same time, what this seems to suggest is also that purely judging by other factors in resumes might effectively act as a proxy for directly judging by gender. As such the more interesting question to me is whether that means that deciding based on such factors might put employers at risk of claims of gender discrimination.
    • belorn 5y ago
      The there are differences in human learning and knowledge and machine learning, one being the kind of errors machines makes compared to humans. A human is unlikely to confuse the side of a truck with the sky (tesla), or the image of an item with the actually item. Humans in turn get effected by stress, metabolism and other biological factors. For an AI to actually predict which employees are going to benefit the company, it need to understand both how the company has evolved over the data period, how it will likely continue to grow, and that some employees that have successful careers are not necessary the people that contribute to the success of the company. An AI might think nepotism is a great indicator for a highly retained and employable employee, but a human would recognize it for what it is but may accept it as an acceptable level of corruption. It would be the same conclusion (ie, hire the person), but for two very different reasons.
  • steinuil 5y ago
    > A lot of machine learning sounds like the computers are just cargo-culting. That's precisely what ML is. Though to be fair, interviewing done by real humans is also largely based on cargo culting.
  • ZeroGravitas 5y ago
    I think the key insight is that the humans the AI replaces were probably also cargo culting. They'd just get a bit more defensive if you suggested that.
    • delusional 5y ago
      Have you ever tried to tell an ML engineer that their model is biased (or even worse sexist). I think the defensiveness is about equal.
  • Zababa 5y ago
    I would call it "pattern recognition" rather than "cargo culting". Cargo culting is an example where pattern recognition fails.
    • yjftsjthsd-h 5y ago
      I mean. This seems like a good example of pattern recognition probably failing?
      • Zababa 5y ago
        That depends on how you measure it. It accurately reproduced how previous recruiters worked, but this approach might not be the best. The thing here is that the metric "the best candidate" is very hard to find, so people use the metric "candidate that were previously recruited". I would call it an instance of the "streelight effect": https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect
• sabas123 5y ago
  Reading the conclusion of the article (not the paper), it sounds like the difference in writing between genders seem to be too strong for current techniques to work.
  • nottorp 5y ago
    I'd say the current techniques work too well :) Also it says something about the current wave of "AI" being able only to mimic the human thought process and not being able to come up with anything different. The I in AI is still out of reach.
  • causi 5y ago
    I wonder what happens if you work with someone of the opposite sex to prepare your resume.
    • nmstoker 5y ago
      Yes! Or use the AI to identify the gender indicating words of the opposite gender that are synonyms of words you've used and swap those in - might be a start (although deeper things like sentence structure could still influence the classification)
• ClumsyPilot 5y ago
  If we grant as correvt the reseatch saying upper eshelons of society is filled with sociopaths and psychopaths, https://www.forbes.com/sites/victorlipman/2013/04/25/the-disturbing-link-between-psychopathy-and-leadership/?sh=337397764104 https://www.forbes.com/sites/victorlipman/2013/04/25/the-dis... Can we train ML to recognise them from a CV? Do they revognise each-other?
• skinkestek 5y ago
  FWIW at least at the company I work this gives women an advantage. I get a nice bonus for anyone I recruit but recruiting a woman also increases my managers KPI.
  • geraldwhen 5y ago
    We have quotas we need to fill. Hiring a white male is basically a nonstarter without prostrating yourself before HR
    • skinkestek 5y ago
      We happily take anyone qualified and I get my bonus. But as mentioned, percentage of female employees is a KPI and it makes my boss looks good and it makes the CEO happy in all hands meetings.
      • bellyfullofbac 5y ago
        My properties ensure that I won't face quotas that would be a barrier for emplyoment (well maybe my gender, I'm male), so, I'm just curious, so what if there's one position to fill, the male candidate scores 60%, and the female one scores 58% (using my hypothetical ranking system which actually works and is fair, etc, etc), so just slightly lower. Would your company pick the female candidate in this case? I guess answering this question "the wrong way" would get a lot of butthurt people showing up to this thread, so feel free to not answer.
        • BoorishBears 5y ago
          People need to realize is actual support for diversity means internalizing there is a benefit. So if you actually believe there is a benefit to having a team of men and women (I do), then it's not "wrong" to let that guide you to the woman when any score above a 50% indicates someone who will be able to actually do the work.
          • 5y ago
• deleted 5y ago
  [deleted]
• wayanon 5y ago
  Automation should be removed from the hiring process.
  • c0balt 5y ago
    But to which level? Basic automation, i.e. sorting by proficency is not really done better by a human than machine. The problem doesn't lie in automation but how we apply it. Don't blame the tool, blame the user. I think there is a lot of room for removing overautomation and creating transparency. Though just outright abandoning automation would hurt even more since human routines are even harder to evaluate/ make transparent.
    • Aeolun 5y ago
      If you have too many CV’s to look at, you can simply trash a random set (or automatically send a rejection email), then look at the remainder. You should still have roughly the same ratio of skilled candidates.
      • Bromeo 5y ago
        To still have the same ratio of skilled candidates would mean the automatic rejection has zero merit. For example, what about filtering for a minimum of x years of work experience for a senior position? Surely your ratio of skilled and appropriate candidates will increase.
    • etripe 5y ago
      What kind of proficiency do you mean, exactly? To me, the concept sounds too nebulous for a machine to analyse, as it needs precise instructions/criteria.
    • formerly_proven 5y ago
      How do you "sort by proficency"?
• icare_1er 5y ago
  Why is that such a big problem for leftists that men and women have slightly different ways of expressing themselves; behaving; interacting socially, etc... ?
  • c048 5y ago
    Because it is religious dogma at this point, and not based on facts.
  • pomdapi 5y ago
    There was a Star Trek episode about that. Some cyborg there said (I'm paraphrasing) : "We are the Oligarchical Collectivists. Lower your attentiveness and surrender your thinking. We will erase your biological and individual distinctiveness to conform our own. Your culture will adapt to service us. You will be undifferentiated. Resistance is futile."
• Aeolun 5y ago
  But you are not trying to de-gender it for AI’s right? It’s humans that still have to distinguish.
• vnorilo 5y ago
  I'm not sure gender inequality can be solved by some magic machine making gender disappear. Rather we can progress by making inequality visible and challenging it: only when attitudes change we will slowly improve. The HN crowd might like Lauren Klein's approach to this via data science [1] - I do. 1: https://data-feminism.mitpress.mit.edu/ https://data-feminism.mitpress.mit.edu/
  • odajay 5y ago
    Thank you, it seems to be an incredible resource, really appreciate you sharing this.
• Zababa 5y ago
  Link to the actual paper here: https://arxiv.org/pdf/2112.08910.pdf https://arxiv.org/pdf/2112.08910.pdf Edit after having read it: It's short and easy to read, I would suggest reading it instead of the article linked. I think that more work should be needed before making suggestions. For example, is there a correlation between the "number of words" and the gendering of a resume? Maybe something like "I was the manager of the soap team for 3 years, during which sales increased by 25.6%" compared to "2012-2015: soap team manager". Though this convey less informations. Maybe a KPI box for each job? Another thing that I would have liked would be to compare that to what actual recruiters can do. If a simple model can do 0.75 and a regular recruiter 0.53, it's not the same as if a recruiter does 0.92. Lastly, the "list of gender indicating words" seems really small. Just 26 words for 348k resume.
• throwawayay02 5y ago
  The obsession with equality (of outcome) is a blight upon our world.
  • Bromeo 5y ago
    What about this is promoting equality of outcome? The paper is investigating whether information preventing equality of opportunity can be successfully obfuscated.
• shimonabi 5y ago
  In my native language every verb has a masculine or feminine ending depending on the subject's gender. I'm also sure a woman has a far better chance of being hired into tech than a man with the same qualifications.
  • raxxorrax 5y ago
    To be fair, getting hired in tech is still pretty easy in most cases and countries. My country puts way too much weight on formal qualification, but in the current economy you find work nearly everywhere.
• severak_cz 5y ago
  It's bit sad that there are no examples of processed CVs, especially those where gender was masked by removing some words. I wonder how the NLP models did their job and I am feeling there is NLP looking for tanks using clouds instead (see https://www.gwern.net/Tanks https://www.gwern.net/Tanks).
• quanto 5y ago
  Original paper: https://arxiv.org/pdf/2112.08910.pdf https://arxiv.org/pdf/2112.08910.pdf The entire premise of the research relies on the fact that the resume samples were well matched: in other words, the quality of male candidates was equal to that of female counterparts. In the paper, it says Specifically, we perform 1-1 matching without replacement such that for each male resume, we find a female resume that is within 2 years of experience, has the same degree, field of study, and has a resume similarity score (i.e. cosine similarity of resume vector representations) of at least 0.7. I am not sure if this method is sufficient to create a match dataset. In other words, it is entirely possible that the (legitimate) experience or quality differences in resumes may act as the signal for the ML algorithm. It would be interesting to see descriptive analytics on male and female applicants. IIRC men and women have differences in self-selection for precisely things like this. It is plausible (or even probable) that such difference in self-selection crept into the dataset despite the research's resume matching process. (As a clarification, this is a critique on the research methodology, and I am not claiming one gender is less qualified than the other in general.)
  • deleted 5y ago
    [deleted]
• kypro 5y ago
  I assume this controlled for the fact the genders don't apply to certain roles at the same rate? I mean I assume most people reading this right now are men simply because this site is a business/tech forum and men tend to have more of an interest in these two things. Similarly, if I got an empty job application for a rust developer role I would expect 90%+ of time it's a man applying. If it were a UI/UX designer I'd be less confident. Maybe a 50/50 split based on my experience. Were it a social media manager role, I'd then assume it was a woman.
  • shultays 5y ago
    The article says Male and female samples were matched 1-1, and a subset obtained by pairing up the best objectively job-appropriate male and female candidates, with a margin-of-error of 2 years, in terms of experience in their field. Thus the dataset consists of 174,000 male and 174,000 female résumés. but that is assuming you are able to match two candidates objectively. Also "with a margin-of-error of 2 years, in terms of experience in their field" is a quite a margin for junior-mid positions.
• vladmiller 5y ago
  What if we take something like GPT-3 and have it rewrite all the resumes before they sent to processing into NLP models?
• odajay 5y ago
  On the importance of de-gendering the recruitment process, I have a parallel to submit with a discipline I know well, the professional classical music industry. A few decades ago, it was commonly thought that the brain of women couldn't understand music. "Women have other preferences," "Their brains are not made for that." If it reminds you of some rhetoric seen here and there, trust your instincts. Here is what happened in the 70s, using extracts from the book "Blindspot": "In 1970, fewer than 10 percent of the instrumentalists in America's major symphony orchestras were women, and women made up less than 20 percent of new hires." "Starting in the 1970s, several major American symphony orchestras experimented with a new procedure that involved interposing a screen between the auditioning instrumentalists and the committee, leaving the applicants audible but not visible to the judges." "The next twenty years provided interesting evidence. After the adoption of blind auditions, the proportion of women hired by major symphony orchestras doubled—from 20 percent to 40 percent." Fun fact: those blind auditions didn't start because of gender ethics; they began because the classical music industry was rife with clannishness and nepotism at all levels, which gave the incitive for orchestras to limit the impact from influential professors.
  • raxxorrax 5y ago
    But that does not mean we have the same situation here. We are living 50 years later and the pipeline for IT looks different than the pipeline for music. Also: https://statmodeling.stat.columbia.edu/2019/05/11/did-blind-orchestra-auditions-really-benefit-women/ https://statmodeling.stat.columbia.edu/2019/05/11/did-blind-...
    • odajay 5y ago
      Thank you, you are totally correct; we are living 50 years after, and those pipelines are different. I tried, without great success, to create perspective in a very one-sided thread. This link is interesting indeed. While there was some fair criticism of this study, I do agree with the conclusion: "I agree that blind auditions can make sense—even if they do not have the large effects claimed in that 2000 paper, or indeed even if they have no aggregate relative effects on men and women at all."
• stopglobalism 5y ago
  Men != Women Women != Men Sorry if that offends you. Youtube: "Brainwash: The Gender Equality Paradox" https://www.youtube.com/watch?v=tiJVJ5QRRUE https://www.youtube.com/watch?v=tiJVJ5QRRUE
• yololol 5y ago
  My boss recently transfered to my team a lady that was lacking very basic skills. I was, and still am happy to mentor her, and have her in our team, like everyone else who joins us. But I suspect that the only reason we keep her is my manager's KPI that is driven by the (well known) views of higher managent in our company. This makes me feel very bad towards all those candidates I rejected that had much more experience than her, but still not enough for the roles we are recruiting for (we set the bar pretty high). This favoritism is implicitly discrimination against these candidates. But you don't fix discrimination with more discrimination. Instead of moving forward, we repeat the same mistakes with different decoration. I want to move somewhere far away :(