24 ms·

Amazon scraps secret AI recruiting tool that showed bias against women

• trustmath 8y ago
  I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
  • ry_ry 8y ago
    Your comment will probably end up buried, but it does raise the question - if they want more female employees, was the issue in the training data, or their recruitment process?
    • trustmath 8y ago
      Buried, yes, I'm sure. This should be obvious when testing. Whether the algorithm discriminates should be a top priority for designing these algorithms. That's half the damn math of machine learning. If you can construct an AI, you should know how to test it for flaw in reasoning. It's just another layer of ML to do that. Outliers. It's short sighted to push these things out assuming their output is correct just because it looks 'normal'.
  • exo762 8y ago
    Why not "women are just not as interested in math, logic and computer science to pursue it AS OFTEN as men"? Why are you not considering this possibility?
    • trustmath 8y ago
      That's always the excuse but an algorithm that shows bias against women has nothing to do with who has what interest unless that's a variable included in the data set initially. You are inferring that relation implicitly due to the result of the algorithm, but it doesn't mean that value is measured in the original set of data. If the algorithm is skewed to imply that, there is at least the possiblity that the algorithm has been trained to yield that result. I wouldn't know unless I looked at all the data. But I'm not going to default to the popular opinion because that's literally half or more of the problem.
    • village-idiot 8y ago
      They did in the medium past, they didn’t in the recent past, and maybe they will again in the near future.
    • komali2 8y ago
      Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models). Basically the idea that if you stack the cards against a population, you're gonna see population-wide effects. Given these data points, a biased hiring AI contributes to the problem. Therefore, it should be fixed, along with the above points. [0]https://www.bbc.co.uk/news/world-40865261 https://www.bbc.co.uk/news/world-40865261 [0]https://www.theguardian.com/technology/2017/aug/13/james-damore-google-memo-youtube-white-men-radicalization https://www.theguardian.com/technology/2017/aug/13/james-dam...
      • 21 8y ago
        > There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models) This one is a bit weird, computer guys were always "nerds" and "geeks" to stay away from.
        • komali2 8y ago
          Wouldn't being an outcast make you even more attracted to heroes of your "outcast class?" Because, presumably, the hero had to overcome so much more for society to recognize them. Depending on the era, we had Einstein, Turing, feinman. Kids my age had Gates (literally the richest man on the planet for my entire formative years), Jobs, Bill Nye. Little further along are the myth busters crew, musk... We have plenty of heroes to pick from :)
        • SolaceQuantum 8y ago
          The rhetorical context here is that human children look for other human adults that they could potentially grow into in order to aim their own dreams and hopes for their adulthood. If a young human boy sees an adult human man pursuing computers, the young human boy learns that being interested in computers is a socially viable construct and this will affect how he pursues his interests in the future. In consequence, if a young human girl does not see any adult human women in computers, she may not understand that that option is available to her and this will affect how she pursues her interests. Although there is some fuzziness in determining this (some children grow up to be trailblazers, others pursue passions regardless of examples).
    • kaitai 8y ago
      If these women are applying for tech jobs at Amazon, they're by definition interested. The uninterested (male or female) are not relevant to this discussion. I must say, I am frustrated by this being brought up in every discussion of women & STEM. Want to discuss the leaky pipeline from physics PhDs to full professor in physics? "Maybe those women did a PhD in physics despite not being interested, and they just didn't notice before!"
    • thoughtexplorer 8y ago
      Because that's an inconvenient truth that doesn't fit the oppressor vs oppressed narrative.
      • deleted 8y ago
        [deleted]
      • trustmath 8y ago
        Truth over the belief of who is actually 'the best fit' for a job doesn't exist until the job is complete. This isn't about oppression. It's about discrimination magically becoming automated because no one bothers to look for these things pre-deployment.
    • mcguire 8y ago
      Let's say you have two otherwise-identical resumes in front of you. * One says "executed concentration camp prisoners in Kosovo". (Yeah, ok, I'm kidding. How about "Executed a plan to reduce production costs by 30%"?) * The other says, "Won the Women's World Chess Championship 3 years in a row". The first has five stars (thanks to "executed") and the second has three (courtesy of "women's"). Which are you more interested in?
    • tzs 8y ago
      I haven't seen anything suggesting women are less interested, but there is some support for the idea that before college girls who are interested in math., etc., are more likely than boys to have other subjects that interest them more. There was a study published a while back that looked at PISA data and found that girls and boys were pretty evenly represented among the kids who were at the top in STEM [1]. But it also found that for the boys in that group quite often STEM was the only thing they were outstanding at. In other areas they were average to good. For the girls, on the other hand, they were often excellent at something else in addition to STEM, with them often even being better at that something else than they were in STEM. People have a tendency to pursue a career in one of the areas they are very good at. This suggests that boys who are very good in math, etc., are more likely than similarly good girls to pursue it as a career because that is their only choice if they want to go into something they are very good at. The girls are more likely to have math, etc., as one of two or more possible careers in areas they are very good at. In pop culture terms, STEM boys are more like Martin Prince, and STEM girls are more like Lisa Simpson. [1] I didn't save the link and have failed to find it with Google. Anyone have it?
    • UncleMeat 8y ago
      We did consider this possibility. Then we did research and the experts and they found that this isn't supported by evidence.
  • komali2 8y ago
    I understand your frustration, but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. Maybe in the 90s, but in the early 10s there was tons of policy around it, bosses were setting the culture, we were doing everything "right." But we were still not hiring that many women, simply because hardly any women ever applied. For chem e, Mech e, EE roles with 100 applicants, usually I'd see at most one female applicant. It was rare to getm but when we did we'd push for the interview and they'd get through with an average success rate (compared to make applicants). I'm hoping industries that hire young are seeing different numbers than I did, because that should signal a shift in older ones that hire senior discipline engineers after a decade or so. Edit: that said, companies should continue to do what they can to remediate this, but I am furious that the government has done almost nothing about the issue. The underrepresented remain exactly that.
    • trustmath 8y ago
      Yes, I am female, I am aware of the statistics. It's frustrating because my life literally gets impacted by automated reasoning such as this. It's not frustrating for anyone who says 'there just aren't enough of you'. That's something that is very easy to say by people who never have to experience that sort of discriminatory practice. The painful stuff is when it's obvious and provable, because it highlights all the times it can be questionable as to whether it occurs.
    • arbitrage 8y ago
      > but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. In my experience in the industry, this is a laughable statement to make. It's a shame that unless one is a victim of unconscious, systemic bias, one is so much less likely to acknowledge it as a problem, that actually exists, and hurts people all the time.
      • komali2 8y ago
        I don't understand what you're saying - is your argument that only victims of unconscious bias will recognize it as a problem? If that's your point, I guess my counter argument is myself, not a victim, very aware of the problem, and acknowledging it as a problem as my post. There's also the victims of conscious bias that would probably be able to acknowledge the problem... Am I misreading your post?
  • Skrillex 8y ago
    Is this not Amazon trying to not discriminate against women?
    • trustmath 8y ago
      Sure. It just sounds like their code and it's results got too complicated to reason about.
  • thoughtexplorer 8y ago
    > if people would just let them. People do let them. You can't force what people are interested in and you can't let in that which does not exist. In fact many places in tech give preference to women applicants, because they don't apply often and the companies want more women. They're just rare to see. :( There's no grand conspiracy. The truth is much less exciting: Women and men have different preferences, generally speaking. Women are more interested in people (i.e. healthcare). Men are more interested in things (i.e. engineering). Most nurses are women. That's not because women are activity trying to keep men out. It's because fewer men are interested or apply! We also don’t see women in the most dangerous jobs. No one seems to have a problem with that, just as no one has a problem with most healthcare jobs being dominated by women. As they shouldn't, because people should be allowed to pursue and apply to what they want to.
    • trustmath 8y ago
      Please read the article. This article is about automated reasoning that discards resumes that are strongly correlated to resumes of women.
      • sakopov 8y ago
        I think you need to read the comment again. The author's reply was to your comment saying "if people would just let them", not what the ML algorithm does in the article.
        • trustmath 8y ago
          That's somewhat appropriate, but, I think having higher standards for identifying discriminatory practices is covered under the umbrella of 'if people would just let them'. Achieving that level of a standard is a balance. There shouldn't be excuses being made. All that can do is contribute to the perpetuation of the conditions that presently exist, because the core issue isn't being identified. Furthermore, if the core issue is the excuse itself, then again, this is covered under the umbrella of 'if people would just let them'. The secondary issue would then be that the core issue isn't being questioned.
• ncallaway 8y ago
  I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
  • throwawaymath 8y ago
    > Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.
    • hanselot 8y ago
      > tech companies hire more men than women for technology-focused roles. More men are hired because more men eventually end up taking that branch in the RPG of life. Gender studies is not likely to land you a job at amazon, unless it is to feed bias into algorithms.
    • Spooky23 8y ago
      I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15. The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc. In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better. Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.
• ummonk 8y ago
  Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.
  • mcherm 8y ago
    Why not? Seriously: let us take as given that the AI models are biased. Will you also admit that the existing processes are biased? If so, then what we need to ask is which is MORE biased. It might be complaining that we shouldn't release self-driving cars because on rare occasions they cause accidents. There is, however, another criterion besides how biased it is: how biased it will be in the future. Human-driven processes have the opportunity to become less biased in the future (also the chance to become more biased, but overall things tend to improve). AI processes that are opaque might lock in bias in a fashion that is unreviewable. I believe that the solution is to build AI models that are more transparent -- that could be BETTER (in terms of avoiding bias) than the human-driven processes we use today.
    • ummonk 8y ago
      I think we basically have the same view. I don't support black box machine learning models. I do support using automated tests and simple well defined objective criteria though, which is basically a transparent AI model. It's just that generally what seems to distinguish whether something is called "machine learning" rather than "data science and modelling" is that the former is black box and the latter is not.
    • Novashi 8y ago
      >Why not? It has to be better than humans. That requires the rare genius to fix the problem, or we have to accidentally invent something better than ourselves. This is a common defense of autopilot systems in cars: it's not perfect, but it's better than the average.
    • linuxftw 8y ago
      Machines are not intuitive or nuanced. They are incapable of learning and formulating abstract though, only identifying patterns and optimizing for some desired outcome.
• zdragnar 8y ago
  I'm reminded of why Watson failed, and the problem with ml and ai in general- you can't peek under the hood to see why something happened, or how to keep it from happening without a lot of time, a lot of hard work, and a whole lot of carefully groomed data.
  • EeThahs2 8y ago
    > you can't peek under the hood to see why something happened That's myth. There are approaches to analyze and debug NNs, deep dream basically fell out from one of those.
  • bglusman 8y ago
    This is especially memorable/well treated in the Cory Doctorow novella/short story "Human Readable" https://craphound.com/stories/2005/10/12/human-readable/ https://craphound.com/stories/2005/10/12/human-readable/
• strict9 8y ago
  At best ai amplifies existing patterns and biases when handling repetitive work. Over and over we hear how Facebook, Twitter, Google, and others will solve the problem of problematic content and bad actors through ai and neural networks. It's a fraud and the digital potemkin village of our era.
  • fx32s 8y ago
    AI learns from the training data it's given and copies any biases this data exhibits. Pretty much all software today uses ML in some form to improve their services. I feel it's here to stay and not bad by default. We just have to make sure we are aware of its current limitations. Facebook is already auto-flagging content this way but it's just a very hard problem (even for humans).
    • mindcrime 8y ago
      AI learns from the training data it's given and copies any biases this data exhibits I hate to sound like "that pedantic guy", but I'd argue that the quote above is only partially true. It's the case that some subset of AI techniques "learn from the training data it's given and copies any biases this data exhibits". There are AI techniques that aren't based on supervised learning from a pre-existing training set. That doesn't mean that those techniques can't wind up adopting the biases of their human overlords, but I believe some aspects of AI are less susceptible to this kind of bias, than others.
• gfodor 8y ago
  This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class? If we're being honest, a system only needs to be in a decision-making capacity for discriminatory behavior to be scrutinized, since in many cases human operators will not be able to identify the specific features being used to make decisions about people -- the features could be highly correlated with some subpopulation of protected class. If you take that to be true, the question reduces onto what decision-making roles ML algorithms have that could be discriminatory, and it's hard to argue this is not a massive part of their current and expected roles. I think this is going to be a long, winding ethical nightmare that is probably just getting started by human-digestible examples such as these. One can imagine things like this one being looked back on as quaint in the naivety to which we assume we can understand these systems. Where do we draw the line, and how much control do we give up to an optimization function? Surely there is a balance -- how do we categorize and made good decisions around this? As far as I know, a cohesive ethical framework around this is pretty much non-existent -- the current regime is simply "someone speaks up when something absurdly and overtly bad happens."
  • abdullahkhalids 8y ago
    > What about cases where the features just merely correlated with a subpopulation of a protected class? This is just Simpson's paradox [1] which is notoriously hard to identify because you have to compare the overall with the breakdown. As you say, current-AI probably already has such biases. [1] https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
  • jakelazaroff 8y ago
    > What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.
• kgwgk 8y ago
  I’m curious: how is it that this submission has not “reused” the submission I made eight hours before? In my experience, sending a link that is already there simply upvotes the existing submission.
  • deleted 8y ago
    [deleted]
  • adventured 8y ago
    After a certain amount of time, the same link can be posted by a different user. I'm not sure if there is a point or point vs time qualification on that. You can post old, popular stories again for example. Your submission was made 11 hours ago. If they merely applied an upvote to that existing story after 10 hours, the point time value rot would be such that it would have greatly reduced impact at ranking the story to the front page (ie it would be nearly useless for discovery purposes).
• village-idiot 8y ago
  We’re going to keep seeing stuff like this until people finally realize that AI isn’t some magic tool that solves every problem. It still reflects the biases and assumptions of its creators and its training data set.
• fuscy 8y ago
  The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the resume amounting to zilch, gender and the aggressiveness of the resume's language being the most important (because that's how the human recruiters actually hired people when someone put a resume). The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). What matters is the rejection rate which it learned from the data.. The hiring process is inherently biased against women. Technically one could say that the AI was successful because it emulated the current Amazon hiring status.
  • hkai 8y ago
    How did you come to the conclusion that gender was being the most important, rather than skills or aggressiveness?
    • kaitai 8y ago
      I don't think that's what the parent was claiming; the parent says "gender and aggressiveness" were most important and skills listed on the resume as providing such an unclear signal for actual hires that they were not picked up by the AI.
  • lalaland1125 8y ago
    > The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). This is incorrect. The key thing to keep in mind is that they are not just predicting who is a good candidate, they are also ranking by the certainty of their prediction. Lower numbers of female candidates could plausibly lead to lower certainty for the prediction model as it would have less data on those people. I've never trained a model on resumes, but I definitely often see this "lower certainty on minorites" thing for models I do train. The lower certainty would in turn lead to lower rankings for women even without any bias in the data. Now, I'm not saying that Amazon's data isn't biased. I would not be surprised if it were. I'm just saying we should be careful in understanding what is evidence of bias and what is not.
• Novashi 8y ago
  I think there's also a subtle point here that an AI figured out that Amazon hiring was biased more quickly than Amazon itself. I wouldn't go so far to say that running your history through an AI is necessarily proof of anything though (esp. in court). Imagine using another company's historical data to suggest that they're discriminatory.
• deleted 8y ago
  [deleted]
• patwolf 8y ago
  The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed. I see four possibilities here: 1. The algorithm was designed in a completely inept fashion 2. The algorithm design was sound, but ultimately ineffective 3. The algorithm was sound and effective, but results were considered discriminatory. 4. There's something biased about how employees are rated--the data that would feed into the algorithm, which is possibly more of a human element. Edit: Added fourth possibility
  • sanxiyn 8y ago
    My current hypothesis is that training data was too noisy.
  • bicubic 8y ago
    We're talking about Amazon, one of the biggest powerhouse ML employers. I don't buy that the model was poorly designed or ineffective. They also didn't just scrap the model without understanding how or why it failed to meet its objectives. And whatever the cause was, it was not the poor quality of the training data. They tried to stop the model from downranking women based on obvious keywords, only to find it learning to downrank them based on more subtle language cues: > Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. So the answer is 3 or 4. If the answer was 4 then they would have probably mentioned the cause of the bias somewhere in that otherwise detailed article. But they didn't, possibly because the cause is controversial - probably option 3 but possibly still option 4. And then there's the subtle cop-out: > Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said. If the model was actually useless and returning random noise, then there wouldn't be any bias, and the article wouldn't need to talk about discrimination. This paragraph reads to me like they decided to mention long-tail results (that you'd find in any ML model) as supportive 'evidence' that the model was somehow broken rather than producing valid but controversial results.
• electrograv 8y ago
  I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn’t work for every kind of engineering/science (edit: and also would introduce bias against professionals too busy for open source.) Regarding bias — it seems the only way to truly eliminate it (including unconscious bias) is author-blind reviews, i.e. reviewing code written by a candidate without knowing anything about that candidate’s identity. (And the nice thing about code is it usually doesn’t signal any identity traits of the author via side channels.)
  • fhbdukfrh 8y ago
    It doesn't work for the vast majority of professional work because it's not open source and not done in the public
    • deleted 8y ago
      [deleted]
  • kentm 8y ago
    Unfortunately using open source work, even if only for programming, introduces all sorts of biases as well. A lot of very competent programmers work at jobs that do not have open source contributions and also have families which limit the time they can spend coding after work.
    • electrograv 8y ago
      Good point! I didn’t mean to propose it as a solution, but rather as an enumeration or one possibility I’ve considered, and ruled out because it doesn’t work for everyone.
      • kentm 8y ago
• deleted 8y ago
  [deleted]
• jkingsbery 8y ago
  (Disclaimer: I am an Amazon employee sharing his own experience, but do not speak in any official capacity for Amazon. I don't know anything about the system mentioned in this article.) I am a frequent interviewer for engineering roles at Amazon. As part of the interview training and other forums, we often discuss the importance of removing bias, looking out for unconscious bias, and so on. The recruiters I know at Amazon all take reaching out to historically under-represented groups seriously. I don't know anything about the system described in the article (even that we had such a system), but if it was introducing bias I'm glad it's being shelved. Hopefully this article doesn't discourage people from applying to work at Amazon - I've found it a good place to work. To say something about the AI/ML aspect of the article: I think as engineers our instinct is "Here's some data that's been classified for me, I can use ML/AI on it!" without thinking through all that follows, including doing quality assurance. I think a lot of focus in ML (at least in what I've read) has been on generating models, and not nearly enough focus has been on generating models that interpretable (i.e., give a reason along with a classification).
  • darawk 8y ago
    It seems like they did think it through, though? And that's why it's being shelved. I don't really see what the story is here. It seems like the whole process worked exactly as it should - Amazon tried something, it had some unintended consequences, they caught it, and shelved it.
    • weliketocode 8y ago
      Agreed. There is no story here.
      • hannasanarion 8y ago
        The story is, some ML researchers did their job properly and detected ethical issues before they became a problem. That's more rare than you'd think.
        • 8y ago
• nobody271 8y ago
  This never has been a problem about gender issues. It's about finding bugs in magical machine learning algorithms. Im sure the algorithm suffers from many other deficiencies but gender bias is the one people write articles about.
• izzydata 8y ago
  It is no secret that more men pursue careers in this field. How can you expect any algorithm to produce an equal number of men and woman applicants? If it did this then that would be actually biased in favor of woman. If this is what they want then they have to feed their neutral algorithm pre-biased data to get their expected results.
  • briandear 8y ago
    I think it isn’t equality in numbers but in activity screening out women. So there may still be a 8:1 ratio of men:women but according to the article, it would seem than even that 1 women would have been negatively impacted. I am not arguing the validity of their conclusions, just saying that I don’t think that it’s about equal numbers: given a male and female applicant, according to the article, women would have had an unequal chance of passing the screening. (Now if that inequality was for valid reasons, that to me, is an open question, but the article indicates that there was unjustifiable bias.)
    • izzydata 8y ago
      Fair enough. I'm still skeptical of their interpretation without more details, but anything is possible.
• mankypro 8y ago
  Probably being used for age bias as well...
• sanitycheck 8y ago
  Given that Amazon is so far unable to successfully recommend any product which I actually want, even given the vast dataset of my Amazon purchase history, I am not remotely surprised that their engineers can't successfully develop a people recommendation engine either.
  • maym86 8y ago
    - If you hired this male candidate you may be interested in this identical male candidate.
• LinuxBender 8y ago
  What would happen if you removed gender from all HR and recruiting systems and then retrain the AI? Or for that matter, remove ethnicity, age, creed, etc... Is there any reason we need to be more specific? Race: Human Gender: Yes Age of legal contractual consent: Yes
  • TangoTrotFox 8y ago
    The AI did not have access to gender. It was just word weighting and it turned out that words that could be linked to females ended up with applicants that had a negative outcome. Like the article says the AI ended up giving a negative weight to any resume containing the word women's as in women's [---] club, or those that mentioned certain all women's colleges.
    • LinuxBender 8y ago
      It should be fairly easy to filter / replace all of that. The same logic can apply. Can we add some simple filters and re-train it?
      • daenz 8y ago
        >Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. If you've ever trained a NN, you'll know that they are exceedingly clever in finding patterns that fit what you're training for. You can remove the word "women's" and other obvious things from being considered, but I promise you, if there's another non-obvious patterns that are more likely to apply to the women candidates, the AI will find them and use them.
• vishal_pym 8y ago
  This is not very surprising - Machine Learning algorithms trained on biased datasets tend to pick up the hidden biases in the training data. It’s important that we be transparent about the training data that we are using, and are looking for hidden biases in it, otherwise we are building biased systems. Fortunately, there are open source tools out there that help audit machine learning models for bias, such as Audit AI, released by pymetrics - https://github.com/pymetrics/audit-ai https://github.com/pymetrics/audit-ai
• mankypro 8y ago
  Wouldn't an easy way to eliminate bias be to remove any algorithms that use name recognition and gender? If the Ai doesn't have this data to "reason" from wouldn't it level the playing field?
  • falcolas 8y ago
    If, and only if, those are the only differences between candidate resumes. I don't think that's a reasonable assumption to make. Work history differences, sentence structure, word choices - all of these can quietly reflect gender differences.
  • deleted 8y ago
    [deleted]
• paulcnichols 8y ago
  Stupid question probably but why not have two models.
  • T-hawk 8y ago
    Because evaluating differently based on gender actually is discriminatory.
    • commandlinefan 8y ago
      Well, that sounds about right, then - every proposed solution to subtle, unconscious bias inevitably turns out to be explicit bias in the other direction.
• kaitai 8y ago
  I'm doing a bunch of ML on a very different data set -- looking at what people eat (survey data). What's interesting to me is that if you do principal component analysis, for instance, there are some differences between the boys & girls in the sample, but they're not very distinct. If you do clustering or random forests on the dietary intakes of the whole cohort, you get mushy and unclear signals. If you split the survey respondents and bin by age and gender, and run different models for each, suddenly signals jump out of clustering incredibly clearly! What's weirdest is that you get some of the same dietary clusters for the different demographic groups -- but those clusters were not evident when you did clustering across the cohort. It's surprising to me that Amazon didn't (apparently) try different models for different populations. Sure, it might open you up to criticism, but there are some good data-driven reasons to do so. Women's colleges won't show up with regularity on men's resumes, for instance. Similarly, there are fraternities and sororities around engineering and STEM that may provide different signals, but won't appear equally distributed on men's & women's resumes. Language use on resumes does differ by gender, and using "Captured value of $100 million by..." rather than "Created value of $100 million by..." may describe the same project. (I gotta say, using verbs at all seems silly, since it really is about how well you market, rather than what you did.) So, curious about the model. Different models for different subsets of the training data can lead to big wins.
  • rfeather 8y ago
    Off topic, but I'm looking at similar data. Are you talking about a public source (eg NHANES) or something else?
• foolfoolz 8y ago
  this sounds like a bad ML tool to begin with because the input data sucks. it’s probay judging who can write a “good” resume. plenty of bad candidates write good resumes. and good hires write bad ones
• whatgoodisaroad 8y ago
  I feel like many comments here are taking this story on face value, but the cynic in me reads this as a planned leak to skapegoat a (hitherto unknown) AI system for their existing hiring biases. Public perception of AI is more aware of model biases nowadays, and we seem all to willing to accept this explanation over the simpler explanation that tech hiring at Amazon is broken in the same way it is everywhere.
• al_chemist 8y ago
  Alice: Our AI says that graduates from that Boston college are poor job candidates. Bob: Seems reasonable, that college's teaching quality is bad. -- Alice: Our AI says that graduates from that all-women college are poor job candidates. Bob: That can't be right, we all know that women are great job candidates whichever college they graduated.
  • griffinmb 8y ago
    *college Also, it isn't entirely clear what you are trying to say. If you are suggesting that school-based screening is also problematic, then you are probably on the right track.
• macinjosh 8y ago
  This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent, smart, and their differing perspectives invaluable. If there was an untapped market of engineers you'd better believe every tech company would be taking advantage of it.
  • briandear 8y ago
    That is very interesting. What are the stats for unemployed female engineers? Is there a shortage of female representation because female engineers aren’t being hired or is there a shortage because there are actually fewer female engineers? There is a shortage of male therapists and kindergarten teachers: is that because males aren’t being hired or because there are fewer of them in existence?
    • mrnobody_67 8y ago
      Also airline pilots, nurses, michelin-starred chefs...
    • petsormeat 8y ago
      A lot of us drop out mid-career. I'd argue there's more a shortage of female engineering directors, VPs, CTOs.
    • deleted 8y ago
      [deleted]
  • commandlinefan 8y ago
    Yeah - I'm not even expressing an unpopular opinion, just asking a (leading) question: where are all these women who are chomping at the bit to get into _technical_ positions like programming but find themselves being turned away by biased recruiters? I've never even seen somebody _claim_ that they were a woman who couldn't find a tech job, just people wondering where all the women were.
• briandear 8y ago
  “Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured” This is the most interesting part to me. It’s suggesting that men and women think about things differently. Which, if true, suggests a lot of other possibilities that have relevance when hiring for certain positions.
• 40acres 8y ago
  I know diversity in tech is a hot button issue but I think hidden underneath the politics of it all the most important reason to champion more diversity is AI. I don't think we're anywhere close to general AI so any AI system out there is built on what we feed it. If your music suggestion algorithm has a biased AI that's one thing, but when you're using AI to make critical decisions in society like who gets hired/recurited, medical diagnoses and other things you need to be extremely careful. When designing AI models the broadest set of viewpoints should be considered. Any piece of AI is simply a reflection of it's creators, we need to make sure that some sort of equitable consensus is reached before deploying AI to critical human scale issues.
• patrickg_zill 8y ago
  What if we trained a bunch of robots with AIs embedded and they turned out to be racist misogynists? There has got to be a science fiction story about that...
  • castlecrasher2 8y ago
    You mean like Tay?
• daenz 8y ago
  It seems that the only socially acceptable output for the AI would have been hiring women 50% or more and hiring minorities at a rate greater than or equal to their representation in the populations. Anything else is clear bias and discrimination. The project was doomed from the start.
• HashHishBang 8y ago
  Hold on here. This article seems to have buried a pretty important piece of information wayyy down in the middle of the text. > Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said. Granted an article isn't going to get as much attention without an attractive headline but that seems a far more likely reason to have an AI based recruiting recommendation scrapped. The discovery of a negative weight associated with "women's" or graduates of two unnamed women's colleges is notable but if it's tossing out results "almost at random" then...well there seems to be bigger problems?
  • user812 8y ago
    The media is no longer reporting things. You can't make money with reporting. The media is actively creating narratives, and one of the narratives that people are fed nowadays is that women are victims. Men and women are pitted against each other. Due to the way the media has evolved people consume their own biases and most often just read the headlines.
    • HashHishBang 8y ago
      I mean, yeah you're not wrong. I try not to be too cynical about the whole thing even if I think the narrative is suspect. Yes women and minority representation in tech is a potential issue but I really want to know more about the AI recommendation system for potential hires. Especially if it was giving out spurious recommendations. It's amazon, I can't imagine how many millions went into something like that. We'll almost certainly not get a postmortem but it's definitely intriguing.
    • njharman 8y ago
      No longer? Read some history, start with yellow journalism and muckraking. Learning history will teach you more about things today than any news source.
    • rrcaptain 8y ago
• leeny 8y ago
  I tried to do something similar a while ago (for eng hiring specifically). It turned out that the number of grammatical errors and typos mattered way more than anything else on a resume. http://blog.alinelerner.com/lessons-from-a-years-worth-of-hiring-data/ http://blog.alinelerner.com/lessons-from-a-years-worth-of-hi... That aside, what sucks is that attempts to automate resume scoring rarely look at harder-to-quantify features and focus on low-hanging fruit like keyword occurrences... though in my experience it's such a low-signal document for engineering hiring that the whole thing is a fool's errand.
• lawnchair_larry 8y ago
  So why is it not ok that women are penalized by being statistically less likely to get an offer yet it’s just fine to penalize men on auto insurance for being statistically more likely to cause an accident?
  • maym86 8y ago
    Because men are statistically more likely to cause auto accidents. There is no evidence that women are not statistically more likely to be bad at getting good work done at companies like Amazon. It's pretty simple.
    • crimsonalucard 8y ago
      Do you have a source for your second statement?
    • zakm 8y ago
      I'm sure that's true, but do you have any evidence for it?
• deleted 8y ago
  [deleted]
• arandr0x 8y ago
  The idea that Amazon is trying to enforce diversity by using an algorithm that is made to detect, and match, patterns boggles the mind. Why, yes, if your recruiting cost function is "is that person just like all the others we hired", you will end-up with a non-diverse workforce, no matter whether the model optimized with this function has 20 layers, 250 hyperparameters, or two legs, two arms and a fast-receding hairline. You can de-bias by explicitly controlling for gender, but now everyone in your company went to CMU and likes dogs. The more I see news about what recruiting for ultra-large corporations, the more I think one of two things is true: * ultra-large corporations are doomed to hire less and less well in a way that is more and more biaised, and we should regulate against such corporations in a way that forces them to redistribute their wealth to SMBs; * ultra-large corporations need to start exclusively growing through acquisitions, which will have the effect of redistributing their wealth to SMBs, and also of hiring a more diverse base of employees because there is a priori a greater diversity of backgrounds leading to success in the free market than the diversity of backgrounds leading to success in the Amazon interview.
• thoughtexplorer 8y ago
  Anyone who has worked at tech companies and has been involved in the hiring process knows the following: The best thing to be right now is a woman engineer. You can easily get hired within the week. Unfortunately this doesn't seem to be well known outside of those involved in hiring.
  • fizwhiz 8y ago
    Is this anecdotal? Are you insinuating that women have an upper hand now for reasons beyond their skillset?
    • thoughtexplorer 8y ago
      Yes. There is a bonus to being a woman in this situation, all other things being equal. Silicon Valley did a good bit on this https://www.youtube.com/watch?v=Dek5HtNdIHY https://www.youtube.com/watch?v=Dek5HtNdIHY It's funny because it's true.
  • InclinedPlane 8y ago
    This is simply silly. The reason why there's a bias toward hiring women in some roles at some corporations is because they are trying to course correct for the massive amount of systemic bias pushing the other way. For some reason many people (mostly men) seem to easily spot the one type of bias while never being able to see the other.
• darawk 8y ago
  There's something I don't understand in stories like this. It ought to be relatively straightforward to correct biases like this. All you need to do is train a model to explicitly classify gender from resume's, and then use that model to de-gender resume's before passing them on to the hiring model. Is there some reason people aren't doing this?
• xarill 8y ago
  If the gender was passed as a "feature" in the machine learning model, then the company may have been biased. If not, then the majority of the women's CVs were inferior. This important information is not present in the article.
  • core-questions 8y ago
    This entire article is a strong hint that the AI system simply picked the better resumes, where better = more likely to get hired, exactly as it was designed to do. The fact that more men than women satisfy these requirements should come as a surprise to precisely nobody. Fact is, there are more men than women working in tech, especially seriously hard-core stuff like these big companies need. This is most likely out of their own personal volition, and no matter how much outreach we do, this is likely to stay the case for generations. Of course, that's also an egregiously wrongthink position to take. Double plus ungood.
• Proven 8y ago
  Crap, now they'll have to discriminate manually!
• beat 8y ago
  Garbage in, garbage out. The system failed because they were trying to solve the wrong problem, or maybe more specifically, didn't solve the problem that led to the problems with the AI. Amazon was treating the hiring problem as an efficiency problem alone, and ignoring the bias problem. So they wound up training the AI to do a shitty job much faster than humans ever could be shitty - and, by analyzing the data in a way the human results weren't analyzed, showed the failings of the human hiring process. Existing process is sexist. Automate to "improve" it, and you wind up with something even more sexist. What this means is that Amazon needs to go back and revamp their whole hiring process to make it fair, before trying to make it faster.
  • pcmaffey 8y ago
    > So they wound up training the AI to do a shitty job much faster than humans ever could be shitty If nothing else, modeling our existing behavior in this way is a great use case for ML. As it allows us to "fast forward" and thus--hopefully!--identify our flaws based on modeled iterations.
• fromthestart 8y ago
  Things are going to get very interesting when AI inevitably reveals consistent differences in performance among genders. There's only so much you can sweep under a rug.
• jiveturkey 8y ago
  Is it a Microsoft Tay problem?
• otakucode 8y ago
  It seems strange to me that they would train the system on resumes submitted to them over the past 10 years. I presume they judged the ones they hired as 'successes' and those they did not as 'failures'? Or did they judge all submissions as successes, expecting to be testing for 'does this person fit in with the group of people who submitted to us over the past 10 years?' Neither matters much, neither makes sense. What they should have been doing was training it on the resumes submitted by employees who then went on to be very successful within the company. Those are the successes. Those are what you want more of. And, probably far less likely to be weirdly biased by gender or race or whatnot.
• Jyaif 8y ago
  Every product in development is going to be secret, but Reuters tries to make it sounds as if some sort of conspiracy was going on. What's next? "Jeff Bezos secretly goes to the restroom"?
• RangerScience 8y ago
  One of the first good critiques of the "oncoming AI apocalypse" that I've seen is exactly this: the encoding of existing biases into the resulting AI system. But, it also means that bias is now measurable... ...and there were more than a few papers at NIPS that were directly dealing with "fairness" in a NN, aimed at addressing and using these issues and effects.
  • Jtsummers 8y ago
    Weapons of Math Destruction offers good discussion on this problem, if you want to read more about it.
• foxhop 8y ago
  I'm pretty sure this tool reached out to me at one point. Here is the email excerpt: http://pad.yohdah.com/638/amazon-ai-recruiting-email http://pad.yohdah.com/638/amazon-ai-recruiting-email
• crimsonalucard 8y ago
  Once I worked for a startup selling fresh human baby milk to mothers who couldn't produce milk. I was contracted to write a supply side AI that would find people willing to sell the milk they produced to us. The AI had the exact opposite bias as the one in this article... it showed extreme bias against men. I also wrote a similar AI for finding surrogate pregnancy candidates and it also showed bias against men. Goes to show how AI can fail and be incredibly sexist.
• screye 8y ago
  The article never mentions what form of supervision Amazon used for building this model. I can easily see a bias towards a particular gender arising, even when a team had intentions to do, simply because of how the data is selected. ________ Case 1 : They used similarity statistics between candidates that were hired and applicants. This is the easy one. No need to label datasets. The approach is semi-supervised. It will also 100% cause a bias towards employees with similar profiles as those already working at Amazon. (ie. men) ________ Case 2 : They manually labelled/ranked a dataset of resumes and assigned them scores. (more likely) Here the implicit bias of the mechanical turks / rubric would be visible. If higher scores were assigned to traditionally masculine activities, then male resumes would stand out. I doubt this was the case though, as people at Amazon are generally competent enough to not make such a trivial mistake. Also, gender only gets mentioned in non-technical skills (mean's team, women's club, etc), which in general are not the most relevant part of the profile anyways. ________ Speculation: 1. > Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said. I wonder if this has anything to do with gender at all. All good resumes that I've read use action words irrespective of gender. Maybe type A personalities use words like "captured" more often than type B ones, and the % of men in type A categories are greater. Would discrimination against women in such a case be unfair ?....Maybe.....Maybe not. 2. Extracurricular activities are more prominent on weaker resumes than stronger ones. So, the model may be weighing down resumes with too much extracurricular fluff vs technical skills. Men's activities are rarely prefaced with the word "men" in it. (they would just say Football team, Chess team). Women's activities on the other hand, always have the word "women" attached to it. If the extracurricular activities were penalized, then the words inside of them, including "women" would also be penalized. Thus, the model learns a latent gender bias without any bad intentions. _________ In ML, one of my favorite statements is : "The model is only as good as the data it is trained on." If the data is not sufficient, rich enough or prepared in the correct manner, then unintended consequences are nearly guaranteed.
• apercu 8y ago
  The discussion here (versus other channels) is so much better. Not that I expected any different. I'm just glad no one is using this media report to confirm their own biases.
• noetic_techy 8y ago
  The problem is you cant feed the ML algorithm training data based on what your company currently looks like, you have to feed it an idealized set of what you want it to look like. It almost needs to be fictitious training data to hide the ugly bias that's already built in. I don't think this will ever work. There is too much variability in resume wording that correlates to gender and even culture of origin even when you take out names and any other protected class identifying markers. The Dutch tried this and ended up with less diversity. I'm going to go out on a limb and say you almost want to leave all that identifying data in, but put each candidate into buckets with separate rating algorithms trained against only that "type" of candidate. The top candidates from each culture, and the top candidates from each gender, etc etc, however you want to do it. Feed them into a picking algorithm that builds a composite of what you want your team to look like diversity wise based on the top candidates from each bucket, and go from there. Don't take my opinion seriously, I'm not an ML guy.
• gumby 8y ago
  I hate the clickbait way in which this story has spread across the net. "secret AI recruiting tool" sounds like Amazon did something nefarious. Instead they built a tool, found out it was broken, and didn't deploy it. The actual newsworthy part, which is getting slightly stale, is that it was influenced by the data bias. I am not even a fan of Amazon but I think this is unfair to them. They did the right thing here.
• hahan 8y ago
  Call me cynical but I found it amusing that no one points out the fact that engineering work is laborious and dry to say the least for most people. That's the reason why there are so few people who have other options, females, upper middle class people, people of means, in the scene. There are so many lowish paid low status unsought for sectors where majority workers are male, say janitors in Universities, why I never see any discussions on that bias ?
  • anon0007658 8y ago
    Because, though logical, it's a highly unpopular thing to discuss. Narrative has replaced critical thinking in many areas, this being one.
  • zaksoup 8y ago
    Cynical, or wrong? There's a clear history of women in software engineering when it was considered a low-class low-skill job. The first "programmers" in the modern sense of the word were women at NASA. The lead software developer for Apollo 11 was a woman. There's so much evidence to suggest that there is far more at play here than simply "engineering is laborious and dry [so women must not want to do the work]" that I'm willing to state it falls under wikipedia's citing rules for globally known knowledge: I don't need to cite that the capital of France is Paris any more than I need to cite this. This is the same line of already-refuted reasoning behind the "I'm just asking questions" in the infamous Google Memo. To answer my original, rhetorical, question: It's not cynical. It's wrong.
    • _Tev 8y ago
      > There's so much evidence to suggest that there is far more at play here than simply "engineering is laborious and dry [so women must not want to do the work]" that I'm willing to state it falls under wikipedia's citing rules for globally known knowledge: I don't need to cite that the capital of France is Paris any more than I need to cite this. What about citations proving that "software engineering when it was considered a low-class low-skill job" is the same profession as the programming in the past 30years. Or at least that it has the same difficulty / processes. Btw usage of phrases like "clear history", "so much evidence" (especially when you cite one(!) arguable data point), "already-refuted" does not convince anyone about you being right. It is at best annoying.
• arountheworld 8y ago
  Have they not removed protected characteristics from the data? If they had, AI wouldn't know about gender and there would be no bias.
• just_myles 8y ago
  Gender. I am more concerned about race and ethnic background. Why is this never addressed?
• chiefalchemist 8y ago
  > "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said." Doesn't this mean the headline is incorrect, incomplete and/or (intentionally) misleading?
• itry2develop 8y ago
  Maybe the AI is just being honest...
• c3534l 8y ago
  I'm honestly pretty shocked that Amazon went ahead with this which is such an obvious legal liability in an extremely touchy area of business practice. Everything about this should have been a red flag that meant it never should have gotten off the ground, let alone been in place for years.
  • megy 8y ago
    You don't think they should try new things, even though they might not be successful?
    • c3534l 8y ago
      When it systematically and illegally oppresses a class of people in an obviously ethically dubious manner, I would expect a certain level of caution.
• iamleppert 8y ago
  The thought that you can somehow suss out with any consistent performance who will be good or not at their job based solely on a training set consisting of resumes is so banal and inept that its laughable. Team building has so much more to do with assembling the right kinds of people with the right talent and personalities that no present day machine learning could ever hope to come close in effectiveness to a thoughtfully applied interviewing and hiring process. It is clear whatever executives in charge of this project haven't the first clue of whatever it is they are doing. It lacks not only any technical deliberateness but also fails the common sense test. I really wouldn't want to be working for such people.
• ramblerman 8y ago
  How do they measure non-bias? i.e when is it working as intended? My only worry is that they are aiming for 50/50 which doesn't reflect the underlying gender ratio of the developer pool.
• Jeibros 8y ago
  Does data balancing before training the system solve this kind of issues?