25 ms·

MIT Researcher Exposing Bias in Facial Recognition Tech Triggers Amazon’s Wrath

172 points by jonbaer 8y ago

externalreality 8y ago
I heard about different research like this about six years ago. So the algorithm wasn't tested on dark-skinned people. Is this really something new? Product developers in a wide variety of markets still neglect to take into account dark skinned peoples when developing their products. Why should it come as a shock that facial recognition product developers suffer from the same bias?
- Iolaum 8y ago
  We should it be OK that everyone makes mistake A just because some people did it?
- headalgorithm 8y ago
  From the article: "Those disparities can sometimes be a matter of life or death: One recent study of the computer vision systems that enable self-driving cars to “see” the road shows they have a harder time detecting pedestrians with darker skin tones." Just think about the consequences of deploying such systems.
  - js2 8y ago
    The story doesn’t cite the specific paper. But I think I submitted the paper last month: “Predictive Inequality in Object Detection” https://news.ycombinator.com/item?id=19379827 https://news.ycombinator.com/item?id=19379827 https://arxiv.org/abs/1902.11097 https://arxiv.org/abs/1902.11097 I’m not denying the conclusion of the paper, but it does have a lot of limitations. Also, Tesla notwithstanding, isn’t object detection primarily done via lidar?
  - jfk13 8y ago
    I wonder if that's also true for human drivers. I certainly have a harder time detecting pedestrians wearing darker clothing, at least in some circumstances.
    - skookumchuck 8y ago
      That's why road workers, cyclists, and joggers wear brightly colored reflective vests.
    - munchbunny 8y ago
      There is also an important ethical difference between dark skin tones and dark clothing. I see the point you are making, but to the extent that an AI can do better than a human due to the physical aspect, for example, of dark skin reflecting less light at night, I think we should at least try.
  - tropo 8y ago
    The computer vision systems could still beat real human vision systems. This could be by absolute numbers crashed into, or by having less bias than humans, or by both. It's not even a bias problem. It's a lighting, contrast, and camo problem. You could as well claim a bias between asphalt and concrete pavement, based on what skin tones are more visible with the pavement as a background.
    - externalreality 8y ago
      > The computer vision systems could still beat real human vision systems. For pattern recognition, No, the computer vision system can't best a human.
      - tropo 8y ago
        I'm not saying it does today. I'm saying it could do so, while still having bias. The existence of bias doesn't make the computer worse than humans.
      - deogeo 8y ago
        But the entire self-driving system can. It is never tired, or distracted, has far fewer blind spots, and possibly LIDAR and IR to help it.
        scottie_m 8y ago
        It can’t today, and until it proves to do so making claims about future tech in relation to present policy is a bad idea.
    - int_19h 8y ago
      Suppose the system does beat real human vision by absolute numbers , but it turns out that it is objectively worse for someone who is not white, and the higher average is because the majority of people in that society are white. Is that acceptable?
      - deleted 8y ago
  - colllectorof 8y ago
    So maybe we should not be using opaque ML algorithms for life-and-death situations to begin with? Instead of, you know, hoping that whomever is screwed by them has an active advocacy group in places like MIT.
    - mikeash 8y ago
      Using NL for such things hasn’t been working too great, so it seems worthwhile to try for something better.
      - hprotagonist 8y ago
        such as?
        mikeash 8y ago
        Are you asking for examples where NL hasn’t done well or for potential ways to do better?
        hprotagonist 8y ago
        the latter. It’s relatively easy to say “stop doing X”; it’s harder to say “...and have you considered Y instead?”
        mikeash 8y ago
        I’d hope it was obvious from context that ML was the potential way to do better here.
        hprotagonist 8y ago
        I read you as saying that it was the thing doing poorly and that some superior object detection method was required. What that approach could be is totally unobvious to me.
  - sonnyblarney 8y ago
    "Just think about the consequences of deploying such systems." Those systems are already widely deployed - they're called 'humans'. All vision systems are going to be more likely to confuse like colours. This problem is quite a bit different from the 'mugshot' problem, or a problem wherein there was a bias or lack of training data for certain samples. The author I think conflated a lot of the issues and just boiled it down to 'evil AI' which I don't think is the right thing to do. Issues of ethnic orientation are quite a bit different from systems that have trouble literally due to the colour of something.
  - maskymask 8y ago
    I saw a talk by one of the authors of the pedestrian detection study that I think they are talking about, and their results did not seem support the scary racist self-driving car hypothesis that the study clearly implies (only marginal/statistically insignificant effects, with very little controls for confounding factors (I believe they controlled for bounding box size or something, but very little else)). With that said, fairness in ML/AI is a real problem, and some people are doing some really good/important work in this area. I am not familiar with Buolamwini's work, but I'm much more inclined to believe disparities in facial recognition than pedestrian detection where very little skin is visible and almost everything you're seeing is clothing.
  - deleted 8y ago
    [deleted]
- m0zg 8y ago
  Edit: this factual description, written by a practitioner in the field of computer vision, of why the algorithms behave the way they do and what can be done about it, was downvoted and deliberately misinterpreted by the fake outrage mob. So I deleted it. I should have known better than to comment on a topic predisposed to attracting such people.
  - m0zg 8y ago
    Edit: deleted
    - danharaj 8y ago
      > you will get better at recognizing dark skinned people, at the expense of getting worse at recognizing light skinned ones, which generally constitute the majority in the real world Sure, buddy.
      - m0zg 8y ago
        In the US that's not up for debate. That's why you omitted "in most parts of the US" part of my message, I suppose, to manufacture outrage.
        danharaj 8y ago
        You were making a value judgment about optimal utility. That's not a fact. They're just Internet Points, comrade. > I suppose, to manufacture outrage. You're the one who seems outraged =^)
        m0zg 8y ago
        I made no judgment at all. I explained why the algorithms behave the way they do, and that there's currently no real way to make them behave better on darker skinned people without hurting accuracy overall, assuming the data distribution in the wild is not perfectly balanced, which it is not. That's literally all I said. Whatever you make up in your perfervid imagination is just that, your imagination, comrade.
- mannykannot 8y ago
  No-one is saying it is a shock - we're saying it is wrong and should be fixed. As for Amazon's self-serving response, it should be given the same degree of respect as any other statement by an entity that is not prepared to discuss it in an interview.
  - CryptoPunk 8y ago
    Any product is going be a poorer fit for customers fitting a less common profile than those fitting a more common one. You will never get rid of that. Even if performance between races equalized due to a disproportionate amount of resources being expended on adapting the product to fit those belonging to minority racial groups, performance between unusual and common facial types or minority and majority ethnic groups within the races would exist. There will always be some minority group that the product fits less well with, because you can divide people into an infinite number of groups based on an infinite number of traits. This is a consequence of R&D resources being finite and the complexity of the world being unbounded. It's not sign of a moral failing or mis-placed priorities.
    - externalreality 8y ago
      I agree with you. However, without trying to take your points down a slippery slope, your points don't apply to safety concerns. We can't have robo-cars hitting black people because the designers neglected to test the pattern recognition libraries on dark-skinned people. That, you can probably admit, is different from a bar of soap making peoples' skin a bit dry, or lotion not taking into account melanin, or the game designer neglecting the Linux community.
      - CryptoPunk 8y ago
        The accident rate should be decreased in the most efficient way possible, to save as many lives as possible. That might not mean targeting R&D resources at reducing the accident rate for one minority group that experiences a higher-than-average malfunction rate. Here's a thought experiment to illustrate the principle I'm trying to get across: if we find out that members of a very small ethnic group suffer disproportionately from some deficiency in software that does a poor job at recognizing features common to them, and this leads to an extra 50 people dying each year, and that the resources it would take to fix this deficiency would allow the software to be improved to reduce the accident rate by 5%, leading to 5,000 fewer people dying each year, but only 5 fewer members of that ethnic group dying each year, should we target the resources at reducing the accident rate for the ethnic group, just because the group they find themselves in happens to be ethnic?
- klyrs 8y ago
  > So the algorithm wasn't tested on dark-skinned people. Is this really something new? No. Racism in the legal system, and our society overall, is not "new." However it's still a real problem in our society. As such, race- and gender-bias that's "accidentally" hard-coded into software sold to police is problematic. Every new product containing this flaw is news. > Why should it come as a shock that facial recognition product developers suffer from the same bias? At the very least, it should come as a shock that one of the biggest software development firms in the world is attacking critics in defense of buggy behavior. Especially since, as you note, this phenomenon is common knowledge and there's a simple and well-known workaround.
  - Mirioron 8y ago
    Well, here's the question though: should she be naming and shaming the development company or instead the organization that uses the technology? I think the responsibility here lies solely on law enforcement as long as Amazon didn't lie. In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job.
    - klyrs 8y ago
      > ...should she be naming and shaming the development company... Yes, if you're offering a product for sale, it's fair for people to review that product in public. And that's not "naming and shaming," it's a critical review of an irresponsibly-developed product being sold to law-enforcement agencies. > In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job. Yes, that is also their responsibility. But where is the recourse? Law enforcement agencies have abysmal records of self-investigation, and the judicial system is unreliable at best, in holding law enforcement agencies accountable. The public has a right to know what technology is being used to police them. If you want to call investigative journalism "naming and shaming," then yes, absolutely, she made the ethical choice in speaking out.
  - isoskeles 8y ago
    This is pretty low on the list of material problems that affect peoples' lives, in comparison to, for example, actual violence and crime. But I imagine it feels very satisfying to accuse society-at-large of "racism" and use scare quotes around "accidentally" to imply that some white men racistly hard-coded racism into a racist algorithm intended to ruin peoples' lives. I hope you got a good shot of endorphins in your brain by accusing people of intentionally building racist software.
dna_polymerase 8y ago
It is amazing to what length those 'activists' go in order to make sure Amazon's product does work better. Do they know that this stuff will be largely used for surveillance, military uses and other nasty stuff (like exposing adult performers on social media)? If people plan on spending time on activism, there is plenty of real issues with Amazon.
- omegaworks 8y ago
  Well, when Boston Dynamics' dog robot walks into your house with a gun on its back, it's about making sure it knows how to differentiate you from the bad guy it's looking for. We don't want autonomous killer bots to not be able to tell black people apart just because their creators can't.
  - modriano 8y ago
    America would have to become a drastically worse place for any law enforcement agency to deploy an autonomous lethal force that had less than 100% identification accuracy.
    - int_19h 8y ago
      It already deploys a manned lethal force with far less than 100% identification accuracy, so what's the fundamental difference? The arguments in courts would be the same - "an honest mistake", "incident due to unfortunate circumstances" etc, wrapped up with a claim that if such broad latitude to make "mistakes" is not given, then society will descend into lawless anarchy. But it would be applied to people who deploy the robots, instead of the people pulling the trigger. Alternatively, and more likely, measures would be taken to make it not autonomous on paper, e.g. requiring a human operator to approve any action the robot is intending to take. In practice, this would likely be one of those "moral crumple zones" with little practical meaning.
    - Mirioron 8y ago
      I think several decades from now I'd take the Boston dynamics robot walking in rather than a police officer, because the robot won't have to fear for its life, so it's less likely to use lethal force.
FakeComments 8y ago
> Her tests on software created by brand-name tech firms such as Amazon uncovered much higher error rates in classifying the gender of darker-skinned women than for lighter-skinned men. The presumption is that this is racism, but is there any support that it’s not caused by physiological differences? Women have softer bone structure, on average, and having darker skin literally impacts how light reflects off of it. It’s possible that the change in bone structure and contrast for shadowed parts of the face makes it harder to identify those faces — that’s certainly the case for rounder, darker objects in general. As far as this article has shown, the researcher is simply so woke she no longer understands biology and physics — and is race baiting instead of acknowledging genuine variation between humans. Frankly, this seems like a hyper partisan kicking up a fuss over imagined racism than any documented genuine racism. If the researcher lacks evidence that this is racism rather than a result of physiological differences, this is nothing but race baiting slander — and Amazon is completely correct to contest that nonsense.
- mikeash 8y ago
  The researcher said she identified racial bias. The factors you describe are merely reasons why such bias would exist. What’s your basis for saying this researcher is race baiting, hyper partisan, or showing a lack of understanding of anything?
  - FakeComments 8y ago
    A racial bias is not indicative of racism — and is not inherently problematic. The entire tone of the article, and indeed her own comments, is that this racial bias is somehow problematic or must be gotten rid of. > What’s struck a chord about Boulamwini’s work is her method of testing the systems created by well-known companies. She applies such systems to a skin-tone scale used by dermatologists, then names and shames those that show racial and gender bias. Buolamwini, who’s also founded a coalition of scholars, activists and others called the Algorithmic Justice League, has blended her scholarly investigations with activism. The question of differential identification between white and black may merely be a question of physics (ie, any recognition system has an easier time of white people because of how optics work), and the relevant question for implementing facial recognition is if it works well enough at identifying black people to be usable. To name and shame people because their product responds to genuine physical differences in groups of people is absurd. Comparing white people and black people without acknowledging the racial disparity in optical response is race baiting; comparing men and women without acknowledging the broad scale difference in bone structure is gender baiting. Why? Because there is no problem with a racial or gender bias if it extends from a physiological difference, and not willful discrimination. Her research is the equivalent of concluding the grocery store is problematically heightist because tall people have an easier time reaching the top shelf — unseriousness unworthy of the topic, and outrage bait.
deogeo 8y ago
I sure feel privileged that Amazon's software can track and spy on me effectively.
tangue 8y ago
Issues are going beyond face recognition and algorithms : photography has a long history of bias [1]. Besides the issue of insufficient testing, I had to ask myself : what if these algorithms are relying on a biased process thinking it was impartial ? [1] https://jezebel.com/the-truth-about-photography-and-brown-skin-1557656792 https://jezebel.com/the-truth-about-photography-and-brown-sk...
- creato 8y ago
  Can anyone explain exactly what is missing in photography due to "bias"? I'm just an amateur photographer that developed film in high school and plays around with RAW and Lightroom. I can't think of anything in the process that isn't simply a matter of light sensitivity and dynamic range. Film and image sensors don't understand high level semantic features like skin. The hardest thing to take a photo of is a black haired cat, and I don't think that's because Adobe is biased agains black cats.
  - tangue 8y ago
    Following the link I posted might be a good start.
    - creato 8y ago
      Do you really think Kodak didn't want to make a film that could capture dark subjects until furniture companies asked for it? The reason I commented is because that article makes a lot of complaints that drum up outrage. As far as I know are of the complaints all much better explained by either 1) less light is harder to photograph, or 2) photographing bright things and dark things at the same time is harder than photographing just bright things or just dark things.
  - swish_bob 8y ago
    In the days of colour film photography, one of the stages applied was skin tone matching. When trying to set your printer up you printed a fairly standard picture, and checked the skin tone looked right. For a long time nobody thought to do the test with people who weren't white, so often the skin tones of darker people were off - hence the pictures you got developed were more likely to be accurate if your subjects were white. https://www.npr.org/2014/11/13/363517842/for-decades-kodak-s-shirley-cards-set-photography-s-skin-tone-standard https://www.npr.org/2014/11/13/363517842/for-decades-kodak-s...
colllectorof 8y ago
Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical. Also, people should have the right to know when machine-learned models are used to make decisions about their lives. They should be able to ask why a particular decisions was made and get that information. This is real AI ethics.
- YeGoblynQueenne 8y ago
  You know what AI systems had the ability to explain their decisions? Expert systems. Expert systems were the big success of GOFAI, but they fell out of favour in the last AI winter, at the end of the '90s or so, clearing the way for probabilistic inference and statistical machine learning. Since then it seems, we took one step forward (with accuracy in classification) and one step back (with the loss of the ability to explain decisions). Who knows, maybe a new AI winter will wipe out the statistical machine learning dinosaurs of today and leave a clear field of play for the AI Mammals of tomorrow.
  - colllectorof 8y ago
    Well, there are some statistical learning models that are explainable. Decision trees can be induced and can "explain themselves", just like in expert systems. Additive models are self-explanatory. Not sure I'm sold on LIME and other similar approaches, though. Seems like a lot of deep learning people are all too happy to substitute "intepretability" for actual explanations.
    - YeGoblynQueenne 8y ago
      Decision trees are not a statistical model. The learning method is statistical, if you will, but the model itself is a disjunction of conjunctions- a symbolic model (and more specifically, a propositional logic theory). Decision Trees are in fact an example of the early years of machine learning where the trend was towards algorithms and techniques that learned symbolic theories. I believe the effort was driven by the realisation that expert systems had a certain problem with knowlege acquisition [1] which drove people to try and learn production rules from data. I digress- I mean to say that decision trees are explainable because their models are not statistical. To be honest, I don't know much about additive models. _______________ [1] https://en.wikipedia.org/wiki/Knowledge_acquisition https://en.wikipedia.org/wiki/Knowledge_acquisition
YeGoblynQueenne 8y ago
>> Chris Adzima, senior information systems analyst for the Washington County Sheriff’s Office in Oregon, said the agency uses Amazon’s Rekognition to identify the most likely matches among its collection of roughly 350,000 mug shots. But because a human makes the final decision, “the bias of that computer system is not transferred over into any results or any action taken,” Adzima said. The gentleman is saying that the system selects a small subset of the 350,000 mugshots, but because a human selects one mugshot from this small subset, there is no bias. That just makes no sense. [edited to remove stronger language]
- Mirioron 8y ago
  It does make some sense. Yes, technically, the bias is still there, but it's lower. Getting some accuracy on image detection isn't that difficult, but making it more and more accurate is. Going from 50% accuracy to 51% is much easier than 98% to 99%. Of course, with the error rates reported on certain groups a human making the final decision is not enough.
  - Retric 8y ago
    The problem is a human would at best reduce but not eliminate the bias. Let’s assume the photo was not a match for the database. Now a human doing the final step is essentially pick a random face from a biased sample. Worse, people are really bad about assuming whatever option they are considering to be far more likely than the base rate. Read up on an random obscure disease and suddenly you start thinking it’s a real risk. Sadly, police do the same thing with criminal suspects resulting in innocent people in prison.
jothezero 8y ago
Not the first article about this kind of stuff. Police departments can do whatever they want to obtain leads (as long as it is legal (Gray area)). I will be worried when they will start using technology instead of judges.
- acdha 8y ago
  That seems like a shortsighted position because it ignores the cost of selective enforcement. Everyone breaks laws on a daily basis - jaywalking, speeding, etc. - and you could have a severely disproportionate cost by enforcing that more for one group than another, even with every trial being completely fair. Here in DC there was an example awhile back prior to legalizing marijuana: white people apparently used at a higher rate but most of the prosecutions were of black people both due to heavier police presence and because demographics meant that white users tended to have more privacy (limited visibility from the street, more distance between houses/sidewalks to make smell harder to notice, etc.) which made it harder to get evidence clearly showing that a specific person had been the one using. The process could be fair without changing the fact that the results disproportionately impacted one group.
  - jothezero 8y ago
    Eternal debate.
  - Mirioron 8y ago
    I've been saying this for a long time: selective enforcement of laws is not okay. We need to design laws that are not enforced selectively, but consistently. If it means that jaywalking shouldn't be against the law then so be it.
  - CryptoPunk 8y ago
    That's not "selective enforcement". Race is not being selected for. People committing crimes in public is being selected for. Or people committing crimes in high-crime areas. Some races happen to correlate with some selected traits more than others, but race is not the trait selected for. It's completely predictable that not all traits of interest for law enforcement will be distributed equally across all racial groups. To treat this fact as a sign of systemic racism is to guarantee that you will consider every society on Earth systemically rac/sex/[group] ist.
RenRav 8y ago
It seems like every other month, some facial recognition system is being attacked because of this.
salty_biscuits 8y ago
Is this something simple like class imbalance in the training sets? Would be pathetic if that were the case. So easily fixed.
malshe 8y ago
The article doesn't mention papers so I am linking them here: Original study [PDF]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... New study [PDF]: https://www.thetalkingmachines.com/sites/default/files/2019-02/aies-19_paper_223.pdf https://www.thetalkingmachines.com/sites/default/files/2019-...
- ulucs 8y ago
  I read the original paper. Wow, they could have tested for so many factors impacting accuracy but they only looked at mean accuracy for each cluster.
  - DarkWiiPlayer 8y ago
    Haven't read the study myself yet, but that sentence alone makes me not want to waste my time on it.
treis 8y ago
Does anyone have a link to the paper?
apta 8y ago
> Her tests on software created by brand-name tech firms such as Amazon uncovered much higher error rates in classifying the gender of darker-skinned women than for lighter-skinned men. What about lighter-skinned women? This seems to be phrased on purpose to incite bias against white men.
- spinach 8y ago
  Going by the article, it just seems to be because the researcher is a darker-skinned woman and the software is mainly trained on lighter-skinned men. "Darker-skinned women were the most misclassified group, with error rates of up to 34.7%. By contrast, the maximum error rate for lighter-skinned males was less than 1%."
- dosy 8y ago
  sure, but I don't think that's her intention. she's just refitting an old already polarizing topic as a vehicle to troll big tech. polarizing topics like these are like performance-enhancers for messaging if you can weave them into your narrative.
- jake-low 8y ago
  Here's the paper [0]. The answer to your question is in Tables 4 and 5 (page 9). To summarize, darker skinned women are misclassified far more often than any other group, in all three of the tested facial recognition systems. Error rates for lighter skinned women are between 5x and 10x lower, depending on the system. Lighter skinned men have the lowest classification error rates by a wide margin in the IBM and Microsoft systems, and are tied with darker skinned men for lowest error rate in the Face++ system. [0]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
dosy 8y ago
I love this, whether the conclusions are valid or not, using science to troll big tech where they live. ouch, right in the feels.
tdaltonc 8y ago
Beyond the very real intersectional problems highlighted . . . The fact that the Amazon gender classifier miss-identifies 7% of white females as male, but 0% of white males as female is very odd. That seems like a JV level bias tuning mistake. That's not systemic violence so much as it is just sloppy.
- caprese 8y ago
  > miss-identifies 7% of white females as male, but 0% of white males as female is very odd. Sounds like it was trained in San Francisco, I can relate to that
- nitwit005 8y ago
  Most of the time the headline for these articles seems like it should be "Crap product also biased". I have some suspicion they haven't focused on bringing down the error rates for minority groups because they know even their best case isn't good enough.
  - tdaltonc 8y ago
    Check out the academic article.[0] It shows that Microsoft and IBM are way way better. [0] http://www.aies-conference.com/wp-content/uploads/2019/01/AIES-19_paper_223.pdf http://www.aies-conference.com/wp-content/uploads/2019/01/AI...
- DarkWiiPlayer 8y ago
  Just a quick thought: women tend to wear makeup, which is rather uncommon among men. That may be one part of the reason why that happens.
  - tdaltonc 8y ago
    True, if the system was only trained on women wearing make-up, but tested on women not wearing makeup, that would definitely create this kind of bias. That's exactly the kind of thing that a good AI engineer should be looking out for.
Thermolabile 8y ago
What does this article have to do with racism and not the failing of the recognition system? It is supposed to be a facial recognition system, it doesn’t work on specific people at a large percentage.. It’s brought to the attention of the manufacturers and should be worked on being fixed.
- swish_bob 8y ago
  And when the manufacturer says "nuh uh" like they did in this case?
torgian 8y ago
"Her tests on software created by brand-name tech firms such as Amazon uncovered much higher error rates in classifying the gender of darker-skinned women than for lighter-skinned men." Besides the article's obvious anti-man, anti-white bias, I'm not surprised that facial recognition software has a hard time analyzing darker skinned people. With photography, I've always had a hard time photographing someone who had dark skin. Lots of light needs to be used, and even then it needs to be filtered correctly, etc. This goes for any subject that is dark. Unfortunately, this is going to be a tough problem. Cameras used by law enforcement and government agencies, (which the article seems to focus on) are normally pretty shitty; software can only do what it does with whatever input it gets. So if the lighting and image quality is shitty, then your results will be as equally shitty. The article doesnt go into what kind of equipment the MIT researcher was using, but I will assume that it is a high quality camera. If so, and if the software is still failing as the article alludes to, then yes, these companies need to make their software better. Even so, it's a crapshoot from the get-go, due to the hardware being used.
- false_alarm 8y ago
  I did not notice an anti-man or anti-white bias in the article. What stuck out to you in the article that seemed biased?
- Balgair 8y ago
  > With photography, I've always had a hard time photographing someone who had dark skin. Lots of light needs to be used, and even then it needs to be filtered correctly, etc. This goes for any subject that is dark. The issue does come down to a mix of (likely) inadvertent racism and physics. It is very difficult to photograph people with darker skin. This is intentional, per Mother Nature. Very briefly, our atmosphere passes a certain bandwidth of colors to the ground. The peak frequency is in the yellow-green (this is why plants are this color usually). Unfortunately, one of those colors is in the UV frequency that just so happens to be the same frequency as the small channel of eukayotic DNA. Thus, cancer risk is raised when that UV photon energetically breaks up the DNA. So, mother nature evolved to co-opt the melanin molecule, which happens to absorb those UV photons, to help protect skin cells from this damage. Thing is, though, that melanin molecule also absorbs not just that UV color, but a lot of other ones. PhysicsGirl has a good video on how sunscreen works and what a freckle looks like under different wavelengths [0] that is informative here. Now for the (inadvertent) racism. When they were first starting to make photographs, just about any photo sensitive material was 'good enough'. These, of course, were all in black and white. But, the frequencies/colors that these chemicals were photosensitive to, were much more than we can see with our eyes. It went from the low IR to the high UV, with all kinds of various notches and mixes in there. Tin-type photographs are a great example. During the US Civil war, there were a lot of pictures taken of slavery and 'colored' soldiers. It's not hard to distinguish the faces and features of these people [1][2]. One thing to note about these tin-types, look at the clouds, rather, the total lack thereof. That's an easy way to notice that the frequencies you are seeing in the photograph are not the frequencies that your eyes see. When color photography started to become more widespread, the frequencies used for each color (CYM) and the absorptive bandwidth of each part of the color films were not accidents. They were chosen such that attractive young 'white' women in very heavy make-up and under very bright 1930's studio lights would be best seen. These color choices, due to the inherent racism of the times and other factors, were perpetuated into modern CCDs and CMOS chips we use today. There are important differences in the physics of digital and film color photography, but largely those original choices have been conserved. One important difference is the IR spectrum pick-up. In most modern phones, the front facing camera does not have an IR filter on it. Next time you are near a security camera or at a toilet with those IR sensors, take a look at your phone using the front-facing camera. It should be straightforward to see the little blinking IR LEDs of these devices. So, though darker skinned people are inherently difficult to photograph due to melanin and Mother Nature, it is not difficult to find the right frequencies that will work. It's a case of the 'lock in' effect that prevents this. [0] https://www.youtube.com/watch?v=GRD-xvlhGMc https://www.youtube.com/watch?v=GRD-xvlhGMc [1] https://www.pinterest.com/pin/385480049327844109/ https://www.pinterest.com/pin/385480049327844109/ [2] https://atlantablackstar.com/2015/01/05/7-things-about-the-field-workers-during-slavery-that-you-may-not-know/ https://atlantablackstar.com/2015/01/05/7-things-about-the-f... [3] https://www.pinterest.com/pin/126452702010324184/ https://www.pinterest.com/pin/126452702010324184/
myopicgoat 8y ago
Dh
username90 8y ago
Where is the wrath? Seems like Amazon just hand waved away her criticism, they didn't really take any actions against her.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]

172 points by jonbaer 8y ago

I heard about different research like this about six years ago. So the algorithm wasn't tested on dark-skinned people. Is this really something new? Product developers in a wide variety of markets still neglect to take into account dark skinned peoples when developing their products. Why should it come as a shock that facial recognition product developers suffer from the same bias?

Iolaum 8y ago
We should it be OK that everyone makes mistake A just because some people did it?
headalgorithm 8y ago
From the article: "Those disparities can sometimes be a matter of life or death: One recent study of the computer vision systems that enable self-driving cars to “see” the road shows they have a harder time detecting pedestrians with darker skin tones." Just think about the consequences of deploying such systems.
- js2 8y ago
  The story doesn’t cite the specific paper. But I think I submitted the paper last month: “Predictive Inequality in Object Detection” https://news.ycombinator.com/item?id=19379827 https://news.ycombinator.com/item?id=19379827 https://arxiv.org/abs/1902.11097 https://arxiv.org/abs/1902.11097 I’m not denying the conclusion of the paper, but it does have a lot of limitations. Also, Tesla notwithstanding, isn’t object detection primarily done via lidar?
- jfk13 8y ago
  I wonder if that's also true for human drivers. I certainly have a harder time detecting pedestrians wearing darker clothing, at least in some circumstances.
  - skookumchuck 8y ago
    That's why road workers, cyclists, and joggers wear brightly colored reflective vests.
  - munchbunny 8y ago
    There is also an important ethical difference between dark skin tones and dark clothing. I see the point you are making, but to the extent that an AI can do better than a human due to the physical aspect, for example, of dark skin reflecting less light at night, I think we should at least try.
- tropo 8y ago
  The computer vision systems could still beat real human vision systems. This could be by absolute numbers crashed into, or by having less bias than humans, or by both. It's not even a bias problem. It's a lighting, contrast, and camo problem. You could as well claim a bias between asphalt and concrete pavement, based on what skin tones are more visible with the pavement as a background.
  - externalreality 8y ago
    > The computer vision systems could still beat real human vision systems. For pattern recognition, No, the computer vision system can't best a human.
    - tropo 8y ago
      I'm not saying it does today. I'm saying it could do so, while still having bias. The existence of bias doesn't make the computer worse than humans.
    - deogeo 8y ago
      But the entire self-driving system can. It is never tired, or distracted, has far fewer blind spots, and possibly LIDAR and IR to help it.
      - scottie_m 8y ago
        It can’t today, and until it proves to do so making claims about future tech in relation to present policy is a bad idea.
  - int_19h 8y ago
    Suppose the system does beat real human vision by absolute numbers , but it turns out that it is objectively worse for someone who is not white, and the higher average is because the majority of people in that society are white. Is that acceptable?
    - deleted 8y ago
- colllectorof 8y ago
  So maybe we should not be using opaque ML algorithms for life-and-death situations to begin with? Instead of, you know, hoping that whomever is screwed by them has an active advocacy group in places like MIT.
  - mikeash 8y ago
    Using NL for such things hasn’t been working too great, so it seems worthwhile to try for something better.
    - hprotagonist 8y ago
      such as?
      - mikeash 8y ago
        Are you asking for examples where NL hasn’t done well or for potential ways to do better?
        hprotagonist 8y ago
        the latter. It’s relatively easy to say “stop doing X”; it’s harder to say “...and have you considered Y instead?”
        mikeash 8y ago
        I’d hope it was obvious from context that ML was the potential way to do better here.
        hprotagonist 8y ago
        I read you as saying that it was the thing doing poorly and that some superior object detection method was required. What that approach could be is totally unobvious to me.
- sonnyblarney 8y ago
  "Just think about the consequences of deploying such systems." Those systems are already widely deployed - they're called 'humans'. All vision systems are going to be more likely to confuse like colours. This problem is quite a bit different from the 'mugshot' problem, or a problem wherein there was a bias or lack of training data for certain samples. The author I think conflated a lot of the issues and just boiled it down to 'evil AI' which I don't think is the right thing to do. Issues of ethnic orientation are quite a bit different from systems that have trouble literally due to the colour of something.
- maskymask 8y ago
  I saw a talk by one of the authors of the pedestrian detection study that I think they are talking about, and their results did not seem support the scary racist self-driving car hypothesis that the study clearly implies (only marginal/statistically insignificant effects, with very little controls for confounding factors (I believe they controlled for bounding box size or something, but very little else)). With that said, fairness in ML/AI is a real problem, and some people are doing some really good/important work in this area. I am not familiar with Buolamwini's work, but I'm much more inclined to believe disparities in facial recognition than pedestrian detection where very little skin is visible and almost everything you're seeing is clothing.
- deleted 8y ago
  [deleted]
m0zg 8y ago
Edit: this factual description, written by a practitioner in the field of computer vision, of why the algorithms behave the way they do and what can be done about it, was downvoted and deliberately misinterpreted by the fake outrage mob. So I deleted it. I should have known better than to comment on a topic predisposed to attracting such people.
- m0zg 8y ago
  Edit: deleted
  - danharaj 8y ago
    > you will get better at recognizing dark skinned people, at the expense of getting worse at recognizing light skinned ones, which generally constitute the majority in the real world Sure, buddy.
    - m0zg 8y ago
      In the US that's not up for debate. That's why you omitted "in most parts of the US" part of my message, I suppose, to manufacture outrage.
      - danharaj 8y ago
        You were making a value judgment about optimal utility. That's not a fact. They're just Internet Points, comrade. > I suppose, to manufacture outrage. You're the one who seems outraged =^)
        m0zg 8y ago
        I made no judgment at all. I explained why the algorithms behave the way they do, and that there's currently no real way to make them behave better on darker skinned people without hurting accuracy overall, assuming the data distribution in the wild is not perfectly balanced, which it is not. That's literally all I said. Whatever you make up in your perfervid imagination is just that, your imagination, comrade.
mannykannot 8y ago
No-one is saying it is a shock - we're saying it is wrong and should be fixed. As for Amazon's self-serving response, it should be given the same degree of respect as any other statement by an entity that is not prepared to discuss it in an interview.
- CryptoPunk 8y ago
  Any product is going be a poorer fit for customers fitting a less common profile than those fitting a more common one. You will never get rid of that. Even if performance between races equalized due to a disproportionate amount of resources being expended on adapting the product to fit those belonging to minority racial groups, performance between unusual and common facial types or minority and majority ethnic groups within the races would exist. There will always be some minority group that the product fits less well with, because you can divide people into an infinite number of groups based on an infinite number of traits. This is a consequence of R&D resources being finite and the complexity of the world being unbounded. It's not sign of a moral failing or mis-placed priorities.
  - externalreality 8y ago
    I agree with you. However, without trying to take your points down a slippery slope, your points don't apply to safety concerns. We can't have robo-cars hitting black people because the designers neglected to test the pattern recognition libraries on dark-skinned people. That, you can probably admit, is different from a bar of soap making peoples' skin a bit dry, or lotion not taking into account melanin, or the game designer neglecting the Linux community.
    - CryptoPunk 8y ago
      The accident rate should be decreased in the most efficient way possible, to save as many lives as possible. That might not mean targeting R&D resources at reducing the accident rate for one minority group that experiences a higher-than-average malfunction rate. Here's a thought experiment to illustrate the principle I'm trying to get across: if we find out that members of a very small ethnic group suffer disproportionately from some deficiency in software that does a poor job at recognizing features common to them, and this leads to an extra 50 people dying each year, and that the resources it would take to fix this deficiency would allow the software to be improved to reduce the accident rate by 5%, leading to 5,000 fewer people dying each year, but only 5 fewer members of that ethnic group dying each year, should we target the resources at reducing the accident rate for the ethnic group, just because the group they find themselves in happens to be ethnic?
klyrs 8y ago
> So the algorithm wasn't tested on dark-skinned people. Is this really something new? No. Racism in the legal system, and our society overall, is not "new." However it's still a real problem in our society. As such, race- and gender-bias that's "accidentally" hard-coded into software sold to police is problematic. Every new product containing this flaw is news. > Why should it come as a shock that facial recognition product developers suffer from the same bias? At the very least, it should come as a shock that one of the biggest software development firms in the world is attacking critics in defense of buggy behavior. Especially since, as you note, this phenomenon is common knowledge and there's a simple and well-known workaround.
- Mirioron 8y ago
  Well, here's the question though: should she be naming and shaming the development company or instead the organization that uses the technology? I think the responsibility here lies solely on law enforcement as long as Amazon didn't lie. In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job.
  - klyrs 8y ago
    > ...should she be naming and shaming the development company... Yes, if you're offering a product for sale, it's fair for people to review that product in public. And that's not "naming and shaming," it's a critical review of an irresponsibly-developed product being sold to law-enforcement agencies. > In fact, I would expect law enforcement to test this software thoroughly themselves, because that's their job. Yes, that is also their responsibility. But where is the recourse? Law enforcement agencies have abysmal records of self-investigation, and the judicial system is unreliable at best, in holding law enforcement agencies accountable. The public has a right to know what technology is being used to police them. If you want to call investigative journalism "naming and shaming," then yes, absolutely, she made the ethical choice in speaking out.
- isoskeles 8y ago
  This is pretty low on the list of material problems that affect peoples' lives, in comparison to, for example, actual violence and crime. But I imagine it feels very satisfying to accuse society-at-large of "racism" and use scare quotes around "accidentally" to imply that some white men racistly hard-coded racism into a racist algorithm intended to ruin peoples' lives. I hope you got a good shot of endorphins in your brain by accusing people of intentionally building racist software.

dna_polymerase 8y ago

It is amazing to what length those 'activists' go in order to make sure Amazon's product does work better. Do they know that this stuff will be largely used for surveillance, military uses and other nasty stuff (like exposing adult performers on social media)? If people plan on spending time on activism, there is plenty of real issues with Amazon.

omegaworks 8y ago
Well, when Boston Dynamics' dog robot walks into your house with a gun on its back, it's about making sure it knows how to differentiate you from the bad guy it's looking for. We don't want autonomous killer bots to not be able to tell black people apart just because their creators can't.
- modriano 8y ago
  America would have to become a drastically worse place for any law enforcement agency to deploy an autonomous lethal force that had less than 100% identification accuracy.
  - int_19h 8y ago
    It already deploys a manned lethal force with far less than 100% identification accuracy, so what's the fundamental difference? The arguments in courts would be the same - "an honest mistake", "incident due to unfortunate circumstances" etc, wrapped up with a claim that if such broad latitude to make "mistakes" is not given, then society will descend into lawless anarchy. But it would be applied to people who deploy the robots, instead of the people pulling the trigger. Alternatively, and more likely, measures would be taken to make it not autonomous on paper, e.g. requiring a human operator to approve any action the robot is intending to take. In practice, this would likely be one of those "moral crumple zones" with little practical meaning.
  - Mirioron 8y ago
    I think several decades from now I'd take the Boston dynamics robot walking in rather than a police officer, because the robot won't have to fear for its life, so it's less likely to use lethal force.

FakeComments 8y ago

> Her tests on software created by brand-name tech firms such as Amazon uncovered much higher error rates in classifying the gender of darker-skinned women than for lighter-skinned men. The presumption is that this is racism, but is there any support that it’s not caused by physiological differences? Women have softer bone structure, on average, and having darker skin literally impacts how light reflects off of it. It’s possible that the change in bone structure and contrast for shadowed parts of the face makes it harder to identify those faces — that’s certainly the case for rounder, darker objects in general. As far as this article has shown, the researcher is simply so woke she no longer understands biology and physics — and is race baiting instead of acknowledging genuine variation between humans. Frankly, this seems like a hyper partisan kicking up a fuss over imagined racism than any documented genuine racism. If the researcher lacks evidence that this is racism rather than a result of physiological differences, this is nothing but race baiting slander — and Amazon is completely correct to contest that nonsense.

mikeash 8y ago
The researcher said she identified racial bias. The factors you describe are merely reasons why such bias would exist. What’s your basis for saying this researcher is race baiting, hyper partisan, or showing a lack of understanding of anything?
- FakeComments 8y ago
  A racial bias is not indicative of racism — and is not inherently problematic. The entire tone of the article, and indeed her own comments, is that this racial bias is somehow problematic or must be gotten rid of. > What’s struck a chord about Boulamwini’s work is her method of testing the systems created by well-known companies. She applies such systems to a skin-tone scale used by dermatologists, then names and shames those that show racial and gender bias. Buolamwini, who’s also founded a coalition of scholars, activists and others called the Algorithmic Justice League, has blended her scholarly investigations with activism. The question of differential identification between white and black may merely be a question of physics (ie, any recognition system has an easier time of white people because of how optics work), and the relevant question for implementing facial recognition is if it works well enough at identifying black people to be usable. To name and shame people because their product responds to genuine physical differences in groups of people is absurd. Comparing white people and black people without acknowledging the racial disparity in optical response is race baiting; comparing men and women without acknowledging the broad scale difference in bone structure is gender baiting. Why? Because there is no problem with a racial or gender bias if it extends from a physiological difference, and not willful discrimination. Her research is the equivalent of concluding the grocery store is problematically heightist because tall people have an easier time reaching the top shelf — unseriousness unworthy of the topic, and outrage bait.

deogeo 8y ago

I sure feel privileged that Amazon's software can track and spy on me effectively.

tangue 8y ago

Issues are going beyond face recognition and algorithms : photography has a long history of bias [1]. Besides the issue of insufficient testing, I had to ask myself : what if these algorithms are relying on a biased process thinking it was impartial ? [1] https://jezebel.com/the-truth-about-photography-and-brown-skin-1557656792 https://jezebel.com/the-truth-about-photography-and-brown-sk...

creato 8y ago
Can anyone explain exactly what is missing in photography due to "bias"? I'm just an amateur photographer that developed film in high school and plays around with RAW and Lightroom. I can't think of anything in the process that isn't simply a matter of light sensitivity and dynamic range. Film and image sensors don't understand high level semantic features like skin. The hardest thing to take a photo of is a black haired cat, and I don't think that's because Adobe is biased agains black cats.
- tangue 8y ago
  Following the link I posted might be a good start.
  - creato 8y ago
    Do you really think Kodak didn't want to make a film that could capture dark subjects until furniture companies asked for it? The reason I commented is because that article makes a lot of complaints that drum up outrage. As far as I know are of the complaints all much better explained by either 1) less light is harder to photograph, or 2) photographing bright things and dark things at the same time is harder than photographing just bright things or just dark things.
- swish_bob 8y ago
  In the days of colour film photography, one of the stages applied was skin tone matching. When trying to set your printer up you printed a fairly standard picture, and checked the skin tone looked right. For a long time nobody thought to do the test with people who weren't white, so often the skin tones of darker people were off - hence the pictures you got developed were more likely to be accurate if your subjects were white. https://www.npr.org/2014/11/13/363517842/for-decades-kodak-s-shirley-cards-set-photography-s-skin-tone-standard https://www.npr.org/2014/11/13/363517842/for-decades-kodak-s...

colllectorof 8y ago

Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical. Also, people should have the right to know when machine-learned models are used to make decisions about their lives. They should be able to ask why a particular decisions was made and get that information. This is real AI ethics.

YeGoblynQueenne 8y ago
You know what AI systems had the ability to explain their decisions? Expert systems. Expert systems were the big success of GOFAI, but they fell out of favour in the last AI winter, at the end of the '90s or so, clearing the way for probabilistic inference and statistical machine learning. Since then it seems, we took one step forward (with accuracy in classification) and one step back (with the loss of the ability to explain decisions). Who knows, maybe a new AI winter will wipe out the statistical machine learning dinosaurs of today and leave a clear field of play for the AI Mammals of tomorrow.
- colllectorof 8y ago
  Well, there are some statistical learning models that are explainable. Decision trees can be induced and can "explain themselves", just like in expert systems. Additive models are self-explanatory. Not sure I'm sold on LIME and other similar approaches, though. Seems like a lot of deep learning people are all too happy to substitute "intepretability" for actual explanations.
  - YeGoblynQueenne 8y ago
    Decision trees are not a statistical model. The learning method is statistical, if you will, but the model itself is a disjunction of conjunctions- a symbolic model (and more specifically, a propositional logic theory). Decision Trees are in fact an example of the early years of machine learning where the trend was towards algorithms and techniques that learned symbolic theories. I believe the effort was driven by the realisation that expert systems had a certain problem with knowlege acquisition [1] which drove people to try and learn production rules from data. I digress- I mean to say that decision trees are explainable because their models are not statistical. To be honest, I don't know much about additive models. _______________ [1] https://en.wikipedia.org/wiki/Knowledge_acquisition https://en.wikipedia.org/wiki/Knowledge_acquisition

YeGoblynQueenne 8y ago

>> Chris Adzima, senior information systems analyst for the Washington County Sheriff’s Office in Oregon, said the agency uses Amazon’s Rekognition to identify the most likely matches among its collection of roughly 350,000 mug shots. But because a human makes the final decision, “the bias of that computer system is not transferred over into any results or any action taken,” Adzima said. The gentleman is saying that the system selects a small subset of the 350,000 mugshots, but because a human selects one mugshot from this small subset, there is no bias. That just makes no sense. [edited to remove stronger language]

Mirioron 8y ago
It does make some sense. Yes, technically, the bias is still there, but it's lower. Getting some accuracy on image detection isn't that difficult, but making it more and more accurate is. Going from 50% accuracy to 51% is much easier than 98% to 99%. Of course, with the error rates reported on certain groups a human making the final decision is not enough.
- Retric 8y ago
  The problem is a human would at best reduce but not eliminate the bias. Let’s assume the photo was not a match for the database. Now a human doing the final step is essentially pick a random face from a biased sample. Worse, people are really bad about assuming whatever option they are considering to be far more likely than the base rate. Read up on an random obscure disease and suddenly you start thinking it’s a real risk. Sadly, police do the same thing with criminal suspects resulting in innocent people in prison.

jothezero 8y ago

Not the first article about this kind of stuff. Police departments can do whatever they want to obtain leads (as long as it is legal (Gray area)). I will be worried when they will start using technology instead of judges.

acdha 8y ago
That seems like a shortsighted position because it ignores the cost of selective enforcement. Everyone breaks laws on a daily basis - jaywalking, speeding, etc. - and you could have a severely disproportionate cost by enforcing that more for one group than another, even with every trial being completely fair. Here in DC there was an example awhile back prior to legalizing marijuana: white people apparently used at a higher rate but most of the prosecutions were of black people both due to heavier police presence and because demographics meant that white users tended to have more privacy (limited visibility from the street, more distance between houses/sidewalks to make smell harder to notice, etc.) which made it harder to get evidence clearly showing that a specific person had been the one using. The process could be fair without changing the fact that the results disproportionately impacted one group.
- jothezero 8y ago
  Eternal debate.
- Mirioron 8y ago
  I've been saying this for a long time: selective enforcement of laws is not okay. We need to design laws that are not enforced selectively, but consistently. If it means that jaywalking shouldn't be against the law then so be it.
- CryptoPunk 8y ago
  That's not "selective enforcement". Race is not being selected for. People committing crimes in public is being selected for. Or people committing crimes in high-crime areas. Some races happen to correlate with some selected traits more than others, but race is not the trait selected for. It's completely predictable that not all traits of interest for law enforcement will be distributed equally across all racial groups. To treat this fact as a sign of systemic racism is to guarantee that you will consider every society on Earth systemically rac/sex/[group] ist.

RenRav 8y ago

It seems like every other month, some facial recognition system is being attacked because of this.

salty_biscuits 8y ago

Is this something simple like class imbalance in the training sets? Would be pathetic if that were the case. So easily fixed.

malshe 8y ago

The article doesn't mention papers so I am linking them here: Original study [PDF]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... New study [PDF]: https://www.thetalkingmachines.com/sites/default/files/2019-02/aies-19_paper_223.pdf https://www.thetalkingmachines.com/sites/default/files/2019-...

ulucs 8y ago
I read the original paper. Wow, they could have tested for so many factors impacting accuracy but they only looked at mean accuracy for each cluster.
- DarkWiiPlayer 8y ago
  Haven't read the study myself yet, but that sentence alone makes me not want to waste my time on it.

treis 8y ago

Does anyone have a link to the paper?

apta 8y ago

spinach 8y ago
Going by the article, it just seems to be because the researcher is a darker-skinned woman and the software is mainly trained on lighter-skinned men. "Darker-skinned women were the most misclassified group, with error rates of up to 34.7%. By contrast, the maximum error rate for lighter-skinned males was less than 1%."
dosy 8y ago
sure, but I don't think that's her intention. she's just refitting an old already polarizing topic as a vehicle to troll big tech. polarizing topics like these are like performance-enhancers for messaging if you can weave them into your narrative.
jake-low 8y ago
Here's the paper [0]. The answer to your question is in Tables 4 and 5 (page 9). To summarize, darker skinned women are misclassified far more often than any other group, in all three of the tested facial recognition systems. Error rates for lighter skinned women are between 5x and 10x lower, depending on the system. Lighter skinned men have the lowest classification error rates by a wide margin in the IBM and Microsoft systems, and are tied with darker skinned men for lowest error rate in the Face++ system. [0]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...

dosy 8y ago

I love this, whether the conclusions are valid or not, using science to troll big tech where they live. ouch, right in the feels.

tdaltonc 8y ago

Beyond the very real intersectional problems highlighted . . . The fact that the Amazon gender classifier miss-identifies 7% of white females as male, but 0% of white males as female is very odd. That seems like a JV level bias tuning mistake. That's not systemic violence so much as it is just sloppy.

caprese 8y ago
> miss-identifies 7% of white females as male, but 0% of white males as female is very odd. Sounds like it was trained in San Francisco, I can relate to that
nitwit005 8y ago
Most of the time the headline for these articles seems like it should be "Crap product also biased". I have some suspicion they haven't focused on bringing down the error rates for minority groups because they know even their best case isn't good enough.
- tdaltonc 8y ago
  Check out the academic article.[0] It shows that Microsoft and IBM are way way better. [0] http://www.aies-conference.com/wp-content/uploads/2019/01/AIES-19_paper_223.pdf http://www.aies-conference.com/wp-content/uploads/2019/01/AI...
DarkWiiPlayer 8y ago
Just a quick thought: women tend to wear makeup, which is rather uncommon among men. That may be one part of the reason why that happens.
- tdaltonc 8y ago
  True, if the system was only trained on women wearing make-up, but tested on women not wearing makeup, that would definitely create this kind of bias. That's exactly the kind of thing that a good AI engineer should be looking out for.

Thermolabile 8y ago

What does this article have to do with racism and not the failing of the recognition system? It is supposed to be a facial recognition system, it doesn’t work on specific people at a large percentage.. It’s brought to the attention of the manufacturers and should be worked on being fixed.

swish_bob 8y ago
And when the manufacturer says "nuh uh" like they did in this case?

torgian 8y ago

"Her tests on software created by brand-name tech firms such as Amazon uncovered much higher error rates in classifying the gender of darker-skinned women than for lighter-skinned men." Besides the article's obvious anti-man, anti-white bias, I'm not surprised that facial recognition software has a hard time analyzing darker skinned people. With photography, I've always had a hard time photographing someone who had dark skin. Lots of light needs to be used, and even then it needs to be filtered correctly, etc. This goes for any subject that is dark. Unfortunately, this is going to be a tough problem. Cameras used by law enforcement and government agencies, (which the article seems to focus on) are normally pretty shitty; software can only do what it does with whatever input it gets. So if the lighting and image quality is shitty, then your results will be as equally shitty. The article doesnt go into what kind of equipment the MIT researcher was using, but I will assume that it is a high quality camera. If so, and if the software is still failing as the article alludes to, then yes, these companies need to make their software better. Even so, it's a crapshoot from the get-go, due to the hardware being used.

false_alarm 8y ago
I did not notice an anti-man or anti-white bias in the article. What stuck out to you in the article that seemed biased?
Balgair 8y ago
> With photography, I've always had a hard time photographing someone who had dark skin. Lots of light needs to be used, and even then it needs to be filtered correctly, etc. This goes for any subject that is dark. The issue does come down to a mix of (likely) inadvertent racism and physics. It is very difficult to photograph people with darker skin. This is intentional, per Mother Nature. Very briefly, our atmosphere passes a certain bandwidth of colors to the ground. The peak frequency is in the yellow-green (this is why plants are this color usually). Unfortunately, one of those colors is in the UV frequency that just so happens to be the same frequency as the small channel of eukayotic DNA. Thus, cancer risk is raised when that UV photon energetically breaks up the DNA. So, mother nature evolved to co-opt the melanin molecule, which happens to absorb those UV photons, to help protect skin cells from this damage. Thing is, though, that melanin molecule also absorbs not just that UV color, but a lot of other ones. PhysicsGirl has a good video on how sunscreen works and what a freckle looks like under different wavelengths [0] that is informative here. Now for the (inadvertent) racism. When they were first starting to make photographs, just about any photo sensitive material was 'good enough'. These, of course, were all in black and white. But, the frequencies/colors that these chemicals were photosensitive to, were much more than we can see with our eyes. It went from the low IR to the high UV, with all kinds of various notches and mixes in there. Tin-type photographs are a great example. During the US Civil war, there were a lot of pictures taken of slavery and 'colored' soldiers. It's not hard to distinguish the faces and features of these people [1][2]. One thing to note about these tin-types, look at the clouds, rather, the total lack thereof. That's an easy way to notice that the frequencies you are seeing in the photograph are not the frequencies that your eyes see. When color photography started to become more widespread, the frequencies used for each color (CYM) and the absorptive bandwidth of each part of the color films were not accidents. They were chosen such that attractive young 'white' women in very heavy make-up and under very bright 1930's studio lights would be best seen. These color choices, due to the inherent racism of the times and other factors, were perpetuated into modern CCDs and CMOS chips we use today. There are important differences in the physics of digital and film color photography, but largely those original choices have been conserved. One important difference is the IR spectrum pick-up. In most modern phones, the front facing camera does not have an IR filter on it. Next time you are near a security camera or at a toilet with those IR sensors, take a look at your phone using the front-facing camera. It should be straightforward to see the little blinking IR LEDs of these devices. So, though darker skinned people are inherently difficult to photograph due to melanin and Mother Nature, it is not difficult to find the right frequencies that will work. It's a case of the 'lock in' effect that prevents this. [0] https://www.youtube.com/watch?v=GRD-xvlhGMc https://www.youtube.com/watch?v=GRD-xvlhGMc [1] https://www.pinterest.com/pin/385480049327844109/ https://www.pinterest.com/pin/385480049327844109/ [2] https://atlantablackstar.com/2015/01/05/7-things-about-the-field-workers-during-slavery-that-you-may-not-know/ https://atlantablackstar.com/2015/01/05/7-things-about-the-f... [3] https://www.pinterest.com/pin/126452702010324184/ https://www.pinterest.com/pin/126452702010324184/

myopicgoat 8y ago

username90 8y ago

Where is the wrath? Seems like Amazon just hand waved away her criticism, they didn't really take any actions against her.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]