13 ms·

I think the thesis is wrong here. Google has a striking history of removing bias against black women. People make content for the web. People are racist, and

I think the thesis is wrong here. Google has a striking history of removing bias against black women. People make content for the web. People are racist, and make lots of racist assumptions in their writing. Society is racist. This results in Google's algorithms reflecting the corpus they scanned, and the searches that are made. But the search is just a reflection of us, and how terrible we all are. Google gets embarrassed by the results, and exercises editorial control over search -- which leads to them actively removing racist systemic biases. Sure it's reactive and not proactive, but it's movement in a positive direction. Google likely spends more money on editorial control for issues of racism than for any other thing that doesn't make money.

shawndrost 9y ago
Your second paragraph explains why Google, the search tool, has a striking history of bias against black women, by stating "search is a reflection of [..] how terrible we are". Your third paragraph explains that Google, the company, spends time and money on correcting bias. I don't think that's news to the author, or a refutation. She says tech companies produce tools in environments of racism/sexism, without proper safeguards, and the result is damaging to people of color and women. She's saying Google can and should consider this stuff in advance, and take responsibility for bias in their products (even when it's the reflection of ugliness in third parties as you said). So, I don't see your perspective as being in conflict with her. Except inasmuch as you excuse Google for reflecting our biases. Do you think they should feel OK if their next AI winds up with our biases? If so, why? It doesn't feel like you've explained that.
- brighteyes 9y ago
  I didn't see that comment as excusing anything. I think we all agree Google should do as much as possible to improve results. But at the same time, if Google gets it right 99 times out of 100, the 100th is what will get an article like this written on it. That doesn't excuse that, it explains why it happens. > Your second paragraph explains why Google, the search tool, has a striking history of bias against black women The article says "Google has a striking history of racism", which you interpret as "Google results have included things which are strikingly racist." That's one interpretation, but it seems equally valid to interpret "Google has a striking history of racism" with "Google has been very racist" (after all, if I said, "Bob has a striking history of racism" you'd be right to assume I think Bob is a racist). I think some of the controversy around this article stems from these interpretations, one of which is a far worse accusation than the other.
  - forapurpose 9y ago
    > if Google gets it right 99 times out of 100, the 100th is what will get an article like this written on it Is there any substantive basis for this claim? My impression is that Google gets away with a lot.
mirimir 9y ago
Right. The title ought to be something like "English-Speaking Cultures Have a Striking History of Bias Against Black Girls". Or "... Women of Color" if that's your preferred language. Google Search is arguably just a mirror. And ideally, that's what it should be. Otherwise, how do researchers know what's really there, and what Google makes up? But in fact, Google is far from that ideal. Maybe it was initially unbiased. But successive waves of fighting SEO, and responding to social pressure, have taken their toll. Not to mention the increasing focus on what's happening right now, forgetting the past, and showing searchers what they likely want to find.
- danso 9y ago
  > Google Search is arguably just a mirror. And ideally, that's what it should be. I disagree this is the ideal. The first implementation of PageRank makes a very big assumption: that people who write webpages and control websites can be trusted to link to worthwhile content. If webmasters and web producers skew a certain way, then PageRank/BackRub will also skew that way. Google original mission statement was "to organize the world's information and make it universally accessible and useful." Being a mirror of web content would only partially solve that mission, and that's assuming webmasters/producers are acting in good faith. Would it really be ideal for Google to be a "mirror" of link counts for "miserable failure" [0] and "Did the Holocaust Happen?" [1] [0] https://en.wikipedia.org/wiki/Google_bomb https://en.wikipedia.org/wiki/Google_bomb [1] https://searchengineland.com/google-holocaust-denial-site-gone-266353 https://searchengineland.com/google-holocaust-denial-site-go...
  - MichaelGG 9y ago
    No, Google does not try to mirror content. It tries to provide the best results for users. Results the user wants to click on and are relevant. Searching for "professor style", nobody wants it to return a minority professor style. Even she doesn't, at scale. If she and others did, they'd be Googling "professor style", not clicking, then Googling "professor style for black women" and results would improve/personalize. Ditto for "black girls": either your looking for porn or for stock photos. What other intentions do you have with such a query? (Besides making a point?) Google changing results for Holocaust "denial" is likewise wrong. Someone searching for such things probably wants to find all such sites and not just a PC version of history repeating what they've already heard. Otherwise they wouldn't need to be searching in the first place. Same for someone searching "Is the moon real" or "is the earth flat". In a great market, Google might start "cleaning" up their results to the point of annoying people, who would then switch. Unfortunately no one comes close to Google search, let alone overall lock-in. So we're stuck with whatever Google decides reflects their brand, including fixing up well-publicized queries for people that want to be offended and complain about stuff. Edit: I search for "best basketball player" and all I see are black guys -- way over-represented. Why doesn't Google bump up Larry Bird to 4th place so there's some more diversity and fight the stereotype? Even worse, Yao Ming is nowhere to be found, what's that say?
rayiner 9y ago
You’re missing the point, which is that you’re dealing with a coupled system (in the physics sense). Google reflects our racism, but exposure to that racism on the first page of results makes us more racist. Google concentrates it and reflects it back to us. (Google does this with everything else as well. What do people think of the new MacBook Pro? Searching for reviews will immediately spit back a concentrated and distilled version the first mover opinions, which will itself drive peoples’ opinions.)
- brighteyes 9y ago
  You raise an interesting point, but the comment was about the article itself and the thesis in the article.
  - grzm 9y ago
    FWIW, I read the article as addressing the issue in general. The headline and lead-in are Google-specific (as well as many of the examples), which is unfortunate in the sense that it's arguably click-baity and leads to arguments like this, that the thesis is wrong because it's not just Google. I understand why it's done, as providing a familiar, well-understood context helps people better understand what the author is getting at. Examples of how the author is approaching the topic more generally are found in headings such as "Content and Creators" and phrases like "Information monopolies such as Google". The author's book is likewise entitled "Algorithms of Oppression: How Search Engines Reinforce Racism", not "How Google Reinforces Racism". I agree that results in Bing and Duck Duck Go, for example, currently show a lot of the flaws describe in the article. But the point (as much as the headline may lead you to believe) isn't that it's only Google.
    - brighteyes 9y ago
      rayiner wasn't just bringing up Google vs other companies, though. rayiner was saying that there is a "coupled system" and Google just reflects external racism, and then Google's search results amplify racism. The article did not mention anything about those interesting points, which might be worthy of bringing up and discussing, but it seems very unfair to accuse the top comment of "not getting the point" about them, when they are rayiner's insights that were only now presented.
deleted 9y ago
[deleted]
danso 9y ago
I agree that there's too much blame-the-algorithm sentiment here, but if I had to guess, I'd think the author would argue that these racist biases should and could have been caught before going into production -- i.e. it doesn't have to be an inevitable cycle of ad hoc public shaming and repentance. These biases could be identified just as other SERP problems are identified before each algorithm update. Just like the HP "racist" webcams [0] might have been caught had the CV trainers been more aware of what is/isn't in the training data, and/or if HP had a few more testers of darker complexion. And perhaps Facebook's "Year in Review" rare tendency to be "cruel" [1] would have been mitigated with a broader group of devs and testers. [0] http://www.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://www.cnn.com/2009/TECH/12/22/hp.webcams/index.html [1] https://meyerweb.com/eric/thoughts/2014/12/24/inadvertent-algorithmic-cruelty/ https://meyerweb.com/eric/thoughts/2014/12/24/inadvertent-al...
- naasking 9y ago
  > I'd think the author would argue that these racist biases should and could have been caught before going into production I think people who believe this significantly underestimate the difficulty of solving these problems.
  - danso 9y ago
    I think that's likely. But the people tasked to solve these problems aren't always given the right priorities. Detecting/classifying the faces of dark-skinned humans is inextricably more complicated than it is with light-skinned humans. It requires significantly more training data, more computational power, and more testing time to reach the same accuracy as a light-skin-only camera and classifier. We would deride Uber's self-driving vehicles if it turns out their systems can't tell the difference between harmless (plastic bags) and dangerous (jaywalkers, wildlife) objects on the road. Likewise, we can judge a company for releasing a general consumer webcam that fails to fully function for 10-15% of the American population.
    - labster 9y ago
      This is very true. I suppose a mirror isn't quite the right analogy for Google, either. It's like film, a snapshot of what it thinks life is like. And for film, there are lots of technical problems for filming black actors, the likes of which had been solved 60 years prior for white actors[0]. It's very possible that the algorithm, captures a version of reality that doesn't show people of color in a good light. And yes, these problems should be solved, and Google should use its editorial control to do the right thing. I believe it has done so, though perhaps not proactively enough. But these are hard problems, ones that are still not solved in other industries. Web search is still a young industry. [0]: https://mic.com/articles/184244/keeping-insecure-lit-hbo-cinematographer-ava-berkofsky-on-properly-lighting-black-faces https://mic.com/articles/184244/keeping-insecure-lit-hbo-cin...
fipple 9y ago
Google can amplify the result though. Just like a population that votes 51% Republican and 49% Democrat can have a 100% Republican legislature if divided into equally representative districts, Google can amplify society’s latent racism by making a model that is more racist than the training data.
- perl4ever 9y ago
  This is a good point, but I also think that any search engine is inherently an amplifier, so it's simplistic to say they should just fix it somehow.
johnhess 9y ago
> Google likely spends more money on editorial control for issues of racism than for any other thing that doesn't make money. Is there evidence to this effect? There are a lot of things Google spends money on.
- forapurpose 9y ago
  Good point; it's loose speculation, if even that, and I'm surprised to see it get so much attention. I'd add that it does make money. Making your customer experience pleasant is an essential money-maker. If people are loudly saying racist things in your restaurant, the financially essential decision is to exercise editorial control before you lose your other customers.
RangerScience 9y ago
By having a mirror, we can see and correct our flaws. I used to be (well, still am, a bit) worried that the increasing automation of our social structures (credit scores, admission processes, etc) will just encode these biases in ways that cannot be altered. But then saw how much of NIPS was about "fairness". If these biases get encoded, they are measurable, examinable, detectable - and can then be accounted for in ways (presumably) much more effective than existing patterns. So now, I'm not worried about the technology, although as always, I worry about the application of it.
- PeterisP 9y ago
  Is the (assuming it's true, but it seems plausible) situation that the majority of users searching for "black girls" actually do want to find sexual content instead of simply smiling faces a flaw that needs correcting? Is their intent unfair in some way, are their sexual preferences unacceptable? If I don't like what I see in my mirror, will putting makeup on the mirror improve my face? Will removing naughty sites from the search results somehow change the fact that "black girls" (or "asian girls", or "czech girls", etc) is not usually used as an innocent search term?
  - RangerScience 9y ago
    So, I'd dance around the issue. Google's search algorithm is "fair", but society is not: It is not "fair" that these search terms are over sexualized. This is, of course, given some particular morality and ethos; your may not find society unfair in this way. (I do) My point is that now it is visible and you can make more informed choices. If you don't like what you see in mirror, why is makeup the "fix" of choice? Alter diets, exercise patterns; sun exposure, fitness levels, stress levels... > Will removing naughty sites from the search results Potentially. We're feedback loops as well; if these terms result in what someone is after, they will use those terms: if they don't, they won't. Eventually, I would expect people to stop trying to search with them; to then stop communicating with them (as meaning that); to then stop thinking with them (as meaning that).
    - 9y ago
mc32 9y ago
I have to agree with you. You can go to any country with a fetish of some sort and have "surprising results" for innocent searches. Go to google.co.jp and search for "Russian Women" [ロシアの女性] Is that a racist algorithm?
charlescearl 9y ago
Perhaps, but I doubt it. As one anecdote, http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... reports facial recognition algorithm performance on darker-skinned women up to 30% lower than lighter-skinned men. The preliminary results were reported to Google, IBM, and Facebook. IBM was able to tune its algorithm achieve comparable scoring across all skin hues/genders within a week (according to my notes from the talk). It could be that search on that scale is much harder than facial recognition, and its also that a quick fix would introduce other errors, I'm sure it's an NP-complete problem, yada yada. But Google and racialized search is not a new issue https://dataprivacylab.org/projects/onlineads/1071-1.pdf https://dataprivacylab.org/projects/onlineads/1071-1.pdf. Yes the society is racist, but then the U.S. enacted the 13th, 14th, 15th and 19th amendments to address extreme racism and gender discrimination. That is, citizens -- responsible engineers and data scientists among them -- can take it as their obligation to confront systemic racism. Yes, Klansmen are racist, Nazis are racist, but we don't have to accept genocide -- we don't even have to accept biased, discriminatory and derogatory internet services either.

shawndrost 9y ago

Your second paragraph explains why Google, the search tool, has a striking history of bias against black women, by stating "search is a reflection of [..] how terrible we are". Your third paragraph explains that Google, the company, spends time and money on correcting bias. I don't think that's news to the author, or a refutation. She says tech companies produce tools in environments of racism/sexism, without proper safeguards, and the result is damaging to people of color and women. She's saying Google can and should consider this stuff in advance, and take responsibility for bias in their products (even when it's the reflection of ugliness in third parties as you said). So, I don't see your perspective as being in conflict with her. Except inasmuch as you excuse Google for reflecting our biases. Do you think they should feel OK if their next AI winds up with our biases? If so, why? It doesn't feel like you've explained that.

brighteyes 9y ago
I didn't see that comment as excusing anything. I think we all agree Google should do as much as possible to improve results. But at the same time, if Google gets it right 99 times out of 100, the 100th is what will get an article like this written on it. That doesn't excuse that, it explains why it happens. > Your second paragraph explains why Google, the search tool, has a striking history of bias against black women The article says "Google has a striking history of racism", which you interpret as "Google results have included things which are strikingly racist." That's one interpretation, but it seems equally valid to interpret "Google has a striking history of racism" with "Google has been very racist" (after all, if I said, "Bob has a striking history of racism" you'd be right to assume I think Bob is a racist). I think some of the controversy around this article stems from these interpretations, one of which is a far worse accusation than the other.
- forapurpose 9y ago
  > if Google gets it right 99 times out of 100, the 100th is what will get an article like this written on it Is there any substantive basis for this claim? My impression is that Google gets away with a lot.

mirimir 9y ago

Right. The title ought to be something like "English-Speaking Cultures Have a Striking History of Bias Against Black Girls". Or "... Women of Color" if that's your preferred language. Google Search is arguably just a mirror. And ideally, that's what it should be. Otherwise, how do researchers know what's really there, and what Google makes up? But in fact, Google is far from that ideal. Maybe it was initially unbiased. But successive waves of fighting SEO, and responding to social pressure, have taken their toll. Not to mention the increasing focus on what's happening right now, forgetting the past, and showing searchers what they likely want to find.

danso 9y ago
> Google Search is arguably just a mirror. And ideally, that's what it should be. I disagree this is the ideal. The first implementation of PageRank makes a very big assumption: that people who write webpages and control websites can be trusted to link to worthwhile content. If webmasters and web producers skew a certain way, then PageRank/BackRub will also skew that way. Google original mission statement was "to organize the world's information and make it universally accessible and useful." Being a mirror of web content would only partially solve that mission, and that's assuming webmasters/producers are acting in good faith. Would it really be ideal for Google to be a "mirror" of link counts for "miserable failure" [0] and "Did the Holocaust Happen?" [1] [0] https://en.wikipedia.org/wiki/Google_bomb https://en.wikipedia.org/wiki/Google_bomb [1] https://searchengineland.com/google-holocaust-denial-site-gone-266353 https://searchengineland.com/google-holocaust-denial-site-go...
- MichaelGG 9y ago
  No, Google does not try to mirror content. It tries to provide the best results for users. Results the user wants to click on and are relevant. Searching for "professor style", nobody wants it to return a minority professor style. Even she doesn't, at scale. If she and others did, they'd be Googling "professor style", not clicking, then Googling "professor style for black women" and results would improve/personalize. Ditto for "black girls": either your looking for porn or for stock photos. What other intentions do you have with such a query? (Besides making a point?) Google changing results for Holocaust "denial" is likewise wrong. Someone searching for such things probably wants to find all such sites and not just a PC version of history repeating what they've already heard. Otherwise they wouldn't need to be searching in the first place. Same for someone searching "Is the moon real" or "is the earth flat". In a great market, Google might start "cleaning" up their results to the point of annoying people, who would then switch. Unfortunately no one comes close to Google search, let alone overall lock-in. So we're stuck with whatever Google decides reflects their brand, including fixing up well-publicized queries for people that want to be offended and complain about stuff. Edit: I search for "best basketball player" and all I see are black guys -- way over-represented. Why doesn't Google bump up Larry Bird to 4th place so there's some more diversity and fight the stereotype? Even worse, Yao Ming is nowhere to be found, what's that say?

rayiner 9y ago

You’re missing the point, which is that you’re dealing with a coupled system (in the physics sense). Google reflects our racism, but exposure to that racism on the first page of results makes us more racist. Google concentrates it and reflects it back to us. (Google does this with everything else as well. What do people think of the new MacBook Pro? Searching for reviews will immediately spit back a concentrated and distilled version the first mover opinions, which will itself drive peoples’ opinions.)

brighteyes 9y ago
You raise an interesting point, but the comment was about the article itself and the thesis in the article.
- grzm 9y ago
  FWIW, I read the article as addressing the issue in general. The headline and lead-in are Google-specific (as well as many of the examples), which is unfortunate in the sense that it's arguably click-baity and leads to arguments like this, that the thesis is wrong because it's not just Google. I understand why it's done, as providing a familiar, well-understood context helps people better understand what the author is getting at. Examples of how the author is approaching the topic more generally are found in headings such as "Content and Creators" and phrases like "Information monopolies such as Google". The author's book is likewise entitled "Algorithms of Oppression: How Search Engines Reinforce Racism", not "How Google Reinforces Racism". I agree that results in Bing and Duck Duck Go, for example, currently show a lot of the flaws describe in the article. But the point (as much as the headline may lead you to believe) isn't that it's only Google.
  - brighteyes 9y ago
    rayiner wasn't just bringing up Google vs other companies, though. rayiner was saying that there is a "coupled system" and Google just reflects external racism, and then Google's search results amplify racism. The article did not mention anything about those interesting points, which might be worthy of bringing up and discussing, but it seems very unfair to accuse the top comment of "not getting the point" about them, when they are rayiner's insights that were only now presented.

deleted 9y ago

[deleted]

danso 9y ago

I agree that there's too much blame-the-algorithm sentiment here, but if I had to guess, I'd think the author would argue that these racist biases should and could have been caught before going into production -- i.e. it doesn't have to be an inevitable cycle of ad hoc public shaming and repentance. These biases could be identified just as other SERP problems are identified before each algorithm update. Just like the HP "racist" webcams [0] might have been caught had the CV trainers been more aware of what is/isn't in the training data, and/or if HP had a few more testers of darker complexion. And perhaps Facebook's "Year in Review" rare tendency to be "cruel" [1] would have been mitigated with a broader group of devs and testers. [0] http://www.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://www.cnn.com/2009/TECH/12/22/hp.webcams/index.html [1] https://meyerweb.com/eric/thoughts/2014/12/24/inadvertent-algorithmic-cruelty/ https://meyerweb.com/eric/thoughts/2014/12/24/inadvertent-al...

naasking 9y ago
> I'd think the author would argue that these racist biases should and could have been caught before going into production I think people who believe this significantly underestimate the difficulty of solving these problems.
- danso 9y ago
  I think that's likely. But the people tasked to solve these problems aren't always given the right priorities. Detecting/classifying the faces of dark-skinned humans is inextricably more complicated than it is with light-skinned humans. It requires significantly more training data, more computational power, and more testing time to reach the same accuracy as a light-skin-only camera and classifier. We would deride Uber's self-driving vehicles if it turns out their systems can't tell the difference between harmless (plastic bags) and dangerous (jaywalkers, wildlife) objects on the road. Likewise, we can judge a company for releasing a general consumer webcam that fails to fully function for 10-15% of the American population.
  - labster 9y ago
    This is very true. I suppose a mirror isn't quite the right analogy for Google, either. It's like film, a snapshot of what it thinks life is like. And for film, there are lots of technical problems for filming black actors, the likes of which had been solved 60 years prior for white actors[0]. It's very possible that the algorithm, captures a version of reality that doesn't show people of color in a good light. And yes, these problems should be solved, and Google should use its editorial control to do the right thing. I believe it has done so, though perhaps not proactively enough. But these are hard problems, ones that are still not solved in other industries. Web search is still a young industry. [0]: https://mic.com/articles/184244/keeping-insecure-lit-hbo-cinematographer-ava-berkofsky-on-properly-lighting-black-faces https://mic.com/articles/184244/keeping-insecure-lit-hbo-cin...

fipple 9y ago

Google can amplify the result though. Just like a population that votes 51% Republican and 49% Democrat can have a 100% Republican legislature if divided into equally representative districts, Google can amplify society’s latent racism by making a model that is more racist than the training data.

perl4ever 9y ago
This is a good point, but I also think that any search engine is inherently an amplifier, so it's simplistic to say they should just fix it somehow.

johnhess 9y ago

> Google likely spends more money on editorial control for issues of racism than for any other thing that doesn't make money. Is there evidence to this effect? There are a lot of things Google spends money on.

forapurpose 9y ago
Good point; it's loose speculation, if even that, and I'm surprised to see it get so much attention. I'd add that it does make money. Making your customer experience pleasant is an essential money-maker. If people are loudly saying racist things in your restaurant, the financially essential decision is to exercise editorial control before you lose your other customers.

RangerScience 9y ago

By having a mirror, we can see and correct our flaws. I used to be (well, still am, a bit) worried that the increasing automation of our social structures (credit scores, admission processes, etc) will just encode these biases in ways that cannot be altered. But then saw how much of NIPS was about "fairness". If these biases get encoded, they are measurable, examinable, detectable - and can then be accounted for in ways (presumably) much more effective than existing patterns. So now, I'm not worried about the technology, although as always, I worry about the application of it.

PeterisP 9y ago
Is the (assuming it's true, but it seems plausible) situation that the majority of users searching for "black girls" actually do want to find sexual content instead of simply smiling faces a flaw that needs correcting? Is their intent unfair in some way, are their sexual preferences unacceptable? If I don't like what I see in my mirror, will putting makeup on the mirror improve my face? Will removing naughty sites from the search results somehow change the fact that "black girls" (or "asian girls", or "czech girls", etc) is not usually used as an innocent search term?
- RangerScience 9y ago
  So, I'd dance around the issue. Google's search algorithm is "fair", but society is not: It is not "fair" that these search terms are over sexualized. This is, of course, given some particular morality and ethos; your may not find society unfair in this way. (I do) My point is that now it is visible and you can make more informed choices. If you don't like what you see in mirror, why is makeup the "fix" of choice? Alter diets, exercise patterns; sun exposure, fitness levels, stress levels... > Will removing naughty sites from the search results Potentially. We're feedback loops as well; if these terms result in what someone is after, they will use those terms: if they don't, they won't. Eventually, I would expect people to stop trying to search with them; to then stop communicating with them (as meaning that); to then stop thinking with them (as meaning that).
  - 9y ago

mc32 9y ago

I have to agree with you. You can go to any country with a fetish of some sort and have "surprising results" for innocent searches. Go to google.co.jp and search for "Russian Women" [ロシアの女性] Is that a racist algorithm?

charlescearl 9y ago

Perhaps, but I doubt it. As one anecdote, http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... reports facial recognition algorithm performance on darker-skinned women up to 30% lower than lighter-skinned men. The preliminary results were reported to Google, IBM, and Facebook. IBM was able to tune its algorithm achieve comparable scoring across all skin hues/genders within a week (according to my notes from the talk). It could be that search on that scale is much harder than facial recognition, and its also that a quick fix would introduce other errors, I'm sure it's an NP-complete problem, yada yada. But Google and racialized search is not a new issue https://dataprivacylab.org/projects/onlineads/1071-1.pdf https://dataprivacylab.org/projects/onlineads/1071-1.pdf. Yes the society is racist, but then the U.S. enacted the 13th, 14th, 15th and 19th amendments to address extreme racism and gender discrimination. That is, citizens -- responsible engineers and data scientists among them -- can take it as their obligation to confront systemic racism. Yes, Klansmen are racist, Nazis are racist, but we don't have to accept genocide -- we don't even have to accept biased, discriminatory and derogatory internet services either.