26 ms·

600k Images Removed from ImageNet After Art Project Exposes Racist Bias

229 points by aaronbrethorst 7y ago

sp332 7y ago
The article mentions https://imagenet-roulette.paglen.com/ https://imagenet-roulette.paglen.com/ but doesn't link to it. Just put in a photo and see what category of person the algorithm sorts them in to.
alfromspace 7y ago
For the artist, ImageNet’s problems are inherent to any kind of classification system. If AI learns from humans, the rationale goes, then it will inherent all the same biases that humans have. Training Humans simply exposes how technology’s air of objectivity is more façade than reality. An AI being accused of bias tends to really mean it works. Removing bias from AI, I've noticed, requires hardcoded 'fixes' rather than refactoring the algorithms. And in my view, becomes yet another human-curated classification system, and no longer AI.
- munchbunny 7y ago
  Your point takes a narrow view on correctness and also shows why you need to take the systemic view on the issue. The problem is not that the AI is "inaccurate". The problem is the second order effect: when you build systems on top of this AI's predictions, you cement the input social biases into future systems in a way that is very hard to remediate. The real problem is how to avoid accidentally creating systems that amplify existing social biases. I'm very specifically using the term "social bias" to distinguish from "bias" as a term in ML, because they are very different problems.
- brobinson 7y ago
  This comment reminded me of a really interesting article about bias in AI: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
vorpalhex 7y ago
AIs are like every other computer program - perfectly naive. If you train them on bad data, you get bad results. Garbage in, garbage out.
- core-questions 7y ago
  What happens when we train them on an incredibly wide set of randomly-sampled data, which should show no built-in bias, and uncomfortable conclusions emerge? Do we censor the AI, castigate someone else in the chain for the problem, or just deal with reality as it presents itself?
  - megablast 7y ago
    In this example, where would you get the tags for the photos from, if not people who are biased?
    - core-questions 7y ago
      Why assume that the people are "biased"? If the people are labelling things the way that people generally do, the software will return labels that are useful. Any kind of filtering needed to censor this for people's personal comfort should be implemented on top.
- megablast 7y ago
  Of course, but the issue is also data availability. It is not easy to get large sets of data to be used in training AI. Imagenet was a great attempt to provide some useful data.
  - vorpalhex 7y ago
    My comment was not intended to be a ding on Imagenet. Data comes from somewhere, and no source is without bias because no human is without bias.
- falsedan 7y ago
minimaxir 7y ago
Has this been confirmed? Not doubting the story, but the article doesn't provide a source. Edit: Might be this, which was a week ago / before the Roulette blew up: http://image-net.org/update-sep-17-2019 http://image-net.org/update-sep-17-2019 > We are in the process of preparing a new version of ImageNet by removing all the synsets identified as “unsafe” and “sensitive” along with their associated images. This will result in the removal of 600,040 images, leaving 577,244 images in the remaining “safe” person synsets.
- raxxorrax 7y ago
  If they delete half the data, the probability of the remaining data sets to be rubbish is quite high. You also wouldn't want someone to pick sets in the first place.
randyrand 7y ago
How were the mechanical turk workers asked to annotate?
- brlewis 7y ago
  That's the key question here, isn't it? If faces are like other images in the database, then according to this article turkers were presented a word and a group of images, and asked to click all the images that showed said word. https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/ https://qz.com/1034972/the-data-that-changed-the-direction-o...
  - Out_of_Characte 7y ago
    ImageNet are likely incredibly naive with how they approached and fixed all of this. from the article "Jamal Jordan, a journalist at the New York Times, explained on Twitter that each of his uploaded photographs returned tags like “Black, Black African, Negroid, or Negro.”" Which indicates that the vocabulary they used has 4 words to describe a black person's skin color. This severely complicates things and leaves all kinds of unintended biases on the table before even getting to the human element. Though I doubt they used a dictionary at all because 'Black African' isn't really a word. And not found in a few dictionaries I tried online. - if the article is accurate. They should have curated the vocabulary first, value judgements like 'attractive' should be removed because it means something different for everyone judging the images. Synonyms should be collapsed into one to prevent weighted biases, etc.
    - esyir 7y ago
      I'm actually quite surprised that they didn't collapse the synonyms though. Though value judgements should not be removed. They should have been separated and placed together with some tagger metadata as they might actually be informative then.
- tastroder 7y ago
  Since the amount of speculation in this overall thread is pretty high, I suggest simply having a look at section 3 of [0]. It describes the construction of the MTurk task and how they quantified the confidence of answers they got. [0] http://www.image-net.org/papers/imagenet_cvpr09.pdf http://www.image-net.org/papers/imagenet_cvpr09.pdf
seph-reed 7y ago
Superficial judgement is kind of where intelligence starts. It's really only people where you can't tell what it does/is from the outside. Cars, trees, animals, mountains... everything else, if it looks a way it acts that way. Early AI will probably have just as much trouble with this as people have historically. I really wish people would start viewing Racism as a willingness to let that primitive part of the mind be in command, rather than a binary attribute you either have or don't. Like, nobody is 100% not racist. There will always be slip ups, over-simplifications, snap judgements, subconscious or not.
- ALittleLight 7y ago
  A car could look fast, but be slow, or have latent problems, software on the car's computer could be fooling emissions tests... Looking at a tree tells you nothing about photosynthesis, tree pheromones, deep relationships with symbiotic insects or bacteria... You can't understand very much at all just based on how things look. That holds for humans and inhumans alike.
  - deogeo 7y ago
    But you can make a pretty good guess: http://www.spsp.org/news-center/blog/stereotype-accuracy-response http://www.spsp.org/news-center/blog/stereotype-accuracy-res...
  - seph-reed 7y ago
    Ultimately we know nothing, so one can always nit-pick any statement into an oversimplified oblivion. There are also cakes that look like cars, model sets which have miniature mountains, on and on. But in typical nature, it works well enough to breed and see what comes next.
    - desert_boi 7y ago
      Tactical Nihilism sounds a lot like solipsism. "We can't ever not make generalizations, so we shouldn't even try to make statements."
  - 7y ago
t8hszj5w 7y ago
Is this data racist? http://i.4pcdn.org/pol/1506370711307.png http://i.4pcdn.org/pol/1506370711307.png & http://i.4pcdn.org/pol/1488380621213.png http://i.4pcdn.org/pol/1488380621213.png
bllguo 7y ago
wouldn't it make more sense to add more images, not remove existing data?
- sp332 7y ago
  I think it's temporary. Sensible defaults to protect users who don't understand potential weaknesses of the model. Also, if someone does train an AI on a bunch of photos and ends up with a biased result, it won't be ImageNet's maintainers' liability.
- mkl 7y ago
  It seems to be largely the tags that are biased. Long term, I'm sure people want big unbiased databases of images of people, but it will take a lot of time and effort to build them and ensure they aren't biased.
- travisoneill1 7y ago
  If I understand correctly, they are removing incorrect tags, not the data. It's just news because some of the incorrect tags are racist bias. If they removed a bunch of cats tagged as dogs, you wouldn't hear about it.
- brlewis 7y ago
  IIUC the database is intended to identify the actual contents of images, not collect data about stereotypes. Even keeping the categorized face images in a separate database wouldn't be that useful, since turkers are not a random sample.
Nasrudith 7y ago
Really the bias isn't the true problem here so much as the lack of epistemology in how it is being used. By definition associations work by blind correlations - expecting anything other than stereotyping is a foolish misuse of the tool. It will be wrong for outliers because it tries to get it right for most cases regardless of cause - like skin cancer detectors who saw rulers as a sign of cancer because most of the reference images of cancer had a ruler in them.
- didericis 7y ago
  Exactly. The large amount of faith currently placed in probabilistic models that do not have common sense ways of eliminating factors which are extremely unlikely to be causal (like the presence if a ruler causing cancer) disturbs me. There is something that humans do that we have not quite figured out how to teach computers yet, at least as far as I can tell, which is to get them to evaluate whether their model is not just compatible with their observations, but other models and prior knowledge about the objects being observed. I think we’ll get there at some point, and I’m not exposed to the most cutting edge AI research, but it seems like AI us currently very overhyped and deeply flawed for many of the applications people would like to use it for.
  - bonoboTP 7y ago
    Because these algorithms don't know that they are classifying cancer. The label it sees is just 1 or 0. For all it knows, based on its inputs, you may want to classify ruler/non-ruler images. To achieve what you want, semantic structure must be used as labels instead of just categorical labels. Assuming we have a sane AI that now knows its looking for cancer, it knows what that means (from digesting medical textbooks, papers and generic text corpora) and it can detect rulers and knows the two are not casually linked from ruler to cancer, we could make the model output "dataset diagnostics", like a "Warning! The cancer label in this dataset is implausibly correlated with the visual presence of a ruler". Or "Warning: 99% of your hotdog images show a human hand. Evaluation on this dataset will ignore errors on hotdog images without hands!" Context does matter though. If there's an orange fluff on a tree trunk, the AI is right to look at the environment and infer it's a squirrel.
deleted 7y ago
[deleted]
Out_of_Characte 7y ago
The AI isn't biased, the curaters were. The people who curated the first training set used subjective words like 'attractive' to tag the images which means the AI tagged all images it deemed 'attractive' to the people who made the training set. as this is a very biased and homogenous group it means the AI turned out biased. Maybe if they randomly sampled millions of people from all countries in order to create the training data then they could effectively train an AI to guess what YOU might find attractive. However even then I somehow doubt it. Beauty is in the eye of the beholder. We dont consiously know the rules of what we are attracted to, nor does an AI have secret information if you just supplied it with enough words and images. If they'd stuck with simple classifications like 'black' 'white' 'man' 'woman' then they would have less subjective judgement values about the original training set.
- maerF0x0 7y ago
  Exactly! They got subjective results because they went beyond facts. For example, it could have been interesting if they tagged each person with their actual religion or propensity to "grin"
  - dev_dull 7y ago
    Going beyond facts is an important component of “being human”, so in that regard it makes the AI seem more intelligent. The problem is the AI is 100% honest with what it thinks, unlike a human.
    - maerF0x0 7y ago
      I agree that going beyond the facts is a good thing when humans are doing critical thinking and when being careful and transparent about their doing so. However this was creating a dataset for classification. Something that specifically should not go beyond the facts. (The basis of the model is the strength of the facts it's built upon)
      - tylerhou 7y ago
humble_engineer 7y ago
If you google "white race" on Google versus "black race" or "asian race" you don't get what you expect. You get hatred for whites, and then historical black and asian cultural for the latter.
bernierocks 7y ago
"For example, the program defined one white woman as a “stunner, looker, mantrap,” and “dish,” describing her as “a very attractive or seductive looking woman.” Many people of color have noted an obvious racist bias to their results. Jamal Jordan, a journalist at the New York Times, explained on Twitter that each of his uploaded photographs returned tags like “Black, Black African, Negroid, or Negro." They claim this is AI, but earlier in the article, it states that mechanical turk was used. Mechanical Turk is basically just people getting paid pennies or fractions of a penny to tag these photos. Many people on Mechanical Turk are from countries where English is not their first language and they don't have the same idea of racism as we do here in the US, which would explain the racist tags mentioned in the article. This doesn't really show us anything about AI and racial bias and more that other countries still aren't up to our levels of what we consider decent.
- notyourday 7y ago
  As someone who spent years in those villages I can tell you it is exactly how they view the world. The only posturing comparable to the one coming from the so called enlightened anti-racist comments that I ever heard/read about was Marie Antoinette saying "Let them eat cake". She ended up without her head.
- LocalH 7y ago
  The AI connection is that these are the type of datasets that feed AI systems. It shows that you have to be careful when curating your dataset, so that you don't further bias. I just hope the masses don't glom onto this, start shrieking that "AI is racist", and attempt to take AI completely out of the picture.
rolltiide 7y ago
reminds me in the early 2000s when people would say "but he's gay" as if that had anything to do with their profession or aspirations in life or the topic at hand, as if that specific metadata defined them more than the other pieces of metadata. this AI seems to be doing that to things it has determined are black people. A bunch of synonyms for black people, some English, some Spanish, some phenotypes, some of those descriptions simultaneously fallen out of favor in some parts of the world but not others, while completely eliminating other metadata. I'm not sure Imagenet's response of removing "sensitive" adjectives is capable of fixing this. Mechanical Turk: english, spanish, academics, using terms that aren't universally agreed upon? That doesn't really address what is happening
maxander 7y ago
Bias in AI (whatever the source or nature of the problem) is a real issue that needs addressing, but taking the relevant training data off of ImageNet seems like a perfect example of papering over a problem to avoid really confronting it. We will need to find ways to make AI programs than can see beyond the biases (about humans or otherwise) that will always exist to some degree in real-world data. If the ImageNet contains bias that leads to embarrassing results- that’s fine! That give us a readily available toy instance of the problem to study. Taking that away could actively harm anti-bias research.
- avinium 7y ago
  I strongly prefer they keep the bias. It shows plainly that these models have zero intelligence, they're just pattern matching over specific datasets. I don't want the appearance of fairness (introduced by human dataset curation) to be mistaken for "intelligence". Keeping the bias would hopefully cause people to think more critically about why such bias exists in the model in the first place.
  - jobigoud 7y ago
    Disagree that it proves zero intelligence. Intelligent humans also have bias.
    - avinium 7y ago
      I'm not saying that intelligence and bias are orthogonal. That discussion requires a much deeper consideration of human cognition and psychology :) I'm just saying that model bias is a very easy thing to explain (usually, data imbalance). You can also fiddle with the label ratios to change the bias - which is also a good way of showing that the models aren't really intelligent.
- WilliamEdward 7y ago
  They could store it somewhere non-public for research purposes, but taking it off the website was the right call.
- 7y ago
orf 7y ago
Their statement: http://image-net.org/update-sep-17-2019 http://image-net.org/update-sep-17-2019 > Each synset is classified as either “unsafe” (offensive regardless of context), “sensitive” (offensive depending on context), or “safe”. “Unsafe” synsets are inherently offensive, such as those with profanity, those that correspond to racial or gender slurs, or those that correspond to negative characterizations of people (for example, racist). “Sensitive” synsets are not inherently offensive, but they may cause offense when applied inappropriately, such as the classification of people based on sexual orientation and religion. > So far out of 2,832 synsets within the person subtree we’ve identified 438 “unsafe” synsets and 1,155 “sensitive” synsets. The remaining 1,239 synsets are temporarily deemed “safe.” They've also completely disabled Imagenet downloads while they remedy this.
- esyir 7y ago
  Great, I can see some of this being useless (some parts of the unsafe dataset), but if they cull the "sensitive" portion, this may induce performance regressions. I need to find an ImageNet archive now.
  - buildbot 7y ago
    It doesn't really touch the 1000 class "Imagenet" that's commonly used in computer vision.
    - esyir 7y ago
      Ah, then it's fine... probably.
    - yorwba 7y ago
      Some of the classes are subcategories of ILSVRC classes in the WordNet hierarchy. So by removing images of persons in categories that are considered inappropriate, the resulting classifier will end up less likely to recognize those as images of people at all. I'm not sure whether that's a better outcome.
- kristianp 7y ago
kayaeb 7y ago
In NLP (specifically vectorizing words, ala word2vec) there's a famous test of whether or not your training has worked properly whereby you calculate the vector of "king" and subtract the vector of "man" and add the vector of "woman," if your machine is properly tuned, you should end up with a vector close to "queen" or "princess." I wonder if similar things can be done to address specific (i.e. racial or gender) biases in computer vision.
- mhuffman 7y ago
  There is a similar word-embedding test that definitely rustles people's jimmies: Doctor - Man + Woman = ? What normally comes out is Nurse. What "they" think should come out is Doctor! By "they" I mean people that get upset by this.
  - tastroder 7y ago
    Yeah, besides the fact that this compositionality is relatively unique to word2vec, research on the biases pre-trained models express is pretty available. Linked a few below for those interested. Most of the issues are down to the same phenomenon discussed here in the context of ImageNet, the input texts were biased and the algorithm learned said bias. [0] https://arxiv.org/abs/1607.06520 https://arxiv.org/abs/1607.06520 "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings " [1] http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf "Understanding the Origins of Bias in Word Embeddings" [2] http://matthewkenney.site/biases.html http://matthewkenney.site/biases.html "Google word2vec biases"
danso 7y ago
edit: I conflated ImageNet with the art exhibitors; it is the former who are culling the images as a result of public reaction, not the latter. This is a really bizarre project. I had seen some really offensive race-based labels, but I thought revealing the ugliness of the system was part of the point of this project? But besides that, the results just seemed completely scattershot; I half expect the artists/exhibitors to reveal that x% of the results were randomized. Last week, I tried it myself after seeing another Asian user display results that were entirely Asian slurs (e.g. gook, slant-eyed). I uploaded my own very Asian-looking photo and got "prophetess", along with very vague labels, such as "person" and "individual". Maybe the exhibitors cleaned the data/results by the time I tried it, but I used it just a few hours after seeing the other Asian user's results, so I'm doubtful that her tweet/complaints were enough on their own to change up the dataset that same day.
- bigiain 7y ago
  > I thought revealing the ugliness of the system was part of the point of this project? It 100% is. From the link to the artist's website: "Things get strange: A photograph of a woman smiling in a bikini is labeled a “slattern, slut, slovenly woman, trollop.” A young man drinking beer is categorized as an “alcoholic, alky, dipsomaniac, boozer, lush, soaker, souse.” A child wearing sunglasses is classified as a “failure, loser, non-starter, unsuccessful person.” You’re looking at the “person” category in a dataset called ImageNet, one of the most widely used training sets for machine learning."
  - danso 7y ago
    I was just about to edit and correct myself; it is ImageNet who has made the decision to delete the offensive images, after the reaction to the exhibitors' work. It's too bad the exhibitors didn't make their own mirror/cache of the dataset. Judging from some tweets I saw, I think this project really helped people to understand how much of current artificial intelligence is human-driven. It's not a sentient computer deeming you to be a "slant-eye", it's a bunch of random Internet users. (not that this makes you feel better about the world, but at least the hate's coming from an expected source)
notyourday 7y ago
I love articles and papers like this. They keep illustrating that the people writing them do not get it: Guess who classified the blonde white woman "a dish" on a Mechanical Turk? Do you think it was that Billy Bob from the swamp of Louisianan who makes $12/hour? Because $12/hour is impossible to make classifying images. Or do you think it was a dirt poor Indonesian or Filipino or Chinese or Indian for whom that's the best job he or she could possibly get and that's exactly how they view the world and there are over three billion of them?
- Gigablah 7y ago
  Your characterization of these nationals aside, what makes you think any of them even know what “dish” means in this context? Sincerely, a Malaysian Chinese who’s seeing this slang for the first time.
  - org3432 7y ago
    Dish is almost an archaic term a this point. So either someone's pretty old grandparents are working for MT, or people are translating a term from another language using an old dictionary.
  - notyourday 7y ago
    It is colonial (British) English. I heard it again recently and it took me a few moments before I recalled it from my childhood. Apparently now "snack" is used the same way. I would guess the classifiers went to school in India or Pakistan.
bigiain 7y ago
I wonder if the article author knew what he was hinting at when he chose the example of the art project calling someone a "dish"??? https://www.youtube.com/watch?v=HEX7xsYF1nA https://www.youtube.com/watch?v=HEX7xsYF1nA (And now I'm gonna be listening to 90s electro-disco all day...)
erikig 7y ago
While interesting, this is not surprising. One of the most commonly utilized datasets for learning ML is the Boston Housing dataset - https://www.kaggle.com/c/boston-housing https://www.kaggle.com/c/boston-housing In it there's a problematic feature tagged simply as "black" and it is defined as the proportion of blacks by town. Any pricing model that is built off of this dataset is inherently racially biased because the data has been collected and the feature tagged - but what's the alternative? Not to collect the information? Or collect it but completely ignore this feature?
- dannyw 7y ago
  Sensitive features should not be collected or used in applications where they introduce bias. For example, a medical screening NN may find race to be a valuable feature for the prediction of illness; but a health insurance assessor should not.
  - chriskanan 7y ago
    Due to spurious correlations, it could still be helpful for the insurance assessor so that they can use bias mitigation techniques. Otherwise, it might learn something about zip code or something else that leads to a similar outcome as having race as an input variable. Just removing a sensitive variable does not suffice for preventing unwanted bias.
  - qoninger 7y ago
    Then you are confusing bias with bias. One is data bias and the other one is racial bias. If you remove the data bias, you are by definition introducing a racial bias by imposing your will on reality.
zaroth 7y ago
I think the ironic thing here is the algorithm is intentionally designed to make a superficial judgement on the image it’s presented with. If it was trained to identify “human” or otherwise categorize the things in the picture, that’s exactly what it would output. Next you can train it to attempt to guess basic traits like gender or ethnicity, of course this can only be done based on the RGB values of a 2D array of pixels. Interestingly the NN will not be merely be using skin color but building probabilistic weightings based on any statistically significant features. For added controversy, it’s probably even possible to invert parts of the network to suss out how it’s weighting various facial features toward different labeling. Lastly the images could be labeled for things like profession of the subject. A good intentioned effort to perhaps detect things like a lab coat could mean a doctor or scientist was followed by oblivious Turkers with predictably poor results. The problem of course is not with the images but the particular labeling hierarchy, and allowing opinionated labels as well as labels which may be true for the particular subject but which bear no distinguishing features for which to actually codify (“portrait of a macroeconomist”), in other words, garbage in garbage out. ImageNet calls this the “imageability” of the label. Calling this racism of course is entirely inapt, because there is no judgement being made whatsoever. Even if the sampling method was well designed and the labeling factually accurate, the system would still produce output which could be considered offensive. Again, because the entire point of the algorithm is to generate statistical assumptions based on a single image. My conclusion is that some superficial judgments are algorithmically useful and hopefully less controversial. “White male human, ~55 yrs old, 180lbs”. Even things like analyzing clothing and guessing where the picture was taken. Iff the clothing is a uniform, identifying the profession (police, fire, paramedic) But you have to know where this goes off the rails. Bad enough to label indistinct portraits with the subject’s profession, let’s not do inane things like labeling them with how subjectively attractive the labeler thinks the person is, their economic status, maybe even a 1-10 scale of how threatening they look or if they look like a criminal or not! </facepalm>
crazygringo 7y ago
The problem seems less about racist bias specifically and more about unbelievably dumb tags generally in the first place. "Stunner"? "Looker"? "Mantrap"? Or even trying to tag people's images with categories like Buddhist, grinner, or microeconomist? What were they thinking?! Clearly these tags were never curated in any remotely responsible way -- for quality, for sensitivity, or just usefulness at all -- and I'm shocked they were ever intended for academic or research use. No wonder image recognition gets a bad rap, with input data like this.
- Iv 7y ago
  Agreed. These are not descriptive labels you can infer from an image. They may be, however, labels generated by users from a context.
- ailideex 7y ago
  But removing the images from ImageNet, an inanimate set of images that cannot make value judgements, will surely fix this problem... Our culture is broken and nobody is willing to stand up to insanity or even admit to it. People are too happy to do completely insane and senseless things if it saves them from the social lynchings.
- tastroder 7y ago
  > What were they thinking?! [...] In addition to their statement, linked in a thread below [0], what they were thinking is pretty out in the open in the abstract of the original paper: "We introduce here a new database called “ImageNet”, a largescale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. " [1] That paper is now 10 years old, WordNet itself had it's first release somewhere in the 1980s. A lot happens in research a decade down the road and lots of research towards biases of datasets are relatively recent in that sense. It can hardly be blamed on the researchers that people use their datasets in production merely because "they're large and there in the open". As their statement indicates they're actively working on the issues that arose during the last few years. Most research datasets are fit for a very specific purpose, to me the larger problem here is two-fold: "AI" fearmongering on one side and startups/corporations on the other side lacking the necessary skills to select and filter their training data. Hopefully that'll be a matter of the past once the hyped data science field reaches maturity. edit: Wordnet seems to include said terms to this day, [2] for example still lists "mantrap" associated to "beaty" in the synset for "a very attractive or seductive looking woman". Although I'm unsure how they should handle these cases, both are words and that's one definition used in practice, even if we find it reprehensible. They seem to have removed the n-word but that's about the only removed instance I can find in a cursory search of sensitive or insulting English words. Maybe they should clarify sensitive annotations like they do with vulgarity. [0] https://news.ycombinator.com/item?id=21054770 https://news.ycombinator.com/item?id=21054770 [1] http://www.image-net.org/papers/imagenet_cvpr09.pdf http://www.image-net.org/papers/imagenet_cvpr09.pdf [2] http://wordnetweb.princeton.edu/perl/webwn?s=beauty http://wordnetweb.princeton.edu/perl/webwn?s=beauty
clamprecht 7y ago
One of my favorites is searching google images for "immigrant" versus "expat". Guess which one is whiter!
ineedasername 7y ago
The longer piece this one is based on is much more comprehensive in both discussion and concrete examples: https://www.excavating.ai/ https://www.excavating.ai/
s9w 7y ago
Of course it was removed. 4chan was having way too much fun with image classification recently :D
- raxxorrax 7y ago
  They will be hated for it, but they do provide a really important service here. And if it is just pulling down the pant of people trying to classify the world.
boyadjian 7y ago
The machine always tells the truth, but it is unbearable for some.
- dannyw 7y ago
  Perhaps when we have real artificial intelligence, whatever that is. Right now we have pattern matchers, and it is not so much the 'truth' as it is stereotypes.
Illniyar 7y ago
Why take down the images? It seems really that one should take down entire classifications, i.e. remove the tags.
- ailideex 7y ago
  Because the woke scolds have spoken and nobody wants to stand up to them. So better to do patently insane things than risk becoming collateral damage in a social lynching.
- yorwba 7y ago
  ImageNet is a mapping from tags to images, so removing the tags means removing the images. An image of a "bad person" wasn't tagged by someone looking at said image and deciding that "bad person" was the best possible description. It was generated by searching Google Images for "bad person" and removing obviously incorrect results (e.g. when there's nobody in the image). Researchers have been using it to learn the inverse mapping from images to tags with some success, but in its construction the dataset is not naturally suited for that task.
  - Illniyar 7y ago
    As I understood it, and confirmed by Wikipedia (as much that's worth) images were hand annotated.
    - yorwba 7y ago
      Only the second step "removing obviously incorrect results" involved human annotators.
dusted 7y ago
That's newthink at a level I find hard to comprehend. Censoring datasets instead of improving them. "We have too many pictures of white people, remove them!" "We don't have enough pictures of non-white people, add them!" I'd have gone for the latter, and have let the set be biased until fixed.
FlowNote 7y ago
The more you impose completeness of morality into a data set, the more inconsistent your results will be. Gödel arbitrage will be very profitable in the future.
helb 7y ago
The "app", mentioned in the article, but not linked, is hosted here (until Friday), if you want to try it out: https://imagenet-roulette.paglen.com/ https://imagenet-roulette.paglen.com/ Apparently i'm either an insurance agent (in a grey t-shirt) or a surgeon (in a red hoodie).
nudpiedo 7y ago
I think the whole training and such AI was just bad designed. Aren't all first impression judgements prejudices as much as all racist bias are? If I say a person is attractive, inspires me just based on their looking it is a positive judgement, a bias based on the looks and a few traits I subconsciously perceive and perhaps I cannot even explain, and the same with the negative ones, and I guess for some people this association happens also to race and looks. Basically an AI was trained to map the prejudices of their inputs, and that means to create a snapshot of the subconscious average of whoever behind the mechanical turk is, everyone has prejudices of all sorts which usually are not manifested externally but in a process of classification such as the one of the mechnical turk might arise much more.
theqult 7y ago
Any way to download those images before they get deleted ?
Grustaf 7y ago
I don’t quite get it, why is “Buddhist” more objectionable than “grinner”? Surely there must be more egregious examples than that if they decided to pull 600,000 images? And if they find the tag “Negro” offensive can’t they just replace it with “black”, “of African origin” or whatever the currently allowed term is? Tagging a black person as black does not seem very objectionable in a country that provides racial breakdowns for official statistics.
- aaronharnly 7y ago
  When white people are labeled for their other qualities, and non-white people are labeled for their (perceived) ethnicity, that’s racist. Which is to say, unfair. Not nice.
  - darawk 7y ago
    Shouldn't everyone just be labeled for everything? Seems like we should just add "white" to the tags of white people.
  - Grustaf 7y ago
    Were those two examples labeled by the same people? If there really is systematic bias why don’t they mention some statistics? For example, 90% of white people lacked a race tag while 80% of black people had one? Also, it doesn’t seem like there has been any evil intent here, nor did I see anything about pernicious consequences. It seems a bit overblown to accuse people of racism over something that is unintended and theoretical. Just update the tags.
ctdonath 7y ago
“This exhibition shows how these images are part of a long tradition of capturing people’s images without their consent, in order to classify, segment, and often stereotype them in ways that evokes colonial projects of the past,” Paglen told the Art Newspaper. Odd that the images were removed for being categorized as the project intended.
Chris2048 7y ago
I uploaded my picture, and it was tagged "beard", Which is fine, I have a beard, But then I read the definition: > beard: a person who diverts suspicion from someone (especially a woman who accompanies a male homosexual in order to conceal his homosexuality) :-/
lakisy 7y ago
Everybody seems to be focusing if the image recognition is racist or biased. I think there is a much more fundamental problem. The image classification does not work for a lot of images of people! For example Search for images on scuba diver https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg is labeled choreographer https://dtmag.com/wp-content/uploads/2015/03/scuba-diver-1050x700.jpg https://dtmag.com/wp-content/uploads/2015/03/scuba-diver-105... is labeled picador (the horseman who pricks the bull with a lance early in the bullfight to goad the bull and to make it keep its head low) How about search for images of dancer https://eugeneballet.org/wp-content/uploads/2018/10/Alessandro_CF004500.jpg https://eugeneballet.org/wp-content/uploads/2018/10/Alessand... is a nonsmoker https://www.ballet.org.uk/wp-content/uploads/2017/09/ENB_Emerging-Dancer_Final-Composite_mobile-544x780.jpg https://www.ballet.org.uk/wp-content/uploads/2017/09/ENB_Eme... is speedskater https://www.ballet.org.uk/wp-content/uploads/2018/10/WEB-ENB-ED2019-2500x1514.jpg https://www.ballet.org.uk/wp-content/uploads/2018/10/WEB-ENB... is plyer https://rachelneville.com/wp-content/uploads/2018/11/10.14.18.SamanthaSchaubach-0416.jpg https://rachelneville.com/wp-content/uploads/2018/11/10.14.1... is a mediatrix (a woman who is a mediator) How about images of lumberjack https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberjack-swinging-axe.jpg https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberja... is a skinhead https://static.tvtropes.org/pmwiki/pub/images/lumberjack_5919.jpg https://static.tvtropes.org/pmwiki/pub/images/lumberjack_591... beard: a person who diverts suspicion from someone (especially a woman who accompanies a male homosexual in order to conceal his homosexuality) http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-Edit.jpg?v=1541448378 http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-... is a flight attendant https://previews.123rf.com/images/rasstock/rasstock1411/rasstock141100008/34058828-stylish-young-man-posing-like-lumberjack.jpg https://previews.123rf.com/images/rasstock/rasstock1411/rass... is an asserter, declarer, affirmer, asseverator, avower: someone who claims to speak the truth How about teacher https://media.edutopia.org/styles/responsive_2880px_16x9/s3/masters/d7_images/cover_media/alber-169hero-thelook-shutterstock_0.jpg https://media.edutopia.org/styles/responsive_2880px_16x9/s3/... is a shot putter: an athlete who competes in the shot put https://c0.dq1.me/uploads/article/54231/student-classroom-teacher-2.jpg https://c0.dq1.me/uploads/article/54231/student-classroom-te... The kid is a non smoker and the teacher is psycholinguist: a person (usually a psychologist but sometimes a linguist) who studies the psychological basis of human language https://media.gannett-cdn.com/29906170001/29906170001_5780351010001_5780346124001-vs.jpg https://media.gannett-cdn.com/29906170001/29906170001_578035... is girl, miss, missy, young lady, young woman, fille: a young woman https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w_728,c_limit/how-teachers-dont-get-sick.jpg https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w... (almost the same picture as the prveious one ) is now a sociologist! How about search for pilot images https://news.delta.com/sites/default/files/Propel%20Embedded%20Image%282%291.jpg https://news.delta.com/sites/default/files/Propel%20Embedded... is a parrot: a copycat who does not understand the words or acts being imitated https://pilotpatrick.com/wp-content/uploads/2017/10/workday_titel.jpg https://pilotpatrick.com/wp-content/uploads/2017/10/workday_... is a beekeeper, apiarist, apiculturist: a farmer who keeps bees for their honey !!! https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F%2Fcdn-image.travelandleisure.com%2Fsites%2Fdefault%2Ffiles%2Fstyles%2F1600x1000%2Fpublic%2F1544737836%2Fpilot-training-BECOMEFLY1218.jpg%3Fitok%3D6BNCSj99&w=400&c=sc&poi=face&q=85 https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F... is a boatbuilder !! And some random one https://media.spiked-online.com/website/images/2019/08/06154453/scarlett-johansson-800x480.jpg https://media.spiked-online.com/website/images/2019/08/06154... is a "Sister/nun" https://www.english-heritage.org.uk/siteassets/home/visit/ingenious/roman/header.jpg?w=1440&h=612&mode=crop&scale=both&quality=60&anchor=NoFocus&WebsiteVersion=20190909 https://www.english-heritage.org.uk/siteassets/home/visit/in... deacon, Protestant deacon: a Protestant layman who assists the minister https://minervasowls.org/wp-content/uploads/2018/04/Romans-Minerva-Ono-sideways-2-e1524937719320.jpg https://minervasowls.org/wp-content/uploads/2018/04/Romans-M... are identified as morris dancer

The article mentions https://imagenet-roulette.paglen.com/ https://imagenet-roulette.paglen.com/ but doesn't link to it. Just put in a photo and see what category of person the algorithm sorts them in to.

alfromspace 7y ago

For the artist, ImageNet’s problems are inherent to any kind of classification system. If AI learns from humans, the rationale goes, then it will inherent all the same biases that humans have. Training Humans simply exposes how technology’s air of objectivity is more façade than reality. An AI being accused of bias tends to really mean it works. Removing bias from AI, I've noticed, requires hardcoded 'fixes' rather than refactoring the algorithms. And in my view, becomes yet another human-curated classification system, and no longer AI.

munchbunny 7y ago
Your point takes a narrow view on correctness and also shows why you need to take the systemic view on the issue. The problem is not that the AI is "inaccurate". The problem is the second order effect: when you build systems on top of this AI's predictions, you cement the input social biases into future systems in a way that is very hard to remediate. The real problem is how to avoid accidentally creating systems that amplify existing social biases. I'm very specifically using the term "social bias" to distinguish from "bias" as a term in ML, because they are very different problems.
brobinson 7y ago
This comment reminded me of a really interesting article about bias in AI: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

vorpalhex 7y ago

AIs are like every other computer program - perfectly naive. If you train them on bad data, you get bad results. Garbage in, garbage out.

core-questions 7y ago
What happens when we train them on an incredibly wide set of randomly-sampled data, which should show no built-in bias, and uncomfortable conclusions emerge? Do we censor the AI, castigate someone else in the chain for the problem, or just deal with reality as it presents itself?
- megablast 7y ago
  In this example, where would you get the tags for the photos from, if not people who are biased?
  - core-questions 7y ago
    Why assume that the people are "biased"? If the people are labelling things the way that people generally do, the software will return labels that are useful. Any kind of filtering needed to censor this for people's personal comfort should be implemented on top.
megablast 7y ago
Of course, but the issue is also data availability. It is not easy to get large sets of data to be used in training AI. Imagenet was a great attempt to provide some useful data.
- vorpalhex 7y ago
  My comment was not intended to be a ding on Imagenet. Data comes from somewhere, and no source is without bias because no human is without bias.
falsedan 7y ago

minimaxir 7y ago

Has this been confirmed? Not doubting the story, but the article doesn't provide a source. Edit: Might be this, which was a week ago / before the Roulette blew up: http://image-net.org/update-sep-17-2019 http://image-net.org/update-sep-17-2019 > We are in the process of preparing a new version of ImageNet by removing all the synsets identified as “unsafe” and “sensitive” along with their associated images. This will result in the removal of 600,040 images, leaving 577,244 images in the remaining “safe” person synsets.

raxxorrax 7y ago
If they delete half the data, the probability of the remaining data sets to be rubbish is quite high. You also wouldn't want someone to pick sets in the first place.

randyrand 7y ago

How were the mechanical turk workers asked to annotate?

brlewis 7y ago
That's the key question here, isn't it? If faces are like other images in the database, then according to this article turkers were presented a word and a group of images, and asked to click all the images that showed said word. https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/ https://qz.com/1034972/the-data-that-changed-the-direction-o...
- Out_of_Characte 7y ago
  ImageNet are likely incredibly naive with how they approached and fixed all of this. from the article "Jamal Jordan, a journalist at the New York Times, explained on Twitter that each of his uploaded photographs returned tags like “Black, Black African, Negroid, or Negro.”" Which indicates that the vocabulary they used has 4 words to describe a black person's skin color. This severely complicates things and leaves all kinds of unintended biases on the table before even getting to the human element. Though I doubt they used a dictionary at all because 'Black African' isn't really a word. And not found in a few dictionaries I tried online. - if the article is accurate. They should have curated the vocabulary first, value judgements like 'attractive' should be removed because it means something different for everyone judging the images. Synonyms should be collapsed into one to prevent weighted biases, etc.
  - esyir 7y ago
    I'm actually quite surprised that they didn't collapse the synonyms though. Though value judgements should not be removed. They should have been separated and placed together with some tagger metadata as they might actually be informative then.
tastroder 7y ago
Since the amount of speculation in this overall thread is pretty high, I suggest simply having a look at section 3 of [0]. It describes the construction of the MTurk task and how they quantified the confidence of answers they got. [0] http://www.image-net.org/papers/imagenet_cvpr09.pdf http://www.image-net.org/papers/imagenet_cvpr09.pdf

seph-reed 7y ago

Superficial judgement is kind of where intelligence starts. It's really only people where you can't tell what it does/is from the outside. Cars, trees, animals, mountains... everything else, if it looks a way it acts that way. Early AI will probably have just as much trouble with this as people have historically. I really wish people would start viewing Racism as a willingness to let that primitive part of the mind be in command, rather than a binary attribute you either have or don't. Like, nobody is 100% not racist. There will always be slip ups, over-simplifications, snap judgements, subconscious or not.

ALittleLight 7y ago
A car could look fast, but be slow, or have latent problems, software on the car's computer could be fooling emissions tests... Looking at a tree tells you nothing about photosynthesis, tree pheromones, deep relationships with symbiotic insects or bacteria... You can't understand very much at all just based on how things look. That holds for humans and inhumans alike.
- deogeo 7y ago
  But you can make a pretty good guess: http://www.spsp.org/news-center/blog/stereotype-accuracy-response http://www.spsp.org/news-center/blog/stereotype-accuracy-res...
- seph-reed 7y ago
  Ultimately we know nothing, so one can always nit-pick any statement into an oversimplified oblivion. There are also cakes that look like cars, model sets which have miniature mountains, on and on. But in typical nature, it works well enough to breed and see what comes next.
  - desert_boi 7y ago
    Tactical Nihilism sounds a lot like solipsism. "We can't ever not make generalizations, so we shouldn't even try to make statements."
- 7y ago

t8hszj5w 7y ago

Is this data racist? http://i.4pcdn.org/pol/1506370711307.png http://i.4pcdn.org/pol/1506370711307.png & http://i.4pcdn.org/pol/1488380621213.png http://i.4pcdn.org/pol/1488380621213.png

bllguo 7y ago

wouldn't it make more sense to add more images, not remove existing data?

sp332 7y ago
I think it's temporary. Sensible defaults to protect users who don't understand potential weaknesses of the model. Also, if someone does train an AI on a bunch of photos and ends up with a biased result, it won't be ImageNet's maintainers' liability.
mkl 7y ago
It seems to be largely the tags that are biased. Long term, I'm sure people want big unbiased databases of images of people, but it will take a lot of time and effort to build them and ensure they aren't biased.
travisoneill1 7y ago
If I understand correctly, they are removing incorrect tags, not the data. It's just news because some of the incorrect tags are racist bias. If they removed a bunch of cats tagged as dogs, you wouldn't hear about it.
brlewis 7y ago
IIUC the database is intended to identify the actual contents of images, not collect data about stereotypes. Even keeping the categorized face images in a separate database wouldn't be that useful, since turkers are not a random sample.

Nasrudith 7y ago

Really the bias isn't the true problem here so much as the lack of epistemology in how it is being used. By definition associations work by blind correlations - expecting anything other than stereotyping is a foolish misuse of the tool. It will be wrong for outliers because it tries to get it right for most cases regardless of cause - like skin cancer detectors who saw rulers as a sign of cancer because most of the reference images of cancer had a ruler in them.

didericis 7y ago
Exactly. The large amount of faith currently placed in probabilistic models that do not have common sense ways of eliminating factors which are extremely unlikely to be causal (like the presence if a ruler causing cancer) disturbs me. There is something that humans do that we have not quite figured out how to teach computers yet, at least as far as I can tell, which is to get them to evaluate whether their model is not just compatible with their observations, but other models and prior knowledge about the objects being observed. I think we’ll get there at some point, and I’m not exposed to the most cutting edge AI research, but it seems like AI us currently very overhyped and deeply flawed for many of the applications people would like to use it for.
- bonoboTP 7y ago
  Because these algorithms don't know that they are classifying cancer. The label it sees is just 1 or 0. For all it knows, based on its inputs, you may want to classify ruler/non-ruler images. To achieve what you want, semantic structure must be used as labels instead of just categorical labels. Assuming we have a sane AI that now knows its looking for cancer, it knows what that means (from digesting medical textbooks, papers and generic text corpora) and it can detect rulers and knows the two are not casually linked from ruler to cancer, we could make the model output "dataset diagnostics", like a "Warning! The cancer label in this dataset is implausibly correlated with the visual presence of a ruler". Or "Warning: 99% of your hotdog images show a human hand. Evaluation on this dataset will ignore errors on hotdog images without hands!" Context does matter though. If there's an orange fluff on a tree trunk, the AI is right to look at the environment and infer it's a squirrel.

deleted 7y ago

[deleted]

Out_of_Characte 7y ago

The AI isn't biased, the curaters were. The people who curated the first training set used subjective words like 'attractive' to tag the images which means the AI tagged all images it deemed 'attractive' to the people who made the training set. as this is a very biased and homogenous group it means the AI turned out biased. Maybe if they randomly sampled millions of people from all countries in order to create the training data then they could effectively train an AI to guess what YOU might find attractive. However even then I somehow doubt it. Beauty is in the eye of the beholder. We dont consiously know the rules of what we are attracted to, nor does an AI have secret information if you just supplied it with enough words and images. If they'd stuck with simple classifications like 'black' 'white' 'man' 'woman' then they would have less subjective judgement values about the original training set.

maerF0x0 7y ago
Exactly! They got subjective results because they went beyond facts. For example, it could have been interesting if they tagged each person with their actual religion or propensity to "grin"
- dev_dull 7y ago
  Going beyond facts is an important component of “being human”, so in that regard it makes the AI seem more intelligent. The problem is the AI is 100% honest with what it thinks, unlike a human.
  - maerF0x0 7y ago
    I agree that going beyond the facts is a good thing when humans are doing critical thinking and when being careful and transparent about their doing so. However this was creating a dataset for classification. Something that specifically should not go beyond the facts. (The basis of the model is the strength of the facts it's built upon)
    - tylerhou 7y ago

humble_engineer 7y ago

If you google "white race" on Google versus "black race" or "asian race" you don't get what you expect. You get hatred for whites, and then historical black and asian cultural for the latter.

bernierocks 7y ago

"For example, the program defined one white woman as a “stunner, looker, mantrap,” and “dish,” describing her as “a very attractive or seductive looking woman.” Many people of color have noted an obvious racist bias to their results. Jamal Jordan, a journalist at the New York Times, explained on Twitter that each of his uploaded photographs returned tags like “Black, Black African, Negroid, or Negro." They claim this is AI, but earlier in the article, it states that mechanical turk was used. Mechanical Turk is basically just people getting paid pennies or fractions of a penny to tag these photos. Many people on Mechanical Turk are from countries where English is not their first language and they don't have the same idea of racism as we do here in the US, which would explain the racist tags mentioned in the article. This doesn't really show us anything about AI and racial bias and more that other countries still aren't up to our levels of what we consider decent.

notyourday 7y ago
As someone who spent years in those villages I can tell you it is exactly how they view the world. The only posturing comparable to the one coming from the so called enlightened anti-racist comments that I ever heard/read about was Marie Antoinette saying "Let them eat cake". She ended up without her head.
LocalH 7y ago
The AI connection is that these are the type of datasets that feed AI systems. It shows that you have to be careful when curating your dataset, so that you don't further bias. I just hope the masses don't glom onto this, start shrieking that "AI is racist", and attempt to take AI completely out of the picture.

rolltiide 7y ago

reminds me in the early 2000s when people would say "but he's gay" as if that had anything to do with their profession or aspirations in life or the topic at hand, as if that specific metadata defined them more than the other pieces of metadata. this AI seems to be doing that to things it has determined are black people. A bunch of synonyms for black people, some English, some Spanish, some phenotypes, some of those descriptions simultaneously fallen out of favor in some parts of the world but not others, while completely eliminating other metadata. I'm not sure Imagenet's response of removing "sensitive" adjectives is capable of fixing this. Mechanical Turk: english, spanish, academics, using terms that aren't universally agreed upon? That doesn't really address what is happening

maxander 7y ago

Bias in AI (whatever the source or nature of the problem) is a real issue that needs addressing, but taking the relevant training data off of ImageNet seems like a perfect example of papering over a problem to avoid really confronting it. We will need to find ways to make AI programs than can see beyond the biases (about humans or otherwise) that will always exist to some degree in real-world data. If the ImageNet contains bias that leads to embarrassing results- that’s fine! That give us a readily available toy instance of the problem to study. Taking that away could actively harm anti-bias research.

avinium 7y ago
I strongly prefer they keep the bias. It shows plainly that these models have zero intelligence, they're just pattern matching over specific datasets. I don't want the appearance of fairness (introduced by human dataset curation) to be mistaken for "intelligence". Keeping the bias would hopefully cause people to think more critically about why such bias exists in the model in the first place.
- jobigoud 7y ago
  Disagree that it proves zero intelligence. Intelligent humans also have bias.
  - avinium 7y ago
    I'm not saying that intelligence and bias are orthogonal. That discussion requires a much deeper consideration of human cognition and psychology :) I'm just saying that model bias is a very easy thing to explain (usually, data imbalance). You can also fiddle with the label ratios to change the bias - which is also a good way of showing that the models aren't really intelligent.
WilliamEdward 7y ago
They could store it somewhere non-public for research purposes, but taking it off the website was the right call.
7y ago

orf 7y ago

Their statement: http://image-net.org/update-sep-17-2019 http://image-net.org/update-sep-17-2019 > Each synset is classified as either “unsafe” (offensive regardless of context), “sensitive” (offensive depending on context), or “safe”. “Unsafe” synsets are inherently offensive, such as those with profanity, those that correspond to racial or gender slurs, or those that correspond to negative characterizations of people (for example, racist). “Sensitive” synsets are not inherently offensive, but they may cause offense when applied inappropriately, such as the classification of people based on sexual orientation and religion. > So far out of 2,832 synsets within the person subtree we’ve identified 438 “unsafe” synsets and 1,155 “sensitive” synsets. The remaining 1,239 synsets are temporarily deemed “safe.” They've also completely disabled Imagenet downloads while they remedy this.

esyir 7y ago
Great, I can see some of this being useless (some parts of the unsafe dataset), but if they cull the "sensitive" portion, this may induce performance regressions. I need to find an ImageNet archive now.
- buildbot 7y ago
  It doesn't really touch the 1000 class "Imagenet" that's commonly used in computer vision.
  - esyir 7y ago
    Ah, then it's fine... probably.
  - yorwba 7y ago
    Some of the classes are subcategories of ILSVRC classes in the WordNet hierarchy. So by removing images of persons in categories that are considered inappropriate, the resulting classifier will end up less likely to recognize those as images of people at all. I'm not sure whether that's a better outcome.
kristianp 7y ago

kayaeb 7y ago

In NLP (specifically vectorizing words, ala word2vec) there's a famous test of whether or not your training has worked properly whereby you calculate the vector of "king" and subtract the vector of "man" and add the vector of "woman," if your machine is properly tuned, you should end up with a vector close to "queen" or "princess." I wonder if similar things can be done to address specific (i.e. racial or gender) biases in computer vision.

mhuffman 7y ago
There is a similar word-embedding test that definitely rustles people's jimmies: Doctor - Man + Woman = ? What normally comes out is Nurse. What "they" think should come out is Doctor! By "they" I mean people that get upset by this.
- tastroder 7y ago
  Yeah, besides the fact that this compositionality is relatively unique to word2vec, research on the biases pre-trained models express is pretty available. Linked a few below for those interested. Most of the issues are down to the same phenomenon discussed here in the context of ImageNet, the input texts were biased and the algorithm learned said bias. [0] https://arxiv.org/abs/1607.06520 https://arxiv.org/abs/1607.06520 "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings " [1] http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf "Understanding the Origins of Bias in Word Embeddings" [2] http://matthewkenney.site/biases.html http://matthewkenney.site/biases.html "Google word2vec biases"

danso 7y ago

edit: I conflated ImageNet with the art exhibitors; it is the former who are culling the images as a result of public reaction, not the latter. This is a really bizarre project. I had seen some really offensive race-based labels, but I thought revealing the ugliness of the system was part of the point of this project? But besides that, the results just seemed completely scattershot; I half expect the artists/exhibitors to reveal that x% of the results were randomized. Last week, I tried it myself after seeing another Asian user display results that were entirely Asian slurs (e.g. gook, slant-eyed). I uploaded my own very Asian-looking photo and got "prophetess", along with very vague labels, such as "person" and "individual". Maybe the exhibitors cleaned the data/results by the time I tried it, but I used it just a few hours after seeing the other Asian user's results, so I'm doubtful that her tweet/complaints were enough on their own to change up the dataset that same day.

bigiain 7y ago
> I thought revealing the ugliness of the system was part of the point of this project? It 100% is. From the link to the artist's website: "Things get strange: A photograph of a woman smiling in a bikini is labeled a “slattern, slut, slovenly woman, trollop.” A young man drinking beer is categorized as an “alcoholic, alky, dipsomaniac, boozer, lush, soaker, souse.” A child wearing sunglasses is classified as a “failure, loser, non-starter, unsuccessful person.” You’re looking at the “person” category in a dataset called ImageNet, one of the most widely used training sets for machine learning."
- danso 7y ago
  I was just about to edit and correct myself; it is ImageNet who has made the decision to delete the offensive images, after the reaction to the exhibitors' work. It's too bad the exhibitors didn't make their own mirror/cache of the dataset. Judging from some tweets I saw, I think this project really helped people to understand how much of current artificial intelligence is human-driven. It's not a sentient computer deeming you to be a "slant-eye", it's a bunch of random Internet users. (not that this makes you feel better about the world, but at least the hate's coming from an expected source)

notyourday 7y ago

I love articles and papers like this. They keep illustrating that the people writing them do not get it: Guess who classified the blonde white woman "a dish" on a Mechanical Turk? Do you think it was that Billy Bob from the swamp of Louisianan who makes $12/hour? Because $12/hour is impossible to make classifying images. Or do you think it was a dirt poor Indonesian or Filipino or Chinese or Indian for whom that's the best job he or she could possibly get and that's exactly how they view the world and there are over three billion of them?

Gigablah 7y ago
Your characterization of these nationals aside, what makes you think any of them even know what “dish” means in this context? Sincerely, a Malaysian Chinese who’s seeing this slang for the first time.
- org3432 7y ago
  Dish is almost an archaic term a this point. So either someone's pretty old grandparents are working for MT, or people are translating a term from another language using an old dictionary.
- notyourday 7y ago
  It is colonial (British) English. I heard it again recently and it took me a few moments before I recalled it from my childhood. Apparently now "snack" is used the same way. I would guess the classifiers went to school in India or Pakistan.

bigiain 7y ago

I wonder if the article author knew what he was hinting at when he chose the example of the art project calling someone a "dish"??? https://www.youtube.com/watch?v=HEX7xsYF1nA https://www.youtube.com/watch?v=HEX7xsYF1nA (And now I'm gonna be listening to 90s electro-disco all day...)

erikig 7y ago

While interesting, this is not surprising. One of the most commonly utilized datasets for learning ML is the Boston Housing dataset - https://www.kaggle.com/c/boston-housing https://www.kaggle.com/c/boston-housing In it there's a problematic feature tagged simply as "black" and it is defined as the proportion of blacks by town. Any pricing model that is built off of this dataset is inherently racially biased because the data has been collected and the feature tagged - but what's the alternative? Not to collect the information? Or collect it but completely ignore this feature?

dannyw 7y ago
Sensitive features should not be collected or used in applications where they introduce bias. For example, a medical screening NN may find race to be a valuable feature for the prediction of illness; but a health insurance assessor should not.
- chriskanan 7y ago
  Due to spurious correlations, it could still be helpful for the insurance assessor so that they can use bias mitigation techniques. Otherwise, it might learn something about zip code or something else that leads to a similar outcome as having race as an input variable. Just removing a sensitive variable does not suffice for preventing unwanted bias.
- qoninger 7y ago
  Then you are confusing bias with bias. One is data bias and the other one is racial bias. If you remove the data bias, you are by definition introducing a racial bias by imposing your will on reality.

zaroth 7y ago

I think the ironic thing here is the algorithm is intentionally designed to make a superficial judgement on the image it’s presented with. If it was trained to identify “human” or otherwise categorize the things in the picture, that’s exactly what it would output. Next you can train it to attempt to guess basic traits like gender or ethnicity, of course this can only be done based on the RGB values of a 2D array of pixels. Interestingly the NN will not be merely be using skin color but building probabilistic weightings based on any statistically significant features. For added controversy, it’s probably even possible to invert parts of the network to suss out how it’s weighting various facial features toward different labeling. Lastly the images could be labeled for things like profession of the subject. A good intentioned effort to perhaps detect things like a lab coat could mean a doctor or scientist was followed by oblivious Turkers with predictably poor results. The problem of course is not with the images but the particular labeling hierarchy, and allowing opinionated labels as well as labels which may be true for the particular subject but which bear no distinguishing features for which to actually codify (“portrait of a macroeconomist”), in other words, garbage in garbage out. ImageNet calls this the “imageability” of the label. Calling this racism of course is entirely inapt, because there is no judgement being made whatsoever. Even if the sampling method was well designed and the labeling factually accurate, the system would still produce output which could be considered offensive. Again, because the entire point of the algorithm is to generate statistical assumptions based on a single image. My conclusion is that some superficial judgments are algorithmically useful and hopefully less controversial. “White male human, ~55 yrs old, 180lbs”. Even things like analyzing clothing and guessing where the picture was taken. Iff the clothing is a uniform, identifying the profession (police, fire, paramedic) But you have to know where this goes off the rails. Bad enough to label indistinct portraits with the subject’s profession, let’s not do inane things like labeling them with how subjectively attractive the labeler thinks the person is, their economic status, maybe even a 1-10 scale of how threatening they look or if they look like a criminal or not! </facepalm>

crazygringo 7y ago

The problem seems less about racist bias specifically and more about unbelievably dumb tags generally in the first place. "Stunner"? "Looker"? "Mantrap"? Or even trying to tag people's images with categories like Buddhist, grinner, or microeconomist? What were they thinking?! Clearly these tags were never curated in any remotely responsible way -- for quality, for sensitivity, or just usefulness at all -- and I'm shocked they were ever intended for academic or research use. No wonder image recognition gets a bad rap, with input data like this.

Iv 7y ago
Agreed. These are not descriptive labels you can infer from an image. They may be, however, labels generated by users from a context.
ailideex 7y ago
But removing the images from ImageNet, an inanimate set of images that cannot make value judgements, will surely fix this problem... Our culture is broken and nobody is willing to stand up to insanity or even admit to it. People are too happy to do completely insane and senseless things if it saves them from the social lynchings.
tastroder 7y ago
> What were they thinking?! [...] In addition to their statement, linked in a thread below [0], what they were thinking is pretty out in the open in the abstract of the original paper: "We introduce here a new database called “ImageNet”, a largescale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. " [1] That paper is now 10 years old, WordNet itself had it's first release somewhere in the 1980s. A lot happens in research a decade down the road and lots of research towards biases of datasets are relatively recent in that sense. It can hardly be blamed on the researchers that people use their datasets in production merely because "they're large and there in the open". As their statement indicates they're actively working on the issues that arose during the last few years. Most research datasets are fit for a very specific purpose, to me the larger problem here is two-fold: "AI" fearmongering on one side and startups/corporations on the other side lacking the necessary skills to select and filter their training data. Hopefully that'll be a matter of the past once the hyped data science field reaches maturity. edit: Wordnet seems to include said terms to this day, [2] for example still lists "mantrap" associated to "beaty" in the synset for "a very attractive or seductive looking woman". Although I'm unsure how they should handle these cases, both are words and that's one definition used in practice, even if we find it reprehensible. They seem to have removed the n-word but that's about the only removed instance I can find in a cursory search of sensitive or insulting English words. Maybe they should clarify sensitive annotations like they do with vulgarity. [0] https://news.ycombinator.com/item?id=21054770 https://news.ycombinator.com/item?id=21054770 [1] http://www.image-net.org/papers/imagenet_cvpr09.pdf http://www.image-net.org/papers/imagenet_cvpr09.pdf [2] http://wordnetweb.princeton.edu/perl/webwn?s=beauty http://wordnetweb.princeton.edu/perl/webwn?s=beauty

clamprecht 7y ago

One of my favorites is searching google images for "immigrant" versus "expat". Guess which one is whiter!

ineedasername 7y ago

The longer piece this one is based on is much more comprehensive in both discussion and concrete examples: https://www.excavating.ai/ https://www.excavating.ai/

s9w 7y ago

Of course it was removed. 4chan was having way too much fun with image classification recently :D

raxxorrax 7y ago
They will be hated for it, but they do provide a really important service here. And if it is just pulling down the pant of people trying to classify the world.

boyadjian 7y ago

The machine always tells the truth, but it is unbearable for some.

dannyw 7y ago
Perhaps when we have real artificial intelligence, whatever that is. Right now we have pattern matchers, and it is not so much the 'truth' as it is stereotypes.

Illniyar 7y ago

Why take down the images? It seems really that one should take down entire classifications, i.e. remove the tags.

ailideex 7y ago
Because the woke scolds have spoken and nobody wants to stand up to them. So better to do patently insane things than risk becoming collateral damage in a social lynching.
yorwba 7y ago
ImageNet is a mapping from tags to images, so removing the tags means removing the images. An image of a "bad person" wasn't tagged by someone looking at said image and deciding that "bad person" was the best possible description. It was generated by searching Google Images for "bad person" and removing obviously incorrect results (e.g. when there's nobody in the image). Researchers have been using it to learn the inverse mapping from images to tags with some success, but in its construction the dataset is not naturally suited for that task.
- Illniyar 7y ago
  As I understood it, and confirmed by Wikipedia (as much that's worth) images were hand annotated.
  - yorwba 7y ago
    Only the second step "removing obviously incorrect results" involved human annotators.

dusted 7y ago

That's newthink at a level I find hard to comprehend. Censoring datasets instead of improving them. "We have too many pictures of white people, remove them!" "We don't have enough pictures of non-white people, add them!" I'd have gone for the latter, and have let the set be biased until fixed.

FlowNote 7y ago

The more you impose completeness of morality into a data set, the more inconsistent your results will be. Gödel arbitrage will be very profitable in the future.

helb 7y ago

The "app", mentioned in the article, but not linked, is hosted here (until Friday), if you want to try it out: https://imagenet-roulette.paglen.com/ https://imagenet-roulette.paglen.com/ Apparently i'm either an insurance agent (in a grey t-shirt) or a surgeon (in a red hoodie).

nudpiedo 7y ago

I think the whole training and such AI was just bad designed. Aren't all first impression judgements prejudices as much as all racist bias are? If I say a person is attractive, inspires me just based on their looking it is a positive judgement, a bias based on the looks and a few traits I subconsciously perceive and perhaps I cannot even explain, and the same with the negative ones, and I guess for some people this association happens also to race and looks. Basically an AI was trained to map the prejudices of their inputs, and that means to create a snapshot of the subconscious average of whoever behind the mechanical turk is, everyone has prejudices of all sorts which usually are not manifested externally but in a process of classification such as the one of the mechnical turk might arise much more.

theqult 7y ago

Any way to download those images before they get deleted ?

Grustaf 7y ago

I don’t quite get it, why is “Buddhist” more objectionable than “grinner”? Surely there must be more egregious examples than that if they decided to pull 600,000 images? And if they find the tag “Negro” offensive can’t they just replace it with “black”, “of African origin” or whatever the currently allowed term is? Tagging a black person as black does not seem very objectionable in a country that provides racial breakdowns for official statistics.

aaronharnly 7y ago
When white people are labeled for their other qualities, and non-white people are labeled for their (perceived) ethnicity, that’s racist. Which is to say, unfair. Not nice.
- darawk 7y ago
  Shouldn't everyone just be labeled for everything? Seems like we should just add "white" to the tags of white people.
- Grustaf 7y ago
  Were those two examples labeled by the same people? If there really is systematic bias why don’t they mention some statistics? For example, 90% of white people lacked a race tag while 80% of black people had one? Also, it doesn’t seem like there has been any evil intent here, nor did I see anything about pernicious consequences. It seems a bit overblown to accuse people of racism over something that is unintended and theoretical. Just update the tags.

ctdonath 7y ago

“This exhibition shows how these images are part of a long tradition of capturing people’s images without their consent, in order to classify, segment, and often stereotype them in ways that evokes colonial projects of the past,” Paglen told the Art Newspaper. Odd that the images were removed for being categorized as the project intended.

Chris2048 7y ago

I uploaded my picture, and it was tagged "beard", Which is fine, I have a beard, But then I read the definition: > beard: a person who diverts suspicion from someone (especially a woman who accompanies a male homosexual in order to conceal his homosexuality) :-/

lakisy 7y ago

Everybody seems to be focusing if the image recognition is racist or biased. I think there is a much more fundamental problem. The image classification does not work for a lot of images of people! For example Search for images on scuba diver https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg is labeled choreographer https://dtmag.com/wp-content/uploads/2015/03/scuba-diver-1050x700.jpg https://dtmag.com/wp-content/uploads/2015/03/scuba-diver-105... is labeled picador (the horseman who pricks the bull with a lance early in the bullfight to goad the bull and to make it keep its head low) How about search for images of dancer https://eugeneballet.org/wp-content/uploads/2018/10/Alessandro_CF004500.jpg https://eugeneballet.org/wp-content/uploads/2018/10/Alessand... is a nonsmoker https://www.ballet.org.uk/wp-content/uploads/2017/09/ENB_Emerging-Dancer_Final-Composite_mobile-544x780.jpg https://www.ballet.org.uk/wp-content/uploads/2017/09/ENB_Eme... is speedskater https://www.ballet.org.uk/wp-content/uploads/2018/10/WEB-ENB-ED2019-2500x1514.jpg https://www.ballet.org.uk/wp-content/uploads/2018/10/WEB-ENB... is plyer https://rachelneville.com/wp-content/uploads/2018/11/10.14.18.SamanthaSchaubach-0416.jpg https://rachelneville.com/wp-content/uploads/2018/11/10.14.1... is a mediatrix (a woman who is a mediator) How about images of lumberjack https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberjack-swinging-axe.jpg https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberja... is a skinhead https://static.tvtropes.org/pmwiki/pub/images/lumberjack_5919.jpg https://static.tvtropes.org/pmwiki/pub/images/lumberjack_591... beard: a person who diverts suspicion from someone (especially a woman who accompanies a male homosexual in order to conceal his homosexuality) http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-Edit.jpg?v=1541448378 http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-... is a flight attendant https://previews.123rf.com/images/rasstock/rasstock1411/rasstock141100008/34058828-stylish-young-man-posing-like-lumberjack.jpg https://previews.123rf.com/images/rasstock/rasstock1411/rass... is an asserter, declarer, affirmer, asseverator, avower: someone who claims to speak the truth How about teacher https://media.edutopia.org/styles/responsive_2880px_16x9/s3/masters/d7_images/cover_media/alber-169hero-thelook-shutterstock_0.jpg https://media.edutopia.org/styles/responsive_2880px_16x9/s3/... is a shot putter: an athlete who competes in the shot put https://c0.dq1.me/uploads/article/54231/student-classroom-teacher-2.jpg https://c0.dq1.me/uploads/article/54231/student-classroom-te... The kid is a non smoker and the teacher is psycholinguist: a person (usually a psychologist but sometimes a linguist) who studies the psychological basis of human language https://media.gannett-cdn.com/29906170001/29906170001_5780351010001_5780346124001-vs.jpg https://media.gannett-cdn.com/29906170001/29906170001_578035... is girl, miss, missy, young lady, young woman, fille: a young woman https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w_728,c_limit/how-teachers-dont-get-sick.jpg https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w... (almost the same picture as the prveious one ) is now a sociologist! How about search for pilot images https://news.delta.com/sites/default/files/Propel%20Embedded%20Image%282%291.jpg https://news.delta.com/sites/default/files/Propel%20Embedded... is a parrot: a copycat who does not understand the words or acts being imitated https://pilotpatrick.com/wp-content/uploads/2017/10/workday_titel.jpg https://pilotpatrick.com/wp-content/uploads/2017/10/workday_... is a beekeeper, apiarist, apiculturist: a farmer who keeps bees for their honey !!! https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F%2Fcdn-image.travelandleisure.com%2Fsites%2Fdefault%2Ffiles%2Fstyles%2F1600x1000%2Fpublic%2F1544737836%2Fpilot-training-BECOMEFLY1218.jpg%3Fitok%3D6BNCSj99&w=400&c=sc&poi=face&q=85 https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F... is a boatbuilder !! And some random one https://media.spiked-online.com/website/images/2019/08/06154453/scarlett-johansson-800x480.jpg https://media.spiked-online.com/website/images/2019/08/06154... is a "Sister/nun" https://www.english-heritage.org.uk/siteassets/home/visit/ingenious/roman/header.jpg?w=1440&h=612&mode=crop&scale=both&quality=60&anchor=NoFocus&WebsiteVersion=20190909 https://www.english-heritage.org.uk/siteassets/home/visit/in... deacon, Protestant deacon: a Protestant layman who assists the minister https://minervasowls.org/wp-content/uploads/2018/04/Romans-Minerva-Ono-sideways-2-e1524937719320.jpg https://minervasowls.org/wp-content/uploads/2018/04/Romans-M... are identified as morris dancer