9 ms·
This happened to both Google Photos and Flickr too. Which makes it an inexcusable mistake to make in 2021 - how are you not testing for this? Google Photos in
by simonw 5y ago
This happened to both Google Photos and Flickr too. Which makes it an inexcusable mistake to make in 2021 - how are you not testing for this?
Google Photos in 2015: https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/ https://www.wired.com/story/when-it-comes-to-gorillas-google...
Flickr in 2015: https://www.independent.co.uk/life-style/gadgets-and-tech/news/flickr-s-auto-tagging-feature-goes-awry-accidentally-tags-black-people-apes-10264144.html https://www.independent.co.uk/life-style/gadgets-and-tech/ne...
- bpodgursky 5y agoYou can't just "test" a neural network like that. For all you know they tested a thousand pictures of Chimpanzees and Gorillas against the network, but for some reason the NN decided to classify the photo differently because the subject was standing in front of the wrong kind of tree or wearing a funny-colored hat. There's no super reliable way to prevent this (with current tech) other than forbidding that output entirely.
- IncRnd 5y agoThe reason these companies don't fix these systems is because they don't know how. It is easier to remove certain outputs or retire the whole system. There is no line of code they can tweak.
- nolaspring 5y agoRichen the dataset it’s trained on enough so that the model is correct before you release it to prod.
- IncRnd 5y agoThat's sort of obvious. How do you know that wasn't attempted?
- MBCook 5y agoIf it was we wouldn’t expect this problem to occur, correct?
- belltaco 5y agoThat makes it sound even worse that they knowingly released it without fixing it.
- IncRnd 5y agoI agree. It does seem worse that way.
- ummonk 5y agoAre you saying an ML system should never be released if it doesn’t have perfect accuracy?
- belltaco 5y agoTagging black people as monkeys is not a showstopper bug? If so it makes them look even worse than if it was an overlooked bug.
- urthor 5y agoQuite honestly, in a team that's stressed and run down to the line, checking for that particular classification in a model that has hundreds of classification targets can be really tricky. Say you have 1000 classification targets. You have to produce a model that checks, for each target, the odds of it being classified as one of other 999. You have to check, specifically, for "adult male as primate" out of a million potential combinations. And apply secondary business rules or optimizations to prevent that classifications. So yes it's possible, but it's not cheap, simple or easy. Facebook just decided to shove the model out the door and not worry about the consequences. Quality engineering work, costs money and time. Facebook didn't spend it.
- shadowgovt 5y agoWe don't actually know how to do that, or how rich is "rich enough." It's an open avenue of research to be able to extrapolate how well-tuned a neutral net is on data not in its training set. Not to imply the problem is unsolvable, just that if an institution has zero tolerance for this mistake, the fix your describing is no guarantee it won't occur.
- 6gvONxR4sf7o 5y agoThat’s not quite complete, right? It’s that we don’t know how to do that without sacrificing other things.
- dheera 5y ago[self-censored to prevent further downvotes because HN is downvoting me for what I said and the delete button is gone; sorry it seems discussion about solutions to racial issues isn't welcome here]
- anyfactor 5y agoI second this but not for the conventional reasons of diversity but the subtleties diversity could be difficult to understand. You need your consumer base to be represented in your product development system. In training an AI model we first test things with what we are personally have a bias for. For example in training a stock prediction model, I am going to first test it with the most familiar stocks that I know or I have bias for. And I am going to adjust my model until I find the model being correct for my bias then test it across a larger dataset. I don't work in AI but I know that test data is incredibly large and they supposedly systematically cover all bases. But what I am saying is this the chance of random events will go down when you deviate from a homogenous development team where everyone is biased towards the same things because they share cultural and racial overlaps.
- fighterpilot 5y ago"You need your consumer base to be represented in your product development system." This is not the real motivation for diversity initiatives since otherwise you would see a push to hire more old Republicans.
- opinion-is-bad 5y agoIs there a sizable population of unemployed black engineers living in the United States to hire from? What if the qualified candidates simply don’t exist to fill the seats?
- hnxs 5y agoare you familiar with the h-1b visa system? there’s a shortage of qualified candidates across every metric you can name for top-band engineering roles in the US.
- hyperinflation 5y agoThe uncomfortable truth is that black people more closely resemble "primates" than other races by virtue of their pigmentation. This isn't racist, it's reality.
- wpietri 5y agoThis reminds me of a favorite tweet from 2013: "Then Google Maps was like, 'turn right on Malcolm Ten Boulevard' and I knew there were no black engineers working there" -- https://twitter.com/alliebland/status/402990270402543616 https://twitter.com/alliebland/status/402990270402543616 Facebook, like a lot of tech companies, has long had problems with diversity in engineering. Here's an article from April that discusses specific incidents and the broader background: https://www.washingtonpost.com/technology/2021/04/06/facebook-discrimination-hiring-bias/ https://www.washingtonpost.com/technology/2021/04/06/faceboo...
- dd444fgdfg 5y agogoogle maps is made in Australia, and the diversity there is different
- oreilles 5y agoThis isn't a problem with diversity. Everybody knows how to pronounce Malcom X. And it's not like just because a google engineer was black that he was like "oh, let's try and see if Malcom X is pronounced correctly because he's black and I'm black too". This only happens in white people's brain.
- sinyug 5y ago> This only happens in white people's brain. 'Eleven Jinping': Indian TV fires anchor over blooper.[1] [1] https://www.bbc.com/news/world-asia-india-29274792 https://www.bbc.com/news/world-asia-india-29274792
- travisporter 5y agoTo play devil advocate, maybe the station fired her for ignorance of current events.
- sinyug 5y ago> maybe the station fired her for ignorance of current events. That would be a valid reason, but I suspect a more culturally appropriate one: loss of reputation. We are sensitive to that. My point was this isn't something that only goes on in 'white' brains but more of a cultural issue. Most people in the West are incapable of pronouncing Asian names. I don't see people making a big issue out of it.
- gumboshoes 5y agoGoogle Photos solved the problem by simply returning no results for words like gorilla, monkey, primate, etc.
- jcims 5y agoI was just thinking about that. Unfortunately it just makes the bias harder to detect. Once you search for these: https://www.google.com/search?q=human+female+face&tbm=isch https://www.google.com/search?q=human+female+face&tbm=isch https://www.google.com/search?q=human+male+face&tbm=isch https://www.google.com/search?q=human+male+face&tbm=isch You can see that 'human face' has a bit of post-hoc tuning. https://www.google.com/search?q=human+face&tbm=isch https://www.google.com/search?q=human+face&tbm=isch
- silisili 5y agoSo disappointing. I was legitimately looking for a monkey pic I took years ago to no avail because of no searchability. One of the richest companies in the world prefers to just remove ability than to solve hard problems. But hey, at least we all get ads.
- nostromo 5y agoIt’s an inevitable result of angry mobs (like this article and entire HN thread) and risk-intolerant corporations. It’s impossible to test every image for accuracy and to guarantee it won’t happen again, so they just sidestep it entirely.
- hdjjhhvvhga 5y agoBut what would you do if you were them? You solved 95% of the problem, you are left with 5% that are extremely hard. Would you throw a large amount of resources to solve that? And, given that you basically deal with probabilities and that the system will never work 100% anyway, and that even one mistake of that kind will cause uproar - is there any other feasible solution?
- jcims 5y agoWhy are you so sure they aren't testing for it? Bias finds a way.
- deleted 5y ago[deleted]
- jcims 5y agoCurious if anyone on HN has built a testing framework to catch this kind of issue.
- godelski 5y ago> Which makes it an inexcusable mistake to make in 2021 - how are you not testing for this? They probably are, but not good enough. These things can be surprisingly hard to detect. Post hoc it is easy to see the bias, but it isn't so easy before you deploy the models. If we take racial connotations out of it then we could say that the algorithm is doing quite well because it got the larger hierarchical class correct, primate. The algorithm doesn't know the racial connotations, it just knows the data and what metric you were seeking. BUT considering the racial and historical context this is NOT an acceptable answer (not even close). I've made a few comments in the past about bias and how many machine learning people are deploying models without understanding them. This is what happens when you don't try to understand statistics and particularly long tail distributions. gumboshoes mentioned that Google just removed the primate type labels. That's a solution, but honestly not a great one (technically speaking). But this solution is far easier than technically fixing the problem (I'd wager that putting a strong loss penalty for misclassifiying a black person as an ape is not enough). If you follow the links from jcims then you might notice that a lot of those faces are white. Would it be all that surprising if Google trained from the FFHQ (Flickr) Dataset?[0] A dataset known to have a strong bias towards white faces. We actually saw that when Pulse[1] turned Obama white (do note that if you didn't know the left picture was a black person and who they were that this is a decent (key word) representation). So it is pretty likely that _some_ problems could simply be fixed by better datasets (This part of the LeCunn controversy last year). Though datasets aren't the only problems here. ML can algorithmically highlight bias in datasets. Often research papers are metric hacking, or going for the highest accuracy that they can get[2]. This leaderboardism undermines some of the usage and often there's a disconnect between researchers and those in production. With large and complex datasets we might be targeting leaderboard scores until we have a sufficient accuracy on that dataset before we start focusing on bias on that dataset (or more often we, sadly, just move to a more complex dataset and start the whole process over again). There's not many people working on the biased aspects of ML systems (both in data bias and algorithmic bias), but as more people are putting these tools into production we're running into walls. Many of these people are not thinking about how these models are trained or the bias that they contain. They go to the leaderboard and pick the best pre-trained model and hit go, maybe tuning on their dataset. Tuning doesn't eliminate the bias in the pre-training (it can actually amplify it!). ~~Money~~Scale is NOT all you need, as GAMF often tries to sell. (or some try to sell augmentation as all you need) These problems won't be solved without significant research into both data and algorithmic bias. They won't be solved until those in production also understand these principles and robust testing methods are created to find these biases. Until people understand that a good ImageNet (or even JFT-300M) score doesn't mean your model will generalize well to real world data (though there is a correlation). So with that in mind, I'll make a prediction that rather than seeing fewer cases of these mistakes rather we're going to see more (I'd actually argue that there's a lot of this currently happening that you just don't see). The AI hype isn't dying down and more people are entering that don't want to learn the math. "Throw a neural net at it" is not and never will be the answer. Anyone saying that is selling snake oil. I don't want people to think I'm anti-ML. In fact I'm a ML researcher. But there's a hard reality we need to face in our field. We've made a lot of progress in the last decade that is very exciting, but we've got a long way to go as well. We can't just have everyone focusing on leaderboard scores and expect to solve our problems. [0] https://github.com/NVlabs/ffhq-dataset https://github.com/NVlabs/ffhq-dataset [1] https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801 [2] https://twitter.com/emilymbender/status/1434874728682901507 https://twitter.com/emilymbender/status/1434874728682901507
- silisili 5y agoIs it inexcusable that if I search 'Japan' to look for pics from my trip to Japan, it shows me pictures containing any Asian person at all? If I search Japan today, I get mostly pics of my not Japanese wife. But I guess we don't complain enough for anyone to care. https://i.ibb.co/Mf6rVdf/Screenshot-20210907-002516-Photos.jpg https://i.ibb.co/Mf6rVdf/Screenshot-20210907-002516-Photos.j... Nobody who has traveled at all would mistake my wife and child as Japanese. And doing so is especially insidious considering the Bataan death march.
- trhway 5y ago>how are you not testing for this? i wonder how testing for that looks and sounds in corporate environment. It may as well be an area similar to patents - you pretend that you never heard, never discussed, God forbid any mentioning in corporate email/chat/etc. or clicking on a link from inside a corporate network,...