5 ms·
OK, I'll try to tread carefully here, but I think it's really helpful to separate the technical issues from the societal issues. In the Google photo case, I've
by hn_throwaway_99 5y ago
OK, I'll try to tread carefully here, but I think it's really helpful to separate the technical issues from the societal issues.
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
- ant6n 5y agoThe problem is the training set, which apparently skews too much towards white ppl, and doesn't include checks for the primates/non-white humans differentiation. One could argue this is due to white people building things for white ppl, which does indeed point towards systemic racism in the overall development process.
- fallingknife 5y agoBut the problem is that there isn't any primates/non-white humans differentiation because humans are primates. I doubt anybody smart enough to be an AI engineer of any race would care if they were called a primate, because they are.
- hn_throwaway_99 5y agoI feel like this is a very "engineer" response that completely ignores how society in general would view this. Yes, humans are primates. But this wasn't a discussion about taxonomy in biology class. There are lots of cases where primates is used in everyday language where it is clearly intended to refer solely to monkeys and apes. When I go to the "primate house" at the zoo I'm not going to expect to see naked humans behind the glass.
- fallingknife 5y agoBut an AI image recognition system is a very "engineer" kind of thing.
- adventured 5y agoAnd in cases where that engineer thing is used on people - it matters when you label a black person as a primate on a live social media network, since it's understood that would be likely to cause significant offense.
- tartoran 5y agoRight and that’s why we should curtail its use because thinking in an engineering box suits very few real world examples. This algorithm is clearly broken, it doesn’t label all people as primates, it labels Black people wearing a green T-shirt possibly as such, I mean I’m sorry but that algorithm is definitely broken. The person in question should not be labeled as primate by any means not should any other humans unless the context is specifically some form of species taxonomy. Also your reaction seems to be all daisies and roses, and I don’t blame you for your social context ignorance you displat, I guess you are just an engineer after all..
- ant6n 5y agoI was gonna write "monkey-blackppl-differentiation", but decided it was too offensive and aimed for more neutral language, thinking for anybody who doesn't want to be an asshole it's clear what it's about.
- 5y ago
- rscoots 5y agoInterested in hearing how you obtained the proprietary training material for this algo from Apple. Can we see it? Also which teams at apple made the algorithm? Since you claim to know they were all 'white'. In reality, I'm sure everyone working on this team was well aware of these potentials for training set bias before most of us even knew what ML was. Even if this was considered it'd be pretty risque to have an employee dedicate time specifically to differentiating black ppl from apes in the algorithm.
- jjj123 5y agoI worked closely with an AI ethicist at $100B+ tech company company and their findings were that all of our AI training data was biased and the ML teams had no plans to de-bias their training data because they had no incentive to do so. I have no reason to believe this finding doesn’t hold for the FAANG companies. AI ethics are mostly an afterthought in tech today. And to your second point about how would we know if the team was “all white,” of course we don’t know that and it’s probably not true. What we do know is that tech in the us is disproportionately whiter than the general public, and ~10 person teams with no black people on them are the norm, not the exception. Edit: I should say that 10 person teams without black people on them are the norm in FAANG companies and others headquartered in the Bay Area. Other tech hubs in the US are much more diverse.
- rscoots 5y agoClearly my point was to highlight your prejudicial instincts around this issue. For all you know the entire team that worked on this project could've been black. Or zero of them could've been white etc. We can and should have a conversation about training sets and ML ethics. Resorting to unprovoked racist attacks is quite a counterproductive approach imo.
- Melchizedek 5y agoTech is not disproportionately white. It’s disproportionately Asian.
- kroltan 5y agoGoogle's street directions also fail miserably with international road names. Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible. For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101". There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406". A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
- com2kid 5y agoGoogle still fails with American roads that have names in the local Natives American language. It has gotten better, but even in 2018 it should have been in- excusable to be mispronouncing names that badly.
- delecti 5y agoSeparating technical issues from social ones is the source of the problem. I don't think it'll be the source of the solution. One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things. Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
- carlmr 5y ago>I've never heard of a location with Roman numerals in the name. At least in Europe, extremely common. In Paris you have Boulevard Henry IV, in Lyon the street named after the same king. There's thousands of kings and queens and many more roads named after them. There's also schools, buildings and such named after them.
- dontreact 5y agoIf you know in advance that Primate vs. Human confusions are going to be a potential issue that upset a lot of people (say, because it already happened with a similar system at another company) there are certainly ways of trying to ensure that same confusion doesn't happen. Ways to actually fix the problem: 1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected. 2. Boost your training set by labeling more unlabeled images that are near this decision boundary. So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration. I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision. At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described. You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.