12 ms·
How we use big data can reinforce our worst biases, or help fix them
- merlincorey 9y agoNo, they're not[0]. Correlation is not causation, and this article makes many such correlations. It also appeals heavily to fairness. Business is not about fairness, it is about profit. I don't see the potential profits in racism - if it is possible to provide services to people profitably, it will be done. Arbitrary clusterings such as zip codes, while certainly statistically significant, do not translate to intent by algorithms or algorithm makers. Fairness is somewhat subjective, and I think it's fair for companies to pursue profits such as "same day shipping", even if that cannot be simultaneously rolled out to all markets. If it is profitable, eventually, it will be. [0] Invoking Betteridge's Law of Headlines ( https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline... )
- lallysingh 9y ago"At the center of Northpointe’s argument was an essentialist fallacy: Because persons that police classified as Afro-Americans were re-arrested more often in the training dataset, they claimed, COMPAS is justified in predicting that other persons classified as Afro-Americans by police—even in a different city, state, and time period—are more likely to be re-arrested. The cycling of classification into data then back into classification echoes W.E.B. Dubois’ 1923 definition, “the black man is the man who has to ride Jim Crow in Georgia.”" Sounds like the answer is yes.
- Ygg2 9y agoThen I would not say they are building new infrastructure of racism. They are getting garbage input and returning garbage output.
- Cthulhu_ 9y agoThe accusation here is that they're maintaining the status quo, in a way, instead of working on making things better.
- Ygg2 9y agoTrue, but the research they are doing - "improving" language to reduce bias, reeks to me of 1984's Doublespeak. What are the chances this won't be co-opted by the other side, to enhance divisions?
- mkane848 9y agoYou're right, let's just do nothing and let the problem get even worse as we push blame onto an algorithm rather than deeply ingrained societal issues
- Ygg2 9y agoI think you are misinterpreting my statement. I never said blame algorithm. I just said, what happens if algorithm (for eliminating some thought patterns using language inference) is used by the other side?
- aninhumer 9y agoThe problem which the quote is pointing to here, is that then people use the old garbage to justify the new reprocessed garbage and vice versa, and it becomes harder for people to realise that it's garbage.
- merlincorey 9y agoHow does that answer the question as an exclusive yes without any room for no?
- throwawayReply 9y ago> Correlation is not causation That's particularly ironic, since a lot of ML can treat it that way.
- jacobinofnone 9y ago"... do not translate to intent ..." intention is orthogonal to structural bias from the last section of the article: "infrastructure beats intent"
- shaki-dora 9y agoWhat a tired, lazy cliché... Why don’t you engage with the content of the article: But when COMPAS’ prediction was inaccurate (either predicting re-arrest when none happened, or not predicting an actual re-arrest) it routinely underestimated the probability of white recidivism and over-estimated the probability of black recidivism. In other words, it contained a bias hidden from the perspective of one set of statistics, but plainly visible in another.. Edit, because you noticed how unhelpful “ha ha, Betteridge” by itself is: intent may not be necessary for discrimination, nor can you deny intent when you continue to use methods shown to produce discriminatory outcomes. A profit motive is similarly useless as a defense of discrimination. Maybe young, white, heterosexual female employees are better for sales, because you operate in a xeno-, homo-, gymno-, gerontophobic neigbourhood. But it’s no excuse to limit your job ads to that demographic. One reason is that such logic perpetuates its own causes: if the police is more likely to stop&frisk non-whites, you will have fewer whites in your arrest statistics, even if they commit the same number of crimes. To then turn around and use those statistics to justify your racist police procedures is both morally and scientifically bankrupt.
- merlincorey 9y agoI feel like intent is necessary for racial discrimination. To discriminate between racial discrimination and mere ignorance or ineptitude one must look at the intent.
- aalleavitch 9y agoWhy would that matter if the outcome is the same? People are being disadvantaged solely on the basis of race, whether this is being done intentionally or not it's harmful and needs to be rectified.
- shaki-dora 9y agoWhy should intent matter? These are not criminal cases, but civil. If an employer is discriminating against a group because of unconscious biases, or using methods that lead to Discriminstory outcomes, there are real people who were harmed. Civil law isn’t about morality. It’s about righting a wrong. If you die because a mechanic negligently forgot to reconnect your car’s brake lines, you (and your family) will be harmed. The question for the courts then becomes shifting that harm to the party responsible, as far as that is possible, and your family will likely be awarded damages.
- perfmode 9y agoYou use scientific buzzwords, but you’re coming into the conversation with unscientific assumptions.
- sgift 9y ago> Business is not about fairness, it is about profit. I don't see the potential profits in racism - if it is possible to provide services to people profitably, it will be done. That assumes companies are completely logical, singular entities which all follow the same motivation without deviation. As companies are the sum of the humans working there with all their personal problems, faults, opinions and attitudes (plus the organizational framework itself) that's rather unlikely. It also assumes companies always know beforehand what the best strategy is to make more profit (just look at VW for a recent example where that didn't pan out).
- EliRivers 9y agoBusiness is not about fairness, it is about profit. Yet historically, a number of businesses in various parts chose not to serve certain sections of the population, and sometimes still do (off the top of my head, wedding cake companies choosing not to sell to gay couples).
- saas_co_de 9y agoThis is a crucial question, especially anywhere that algorithms will be applied to areas where there are laws against discrimination. If you want to use algorithms in these areas you will eventually have to prove that they don't illegally discriminate.
- fein 9y agoIt is impossible for an algorithm to "illegally" discriminate, as it's a machine devoid of the human concept of "intent"; It doesn't know what "race" even means. So this narrows to these basic outcomes if we assume any given algorithm isn't purpose built to always have a discriminatory output: 1. The input data is junk data, so the output is junk data. 2. The input data is fine and humans don't like the output for human reasons. The problem I'm hitting on here is that if we assume any sort of "discrimination" based on a data set is "bad", then we don't really care what the outcome of a given algo is, and we're just trying to massage data sets into an "acceptable" output. Seems like the opposite of good science to just throw away results that we don't like, because reasons.
- CPLX 9y agoThis is nonsense. It's quite possible for an algorithm that doesn't know what race is to be biased despite ignoring race if the real world is full of racism, and that racism manifests in dependent variables that the algorithm bases its decision on. This concept isn't even a little novel. Google the phrase "previous servitude" for an example of just such a dependent variable in action.
- fein 9y agoIn this case, you seem to be implying garbage in/ garbage out. My point is that when you get a good input dataset and there is still some sort of bias in the output, you have to start asking questions on why the results are biased towards one group by examining the variables pertaining to that group. The position you're starting with seems to be that the output should never be biased, which means you're not really approaching this scientifically and won't stop messing with input sets until you get the output you agree with; not the output which is necessarily true but you disagree with for personal reasons. You can't just claim "bad data" every time you get a result that doesn't jive with your politics.
- pornel 9y ago> Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It's a clean, mathematical apparatus that gives the status quo the aura of logical inevitability. The numbers don't lie. http://idlewords.com/talks/sase_panel.htm http://idlewords.com/talks/sase_panel.htm
- fiatpandas 9y ago>Machine learning is like money laundering for bias A potent analogy. The cycle is truly terrifying: 1. deep institutional racism compromises training data 2. data classified through that lens 3. ML insights and patterns inform institutional policy 4. goto 1
- icebraining 9y agoThe book Weapons of Math Destruction raises similar questions - not just perpetuating racism, but also other discrimination (such as against the poor). There was a previous discussion on HN regarding an interview with the author: https://news.ycombinator.com/item?id=12642432 https://news.ycombinator.com/item?id=12642432
- jakecopp 9y agoThe writer of this book did a great 99 Percent Invisible interview, well worth the 15 minutes. https://99percentinvisible.org/episode/the-age-of-the-algorithm/ https://99percentinvisible.org/episode/the-age-of-the-algori...
- cm2187 9y agoThe problem is that algorithms are like a 5y old child. They know no such thing as politeness, political correctness or politically charged environment unless that is implemented with additional code. So they will do the equivalent of a 5y old child saying "you smell bad" to the old aunt, which might be true nevertheless. The question then becomes should we write politically correct code or bias data so that it doesn't offend our sensitivity? Or on the other hand should we not anthropomorphize a mere algorithm and not interpret as racist what is the mechanical reflection of our own problems (in the article example, poor neighborhoods in the US don't order much on amazon, poor neighborhoods tend to be non white, non white neighborhoods end up being less served by amazon's algorithms).
- carlmcqueen 9y agoAlso, just like a 5 year old child, the bias of the parents is coded into the algorithm. So the 'bad smell' of the old aunt may not be bad to many people's standard of smell but the algorithm still identified it as bad.
- bobcostas55 9y agoThis same article keeps popping up about once a week, with the words slightly switched around and in a different magazine. A couple of weeks ago it was the NYT (https://www.nytimes.com/2017/12/02/opinion/sunday/intelligent-policing-and-my-innocent-children.html https://www.nytimes.com/2017/12/02/opinion/sunday/intelligen...), before that in the Atlantic, Politico, ProPublica, etc. The error they make is always the same. I highly recommend this piece: AI 'Bias' Doesn't Mean What Journalists Say it Means https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-journalists-want-you-to-think-it-means/ https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...
- fein 9y agoSurprised at the downvotes you're getting. That's the first time I've ever seen that jacobitemag article and it was a very interesting read.
- ufo 9y agoThe majority of that article seems to report on the same stories as the Nautilus article. The meat of the disagreement is in the conclusion section. > However, there is a significant cost to forcing algorithm outputs to reflect wishful thinking. If we are issuing loans, we will issue more loans to people who do not pay them back. If we are making parole/sentencing decisions for convicted criminals, we will release more dangerous criminals who go on to commit new crimes. > This may not be a concern for the journalist, but it should be a concern for the rest of us. I agree that this is the heart of the issue but by leaving this to the conclusion, I think the author didn't end up arguing for why we should optimize for objective cost instead of for our ideal reality. It might be financially profitable for a bank to use a racist lending policy but as a society we think that it is wrong so we passed regulations forbidding the banks to do so. Similarly, the criminal justice system is supposed to serve society and follow our ideal sense of justice and shouldnt be blindly optimizing recidivism rate. As the Nautilus article said, "What gets chosen is usually whatever is easiest to quantify, rather than the fairest".
- watashida 9y agoI wish we stopped looking downstream, at this incident level data (arrests, recidivism, criminality in general) and more upstream. How does criminality play out as a function of the make up congress, historical generations, political leans of the administration, etc, etc?
- ponderatul 9y agoFinally someone addressing real issues.
- grawprog 9y agoComputers aren't racist, algorithms aren't racist, data aren't racist. Computers are inanimate objects, algorithms are instructions and data are bytes. None of those things can possibly be racist.
- ufo 9y agoThis comment is missing the point. The problem is that the algorithms can reinforce the disadvantaged situation of some groups of people, instead of help improve it. And the thin veneer of objectivity that the algorithms provide can legitimize something that might be effectively a racist policy.
- grawprog 9y agoThen they're shitty algorithms that don't work properly or your data is incomplete. If your data is showing biases it's because they exist in the world not because the data is racist.
- ufo 9y agoYes, the problem has always been about what we do with the data instead of the data itself. The issue now is that people currently put a lot of blind trust on machine learning, even if the input data is garbage or the output doesn't have the intended result.
- grawprog 9y agoNot just machine learning. I remember learning in ecology that if the computer models didn't match with reality. Reality was probably wrong the model just needed better variables. I remember thinking how incredibly stupid that was at the time. Now I realize that's the way it works with everything and has for a while. The problem with modelling real world systems is there are too many variables and we have no way of knowing just how many their are and what effects they all actually have on a system. We can never make a complete model yet we act and make decisions as a society as though our models are complete and infallible.
- cJ0th 9y ago/offtopic Are Algorithms Building the New Infrastructure of Racism? Possibly. But what I find even more worrying: One day algorithms may take so many factors into account that they can discriminate against minorities so small that they neither have a name nor a voice. If an algorithm like that affects you your life could turn quite Kafkaesque.
- NPV_Computers 9y agoWow, slick. Looks like a really well put together kit. Is he happy with the manufacturing quality? How much allowances is there for variations in router diameter?
- justaman 9y agoI will not deny that Israel's violence. It seems you are ignoring Palestine's violence. When is violence just violence. When is it racism?
- ivanbakel 9y agoNautilus is a US website. At the same time, to imagine that every Jewish person is somehow required to criticise Israel with every second breath - no matter who or what else is actually being discussed - echoes the "Jew second" obsession of anti-semitic nationalists. Is it possible that the writer is Jewish and has no interest in writing about Israel in the article, accusingly or defensively?
- onlyfacts___ 9y agoif your first thought to anyone questioning the BS of this article is to start crying about anti-semitism, im guessing you harbor some racist biases inside as well
- Houshalter 9y agoNo, they aren't. They are the one chance we have to escape it. You see humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker. It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. Way back in the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists. Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all. But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". (http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...) The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them. This is why they aren't everywhere already. The last thing we need is fear mongering articles like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not. A big part of the case this article makes is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant, which is a pretty low bar to begin with. And yet it always comes up as the main reference in these discussions. It's the only piece of evidence they can find of this nonissue ever happening. Stop. You are just making the problems you claim to care about worse.
- ivanbakel 9y ago
- turc1656 9y agoThis article is catnip for the HN community. Talk of algorithms and institutional bias? And a way to improve humanity using Silicon Valley machine learning technology? Many of you probably needed a change of pants. This article is trying real hard to find victim's anywhere and everywhere. Not that long ago there were many people talking about how wonderful the future was going to be with algorithms and big data to make unbiased decisions. Now we are basically there and they still find a problem with it because of "calibration issues" which were mentioned to be due to mathematical incompatibility. You'll also note the wording about the improved algorithm being "independent" of race. Not that the results were more fair or accurate, just "independent". Then there's the ever popular gender pronouns that were mentioned. Guess what. In nearly every country in the world doctors are majority male and nurses are majority female. Funny that we never hear these "gender biases" mentioned with things like "___ is a construction worker" or "___ is a bus driver", isn't it? Those are male dominated fields as well yet no one ever talks about gender imbalance or discrimination. It's only ever with the higher status/income jobs that you see this talk of imbalance. Many of these aren't "biases" but rather statistical probabilities.