11 ms·

Algorithmic decision making and the cost of fairness (2017)

86 points by shawndumas 9y ago

candiodari 9y ago
If justice was truly blind (which is what I would consider fair) then the number of defendants incorrectly classified as dangerous would depend on the likelihood of all the groups they're part of committing dangerous crimes. You want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?
- rspeer 9y ago
  You could try reading this paper or any other paper on the topic.
- mc32 9y ago
  I believe France does not collect race information from its citizens (or perh inhabitants in general). If true, it'd be interesting for an outside observer[1] to see if the omission of race makes a difference. That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc. [1]Given they themselves don't officially collect race information.
  - halflings 9y ago
    France does not use any algorithmic decision making in its court system as far as I am aware, but like many people said here, in the end, the cause is not "race" itself but the correlations it comes with (e.g: in France, people that commit crimes are more likely to be from poor suburbs, which are predominantly populated by people whose parents came from North Africa/Subsaharan Africa).
    - bluecalm 9y ago
      I am not convinced. Including address sounds like a very bad idea as well. We don't want to punish people for being black just as much as we don't want to punish people for living in poor neighborhoods. It seems to me you should only include data you are ok to make decisions on. This will require collecting more information during court proceeding than you have in general statistics though.
      - mc32 9y ago
        But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping, in say a country with less regulation? Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
        goialoq 9y ago
        Geography is a good place to look for clues to causes, but a not good justification for locking people in jail without trial.
        bluecalm 9y ago
        >>But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping I don't want to punish people for living in polluted areas therefore I don't include that data in the training set. >>Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site. I am not ok with doing that therefore I don't include address in the data. Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there.
        mc32 9y ago
        >"Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there." I'm not sure I can reconcile that. 1. in the first option, we're not applying the consequences random people, we'd be affecting those already found to have committed a crime. 2. in the second option, if we think we'd be affecting innocent people in option 1. why do we think we would not be affecting innocent people (good drivers who happen to live among poor drivers) in options 2? That's to say, either both are right or both are wrong. I see the same effects in both.
- dspoka 9y ago
  So this idea seems intuitive at first but turns out to be one of the worst things to treat unfairness. There are several reasons for this from both technical and legal perspective. It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands. If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe. from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing. most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.
  - bluecalm 9y ago
    >>It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands. I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing. "You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with: "You are not getting parole because you willingly associated with people who committed crime".
  - candiodari 9y ago
    The necessary result is that you are no longer judged on your crimes, but on your genes. Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal. Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack. Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.
- kosmet 9y ago
  Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters. What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.
  - bluecalm 9y ago
    >>Then, the algorithm gives negative weight to locations where dangerous groups tend to live. Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"? >>What you should do is to keep the ratio of people classified as dangerous same among different groups. Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
    - Larrikin 9y ago
      >>Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished. Now imagine if people were unfairly punished based on something they were born with and not where they lived.
      - lagadu 9y ago
        The justice system is meant to treat every person using the exact same criteria, as long as it's statistically relevant. That people are born in places where crime is higher isn't a problem the justice can or should handle; it's completely outside its domain of responsibility.
  - taneq 9y ago
    > Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters. No, the algorithm just gives negative weight to locations where dangerous groups tend to live. You infer race from that measure.
- dspoka 9y ago
  Europe has also passed “right to explanation” to help with this issue which is great. unfortunately it seems they came up with it in isolation from researchers on this topic and went with the don't record it philosophy. here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf https://arxiv.org/pdf/1606.08813.pdf European Union regulations on algorithmic decision-making and a “right to explanation”
- goialoq 9y ago
  Because correlation != causation. Here's a very simple refutation: Is past arrest history a valid factor in the model? What if past arrest history is biased due to biased policing (Hint: it is, well proven) How about zip code?
  - TruthBeDum123 9y ago
    So you are going to argue with statistical probability? It is our best determining factor. Have any better ideas? If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC. Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.
sean_anandale 9y ago
So in sum: "analyzing data from Broward County, we find that optimizing for public safety yields stark racial disparities; conversely, satisfying past fairness definitions means releasing more high-risk defendants, adversely affecting public safety." In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.
- hateduser2 9y ago
  On a topic like this it’s important I think to address the elephant in the room. I don’t think this implies a genetic issue! I’m not an expert but from what I know this shouldn’t lead me to the conclusion that “black Americans are predisposed to violence” and, if you know approximately the same things I do, I think that’s probably fair for you too!
  - lagadu 9y ago
    But they are, according to virtually all the data that we have. However there's no indication at all that the cause is genetic but rather cultural in the US.
  - deleted 9y ago
    [deleted]
  - toomanybeersies 9y ago
    I don't think anyone here on HN thinks that it implies a genetic issue, but rather an issue caused by generations of discrimination and failed attempts to stop the cycle of poverty and crime that plagues many of these communities.
    - trowawee 9y ago
      I am 100% certain, based on 5 years of observation, that there is a significant portion of users on this site who absolutely believe that racial disparities are genetic and immutable.
thundergolfer 9y ago
Bernhard Scholkopf explored this at the hour mark in his 2017 ICML keynote. https://vimeo.com/238274659 https://vimeo.com/238274659 Was interesting, and worth a look.
avinium 9y ago
I've only had a chance to skim this, so maybe I misunderstood. That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race." In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B? Pretty odd concept of "fairness" if you ask me. Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.
- goialoq 9y ago
  > the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem. Why should I be jailed because other people with similar skin color to me committed crimes? Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?
  - avinium 9y ago
    I don't quite understand your point. Race is explicitly not a factor in individual pre-trial detention decisions. People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc. As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair. In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations. Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black. As I said, I don't think many people would agree with that definition of "fair." * I don't know exactly what the model inputs are, I just made these up for demonstration purposes.
crb002 9y ago
This hits home in Iowa. They just started using it to determine bail. https://businessrecord.com/Content/Law-Government/Law-Government/Article/Polk-County-courts-begin-using-pretrial-risk-assessments-/164/788/80952 https://businessrecord.com/Content/Law-Government/Law-Govern...
adjkant 9y ago
It seems like the problem is that any data that works statistically will not be fair to individuals who are intuitively exceptions to the general trend. It sounds like algorithms relying on statistics are pretty much guaranteed to fail the fairness test. It's the wrong tool for this problem I think. There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways. Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.
- quotemstr 9y ago
  > There are so many factors in morality that I think individually, case by case judgments are likely the best option. What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth. There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.
  - adrianN 9y ago
    While I agree with your general point, artificial neural networks and biological neural networks are sufficiently different that it's not really fair to compare them just because they happen to have the same name. A biological neuron does a lot more than computing some affine transformation of its inputs. Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.
femto 9y ago
Statistics tell you something significant about a group, but little of significance about an individual. Statistical measures should not be used to make decisions about individuals. Other names for doing so are: "Sacrificing the one for the many" or "Presumption of guilt."
jimmywanger 9y ago
There is no cost of fairness here. They're playing percentages. I'd rather have an algorithm make predictions than a human judge/prosecutor. If a certain group of people are disposed towards recidivism or not showing up for trial, it doesn't matter how much they've suffered in the past. That's a policy decision to reach out to them.
slavik81 9y ago
I just saw another article[1] on a study using the same data set. It concluded that the COMPAS algorithm is no more accurate than random people given the task on Mechanical Turk. Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much. [1]: https://www.economist.com/news/science-and-technology/21734986-short-answer-two-are-about-same-are-programs-better https://www.economist.com/news/science-and-technology/217349...
- noobermin 9y ago
  Off topic, I can't access the economist article, could you post a link to the study so I and/or others can read its abstract?
  - indubitable 9y ago
    http://advances.sciencemag.org/content/4/1/eaao5580.full http://advances.sciencemag.org/content/4/1/eaao5580.full Not paywalled.
- AnthonyMouse 9y ago
  > Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much. That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement. Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people. And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions. If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.
Houshalter 9y ago
Getting rid of algorithms means replacing them with humans. And humans are far worse. Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker. It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists. Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all. But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them. This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not. A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Also, A.I. ‘Bias’ Doesn’t Mean What Journalists Say it Means: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-journalists-want-you-to-think-it-means/ https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...
- 9y ago
extension 9y ago
Imprisoning people without trial, for predicted future crimes, is what's unfair. Letting an algorithm make the prediction instead of a judge only punctuates the injustice. What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color? These are exactly the things that justice is supposed to be blind to. I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.
- quotemstr 9y ago
  Is there any circumstance in which you would support detention before trial? If not, would you oppose the detention of a serial killer caught in the act? If you do support pre-trial detention in some cases, how do you distinguish these cases from those cases for which pre-trial detention is unjustified? Could such a decision scheme be "fair" in principle? What would make it fair?
  - lmm 9y ago
    You could apply pre-trial detention to anyone accused of a crime, not offer bail at all. The upper classes might start to care more about speedy trials and police harassment once it became impossible for them to buy their way out of pre-trial detention.
  - extension 9y ago
    I think detention before trial is always unfair in principle, but likely unavoidable in practice. I would like to see the issue acknowledged and taken more seriously, but it's a tricky problem and I have no easy solutions to offer. Practical mitigations may be the best we can do, which I'll grant may be expensive, non-trivial to implement, and allow more criminals to roam free. Here are some vague ideas off the cuff: * Base decisions only on things that would be relevant in a trial, like evidence and criminal history. * Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect. * Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle. * Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy. If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.
nabla9 9y ago
Assume that you are insurance company with access to customers social media profiles. You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself. What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence? Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice. In some other cases it might be justifiable.
- taneq 9y ago
  Don't they do precisely that, across a whole bunch of data? Not "friends on facebook" but location, habits, age, etc.
  - nabla9 9y ago
    Yes. That's the idea when managing insurance risk. The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.
- Houshalter 9y ago
  This is a different problem than what people are complaining about. In fact it may be the exact opposite problem in some ways. The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive. But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned. >Justice should be individualized or it's not justice. There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.
dblotsky 9y ago
Personal two cents: the actual maximally "fair" and maximally safe strategy is not to release anyone on bail. True positives are 100%, and false negatives are 0%. Beyond that, I agree with nabla9: > I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.

86 points by shawndumas 9y ago

If justice was truly blind (which is what I would consider fair) then the number of defendants incorrectly classified as dangerous would depend on the likelihood of all the groups they're part of committing dangerous crimes. You want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?

rspeer 9y ago
You could try reading this paper or any other paper on the topic.
mc32 9y ago
I believe France does not collect race information from its citizens (or perh inhabitants in general). If true, it'd be interesting for an outside observer[1] to see if the omission of race makes a difference. That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc. [1]Given they themselves don't officially collect race information.
- halflings 9y ago
  France does not use any algorithmic decision making in its court system as far as I am aware, but like many people said here, in the end, the cause is not "race" itself but the correlations it comes with (e.g: in France, people that commit crimes are more likely to be from poor suburbs, which are predominantly populated by people whose parents came from North Africa/Subsaharan Africa).
  - bluecalm 9y ago
    I am not convinced. Including address sounds like a very bad idea as well. We don't want to punish people for being black just as much as we don't want to punish people for living in poor neighborhoods. It seems to me you should only include data you are ok to make decisions on. This will require collecting more information during court proceeding than you have in general statistics though.
    - mc32 9y ago
      But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping, in say a country with less regulation? Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
      - goialoq 9y ago
        Geography is a good place to look for clues to causes, but a not good justification for locking people in jail without trial.
      - bluecalm 9y ago
        >>But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping I don't want to punish people for living in polluted areas therefore I don't include that data in the training set. >>Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site. I am not ok with doing that therefore I don't include address in the data. Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there.
        mc32 9y ago
        >"Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there." I'm not sure I can reconcile that. 1. in the first option, we're not applying the consequences random people, we'd be affecting those already found to have committed a crime. 2. in the second option, if we think we'd be affecting innocent people in option 1. why do we think we would not be affecting innocent people (good drivers who happen to live among poor drivers) in options 2? That's to say, either both are right or both are wrong. I see the same effects in both.
dspoka 9y ago
So this idea seems intuitive at first but turns out to be one of the worst things to treat unfairness. There are several reasons for this from both technical and legal perspective. It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands. If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe. from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing. most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.
- bluecalm 9y ago
  >>It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands. I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing. "You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with: "You are not getting parole because you willingly associated with people who committed crime".
- candiodari 9y ago
  The necessary result is that you are no longer judged on your crimes, but on your genes. Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal. Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack. Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.
kosmet 9y ago
Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters. What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.
- bluecalm 9y ago
  >>Then, the algorithm gives negative weight to locations where dangerous groups tend to live. Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"? >>What you should do is to keep the ratio of people classified as dangerous same among different groups. Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
  - Larrikin 9y ago
    >>Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished. Now imagine if people were unfairly punished based on something they were born with and not where they lived.
    - lagadu 9y ago
      The justice system is meant to treat every person using the exact same criteria, as long as it's statistically relevant. That people are born in places where crime is higher isn't a problem the justice can or should handle; it's completely outside its domain of responsibility.
- taneq 9y ago
  > Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters. No, the algorithm just gives negative weight to locations where dangerous groups tend to live. You infer race from that measure.
dspoka 9y ago
Europe has also passed “right to explanation” to help with this issue which is great. unfortunately it seems they came up with it in isolation from researchers on this topic and went with the don't record it philosophy. here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf https://arxiv.org/pdf/1606.08813.pdf European Union regulations on algorithmic decision-making and a “right to explanation”
goialoq 9y ago
Because correlation != causation. Here's a very simple refutation: Is past arrest history a valid factor in the model? What if past arrest history is biased due to biased policing (Hint: it is, well proven) How about zip code?
- TruthBeDum123 9y ago
  So you are going to argue with statistical probability? It is our best determining factor. Have any better ideas? If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC. Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.

sean_anandale 9y ago

So in sum: "analyzing data from Broward County, we find that optimizing for public safety yields stark racial disparities; conversely, satisfying past fairness definitions means releasing more high-risk defendants, adversely affecting public safety." In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.

hateduser2 9y ago
On a topic like this it’s important I think to address the elephant in the room. I don’t think this implies a genetic issue! I’m not an expert but from what I know this shouldn’t lead me to the conclusion that “black Americans are predisposed to violence” and, if you know approximately the same things I do, I think that’s probably fair for you too!
- lagadu 9y ago
  But they are, according to virtually all the data that we have. However there's no indication at all that the cause is genetic but rather cultural in the US.
- deleted 9y ago
  [deleted]
- toomanybeersies 9y ago
  I don't think anyone here on HN thinks that it implies a genetic issue, but rather an issue caused by generations of discrimination and failed attempts to stop the cycle of poverty and crime that plagues many of these communities.
  - trowawee 9y ago
    I am 100% certain, based on 5 years of observation, that there is a significant portion of users on this site who absolutely believe that racial disparities are genetic and immutable.

thundergolfer 9y ago

Bernhard Scholkopf explored this at the hour mark in his 2017 ICML keynote. https://vimeo.com/238274659 https://vimeo.com/238274659 Was interesting, and worth a look.

avinium 9y ago

I've only had a chance to skim this, so maybe I misunderstood. That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race." In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B? Pretty odd concept of "fairness" if you ask me. Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.

goialoq 9y ago
> the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem. Why should I be jailed because other people with similar skin color to me committed crimes? Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?
- avinium 9y ago
  I don't quite understand your point. Race is explicitly not a factor in individual pre-trial detention decisions. People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc. As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair. In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations. Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black. As I said, I don't think many people would agree with that definition of "fair." * I don't know exactly what the model inputs are, I just made these up for demonstration purposes.

crb002 9y ago

This hits home in Iowa. They just started using it to determine bail. https://businessrecord.com/Content/Law-Government/Law-Government/Article/Polk-County-courts-begin-using-pretrial-risk-assessments-/164/788/80952 https://businessrecord.com/Content/Law-Government/Law-Govern...

adjkant 9y ago

It seems like the problem is that any data that works statistically will not be fair to individuals who are intuitively exceptions to the general trend. It sounds like algorithms relying on statistics are pretty much guaranteed to fail the fairness test. It's the wrong tool for this problem I think. There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways. Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.

quotemstr 9y ago
> There are so many factors in morality that I think individually, case by case judgments are likely the best option. What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth. There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.
- adrianN 9y ago
  While I agree with your general point, artificial neural networks and biological neural networks are sufficiently different that it's not really fair to compare them just because they happen to have the same name. A biological neuron does a lot more than computing some affine transformation of its inputs. Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.

femto 9y ago

Statistics tell you something significant about a group, but little of significance about an individual. Statistical measures should not be used to make decisions about individuals. Other names for doing so are: "Sacrificing the one for the many" or "Presumption of guilt."

jimmywanger 9y ago

There is no cost of fairness here. They're playing percentages. I'd rather have an algorithm make predictions than a human judge/prosecutor. If a certain group of people are disposed towards recidivism or not showing up for trial, it doesn't matter how much they've suffered in the past. That's a policy decision to reach out to them.

slavik81 9y ago

I just saw another article[1] on a study using the same data set. It concluded that the COMPAS algorithm is no more accurate than random people given the task on Mechanical Turk. Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much. [1]: https://www.economist.com/news/science-and-technology/21734986-short-answer-two-are-about-same-are-programs-better https://www.economist.com/news/science-and-technology/217349...

noobermin 9y ago
Off topic, I can't access the economist article, could you post a link to the study so I and/or others can read its abstract?
- indubitable 9y ago
  http://advances.sciencemag.org/content/4/1/eaao5580.full http://advances.sciencemag.org/content/4/1/eaao5580.full Not paywalled.
AnthonyMouse 9y ago
> Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much. That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement. Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people. And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions. If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.

Houshalter 9y ago

Getting rid of algorithms means replacing them with humans. And humans are far worse. Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker. It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists. Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all. But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them. This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not. A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Also, A.I. ‘Bias’ Doesn’t Mean What Journalists Say it Means: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-journalists-want-you-to-think-it-means/ https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...

9y ago

extension 9y ago

Imprisoning people without trial, for predicted future crimes, is what's unfair. Letting an algorithm make the prediction instead of a judge only punctuates the injustice. What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color? These are exactly the things that justice is supposed to be blind to. I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.

quotemstr 9y ago
Is there any circumstance in which you would support detention before trial? If not, would you oppose the detention of a serial killer caught in the act? If you do support pre-trial detention in some cases, how do you distinguish these cases from those cases for which pre-trial detention is unjustified? Could such a decision scheme be "fair" in principle? What would make it fair?
- lmm 9y ago
  You could apply pre-trial detention to anyone accused of a crime, not offer bail at all. The upper classes might start to care more about speedy trials and police harassment once it became impossible for them to buy their way out of pre-trial detention.
- extension 9y ago
  I think detention before trial is always unfair in principle, but likely unavoidable in practice. I would like to see the issue acknowledged and taken more seriously, but it's a tricky problem and I have no easy solutions to offer. Practical mitigations may be the best we can do, which I'll grant may be expensive, non-trivial to implement, and allow more criminals to roam free. Here are some vague ideas off the cuff: * Base decisions only on things that would be relevant in a trial, like evidence and criminal history. * Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect. * Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle. * Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy. If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.

nabla9 9y ago

Assume that you are insurance company with access to customers social media profiles. You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself. What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence? Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice. In some other cases it might be justifiable.

taneq 9y ago
Don't they do precisely that, across a whole bunch of data? Not "friends on facebook" but location, habits, age, etc.
- nabla9 9y ago
  Yes. That's the idea when managing insurance risk. The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.
Houshalter 9y ago
This is a different problem than what people are complaining about. In fact it may be the exact opposite problem in some ways. The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive. But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned. >Justice should be individualized or it's not justice. There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.

dblotsky 9y ago

Personal two cents: the actual maximally "fair" and maximally safe strategy is not to release anyone on bail. True positives are 100%, and false negatives are 0%. Beyond that, I agree with nabla9: > I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.