7 ms·

Justice.exe: Bias in Algorithmic sentencing

43 points by ahakki 9y ago

Robelius 9y ago
Forgive my ignorance since I don't have a CS degree, but don't all algorithms used for selecting have a human behind the logic. So wouldn't an algorithm always have some bias based on the weights given by a human. Kind of like OKCupid saying you are a 75% match with someone. You may not be compatible with that person, but whoever designed the algorithm declared it to be true with their design. So wouldn't any algorithm always have some biased in it? Sorry if this comes off as a stupid question or is unclear.
- untog 9y ago
  It's not a stupid question at all, and it cuts to the core of a lot of problems with AI and algorithmic-based anything. Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.
  - gravypod 9y ago
    > Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group. Can you break this down more? It doesn't make sense to me. If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program. Unless the programmers are changing the datasets then I don't see how this makes sense.
    - theemathas 9y ago
      Well... why can't the dataset be biased?
      - amitdeshwar 9y ago
        That's not a bias of the programmer
        chongli 9y ago
        Unless the programmer chose the dataset.
    - woodruffw 9y ago
      In the case you presented, the biases are in the dataset itself. Arrest statistics in the US are heavily skewed by race. If you were to take a dataset of all US arrests between 1900 and 2000 and ask which populations are most likely to "commit crimes" (i.e., be arrested), you'd get racially biased model without recording why it's biased (discrimination, enduring poverty, minimum sentencing, &c).
      - dante821 9y ago
        Can't that problem be solved by simply not telling the machine the race of the person who's being judged?
      - mattnewton 9y ago
        Dead sibling comment asks a common questions I hear: why can't we just not tell the algorithm race? And the answer is that, with enough data points, if there is a statistical bias in the dataset, the algorithm will likely learn a substitute for race in the other features. For the sake of argument, perhaps an interaction between home address, annual income and model of car driven is predictive of race, and race is predictive of recidivism in the dataset- then the algorithm will learn this cross of features is predictive of recividism, even though we would like all of them to ideally be irrelevant to sentencing.
        gravypod 9y ago
        > home address, annual income and model of car driven is predictive Why the hell would those be factors? The factors of the case should be things that actually matter in the case. A rich person who murders someone should get the same sentence as a poor person who murders someone. A black person who murders someone should get the same sentence as a white person who murders someone. Adding those factors would be insane in the first place. If you're adding crazy things like that you might as well add factors like "Can Juggle" and "Can Burp the Alphabet" because things like that should have just as much to do with sentencing as what kind of car you drive or where you live.
        9y ago
    - dogecoinbase 9y ago
      How was the dataset produced?
    - deleted 9y ago
      [deleted]
    - randomtask 9y ago
      As with anything, it's hard to generalise without oversimplifying, but here goes. You don't generally just have a data set and a machine learning algorithm that somehow magics outputs from a data set. Usually decisions have to be made by people either in training the model, selecting variables that are included in a model, etc. Here's a simple example. Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know). We have to tell the algorithm whether a given article in the training set is fake or legitimate, otherwise how would it know? Clearly this will reflect the views of whoever is tagging the articles. When we run the model on a training set we need to score how well it did, again this will reflect the opinion of the person doing the scoring. For a real example: https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-.. https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-....
      - gravypod 9y ago
        I've done some machine learning in the past. I get how you tackle machine learning problems and the first thing that I would say that your proposed topic is not currently possible. "*Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know).*" Current AI cannot do this. This would take finding sources, pulling data out of those sources, cross referencing multiple sources, and recurring for those articles to a certain depth. That's not a good facsimile for deciding sentencing. Sentencing is more like a linear regression classification. You have a history of previous cases where the defendant was found guilty. You then have a pile of factors that played into the judge's decision for sentencing. For example: * If they meant to do it * If they feel bad about doing it * If they did do it (Beyond a reasonable doubt) * If they have done it before * What severity this crime is * ... etc The judge then uses their experience in law and previous case law as well as statues to find a proper punishment. This is in the form of: * Time served * Fines * Privileges revoked * Community Service This would then be fed into a classification engine. You leave all of the existing infrastructure in place (Judge, Jury, Lawers) and just use their decision as input into the sentencing. Deciding the validity of claims is not within the scope of modern day machine learning (as of 2017). Classification engines are very much in the scope of machine learning of today. I don't see how case factors could be biased. I don't see how historical cases (when stripped of all identifying information) could be biased. I don't see why a system like this would be bad. All treatment of everyone would converge into a uniform handling of cases.
- zitterbewegung 9y ago
  Not only the algorithm can be biased but also the underlying dataset could be biased. See http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender http://www.sciencemag.org/news/2017/04/even-artificial-intel... And the paper http://opus.bath.ac.uk/55288/ http://opus.bath.ac.uk/55288/
  - Hydraulix989 9y ago
    I can see how the dataset can be biased, but I'm struggling to find any biases in the algorithms themselves. Assuming I'm using some actual machine learning model and not my own hand-coded finite state machine, I just don't see how, say, k-means clustering could be biased (unless there was a bug in the programmer's implementation?).
    - hackuser 9y ago
      It will be intentionally biased; that I'm confident of. https://news.ycombinator.com/item?id=14288232 https://news.ycombinator.com/item?id=14288232
    - Jtsummers 9y ago
      There are two algorithms. Machine learning generates models based off of input data. Machine learning, principally, will not be [biased or] exhibit any direct bias. [It's math, on its own it's not the problem.] The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased. [EDIT: Forgot two words, added a sentence]
      - Hydraulix989 9y ago
        "There are two algorithms." In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g. forward prop), the situation is no different. Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" -- neither does, forward propagation. These algorithms are merely functions that are _parametrized_ by a potentially biased learned model (which really entirely amounts to having biased data). I recommend taking a look at Stanford's Intro to Machine Learning CS 229 course notes for a brief overview of these concepts. I can't speak anymore highly of the material in this course; it's by far the best out there, and it's how I learned ML. "The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased." Right, the "final model" (which comes directly from the data) will be biased, but not the ALGORITHM (the "math," as you call it). For example, I can't make an unbiased data set "biased" by using linear regression instead of logistic regression. It would be quite a tremendous discovery if you somehow figured out a way SGD in itself was biased, but the burden of proof on such an extraordinary claim would be on you. So like I said in my parent post, I can see how the data can be biased, but I'm struggling to find any biases in the algorithms themselves (assuming they are properly implemented and not an Underhanded C Contest [1] entry). Granted, if we're not actually using ML, and we're using something like a hardcoded decision tree or FSM instead (it was pretty strongly implied that we weren't though because we were talking about data-driven algorithms), then sure, that could very easily be a biased algorithm, but I've maintained that all along. [1] http://www.underhanded-c.org/ http://www.underhanded-c.org/
- hackuser 9y ago
  Ultimately, everything is politics, and I predict that if these algorithms become widely used, politics eventually will be the primary determining factor in their calculations. Why? If these algorithms become widely used and thus influence important outcomes, then people who like to control power will seek to obtain control of the algorithms, either to protect their interests or to grow their own influence. As possible examples, politicians who see an algorithm apolitically jail a powerful person might pass a law saying that public service should be weighed more heavily. Law enforcement might want exceptions or special weightings for police officer defendants, or for police officer-given testimony. Powerful social activists will buy the algorithm developers (do you know who is writing these algorithms even now? I don't) as a way of influencing society - the Koch brothers, for example, invest in other areas, such as academia and down-ballot elections (e.g., secretary of state in U.S. states) for society-changing purposes. Special code might even generate special outcomes for particular individuals - who will ever see the code? Who will even ask this question after the sentencing? If you think such corruption couldn't happen, look around. Brazen corruption happens all the time, especially in state and local government. This will be harder to detect, and will be covered by the excuse that non-technical people will believe: I didn't do it; it was the algorithm! It must be objective! I think the algorithms ultimately will be a step backward, reducing transparency by hiding the bias in a software development process, in the back room of a software firm, and in a mountain of code. At least legislation is published and voted on in the open.
Hydraulix989 9y ago
One of the things that computers really have going for in the case of "algorithmic sentencing" is that they are NOT biased; they are just machines that repeatedly and reliably run an ordered set of instructions X, their program, free of any human emotions that can override such logic. Algorithmic sentencing, when implemented correctly (i.e. the X that it runs is not biased), have the potential to be vastly more "fair" and "just" than any human jury could be. For example, from a widely circulated online video of a joint talk given by a lawyer and a cop at a law school: "the [Jury's] perception is that if you're sitting next to a defense attorney, YOU have to prove you're innocent" [1] Juries are comprised of people off the street who have no expertise in determining things like innocence. The average person is subject to all sorts of biases, most of which they aren't even consciously aware they have. Knowing this, if I ever had to stand for trial in front of a jury of my peers as a defendant, I'd be scared shitless, _especially_ if I was innocent. As you can see from the linked video, machines actually have a pretty low bar to live up to in order to outperform humans. [1] https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be&t=16m46s https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be...
- hackuser 9y ago
  People will write that software, select which software to purchase, write the rules for its use, and implement them. Powerful people aren't going to sit back and let those things happen in ways that they don't like, which are threats to them, or which reduce their influence.
  - Hydraulix989 9y ago
    Then we need the software to be implemented as a decentralized smart contract.
    - twblalock 9y ago
      How would that help? More specifically, how would implementing this as a smart contract prevent bias from making its way into the algorithms or the learning datasets? Remember that bias can be inserted accidentally, by well-meaning programmers who don't realize that the data they are using is biased or has other methodological problems.
twiss 9y ago
Small bit of feedback on the game: not being a law student, it took me a minute to figure out what I was supposed to do. From the mention of "Risk Assessment Program", I infer that I'm supposed to assume that everyone was found guilty, and guesstimate whether they are likely to commit more offenses in the future (and choose a sentence based on that). Is that correct?
Houshalter 9y ago
Humans are a thousand times worse than algorithms. Unattractive defendants get prison sentences twice as long as attractive ones. Judges give significantly harsher sentences just before lunch, when they are hungry. And of course the classic biases against race and gender and other protected classes. I bet there are many other biases that we haven't even discovered yet. Some studies have shown that job interviews are worse than just looking at resumes. Humans should never ever be in charge of sentencing decisions, or anything else you want to be "fair". There's nothing remotely fair about human judgement. One particularly weird bias humans have, is we strongly distrust machines. Psychologists have studied this and named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... Even when we know machines do better than humans, we are much more distrustful of them. We can watch humans make mistakes and an algorithm make many fewer mistakes, and still end up trusting the human far more. And it's weird, because in almost every domain that's been studied, algorithms do vastly better than humans. For any statistical problem where there are a few features available to predict the outcome, even simple linear regression typically beats human experts. We suck at probabilistic and statistical problems. Often the predictions of experts are barely better than chance. And this has been known for a long time. The earliest reference I found was from 1928! Where an incredibly crude statistical model (hand trained!) beat a group of prison psychologists, at predicting recidivism of convicts. Even the sources that inspired this game have been shown to be fraudulent. That is the propublica study they cite. See https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Propublica claimed their results were "almost statistically significant." I.e., they not statistically significant. And why would it be so? Algorithms have no particular reason to hate a minority group. They try to make the most accurate predictions possible given the data.
- dsacco 9y ago
  > They try to make the most accurate predictions possible given the data. So what you're saying is that a bias in the data could result in a biased outcome.
  - 9y ago
nrdwavexe 9y ago
this is bizarre in the extreme; out-of-control state censorship is absolutely WRECKING the USA right now and it is at GENOCIDAL levels. frankly we need to tell the jews to get the hell out of the USA and then we need to force the jews to live in fear no matter where they go. either we live in fear of them or they live in fear of us. living in fear of the jews is no different from being a slave of the jews. there can be no justice until the jews are forced to live in fear
tscs37 9y ago
There has been some interesting discussion on this thread already. However, what most seems to fail to see is that an algorithm compared to a human justice system, could inherently be less biased. Having no bias at all is hard but only having less bias is totally fine IMO. When writing the algorithm it is of course impossible to not instill some bias into the datasets, the code itself should be bias-free if written correctly. The end result is a bias-free judgement algorithm based on biased data. Is this worse than what we have now?
magwa101 9y ago
How about the danger of HUMAN sentencing, talk about lack of training, transparency and consistency.

43 points by ahakki 9y ago

Forgive my ignorance since I don't have a CS degree, but don't all algorithms used for selecting have a human behind the logic. So wouldn't an algorithm always have some bias based on the weights given by a human. Kind of like OKCupid saying you are a 75% match with someone. You may not be compatible with that person, but whoever designed the algorithm declared it to be true with their design. So wouldn't any algorithm always have some biased in it? Sorry if this comes off as a stupid question or is unclear.

untog 9y ago
It's not a stupid question at all, and it cuts to the core of a lot of problems with AI and algorithmic-based anything. Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group.
- gravypod 9y ago
  > Any sufficiently complex programming project will end up reflecting some assumptions and biases on the behalf of the programmer. As more programmers contribute to a project, it won't reflect an individual's biases, but will still reflect those programmers as a group. Can you break this down more? It doesn't make sense to me. If you're writing a machine learning application to take a dataset and match future inputs to past results I don't see how these biases can sneak into the program. Unless the programmers are changing the datasets then I don't see how this makes sense.
  - theemathas 9y ago
    Well... why can't the dataset be biased?
    - amitdeshwar 9y ago
      That's not a bias of the programmer
      - chongli 9y ago
        Unless the programmer chose the dataset.
  - woodruffw 9y ago
    In the case you presented, the biases are in the dataset itself. Arrest statistics in the US are heavily skewed by race. If you were to take a dataset of all US arrests between 1900 and 2000 and ask which populations are most likely to "commit crimes" (i.e., be arrested), you'd get racially biased model without recording why it's biased (discrimination, enduring poverty, minimum sentencing, &c).
    - dante821 9y ago
      Can't that problem be solved by simply not telling the machine the race of the person who's being judged?
    - mattnewton 9y ago
      Dead sibling comment asks a common questions I hear: why can't we just not tell the algorithm race? And the answer is that, with enough data points, if there is a statistical bias in the dataset, the algorithm will likely learn a substitute for race in the other features. For the sake of argument, perhaps an interaction between home address, annual income and model of car driven is predictive of race, and race is predictive of recidivism in the dataset- then the algorithm will learn this cross of features is predictive of recividism, even though we would like all of them to ideally be irrelevant to sentencing.
      - gravypod 9y ago
        > home address, annual income and model of car driven is predictive Why the hell would those be factors? The factors of the case should be things that actually matter in the case. A rich person who murders someone should get the same sentence as a poor person who murders someone. A black person who murders someone should get the same sentence as a white person who murders someone. Adding those factors would be insane in the first place. If you're adding crazy things like that you might as well add factors like "Can Juggle" and "Can Burp the Alphabet" because things like that should have just as much to do with sentencing as what kind of car you drive or where you live.
        9y ago
  - dogecoinbase 9y ago
    How was the dataset produced?
  - deleted 9y ago
    [deleted]
  - randomtask 9y ago
    As with anything, it's hard to generalise without oversimplifying, but here goes. You don't generally just have a data set and a machine learning algorithm that somehow magics outputs from a data set. Usually decisions have to be made by people either in training the model, selecting variables that are included in a model, etc. Here's a simple example. Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know). We have to tell the algorithm whether a given article in the training set is fake or legitimate, otherwise how would it know? Clearly this will reflect the views of whoever is tagging the articles. When we run the model on a training set we need to score how well it did, again this will reflect the opinion of the person doing the scoring. For a real example: https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-.. https://mathbabe.org/2016/05/12/algorithms-are-as-biased-as-....
    - gravypod 9y ago
      I've done some machine learning in the past. I get how you tackle machine learning problems and the first thing that I would say that your proposed topic is not currently possible. "*Say you're trying to come up with an algorithm that decides whether articles in a data set are "fake news" (topical, I know).*" Current AI cannot do this. This would take finding sources, pulling data out of those sources, cross referencing multiple sources, and recurring for those articles to a certain depth. That's not a good facsimile for deciding sentencing. Sentencing is more like a linear regression classification. You have a history of previous cases where the defendant was found guilty. You then have a pile of factors that played into the judge's decision for sentencing. For example: * If they meant to do it * If they feel bad about doing it * If they did do it (Beyond a reasonable doubt) * If they have done it before * What severity this crime is * ... etc The judge then uses their experience in law and previous case law as well as statues to find a proper punishment. This is in the form of: * Time served * Fines * Privileges revoked * Community Service This would then be fed into a classification engine. You leave all of the existing infrastructure in place (Judge, Jury, Lawers) and just use their decision as input into the sentencing. Deciding the validity of claims is not within the scope of modern day machine learning (as of 2017). Classification engines are very much in the scope of machine learning of today. I don't see how case factors could be biased. I don't see how historical cases (when stripped of all identifying information) could be biased. I don't see why a system like this would be bad. All treatment of everyone would converge into a uniform handling of cases.
zitterbewegung 9y ago
Not only the algorithm can be biased but also the underlying dataset could be biased. See http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender http://www.sciencemag.org/news/2017/04/even-artificial-intel... And the paper http://opus.bath.ac.uk/55288/ http://opus.bath.ac.uk/55288/
- Hydraulix989 9y ago
  I can see how the dataset can be biased, but I'm struggling to find any biases in the algorithms themselves. Assuming I'm using some actual machine learning model and not my own hand-coded finite state machine, I just don't see how, say, k-means clustering could be biased (unless there was a bug in the programmer's implementation?).
  - hackuser 9y ago
    It will be intentionally biased; that I'm confident of. https://news.ycombinator.com/item?id=14288232 https://news.ycombinator.com/item?id=14288232
  - Jtsummers 9y ago
    There are two algorithms. Machine learning generates models based off of input data. Machine learning, principally, will not be [biased or] exhibit any direct bias. [It's math, on its own it's not the problem.] The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased. [EDIT: Forgot two words, added a sentence]
    - Hydraulix989 9y ago
      "There are two algorithms." In the case of machine learning techniques that employ a learning algorithm (e.g. SGD) and a separate classification algorithm (e.g. forward prop), the situation is no different. Gradient descent in itself doesn't take "unbiased" data and pollute it with "bias" -- neither does, forward propagation. These algorithms are merely functions that are _parametrized_ by a potentially biased learned model (which really entirely amounts to having biased data). I recommend taking a look at Stanford's Intro to Machine Learning CS 229 course notes for a brief overview of these concepts. I can't speak anymore highly of the material in this course; it's by far the best out there, and it's how I learned ML. "The generated model is then incorporated into an algorithm for making decisions or categorizing objects. If the original data contains biases, the final model will be biased." Right, the "final model" (which comes directly from the data) will be biased, but not the ALGORITHM (the "math," as you call it). For example, I can't make an unbiased data set "biased" by using linear regression instead of logistic regression. It would be quite a tremendous discovery if you somehow figured out a way SGD in itself was biased, but the burden of proof on such an extraordinary claim would be on you. So like I said in my parent post, I can see how the data can be biased, but I'm struggling to find any biases in the algorithms themselves (assuming they are properly implemented and not an Underhanded C Contest [1] entry). Granted, if we're not actually using ML, and we're using something like a hardcoded decision tree or FSM instead (it was pretty strongly implied that we weren't though because we were talking about data-driven algorithms), then sure, that could very easily be a biased algorithm, but I've maintained that all along. [1] http://www.underhanded-c.org/ http://www.underhanded-c.org/
hackuser 9y ago
Ultimately, everything is politics, and I predict that if these algorithms become widely used, politics eventually will be the primary determining factor in their calculations. Why? If these algorithms become widely used and thus influence important outcomes, then people who like to control power will seek to obtain control of the algorithms, either to protect their interests or to grow their own influence. As possible examples, politicians who see an algorithm apolitically jail a powerful person might pass a law saying that public service should be weighed more heavily. Law enforcement might want exceptions or special weightings for police officer defendants, or for police officer-given testimony. Powerful social activists will buy the algorithm developers (do you know who is writing these algorithms even now? I don't) as a way of influencing society - the Koch brothers, for example, invest in other areas, such as academia and down-ballot elections (e.g., secretary of state in U.S. states) for society-changing purposes. Special code might even generate special outcomes for particular individuals - who will ever see the code? Who will even ask this question after the sentencing? If you think such corruption couldn't happen, look around. Brazen corruption happens all the time, especially in state and local government. This will be harder to detect, and will be covered by the excuse that non-technical people will believe: I didn't do it; it was the algorithm! It must be objective! I think the algorithms ultimately will be a step backward, reducing transparency by hiding the bias in a software development process, in the back room of a software firm, and in a mountain of code. At least legislation is published and voted on in the open.

Hydraulix989 9y ago

One of the things that computers really have going for in the case of "algorithmic sentencing" is that they are NOT biased; they are just machines that repeatedly and reliably run an ordered set of instructions X, their program, free of any human emotions that can override such logic. Algorithmic sentencing, when implemented correctly (i.e. the X that it runs is not biased), have the potential to be vastly more "fair" and "just" than any human jury could be. For example, from a widely circulated online video of a joint talk given by a lawyer and a cop at a law school: "the [Jury's] perception is that if you're sitting next to a defense attorney, YOU have to prove you're innocent" [1] Juries are comprised of people off the street who have no expertise in determining things like innocence. The average person is subject to all sorts of biases, most of which they aren't even consciously aware they have. Knowing this, if I ever had to stand for trial in front of a jury of my peers as a defendant, I'd be scared shitless, _especially_ if I was innocent. As you can see from the linked video, machines actually have a pretty low bar to live up to in order to outperform humans. [1] https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be&t=16m46s https://www.youtube.com/watch?v=08fZQWjDVKE&feature=youtu.be...

hackuser 9y ago
People will write that software, select which software to purchase, write the rules for its use, and implement them. Powerful people aren't going to sit back and let those things happen in ways that they don't like, which are threats to them, or which reduce their influence.
- Hydraulix989 9y ago
  Then we need the software to be implemented as a decentralized smart contract.
  - twblalock 9y ago
    How would that help? More specifically, how would implementing this as a smart contract prevent bias from making its way into the algorithms or the learning datasets? Remember that bias can be inserted accidentally, by well-meaning programmers who don't realize that the data they are using is biased or has other methodological problems.

twiss 9y ago

Small bit of feedback on the game: not being a law student, it took me a minute to figure out what I was supposed to do. From the mention of "Risk Assessment Program", I infer that I'm supposed to assume that everyone was found guilty, and guesstimate whether they are likely to commit more offenses in the future (and choose a sentence based on that). Is that correct?

Houshalter 9y ago

Humans are a thousand times worse than algorithms. Unattractive defendants get prison sentences twice as long as attractive ones. Judges give significantly harsher sentences just before lunch, when they are hungry. And of course the classic biases against race and gender and other protected classes. I bet there are many other biases that we haven't even discovered yet. Some studies have shown that job interviews are worse than just looking at resumes. Humans should never ever be in charge of sentencing decisions, or anything else you want to be "fair". There's nothing remotely fair about human judgement. One particularly weird bias humans have, is we strongly distrust machines. Psychologists have studied this and named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... Even when we know machines do better than humans, we are much more distrustful of them. We can watch humans make mistakes and an algorithm make many fewer mistakes, and still end up trusting the human far more. And it's weird, because in almost every domain that's been studied, algorithms do vastly better than humans. For any statistical problem where there are a few features available to predict the outcome, even simple linear regression typically beats human experts. We suck at probabilistic and statistical problems. Often the predictions of experts are barely better than chance. And this has been known for a long time. The earliest reference I found was from 1928! Where an incredibly crude statistical model (hand trained!) beat a group of prison psychologists, at predicting recidivism of convicts. Even the sources that inspired this game have been shown to be fraudulent. That is the propublica study they cite. See https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying.... Propublica claimed their results were "almost statistically significant." I.e., they not statistically significant. And why would it be so? Algorithms have no particular reason to hate a minority group. They try to make the most accurate predictions possible given the data.

dsacco 9y ago
> They try to make the most accurate predictions possible given the data. So what you're saying is that a bias in the data could result in a biased outcome.
- 9y ago

nrdwavexe 9y ago

this is bizarre in the extreme; out-of-control state censorship is absolutely WRECKING the USA right now and it is at GENOCIDAL levels. frankly we need to tell the jews to get the hell out of the USA and then we need to force the jews to live in fear no matter where they go. either we live in fear of them or they live in fear of us. living in fear of the jews is no different from being a slave of the jews. there can be no justice until the jews are forced to live in fear

tscs37 9y ago

There has been some interesting discussion on this thread already. However, what most seems to fail to see is that an algorithm compared to a human justice system, could inherently be less biased. Having no bias at all is hard but only having less bias is totally fine IMO. When writing the algorithm it is of course impossible to not instill some bias into the datasets, the code itself should be bias-free if written correctly. The end result is a bias-free judgement algorithm based on biased data. Is this worse than what we have now?

magwa101 9y ago

How about the danger of HUMAN sentencing, talk about lack of training, transparency and consistency.