3 ms·

I know a lot of people here object to systems like these, but I think they're really great, at least in principle. It's certainly true that early iterations wil

by darawk 6y ago

I know a lot of people here object to systems like these, but I think they're really great, at least in principle. It's certainly true that early iterations will have problem of bias, and that's bad. But ultimately having an explicit model allows us to actually interrogate and correct those biases in a permanent way. You can't do that with a subjective system. Having an actual model, even if it starts out biased, that we can actually iterate on and tweak empirically seems like an invaluable step towards solving systemic racial and other forms of inequality in our justice system.

mola 6y ago
I think you are wrong. 1. From a technical point of view, The mere fact of a single model responsible for all decisions is bad. The inherent variance of judges opinions in concrete cases is the best way to fight bias. Bias by definition means less variance. Consolidating all decision making will tremendously worsen the bias. 2. From a moral point of view, law is made by people for people. It is a convention and ritual which gets it's moral validity from it's connection with tradition. We can hardly quantify these things let alone incorporate them to an ML algorithm. We ensure these by having people study for years, take bar exams, and through apprentenship programs. Do you really believe the state of the art in ML can come close to this? As a practitioner of ML, I know we can at most replace human activities that usually don't require any training, and are done by a human in a few seconds. 3. ML is really crappy. But is hyped as fact based and scientific. This is dangerous, a judge seeing a prediction by an algo is forced to comply with the algo, otherwise he'll be scrutinized for dismissing objective facts.
- darawk 6y ago
  > 1. From a technical point of view, The mere fact of a single model responsible for all decisions is bad. The inherent variance of judges opinions in concrete cases is the best way to fight bias. Bias by definition means less variance. Consolidating all decision making will tremendously worsen the bia Who said there has to be one model? Bias also definitely does not mean less variance. If I were to try to flesh out the argument I think you're trying to make, it'd go something like this: Judges have uncorrelated biases, and that low correlation between their biases ultimately results in a lower bias than a unified model. However, whether or not that's actually true in practice hinges on two important things: How uncorrelated those biases actually are, and their relative magnitudes to those of the unified model. > 2. From a moral point of view, law is made by people for people. It is a convention and ritual which gets it's moral validity from it's connection with tradition. We can hardly quantify these things let alone incorporate them to an ML algorithm. We ensure these by having people study for years, take bar exams, and through apprentenship programs. Do you really believe the state of the art in ML can come close to this? As a practitioner of ML, I know we can at most replace human activities that usually don't require any training, and are done by a human in a few seconds. I don't believe that ML is at the point where it should be the sole arbiter of these things, no. But I do believe that a properly calibrated model can be a very useful guide to these sorts of decisions. > 3. ML is really crappy. But is hyped as fact based and scientific. This is dangerous, a judge seeing a prediction by an algo is forced to comply with the algo, otherwise he'll be scrutinized for dismissing objective facts. This is just an education problem though.
dan-robertson 6y ago
The problem is that the system isn’t falsifiable. If the model says someone won’t turn up to court they are locked in cells until their trial, how will you know whether it was right or not? If you randomly release people as a test of the model then this is firstly unfair but secondarily won’t necessarily reveal small scale problems with the model that only affect certain groups of people because the sample will be small. This also ignores the large impact that going to jail tends to have on people’s lives. I don’t think it is reasonable to simply dismiss the likelihood of bias.
- AnHonestComment 6y ago
  There’s certainly parts that are hard to test, but it’s incorrect to say it’s unfalsifiable as a whole. For example, court attendance isn’t a Boolean: we create a confidence score and decide the “yes/no” question via some threshold. Even if you can’t test the model on the people you decide to keep in custody, you can test it on the people you release. The percentage who do appear for trial vs running should match your confidence score in that cohort — ie, if you guess they show up 95% of the time, then they should show up 95% of the time.
- darawk 6y ago
  For any one instance? You can't. But in aggregate, we do it all the time: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diver...
Jabbles 6y ago
Surely it would be better to concentrate on the existing sources of bias, e.g. the judges? This would presumably carry over to their other duties. Perhaps this model could be used to generate a set of rules that were clear and less open to interpretation by judges, and in that way it could be valuable.
- darawk 6y ago
  How are we going to eliminate the human bias of judges?
TuringNYC 6y ago
What about negative loops. If (when) the system consistently makes bad decisions, and people react negatively, the system will re-enforce those bad decisions with newfound evidence it was correct. Seems like the story of law enforcement in this country, except with the shine of a "system" to do it which will be trusted as unbiased.
pavel_lishin 6y ago
> But ultimately having an explicit model allows us to actually interrogate and correct those biases in a permanent way. Who is "us"? And does it? Does the firm writing the software actually allow this? Plenty of software sold to the government comes with contractual stipulations that it may not be reverse-engineered, inspected, or otherwise second-guessed. Besides, principles shminciples. The way to judge these systems is by the actual impact they have. Will this result in a more just system? If not, we can put the principles right into /dev/null. The article does seem to address this: > Among the debates was how to balance accuracy and fairness, said Ojmarrh Mitchell, a professor at Arizona State University who served on the panel. An accurate algorithm would do a good job of predicting whether defendants showed up, and thus whether to recommend release, Mr. Mitchell said, and a fair algorithm wouldn’t result in more release recommendations for white defendants than for others. > More than a year of tinkering ensued. To decrease the differences in outcomes for different races and ethnicities, the researchers excluded data on low-level marijuana offenses and “theft of service,” mainly subway turnstile jumping. Removing fare beating and marijuana arrests lowered the racial disparity in the tool by 0.4%, essentially making it slightly fairer but a little less accurate. > Data released this month show the new tool’s recommendations didn’t have such racial disparities. From Nov. 12 through March 17, the algorithm recommended releasing without conditions 83.9% of Blacks, 83.5% of whites and 85.8% of Hispanics. Defendants with higher scores returned to court more often than those with lower scores, showing the algorithm seemingly made accurate predictions. ...Albeit with mixed results. You can't avoid bias. You can just choose whether you want to bias towards justice, or rigidity. > Judges didn’t like it. “They said, ‘I was loving this, all this data-driven, evidence-based demonstration of what’s predictive. Now you’re putting a policy thumb on this,’ ” said Susan Sommer, general counsel at the Mayor’s Office of Criminal Justice. > The researchers added those charges back in.
- darawk 6y ago
  > Who is "us"? And does it? > Does the firm writing the software actually allow this? Plenty of software sold to the government comes with contractual stipulations that it may not be reverse-engineered, inspected, or otherwise second-guessed. So don't accept contracts with those stipulations. > ...Albeit with mixed results. You can't avoid bias. You can just choose whether you want to bias towards justice, or rigidity. That's my point though. Bias we can actually inspect and understand and iterate on is better than bias that we have no insight into at all.
x86_64Ubuntu 6y ago
And I'm sure the ethnic groups targeted by such systems are just thrilled to have their freedom determined by them for the sake of "research".

by darawk 6y ago

mola 6y ago

I think you are wrong. 1. From a technical point of view, The mere fact of a single model responsible for all decisions is bad. The inherent variance of judges opinions in concrete cases is the best way to fight bias. Bias by definition means less variance. Consolidating all decision making will tremendously worsen the bias. 2. From a moral point of view, law is made by people for people. It is a convention and ritual which gets it's moral validity from it's connection with tradition. We can hardly quantify these things let alone incorporate them to an ML algorithm. We ensure these by having people study for years, take bar exams, and through apprentenship programs. Do you really believe the state of the art in ML can come close to this? As a practitioner of ML, I know we can at most replace human activities that usually don't require any training, and are done by a human in a few seconds. 3. ML is really crappy. But is hyped as fact based and scientific. This is dangerous, a judge seeing a prediction by an algo is forced to comply with the algo, otherwise he'll be scrutinized for dismissing objective facts.

darawk 6y ago
> 1. From a technical point of view, The mere fact of a single model responsible for all decisions is bad. The inherent variance of judges opinions in concrete cases is the best way to fight bias. Bias by definition means less variance. Consolidating all decision making will tremendously worsen the bia Who said there has to be one model? Bias also definitely does not mean less variance. If I were to try to flesh out the argument I think you're trying to make, it'd go something like this: Judges have uncorrelated biases, and that low correlation between their biases ultimately results in a lower bias than a unified model. However, whether or not that's actually true in practice hinges on two important things: How uncorrelated those biases actually are, and their relative magnitudes to those of the unified model. > 2. From a moral point of view, law is made by people for people. It is a convention and ritual which gets it's moral validity from it's connection with tradition. We can hardly quantify these things let alone incorporate them to an ML algorithm. We ensure these by having people study for years, take bar exams, and through apprentenship programs. Do you really believe the state of the art in ML can come close to this? As a practitioner of ML, I know we can at most replace human activities that usually don't require any training, and are done by a human in a few seconds. I don't believe that ML is at the point where it should be the sole arbiter of these things, no. But I do believe that a properly calibrated model can be a very useful guide to these sorts of decisions. > 3. ML is really crappy. But is hyped as fact based and scientific. This is dangerous, a judge seeing a prediction by an algo is forced to comply with the algo, otherwise he'll be scrutinized for dismissing objective facts. This is just an education problem though.

dan-robertson 6y ago

The problem is that the system isn’t falsifiable. If the model says someone won’t turn up to court they are locked in cells until their trial, how will you know whether it was right or not? If you randomly release people as a test of the model then this is firstly unfair but secondarily won’t necessarily reveal small scale problems with the model that only affect certain groups of people because the sample will be small. This also ignores the large impact that going to jail tends to have on people’s lives. I don’t think it is reasonable to simply dismiss the likelihood of bias.

AnHonestComment 6y ago
There’s certainly parts that are hard to test, but it’s incorrect to say it’s unfalsifiable as a whole. For example, court attendance isn’t a Boolean: we create a confidence score and decide the “yes/no” question via some threshold. Even if you can’t test the model on the people you decide to keep in custody, you can test it on the people you release. The percentage who do appear for trial vs running should match your confidence score in that cohort — ie, if you guess they show up 95% of the time, then they should show up 95% of the time.
darawk 6y ago
For any one instance? You can't. But in aggregate, we do it all the time: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diver...

Jabbles 6y ago

Surely it would be better to concentrate on the existing sources of bias, e.g. the judges? This would presumably carry over to their other duties. Perhaps this model could be used to generate a set of rules that were clear and less open to interpretation by judges, and in that way it could be valuable.

darawk 6y ago
How are we going to eliminate the human bias of judges?

TuringNYC 6y ago

What about negative loops. If (when) the system consistently makes bad decisions, and people react negatively, the system will re-enforce those bad decisions with newfound evidence it was correct. Seems like the story of law enforcement in this country, except with the shine of a "system" to do it which will be trusted as unbiased.

pavel_lishin 6y ago

> But ultimately having an explicit model allows us to actually interrogate and correct those biases in a permanent way. Who is "us"? And does it? Does the firm writing the software actually allow this? Plenty of software sold to the government comes with contractual stipulations that it may not be reverse-engineered, inspected, or otherwise second-guessed. Besides, principles shminciples. The way to judge these systems is by the actual impact they have. Will this result in a more just system? If not, we can put the principles right into /dev/null. The article does seem to address this: > Among the debates was how to balance accuracy and fairness, said Ojmarrh Mitchell, a professor at Arizona State University who served on the panel. An accurate algorithm would do a good job of predicting whether defendants showed up, and thus whether to recommend release, Mr. Mitchell said, and a fair algorithm wouldn’t result in more release recommendations for white defendants than for others. > More than a year of tinkering ensued. To decrease the differences in outcomes for different races and ethnicities, the researchers excluded data on low-level marijuana offenses and “theft of service,” mainly subway turnstile jumping. Removing fare beating and marijuana arrests lowered the racial disparity in the tool by 0.4%, essentially making it slightly fairer but a little less accurate. > Data released this month show the new tool’s recommendations didn’t have such racial disparities. From Nov. 12 through March 17, the algorithm recommended releasing without conditions 83.9% of Blacks, 83.5% of whites and 85.8% of Hispanics. Defendants with higher scores returned to court more often than those with lower scores, showing the algorithm seemingly made accurate predictions. ...Albeit with mixed results. You can't avoid bias. You can just choose whether you want to bias towards justice, or rigidity. > Judges didn’t like it. “They said, ‘I was loving this, all this data-driven, evidence-based demonstration of what’s predictive. Now you’re putting a policy thumb on this,’ ” said Susan Sommer, general counsel at the Mayor’s Office of Criminal Justice. > The researchers added those charges back in.

darawk 6y ago
> Who is "us"? And does it? > Does the firm writing the software actually allow this? Plenty of software sold to the government comes with contractual stipulations that it may not be reverse-engineered, inspected, or otherwise second-guessed. So don't accept contracts with those stipulations. > ...Albeit with mixed results. You can't avoid bias. You can just choose whether you want to bias towards justice, or rigidity. That's my point though. Bias we can actually inspect and understand and iterate on is better than bias that we have no insight into at all.

x86_64Ubuntu 6y ago

And I'm sure the ethnic groups targeted by such systems are just thrilled to have their freedom determined by them for the sake of "research".