3 ms·

> Come to think of it, why is the criminal justice system not "tested"? In other words, set up a mock situation where you know in advance who is innocent and wh

by edanm 9y ago

> Come to think of it, why is the criminal justice system not "tested"? In other words, set up a mock situation where you know in advance who is innocent and who is not (because it's a setup), and run a mock trial by all the rules using actual jurors, and compare the outcome of the trial to the actual innocence or guilt of the actor? Probably for the same reason most things are or aren't done - some form of status-quo bias. I mean, someone has to think of the idea, push it, administer it, etc. It's a lot of work. And no one really has an incentive to do that work right now. Btw, your statement isn't really accurate: >> I think people usually go to prison for the legally correct reason >Or not, if most of the incriminating information was eyewitness testimony, the #1 cause of wrongful convictions: The fact that eyewitness testimony is the #1 cause of wrongful convictions doesn't tell us anything about the rate at which wrong convictions occur. It's possible for eyewitness testimony to be the #1 cause of wrongful convictions, but for there to be only e.g. 3 wrong convictions a year out of 100k trials. Which would probably be pretty "acceptable". (I say this because it sounded like you were making the common probability mistake of flipping two things: knowledge of how often a wrongful conviction is caused by eyewitness testimony, and knowledge of how often eyewitness testimony causes a wrongful conviction. Knowing the former doesn't mean we know the latter. Not sure if you're actually claiming this, but wanted to clarify what it sounded like to me).

edanm 9y ago
Also, just to answer your actual hypothetical of "testing" the criminal justice system: I'm not sure how you would test it. You can run a "mock" trial, although I'm not sure it's so easy - to be realistic, I think you need to create a scenario which can plausibly go in either direction (innocent or guilty). But what would that tell you? I mean, "beyond a reasonable doubt" is a pretty high standard and all, but if you make it seem pretty clear that someone is guilty when they're "really" innocent, that doesn't mean much, because this is fiction - you made up a tricky case, and people got it wrong. So what? The question is how often does real life cause a tricky scenario to manifest. You can go the other way and set up a situation that's pretty clear, but I'm not sure that will be very insightful. That's not to say we can't think of some way to test the system which does work - I'm sure smarter and more knowledgeable people than me can think of something. But that brings us to problem number 2 - now what? Let's say people got it wrong. What do you do? It's not like you can just change the way trials work - you need laws for that. The most you can reasonably do is use trial tests as a basis for potentially changing the law, but that's before any politics gets involved. Realistically, I'm not sure you can do much more than gets done now - once in a while an article gets written about a specific bad practice, it causes a public stir, and the law maybe gets changed. I'm not sure systematizing any of the testing steps really changes anything else in this picture. And we might as well ask about easier things, why don't we test them. I.e. why not test economic policies by building "test cities" which have different economic policies, and seeing which ones work? The States kind-of does this with different laws in different States. So there is kind of a natural experiment of the effect of different laws on economics, on justice, etc. This does affect some things, but I'm not sure the effect is very large (but then again, I'm not an American, so I'm not that up to date on the politics there. It's certainly an idea that a lot of lip-service is paid to, but maybe it's more than that).
- frobozz 9y ago
  Part of the point of a jury is to have multiple disinterested individuals ruling on the defendant's guilt. That in itself forms a partial test of the process. You could run a trial with multiple independent juries, or record a trial and play it back to a set of juries and judges to see if they come up with the same verdict and sentence. This would help determine whether a given jury has a particularly forceful and prejudiced member, or if a hanging judge is handing out unfair sentences. I'd be surprised if that sort of thing doesn't already happen occasionally. However, all this does is indicate where a problem might be, rather than validate a potential solution. A/B testing of potential enhancements would be very difficult and limited. There are too many confounding factors. Judge and jury would have to be different each time. Witness testimony may seem unconvincingly rehearsed, or more convincingly thorough, the nth time they give it. Defendants will be able to avoid incriminating traps the prosecuting barrister set in previous iterations of the trial.
  - sorokod 9y ago
    Given the intese scrutiny the litigating side give to jury selection, I'd say that it is well understood that trials may go this way or that based on a particular jury and regardless of the facts
- pmarreck 9y ago
  > I.e. why not test economic policies by building "test cities" which have different economic policies, and seeing which ones work? I would be exactly behind this. See: the old definition of https://en.wikipedia.org/wiki/Technocracy https://en.wikipedia.org/wiki/Technocracy
joeblow9999 9y ago
"3 wrong convictions a year out of 100k trials. Which would probably be pretty "acceptable"." Not if you're one of the 3.
deadmetheny 9y ago
>It's possible for eyewitness testimony to be the #1 cause of wrongful convictions, but for there to be only e.g. 3 wrong convictions a year out of 100k trials. Which would probably be pretty "acceptable". "It is better 100 guilty Persons should escape than that one innocent Person should suffer." Three innocent people being punished is a travesty and in no way acceptable.
- pmarreck 9y ago
  https://en.wikipedia.org/wiki/Blackstone%27s_formulation https://en.wikipedia.org/wiki/Blackstone%27s_formulation actually says that it is better that 10, not 100, guilty go free. And sadly, in a world of imperfect information, it is arguable that 3 wrongly convicted innocents out of 100,000 trials is "acceptable," because you're never going to get to 100% precision/accuracy.
  - deadmetheny 9y ago
    Benjamin Franklin's paraphrase of the principle says 100, which is my direct quote ;D I don't disagree that it's unrealistic to achieve 100% accuracy, but it should never be accepted - we should never, ever stop striving for improvement in this arena, and to err on the side of letting people go free if it can't be conclusively and impartially proven they are guilty.
- earenndil 9y ago
  Let's take this to an extreme. Let's get rid of incarceration and legal consequence entirely. Then you're to never wrongly convict someone, right? All other ways could and likely will end up with false positives.

by edanm 9y ago

edanm 9y ago

Also, just to answer your actual hypothetical of "testing" the criminal justice system: I'm not sure how you would test it. You can run a "mock" trial, although I'm not sure it's so easy - to be realistic, I think you need to create a scenario which can plausibly go in either direction (innocent or guilty). But what would that tell you? I mean, "beyond a reasonable doubt" is a pretty high standard and all, but if you make it seem pretty clear that someone is guilty when they're "really" innocent, that doesn't mean much, because this is fiction - you made up a tricky case, and people got it wrong. So what? The question is how often does real life cause a tricky scenario to manifest. You can go the other way and set up a situation that's pretty clear, but I'm not sure that will be very insightful. That's not to say we can't think of some way to test the system which does work - I'm sure smarter and more knowledgeable people than me can think of something. But that brings us to problem number 2 - now what? Let's say people got it wrong. What do you do? It's not like you can just change the way trials work - you need laws for that. The most you can reasonably do is use trial tests as a basis for potentially changing the law, but that's before any politics gets involved. Realistically, I'm not sure you can do much more than gets done now - once in a while an article gets written about a specific bad practice, it causes a public stir, and the law maybe gets changed. I'm not sure systematizing any of the testing steps really changes anything else in this picture. And we might as well ask about easier things, why don't we test them. I.e. why not test economic policies by building "test cities" which have different economic policies, and seeing which ones work? The States kind-of does this with different laws in different States. So there is kind of a natural experiment of the effect of different laws on economics, on justice, etc. This does affect some things, but I'm not sure the effect is very large (but then again, I'm not an American, so I'm not that up to date on the politics there. It's certainly an idea that a lot of lip-service is paid to, but maybe it's more than that).

frobozz 9y ago
Part of the point of a jury is to have multiple disinterested individuals ruling on the defendant's guilt. That in itself forms a partial test of the process. You could run a trial with multiple independent juries, or record a trial and play it back to a set of juries and judges to see if they come up with the same verdict and sentence. This would help determine whether a given jury has a particularly forceful and prejudiced member, or if a hanging judge is handing out unfair sentences. I'd be surprised if that sort of thing doesn't already happen occasionally. However, all this does is indicate where a problem might be, rather than validate a potential solution. A/B testing of potential enhancements would be very difficult and limited. There are too many confounding factors. Judge and jury would have to be different each time. Witness testimony may seem unconvincingly rehearsed, or more convincingly thorough, the nth time they give it. Defendants will be able to avoid incriminating traps the prosecuting barrister set in previous iterations of the trial.
- sorokod 9y ago
  Given the intese scrutiny the litigating side give to jury selection, I'd say that it is well understood that trials may go this way or that based on a particular jury and regardless of the facts
pmarreck 9y ago
> I.e. why not test economic policies by building "test cities" which have different economic policies, and seeing which ones work? I would be exactly behind this. See: the old definition of https://en.wikipedia.org/wiki/Technocracy https://en.wikipedia.org/wiki/Technocracy

joeblow9999 9y ago

"3 wrong convictions a year out of 100k trials. Which would probably be pretty "acceptable"." Not if you're one of the 3.

deadmetheny 9y ago

>It's possible for eyewitness testimony to be the #1 cause of wrongful convictions, but for there to be only e.g. 3 wrong convictions a year out of 100k trials. Which would probably be pretty "acceptable". "It is better 100 guilty Persons should escape than that one innocent Person should suffer." Three innocent people being punished is a travesty and in no way acceptable.

pmarreck 9y ago
https://en.wikipedia.org/wiki/Blackstone%27s_formulation https://en.wikipedia.org/wiki/Blackstone%27s_formulation actually says that it is better that 10, not 100, guilty go free. And sadly, in a world of imperfect information, it is arguable that 3 wrongly convicted innocents out of 100,000 trials is "acceptable," because you're never going to get to 100% precision/accuracy.
- deadmetheny 9y ago
  Benjamin Franklin's paraphrase of the principle says 100, which is my direct quote ;D I don't disagree that it's unrealistic to achieve 100% accuracy, but it should never be accepted - we should never, ever stop striving for improvement in this arena, and to err on the side of letting people go free if it can't be conclusively and impartially proven they are guilty.
earenndil 9y ago
Let's take this to an extreme. Let's get rid of incarceration and legal consequence entirely. Then you're to never wrongly convict someone, right? All other ways could and likely will end up with false positives.