4 ms·

A false positive rate of 0.1% and a false negative rate of 40-50%? Isn't that... actually pretty useful? You'd never want to convict someone based exclusively o

by ip26 7y ago

A false positive rate of 0.1% and a false negative rate of 40-50%? Isn't that... actually pretty useful? You'd never want to convict someone based exclusively on this, but it could fit into the "preponderance of evidence" standard. No one piece of evidence is foolproof, e.g. the classic examples of DNA & fingerprints- did you even lift the right set of prints or collect the right sample of hair?

exabrial 7y ago
That was my thought. I think the main complaint is that there evidence is presented without the statistics you mentioned, as if it's near effective 100% of the time.
matheusmoreira 7y ago
Contrast that with sensitive drug tests commonly used in the field: https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/ https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/ Without confirmation, false positive rates varied from 2 to 4 percent. That means up to 4 percent of test subjects will end up arrested because of a false positive.
burkaman 7y ago
I'm not sure the article correctly interpreted the findings. The paper is here: https://www.pnas.org/content/pnas/early/2020/02/18/1917222117.full.pdf https://www.pnas.org/content/pnas/early/2020/02/18/191722211... What they found was a correlation between false positive rates and true positive rates, where IF you could accept a false positive rate of 0.1%, THEN you could get a little over 50% true positive rate with the longest seam length under "ideal imaging conditions". They do not attempt to estimate the false positive rate of whatever methodology FBI agents are using in practice. I doubt the FBI has a rigorous well-defined test that could be analyzed; I suspect this is just a general technique that is applied by different agents in different ways. Even if the FBI's methods approach this ideal false positive rate (which they can't), they are not describing it honestly in court. > When, for example, an FBI analyst identified the plaid shirt in a surveillance video as that belonging to Wilbert McKreith, the analyst stated in court that the odds that two different shirts would match were a staggering 1-in-650 billion. The analyst arrived at this number by making eight measurements along two seams, estimating the probability that the plaid stripes at these eight locations were misaligned by a certain distance, and then multiplying all eight of these probabilities to reach the astronom-ically low odds of 1-in-650 billion.
jml7c5 7y ago
It would be, but those numbers are for the absolutely ideal conditions present in the study. It seems unlikely that a photo like this will come up in a real court case: >The analyst excluded any seams that did not satisfy each of the following properties: 1) the length of the seam was at least 30 cm, 2) the resolution of this 30-cm seam was at least 1,500 pixels, 3) the width of the seam was at least 30 pixels, and 4) the final extracted 1D signal clearly showed a distinct pattern of ridges and valleys. Even with photos taken specifically to see these patterns, "[e]nforcement of these criteria reduced the original 211 left/right and inner/outer seams to 164 left/inner, 162 right/inner, 81 left/outer, and 90 right/outer seams." Unfortunately they aren't very clear on what they mean by different orientations/lighting conditions/drapes.
ggggtez 7y ago
With FP and FN you really need to be careful... 1 person committed a crime. So the true positive rate is 1 out of 7 billion or so. So how many false positives would 0.1% lead to? About 7 million pants wearers? And for false negatives, we expect half the time the true criminal's pants would prove them innocent? I don't think you can really trust evidence if the true criminal has a coin-flip chance to show innocence with it.
xorfish 7y ago
The key is to only use evidence in court that was not used to identify the perpetrator. If a piece of evidence is used to identify, you select for the population that matches that property.

by ip26 7y ago

exabrial 7y ago

That was my thought. I think the main complaint is that there evidence is presented without the statistics you mentioned, as if it's near effective 100% of the time.

matheusmoreira 7y ago

Contrast that with sensitive drug tests commonly used in the field: https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/ https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/ Without confirmation, false positive rates varied from 2 to 4 percent. That means up to 4 percent of test subjects will end up arrested because of a false positive.

burkaman 7y ago

I'm not sure the article correctly interpreted the findings. The paper is here: https://www.pnas.org/content/pnas/early/2020/02/18/1917222117.full.pdf https://www.pnas.org/content/pnas/early/2020/02/18/191722211... What they found was a correlation between false positive rates and true positive rates, where IF you could accept a false positive rate of 0.1%, THEN you could get a little over 50% true positive rate with the longest seam length under "ideal imaging conditions". They do not attempt to estimate the false positive rate of whatever methodology FBI agents are using in practice. I doubt the FBI has a rigorous well-defined test that could be analyzed; I suspect this is just a general technique that is applied by different agents in different ways. Even if the FBI's methods approach this ideal false positive rate (which they can't), they are not describing it honestly in court. > When, for example, an FBI analyst identified the plaid shirt in a surveillance video as that belonging to Wilbert McKreith, the analyst stated in court that the odds that two different shirts would match were a staggering 1-in-650 billion. The analyst arrived at this number by making eight measurements along two seams, estimating the probability that the plaid stripes at these eight locations were misaligned by a certain distance, and then multiplying all eight of these probabilities to reach the astronom-ically low odds of 1-in-650 billion.

jml7c5 7y ago

It would be, but those numbers are for the absolutely ideal conditions present in the study. It seems unlikely that a photo like this will come up in a real court case: >The analyst excluded any seams that did not satisfy each of the following properties: 1) the length of the seam was at least 30 cm, 2) the resolution of this 30-cm seam was at least 1,500 pixels, 3) the width of the seam was at least 30 pixels, and 4) the final extracted 1D signal clearly showed a distinct pattern of ridges and valleys. Even with photos taken specifically to see these patterns, "[e]nforcement of these criteria reduced the original 211 left/right and inner/outer seams to 164 left/inner, 162 right/inner, 81 left/outer, and 90 right/outer seams." Unfortunately they aren't very clear on what they mean by different orientations/lighting conditions/drapes.

ggggtez 7y ago

With FP and FN you really need to be careful... 1 person committed a crime. So the true positive rate is 1 out of 7 billion or so. So how many false positives would 0.1% lead to? About 7 million pants wearers? And for false negatives, we expect half the time the true criminal's pants would prove them innocent? I don't think you can really trust evidence if the true criminal has a coin-flip chance to show innocence with it.

xorfish 7y ago

The key is to only use evidence in court that was not used to identify the perpetrator. If a piece of evidence is used to identify, you select for the population that matches that property.