3 ms·

Coming from a statistics background, it seems like there is an easy fix for many of these problems (trigger warning: p-values ahead). For example, if you want

Coming from a statistics background, it seems like there is an easy fix for many of these problems (trigger warning: p-values ahead). For example, if you want to claim someone's fingerprints are a match for a given fingerprint from a crime scene, you need to show that the probability of a random person's fingerprints matching is small. One direct way to prove this is to get the prints of N random people, plus the prints of the suspect, and compare them to the print from the crime scene. If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. Even if matching is done by a human expert, a computer could be used to reduce the set of potential matches before a human was used to select the best match out of these candidates (of course, the expert would not know which one belonged to the suspect). I know all of this is obvious to someone with a thorough knowledge of statistics, but I honestly think this is just lacking in the discipline.

capnrefsmmat 11y ago
Expert witnesses will often quote numbers like that in court, stating that the probability of a random person getting a DNA match is less than 1 in (some large number) million. However, the FBI has been strongly resistant to opening up their DNA databases so these numbers could actually be verified empirically. (I think currently they just multiply some numbers together to get an estimate.) Some preliminary research suggests that partial DNA matches are much more common than claimed. Here's a law review article about the problem: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 Naturally it suffers from the common law review problem that most of its text is in footnotes instead of the main body, but it's pretty good...
- sukilot 11y ago
  The math is still wrong! A million to one odds and a full USA population database means the 300 people are perfect matches, open and shut convictions! There was famous case that actually won (for the good guys) by taking this math to court. These forensic anyalses do bad statistics (wildly wrong priors) and compute wildly wrong posteriors.
  - capnrefsmmat 11y ago
    That's true, but they'll often report random match probabilities like "one in 930 sextillion", which are absurd. Do you have a reference to the case you mention? I'd like to read the history of it.
hackuser 11y ago
IIRC, part of the problem is that fingerprint comparison methods do not produce consistent results. Different experts produce different results with the same set of fingerprints. Also, as the article says, fingerprints 'in the wild' are often incomplete, smudged, etc. EDIT: From the National Academy of Sciences report linked to from the article (the ACE-V method is the standard fingerprint analysis): the ACE-V method does not specify particular measurements or a standard test protocol, and examiners must make subjective assessments throughout. In the United States, the threshold for making a source identification is deliberately kept subjective, so that the examiner can take into account both the quantity and quality of comparable details. As a result, the outcome of a friction ridge analysis is not necessarily repeatable from examiner to examiner. In fact, recent research by Dror23 has shown that experienced examiners do not necessarily agree with even their own past conclusions when the examination is presented in a different context some time later. Don't miss that last sentence.
- formulaT 11y ago
  I don't see how any of those things prevent the methodology I described being applied.
  - hackuser 11y ago
    For example, you say, > If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. There may be no way to objectively determine if the suspect's prints are a better match than others. More generally, I lack your expertise but it seems to me that statistics require data; we may not have reliable data. More colloquially, garbage in, garbage out. Anyway, I recommend the National Academy of Sciences report, which is authored in part by their Committee on Applied and Theoretical Statistics.
woodson 11y ago
At least in academic circles, statisticians working in forensic inference and statistics widely support probabilistic approaches, to give the strength of the evidence in terms of likelihood ratios (or Bayes factors) with respect to prosecution and defense hypothesis. For example, what is the probability of observing the features of a given trace had it come from the suspect versus had it come from another individual in the relevant population. The concept of "matching" is also slightly more complicated in reality. Basically you have to account for uncertainty in measurements and other factors. For example, often you won't get a perfect print from a crime scene but a smudged partial mark, which you then compare to an inked fingerprint. Studies about the validity and reliability of fingermark-to-fingerprint comparison that are based on fingerprint databases of inked fingerprints most likely overstate the accuracy of the method as compared to testing under conditions of real casework. For some additional background on problems with fingerprints in forensics see: National Institute of Standards and Technology, National Institute of Justice, February 2012 report: "Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach" (http://www.nist.gov/manuscript-publication-search.cfm?pub_id=910745 http://www.nist.gov/manuscript-publication-search.cfm?pub_id...) Edit: See this paper for a recent example of academic work in forensic statistics (I'm not affiliated with the work/authors etc.) : http://arxiv.org/abs/1503.08234v1 http://arxiv.org/abs/1503.08234v1 Edit2: fixed link

capnrefsmmat 11y ago

Expert witnesses will often quote numbers like that in court, stating that the probability of a random person getting a DNA match is less than 1 in (some large number) million. However, the FBI has been strongly resistant to opening up their DNA databases so these numbers could actually be verified empirically. (I think currently they just multiply some numbers together to get an estimate.) Some preliminary research suggests that partial DNA matches are much more common than claimed. Here's a law review article about the problem: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 Naturally it suffers from the common law review problem that most of its text is in footnotes instead of the main body, but it's pretty good...

sukilot 11y ago
The math is still wrong! A million to one odds and a full USA population database means the 300 people are perfect matches, open and shut convictions! There was famous case that actually won (for the good guys) by taking this math to court. These forensic anyalses do bad statistics (wildly wrong priors) and compute wildly wrong posteriors.
- capnrefsmmat 11y ago
  That's true, but they'll often report random match probabilities like "one in 930 sextillion", which are absurd. Do you have a reference to the case you mention? I'd like to read the history of it.

hackuser 11y ago

IIRC, part of the problem is that fingerprint comparison methods do not produce consistent results. Different experts produce different results with the same set of fingerprints. Also, as the article says, fingerprints 'in the wild' are often incomplete, smudged, etc. EDIT: From the National Academy of Sciences report linked to from the article (the ACE-V method is the standard fingerprint analysis): the ACE-V method does not specify particular measurements or a standard test protocol, and examiners must make subjective assessments throughout. In the United States, the threshold for making a source identification is deliberately kept subjective, so that the examiner can take into account both the quantity and quality of comparable details. As a result, the outcome of a friction ridge analysis is not necessarily repeatable from examiner to examiner. In fact, recent research by Dror23 has shown that experienced examiners do not necessarily agree with even their own past conclusions when the examination is presented in a different context some time later. Don't miss that last sentence.

formulaT 11y ago
I don't see how any of those things prevent the methodology I described being applied.
- hackuser 11y ago
  For example, you say, > If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. There may be no way to objectively determine if the suspect's prints are a better match than others. More generally, I lack your expertise but it seems to me that statistics require data; we may not have reliable data. More colloquially, garbage in, garbage out. Anyway, I recommend the National Academy of Sciences report, which is authored in part by their Committee on Applied and Theoretical Statistics.

woodson 11y ago

At least in academic circles, statisticians working in forensic inference and statistics widely support probabilistic approaches, to give the strength of the evidence in terms of likelihood ratios (or Bayes factors) with respect to prosecution and defense hypothesis. For example, what is the probability of observing the features of a given trace had it come from the suspect versus had it come from another individual in the relevant population. The concept of "matching" is also slightly more complicated in reality. Basically you have to account for uncertainty in measurements and other factors. For example, often you won't get a perfect print from a crime scene but a smudged partial mark, which you then compare to an inked fingerprint. Studies about the validity and reliability of fingermark-to-fingerprint comparison that are based on fingerprint databases of inked fingerprints most likely overstate the accuracy of the method as compared to testing under conditions of real casework. For some additional background on problems with fingerprints in forensics see: National Institute of Standards and Technology, National Institute of Justice, February 2012 report: "Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach" (http://www.nist.gov/manuscript-publication-search.cfm?pub_id=910745 http://www.nist.gov/manuscript-publication-search.cfm?pub_id...) Edit: See this paper for a recent example of academic work in forensic statistics (I'm not affiliated with the work/authors etc.) : http://arxiv.org/abs/1503.08234v1 http://arxiv.org/abs/1503.08234v1 Edit2: fixed link