9 ms·

The Troubling Reality of Forensic “Science”

65 points by anon1385 12y ago

formulaT 12y ago
Coming from a statistics background, it seems like there is an easy fix for many of these problems (trigger warning: p-values ahead). For example, if you want to claim someone's fingerprints are a match for a given fingerprint from a crime scene, you need to show that the probability of a random person's fingerprints matching is small. One direct way to prove this is to get the prints of N random people, plus the prints of the suspect, and compare them to the print from the crime scene. If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. Even if matching is done by a human expert, a computer could be used to reduce the set of potential matches before a human was used to select the best match out of these candidates (of course, the expert would not know which one belonged to the suspect). I know all of this is obvious to someone with a thorough knowledge of statistics, but I honestly think this is just lacking in the discipline.
- capnrefsmmat 12y ago
  Expert witnesses will often quote numbers like that in court, stating that the probability of a random person getting a DNA match is less than 1 in (some large number) million. However, the FBI has been strongly resistant to opening up their DNA databases so these numbers could actually be verified empirically. (I think currently they just multiply some numbers together to get an estimate.) Some preliminary research suggests that partial DNA matches are much more common than claimed. Here's a law review article about the problem: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 Naturally it suffers from the common law review problem that most of its text is in footnotes instead of the main body, but it's pretty good...
  - sukilot 12y ago
    The math is still wrong! A million to one odds and a full USA population database means the 300 people are perfect matches, open and shut convictions! There was famous case that actually won (for the good guys) by taking this math to court. These forensic anyalses do bad statistics (wildly wrong priors) and compute wildly wrong posteriors.
    - capnrefsmmat 12y ago
      That's true, but they'll often report random match probabilities like "one in 930 sextillion", which are absurd. Do you have a reference to the case you mention? I'd like to read the history of it.
- hackuser 12y ago
  IIRC, part of the problem is that fingerprint comparison methods do not produce consistent results. Different experts produce different results with the same set of fingerprints. Also, as the article says, fingerprints 'in the wild' are often incomplete, smudged, etc. EDIT: From the National Academy of Sciences report linked to from the article (the ACE-V method is the standard fingerprint analysis): the ACE-V method does not specify particular measurements or a standard test protocol, and examiners must make subjective assessments throughout. In the United States, the threshold for making a source identification is deliberately kept subjective, so that the examiner can take into account both the quantity and quality of comparable details. As a result, the outcome of a friction ridge analysis is not necessarily repeatable from examiner to examiner. In fact, recent research by Dror23 has shown that experienced examiners do not necessarily agree with even their own past conclusions when the examination is presented in a different context some time later. Don't miss that last sentence.
  - formulaT 12y ago
    I don't see how any of those things prevent the methodology I described being applied.
    - hackuser 12y ago
      For example, you say, > If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. There may be no way to objectively determine if the suspect's prints are a better match than others. More generally, I lack your expertise but it seems to me that statistics require data; we may not have reliable data. More colloquially, garbage in, garbage out. Anyway, I recommend the National Academy of Sciences report, which is authored in part by their Committee on Applied and Theoretical Statistics.
- woodson 12y ago
  At least in academic circles, statisticians working in forensic inference and statistics widely support probabilistic approaches, to give the strength of the evidence in terms of likelihood ratios (or Bayes factors) with respect to prosecution and defense hypothesis. For example, what is the probability of observing the features of a given trace had it come from the suspect versus had it come from another individual in the relevant population. The concept of "matching" is also slightly more complicated in reality. Basically you have to account for uncertainty in measurements and other factors. For example, often you won't get a perfect print from a crime scene but a smudged partial mark, which you then compare to an inked fingerprint. Studies about the validity and reliability of fingermark-to-fingerprint comparison that are based on fingerprint databases of inked fingerprints most likely overstate the accuracy of the method as compared to testing under conditions of real casework. For some additional background on problems with fingerprints in forensics see: National Institute of Standards and Technology, National Institute of Justice, February 2012 report: "Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach" (http://www.nist.gov/manuscript-publication-search.cfm?pub_id=910745 http://www.nist.gov/manuscript-publication-search.cfm?pub_id...) Edit: See this paper for a recent example of academic work in forensic statistics (I'm not affiliated with the work/authors etc.) : http://arxiv.org/abs/1503.08234v1 http://arxiv.org/abs/1503.08234v1 Edit2: fixed link
ChuckMcM 12y ago
I wonder what became of the National Commission on Forensic Science[1] ? [1] http://www.popsci.com/science/article/2013-02/new-commission-set-standards-troubled-forensic-sciences http://www.popsci.com/science/article/2013-02/new-commission...
hackuser 12y ago
For those interested in these kinds of issues, I recommend the Reference Manual on Scientific Evidence, written by the National Academies of Science for federal judges. http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$fil... Just the chapter "How Science Works" by David Goodstein, Professor of Physics Emeritus (and former Vice Provost) at Cal Tech, is a revelation for anyone outside the scientific community. Highly recommended.
- jacquesm 12y ago
  Warning for those on mobile bandwidth, that's a 1000+ page PDF.
  - hackuser 12y ago
    Sorry. You can read it online, one page at a time: http://www.nap.edu/catalog/13163/reference-manual-on-scientific-evidence-third-edition http://www.nap.edu/catalog/13163/reference-manual-on-scienti... There also used to be a place to download PDFs of individual chapters.
time_is_scary 12y ago
I've had my own bad experience with forensic science. I was a research intern at Oak Ridge National Lab. They put on several educational events for the interns. One event was a guy from the group doing forensic science to come in and talk about it. He talked about rate of decay of corpses and things like that. Then he brought out dowsing sticks. Said something about the bones of buried bodies giving off magnetic fields that affect the sticks. He then took a leg bone out of a trash bag, put it on the ground, and walked passed it holding the sticks. The sticks moved when he did it, but of course when anyone else in the room tried nothing happened. Honestly, all the interns in the room were dead quiet. I think we were all really confused, thinking that maybe it was some sort of joke or hazing. Or maybe a test of some sort? It was really weird, but destroyed my regard for forensic "science".
- rjsw 12y ago
  He could have been a complete nut, or... I have tried dowsing for water and found it very accurate. We had a problem with the water supply to a house. There was a spring that fed a collection tank through some pipes under agricultural land, we guessed that a tractor had squashed the pipes somewhere along their length. A backhoe was booked for the following day so it didn't really matter whether we got any results or not but we decided to try dowsing to see if we could work out the path of the pipes, just for a bit of fun more than expecting it to really work. We used some bent aluminium rods and walked about a bit in the field sticking twigs into the ground wherever the rods moved. This worked for some family members, not for others. The next day the backhoe dug out the old wooden pipes and they were exactly underneath where we had placed the twigs on the surface but several feet down.
  - EliRivers 12y ago
    Well then, can I interest your successful family members in a million dollars? It's a genuine offer. One million dollars if they can do it. Dowsers have tried and failed in the past, but perhaps your family really do have supernatural powers.
    - rjsw 12y ago
DanBC 12y ago
There's a BBC radio programme talking about bad forensics. Only one episode so far, but it talks about some of the same cases. "Forensics in Crisis" http://www.bbc.co.uk/programmes/b05r3tf1 http://www.bbc.co.uk/programmes/b05r3tf1 Edit this submission links to http://www.washingtonpost.com/news/the-watch/wp/2015/02/18/attack-of-the-bite-mark-matchers-2/ http://www.washingtonpost.com/news/the-watch/wp/2015/02/18/a... which is a better read.
mathgenius 12y ago
All of this points to the fact that it is just so hard to find out about "reality". And when egos are involved triple that. Actually, you could probably define ego to be the denial of reality. I'm still waiting for particle physicists to do double-blind experiments. (Not going to happen.)
- TheCoelacanth 12y ago
  I curious what a double-blind experiment would mean in the context of particle physics. The particles don't know about the experiment?

65 points by anon1385 12y ago

Coming from a statistics background, it seems like there is an easy fix for many of these problems (trigger warning: p-values ahead). For example, if you want to claim someone's fingerprints are a match for a given fingerprint from a crime scene, you need to show that the probability of a random person's fingerprints matching is small. One direct way to prove this is to get the prints of N random people, plus the prints of the suspect, and compare them to the print from the crime scene. If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. Even if matching is done by a human expert, a computer could be used to reduce the set of potential matches before a human was used to select the best match out of these candidates (of course, the expert would not know which one belonged to the suspect). I know all of this is obvious to someone with a thorough knowledge of statistics, but I honestly think this is just lacking in the discipline.

capnrefsmmat 12y ago
Expert witnesses will often quote numbers like that in court, stating that the probability of a random person getting a DNA match is less than 1 in (some large number) million. However, the FBI has been strongly resistant to opening up their DNA databases so these numbers could actually be verified empirically. (I think currently they just multiply some numbers together to get an estimate.) Some preliminary research suggests that partial DNA matches are much more common than claimed. Here's a law review article about the problem: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551467 Naturally it suffers from the common law review problem that most of its text is in footnotes instead of the main body, but it's pretty good...
- sukilot 12y ago
  The math is still wrong! A million to one odds and a full USA population database means the 300 people are perfect matches, open and shut convictions! There was famous case that actually won (for the good guys) by taking this math to court. These forensic anyalses do bad statistics (wildly wrong priors) and compute wildly wrong posteriors.
  - capnrefsmmat 12y ago
    That's true, but they'll often report random match probabilities like "one in 930 sextillion", which are absurd. Do you have a reference to the case you mention? I'd like to read the history of it.
hackuser 12y ago
IIRC, part of the problem is that fingerprint comparison methods do not produce consistent results. Different experts produce different results with the same set of fingerprints. Also, as the article says, fingerprints 'in the wild' are often incomplete, smudged, etc. EDIT: From the National Academy of Sciences report linked to from the article (the ACE-V method is the standard fingerprint analysis): the ACE-V method does not specify particular measurements or a standard test protocol, and examiners must make subjective assessments throughout. In the United States, the threshold for making a source identification is deliberately kept subjective, so that the examiner can take into account both the quantity and quality of comparable details. As a result, the outcome of a friction ridge analysis is not necessarily repeatable from examiner to examiner. In fact, recent research by Dror23 has shown that experienced examiners do not necessarily agree with even their own past conclusions when the examination is presented in a different context some time later. Don't miss that last sentence.
- formulaT 12y ago
  I don't see how any of those things prevent the methodology I described being applied.
  - hackuser 12y ago
    For example, you say, > If the suspect is a better match than N(1-p) of the other fingerprints, then the match has a p-value of p. There may be no way to objectively determine if the suspect's prints are a better match than others. More generally, I lack your expertise but it seems to me that statistics require data; we may not have reliable data. More colloquially, garbage in, garbage out. Anyway, I recommend the National Academy of Sciences report, which is authored in part by their Committee on Applied and Theoretical Statistics.
woodson 12y ago
At least in academic circles, statisticians working in forensic inference and statistics widely support probabilistic approaches, to give the strength of the evidence in terms of likelihood ratios (or Bayes factors) with respect to prosecution and defense hypothesis. For example, what is the probability of observing the features of a given trace had it come from the suspect versus had it come from another individual in the relevant population. The concept of "matching" is also slightly more complicated in reality. Basically you have to account for uncertainty in measurements and other factors. For example, often you won't get a perfect print from a crime scene but a smudged partial mark, which you then compare to an inked fingerprint. Studies about the validity and reliability of fingermark-to-fingerprint comparison that are based on fingerprint databases of inked fingerprints most likely overstate the accuracy of the method as compared to testing under conditions of real casework. For some additional background on problems with fingerprints in forensics see: National Institute of Standards and Technology, National Institute of Justice, February 2012 report: "Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach" (http://www.nist.gov/manuscript-publication-search.cfm?pub_id=910745 http://www.nist.gov/manuscript-publication-search.cfm?pub_id...) Edit: See this paper for a recent example of academic work in forensic statistics (I'm not affiliated with the work/authors etc.) : http://arxiv.org/abs/1503.08234v1 http://arxiv.org/abs/1503.08234v1 Edit2: fixed link

ChuckMcM 12y ago

I wonder what became of the National Commission on Forensic Science[1] ? [1] http://www.popsci.com/science/article/2013-02/new-commission-set-standards-troubled-forensic-sciences http://www.popsci.com/science/article/2013-02/new-commission...

hackuser 12y ago

For those interested in these kinds of issues, I recommend the Reference Manual on Scientific Evidence, written by the National Academies of Science for federal judges. http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$fil... Just the chapter "How Science Works" by David Goodstein, Professor of Physics Emeritus (and former Vice Provost) at Cal Tech, is a revelation for anyone outside the scientific community. Highly recommended.

jacquesm 12y ago
Warning for those on mobile bandwidth, that's a 1000+ page PDF.
- hackuser 12y ago
  Sorry. You can read it online, one page at a time: http://www.nap.edu/catalog/13163/reference-manual-on-scientific-evidence-third-edition http://www.nap.edu/catalog/13163/reference-manual-on-scienti... There also used to be a place to download PDFs of individual chapters.

time_is_scary 12y ago

I've had my own bad experience with forensic science. I was a research intern at Oak Ridge National Lab. They put on several educational events for the interns. One event was a guy from the group doing forensic science to come in and talk about it. He talked about rate of decay of corpses and things like that. Then he brought out dowsing sticks. Said something about the bones of buried bodies giving off magnetic fields that affect the sticks. He then took a leg bone out of a trash bag, put it on the ground, and walked passed it holding the sticks. The sticks moved when he did it, but of course when anyone else in the room tried nothing happened. Honestly, all the interns in the room were dead quiet. I think we were all really confused, thinking that maybe it was some sort of joke or hazing. Or maybe a test of some sort? It was really weird, but destroyed my regard for forensic "science".

rjsw 12y ago
He could have been a complete nut, or... I have tried dowsing for water and found it very accurate. We had a problem with the water supply to a house. There was a spring that fed a collection tank through some pipes under agricultural land, we guessed that a tractor had squashed the pipes somewhere along their length. A backhoe was booked for the following day so it didn't really matter whether we got any results or not but we decided to try dowsing to see if we could work out the path of the pipes, just for a bit of fun more than expecting it to really work. We used some bent aluminium rods and walked about a bit in the field sticking twigs into the ground wherever the rods moved. This worked for some family members, not for others. The next day the backhoe dug out the old wooden pipes and they were exactly underneath where we had placed the twigs on the surface but several feet down.
- EliRivers 12y ago
  Well then, can I interest your successful family members in a million dollars? It's a genuine offer. One million dollars if they can do it. Dowsers have tried and failed in the past, but perhaps your family really do have supernatural powers.
  - rjsw 12y ago

DanBC 12y ago

There's a BBC radio programme talking about bad forensics. Only one episode so far, but it talks about some of the same cases. "Forensics in Crisis" http://www.bbc.co.uk/programmes/b05r3tf1 http://www.bbc.co.uk/programmes/b05r3tf1 Edit this submission links to http://www.washingtonpost.com/news/the-watch/wp/2015/02/18/attack-of-the-bite-mark-matchers-2/ http://www.washingtonpost.com/news/the-watch/wp/2015/02/18/a... which is a better read.

mathgenius 12y ago

All of this points to the fact that it is just so hard to find out about "reality". And when egos are involved triple that. Actually, you could probably define ego to be the denial of reality. I'm still waiting for particle physicists to do double-blind experiments. (Not going to happen.)

TheCoelacanth 12y ago
I curious what a double-blind experiment would mean in the context of particle physics. The particles don't know about the experiment?