6 ms·

There is one simple technique that would allow nearly any forensic analysis technique to be assessed. It would detect the difference between good science (DNA)

by mcherm 9y ago

There is one simple technique that would allow nearly any forensic analysis technique to be assessed. It would detect the difference between good science (DNA) and junk science (forensic hair matches). It would work down to the level of the individual laboratory (as in this example) or individual laboratory researcher (Annie Dookhan). And it would only cost 10% extra. That technique is blind testing with published results. 10% of all submissions to the testing lab would be test samples -- a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) and a few known positive samples (perhaps from someone already convicted with overwhelming evidence). The lab would not be told which samples were tests and which were "real", and the test samples would be evenly distributed (10% of the submissions from each department; 10% of the "rush" submissions, etc.). Finally, the results reported on these test samples would be made publicly available. Although this simple, effective, and cheap method would completely transform forensic practice, it will not be implemented. The reason is incentives. The parties involved in forensics are the police or prosecutors and the labs. Both have an incentive to identify the guilty. The accused have a countervailing interest (in correctly determining innocence), but they are not part of the transaction, so their interests are not represented. In fact, the entire system is incented to find people guilty, not to correctly assess people's guilt. So most in the system are actually DISincented to discover and debunk junk science.

dsfyu404ed 9y ago
>That technique is blind testing with published results. 10% of all submissions to the testing lab would be test samples -- a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) and a few known positive samples (perhaps from someone already convicted with overwhelming evidence). The lab would not be told which samples were tests and which were "real", and the test samples would be evenly distributed (10% of the submissions from each department; 10% of the "rush" submissions, etc.). Finally, the results reported on these test samples would be made publicly available. And not only would it let you weed out the pseudoscience but it would let you weed out the labs that are practicing the real science well from the ones that are sloppy (which in my opinion is an equally large problem).
Zigurd 9y ago
That, and the labs should not be a branch of law enforcement or prosecutors' offices. They should be more like a weights and measures department, and be at the service of both prosecutors and defendants.
- mabbo 9y ago
  These labs are private businesses. Prosecutors and police can choose which ones to use and they will use, and they'll likely make that choice based on who gives them the results they want rather than the results that are accurate.
  - Zigurd 9y ago
    In Massachusetts, there is a state police crime lab: https://www.mass.gov/orgs/massachusetts-state-police-crime-laboratory https://www.mass.gov/orgs/massachusetts-state-police-crime-l... It is a department of the state police, and, obviously enough, is structurally incentivized to make state police performance look good.
    - NutriSugar 9y ago
      So there is incentive for corruption with both public and private labs. The solution will require something far more drastic, which I'm not sure the population is willing to support given how many people don't like to use science when it disagrees with their emotional stances. As long as the population wants tough on crime stances, and are willing to dehumanize anyone charged (much less wrongly convicted), then the system is giving them what they want. So how can we change the voters so that they demand a more just legal system?
    - xkcd-sucks 9y ago
      The Massachusetts state crime lab is an actual madhouse, and the state government is shockingly corrupt. The following article is one of the more detailed descriptions of how the lab scandals were managed https://www.rollingstone.com/culture/news/did-falsified-drug-tests-lead-to-wrongful-convictions-w514801 https://www.rollingstone.com/culture/news/did-falsified-drug...
bmelton 9y ago
I think that secret shopping is a great idea for forensics, but I worry that in areas of criminality, just pushing fake requests over to forensics won't work unless it's accompanied by the usual pleas from the cops and DAs actively working the cases to which those data are associated. Similarly, a rush order is a rush order, but a rush order with a group of people carping on how it's a rush order tends to get more rushed. When the queues are effectively bottomless, stamping 'rush' on an order without the associated pressure seems likely to relegate its status to regular.
- lttlrck 9y ago
  Maybe submit multiple controls in the same request.
darkerside 9y ago
Something could trivially be done to the same effect. A private party, say an investigative journalist, could submit a number of samples and report on the results (perhaps that happened in this case and that's why we know about it?). But to keep costs under control, instead of testing like this on every lab on a regular basis, that private party could randomly select labs for testing. While they're at it, they'd probably prefer to select labs with complaints, rather than a random sample. And at that point, you have something resembling the system we currently operate under, which has--to be fair--been completely successful (if a bit later than ideal) in identifying this problem lab. But as taxpayers, is it worth instituting a new regulated program over this essentially free solution? Besides, this seems like a case of closing the barn door after the horse has escaped. Successfully identifying a problem lab--albeit tardily--is a signal the system is working; not that it needs a complete overhaul. Rather consider what completely unrelated problems would benefit from regulatory oversight at a higher return on investment. Focus on those, not on a solved problem.
- testvox 9y ago
  The essentially free solution cost 25 years worth of bad results.
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- sbov 9y ago
  You're assuming there aren't any other problem labs that haven't or won't be discovered. How expensive are these tests? How much of the police's budget is it? Surely you know these things if you are talking return on investment, because you must know these things to actually know what your return on investment is. 25 years is a long time to be improperly ripping apart families. Is there a point at which it becomes a problem? Death tardily solves all your problems after only 80 years, so I assume somewhere between 25 and 80.
beached_whale 9y ago
It is different in Canada. These kinds of things will be fixed because the SCC will rule that the laws enabling it are without force and the government will get a year to fix it. It
- slededit 9y ago
  This was a failure of the court itself to enforce proper standards of evidence. For once you can't blame parliament.
  - beached_whale 9y ago
    It is really sad that it took over 30 years and the person is still working, albeit in Israel. But wouldn't the Evidence Act(provincial and federal) set the standards that allowed this?
    - slededit 9y ago
      The Evidence Act can further restrain the court but in a scenario that is ambiguous the court will rely on case law (e.g. law the court itself made up based on prior decisions). Should parliament disagree with the decisions of the court they may amend the law to force a change in future trials. We're used to hearing about the supreme court restraining the actions of parliament by declaring something unconstitutional but in actuality the parliament exercises far more control over the court than vice versa. Even in the case of the court declaring something unconstitutional parliament could overrule them by changing the constitution itself (although that is extremely unlikely given the political climate).
Clanan 9y ago
If someone had enough money, could they conduct this independently of the government parties?
- sigstoat 9y ago
  many of the government labs won't take samples from private parties, and the ones that do frequently don't advertise the fact. the private labs, sure, you could spike (or not) some blood samples and pay for them to be tested.
sigstoat 9y ago
accredited labs already have to do some of that. the organizations chosen (college of american pathologists, for example) to administer the tests are fucking garbage. so it's a bit harder to get this kind of validation going than just deciding to test the labs. > ...a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) haaah. you have no idea how many people are on something. getting known blank samples is nontrivial. > Although this simple, effective, and cheap method would completely transform forensic practice, it will not be implemented. The reason is incentives. The parties involved in forensics are the police or prosecutors and the labs. Both have an incentive to identify the guilty. there are independent labs that work for prosecution, defense, civil cases, etc etc every form of case you can imagine, and basically throw away all case information for the middle part of the testing process, so the folks who could hypothetically swing things one way or the other don't even know which way they'd be throwing cases if they tampered with it.
abecedarius 9y ago
Feynman gave a talk once which included a by-the-way remark like "this is not a scientific age". It stuck in my mind because of course that wasn't the common view of the 1950s or 60s when he said it. The obvious reason to say it is that some people still consume astrology, etc. But I came to realize that this is the sort of non-scientific-ageness that stands out. About all sorts of important things in daily life we don't collectively act as if we're curious about the truth and ready to do what it takes to learn it. If this were a scientific age then not routinely checking practices like this would stand out. Everyone would wonder if they had something to hide. Programming was less routinely scientific like that in my own experience earlier in my career when automated testing was less common. I think the Feynman talk was the one that turned into his posthumous book The meaning of it all, though I'm not sure.

by mcherm 9y ago

dsfyu404ed 9y ago

>That technique is blind testing with published results. 10% of all submissions to the testing lab would be test samples -- a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) and a few known positive samples (perhaps from someone already convicted with overwhelming evidence). The lab would not be told which samples were tests and which were "real", and the test samples would be evenly distributed (10% of the submissions from each department; 10% of the "rush" submissions, etc.). Finally, the results reported on these test samples would be made publicly available. And not only would it let you weed out the pseudoscience but it would let you weed out the labs that are practicing the real science well from the ones that are sloppy (which in my opinion is an equally large problem).

Zigurd 9y ago

That, and the labs should not be a branch of law enforcement or prosecutors' offices. They should be more like a weights and measures department, and be at the service of both prosecutors and defendants.

mabbo 9y ago
These labs are private businesses. Prosecutors and police can choose which ones to use and they will use, and they'll likely make that choice based on who gives them the results they want rather than the results that are accurate.
- Zigurd 9y ago
  In Massachusetts, there is a state police crime lab: https://www.mass.gov/orgs/massachusetts-state-police-crime-laboratory https://www.mass.gov/orgs/massachusetts-state-police-crime-l... It is a department of the state police, and, obviously enough, is structurally incentivized to make state police performance look good.
  - NutriSugar 9y ago
    So there is incentive for corruption with both public and private labs. The solution will require something far more drastic, which I'm not sure the population is willing to support given how many people don't like to use science when it disagrees with their emotional stances. As long as the population wants tough on crime stances, and are willing to dehumanize anyone charged (much less wrongly convicted), then the system is giving them what they want. So how can we change the voters so that they demand a more just legal system?
  - xkcd-sucks 9y ago
    The Massachusetts state crime lab is an actual madhouse, and the state government is shockingly corrupt. The following article is one of the more detailed descriptions of how the lab scandals were managed https://www.rollingstone.com/culture/news/did-falsified-drug-tests-lead-to-wrongful-convictions-w514801 https://www.rollingstone.com/culture/news/did-falsified-drug...

bmelton 9y ago

I think that secret shopping is a great idea for forensics, but I worry that in areas of criminality, just pushing fake requests over to forensics won't work unless it's accompanied by the usual pleas from the cops and DAs actively working the cases to which those data are associated. Similarly, a rush order is a rush order, but a rush order with a group of people carping on how it's a rush order tends to get more rushed. When the queues are effectively bottomless, stamping 'rush' on an order without the associated pressure seems likely to relegate its status to regular.

lttlrck 9y ago
Maybe submit multiple controls in the same request.

darkerside 9y ago

Something could trivially be done to the same effect. A private party, say an investigative journalist, could submit a number of samples and report on the results (perhaps that happened in this case and that's why we know about it?). But to keep costs under control, instead of testing like this on every lab on a regular basis, that private party could randomly select labs for testing. While they're at it, they'd probably prefer to select labs with complaints, rather than a random sample. And at that point, you have something resembling the system we currently operate under, which has--to be fair--been completely successful (if a bit later than ideal) in identifying this problem lab. But as taxpayers, is it worth instituting a new regulated program over this essentially free solution? Besides, this seems like a case of closing the barn door after the horse has escaped. Successfully identifying a problem lab--albeit tardily--is a signal the system is working; not that it needs a complete overhaul. Rather consider what completely unrelated problems would benefit from regulatory oversight at a higher return on investment. Focus on those, not on a solved problem.

testvox 9y ago
The essentially free solution cost 25 years worth of bad results.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
sbov 9y ago
You're assuming there aren't any other problem labs that haven't or won't be discovered. How expensive are these tests? How much of the police's budget is it? Surely you know these things if you are talking return on investment, because you must know these things to actually know what your return on investment is. 25 years is a long time to be improperly ripping apart families. Is there a point at which it becomes a problem? Death tardily solves all your problems after only 80 years, so I assume somewhere between 25 and 80.

beached_whale 9y ago

It is different in Canada. These kinds of things will be fixed because the SCC will rule that the laws enabling it are without force and the government will get a year to fix it. It

slededit 9y ago
This was a failure of the court itself to enforce proper standards of evidence. For once you can't blame parliament.
- beached_whale 9y ago
  It is really sad that it took over 30 years and the person is still working, albeit in Israel. But wouldn't the Evidence Act(provincial and federal) set the standards that allowed this?
  - slededit 9y ago
    The Evidence Act can further restrain the court but in a scenario that is ambiguous the court will rely on case law (e.g. law the court itself made up based on prior decisions). Should parliament disagree with the decisions of the court they may amend the law to force a change in future trials. We're used to hearing about the supreme court restraining the actions of parliament by declaring something unconstitutional but in actuality the parliament exercises far more control over the court than vice versa. Even in the case of the court declaring something unconstitutional parliament could overrule them by changing the constitution itself (although that is extremely unlikely given the political climate).

Clanan 9y ago

If someone had enough money, could they conduct this independently of the government parties?

sigstoat 9y ago
many of the government labs won't take samples from private parties, and the ones that do frequently don't advertise the fact. the private labs, sure, you could spike (or not) some blood samples and pay for them to be tested.

sigstoat 9y ago

accredited labs already have to do some of that. the organizations chosen (college of american pathologists, for example) to administer the tests are fucking garbage. so it's a bit harder to get this kind of validation going than just deciding to test the labs. > ...a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) haaah. you have no idea how many people are on something. getting known blank samples is nontrivial. > Although this simple, effective, and cheap method would completely transform forensic practice, it will not be implemented. The reason is incentives. The parties involved in forensics are the police or prosecutors and the labs. Both have an incentive to identify the guilty. there are independent labs that work for prosecution, defense, civil cases, etc etc every form of case you can imagine, and basically throw away all case information for the middle part of the testing process, so the folks who could hypothetically swing things one way or the other don't even know which way they'd be throwing cases if they tampered with it.

abecedarius 9y ago

Feynman gave a talk once which included a by-the-way remark like "this is not a scientific age". It stuck in my mind because of course that wasn't the common view of the 1950s or 60s when he said it. The obvious reason to say it is that some people still consume astrology, etc. But I came to realize that this is the sort of non-scientific-ageness that stands out. About all sorts of important things in daily life we don't collectively act as if we're curious about the truth and ready to do what it takes to learn it. If this were a scientific age then not routinely checking practices like this would stand out. Everyone would wonder if they had something to hide. Programming was less routinely scientific like that in my own experience earlier in my career when automated testing was less common. I think the Feynman talk was the one that turned into his posthumous book The meaning of it all, though I'm not sure.