3 ms·

Are forensic labs often get blind tested? If there is a bias for guilty cases then it should turn out in those blind tests. Source code is a red herring here, t

Are forensic labs often get blind tested? If there is a bias for guilty cases then it should turn out in those blind tests. Source code is a red herring here, there should be independent evaluations of forensic laboratories/methodologies/etc... regardless of software source code availability. Maybe these checks are already in place, I genuinely don't know.

cmpb 6y ago
>Source code is a red herring here If there were a way to ensure that the test suite applied to these forensic labs was all-encompassing w.r.t. the genetic variables at play, then maybe. But that sounds impossible. What if there's a coding error that causes the software to operate differently/incorrectly only for people with a certain (rare) genetic abnormality? For what it's worth, I'm totally unversed in genetics, though I have a great deal of experience writing software tests (and seeing them come up short in adequately modelling real-world data).
scott00 6y ago
I don't know the answer to your question, but blind testing is complementary to (not a substitute for) source code review. It's very common for software to work correctly a high percentage of the time, but fail on rare input data. If, say, the software works correctly 999,999 times out of a million, you're going to be very unlikely to discover that error by throwing random samples at it, especially if you need a physical process (ie, drawing blood) in order to generate a test case. On the other hand, once you have a known failing case (as you would if the defendant knows the result must be in error because he didn't commit the crime), it's often fairly straightforward to identify the error by reviewing the source and/or using a debugger to examine the progress of the algorithm.
- fritzo 6y ago
  Agreed, blind testing is important for statistical correctness and code review is important to avoid adversarial backdoors like dieselgate.
WaitWaitWha 6y ago
Not in my experience. Most labs of various types supposed to get certification, but these certifications primarily about chain of custody, operation protocols, record keeping, and such. It has little to do with the veracity of their conclusions.

cmpb 6y ago

>Source code is a red herring here If there were a way to ensure that the test suite applied to these forensic labs was all-encompassing w.r.t. the genetic variables at play, then maybe. But that sounds impossible. What if there's a coding error that causes the software to operate differently/incorrectly only for people with a certain (rare) genetic abnormality? For what it's worth, I'm totally unversed in genetics, though I have a great deal of experience writing software tests (and seeing them come up short in adequately modelling real-world data).

scott00 6y ago

I don't know the answer to your question, but blind testing is complementary to (not a substitute for) source code review. It's very common for software to work correctly a high percentage of the time, but fail on rare input data. If, say, the software works correctly 999,999 times out of a million, you're going to be very unlikely to discover that error by throwing random samples at it, especially if you need a physical process (ie, drawing blood) in order to generate a test case. On the other hand, once you have a known failing case (as you would if the defendant knows the result must be in error because he didn't commit the crime), it's often fairly straightforward to identify the error by reviewing the source and/or using a debugger to examine the progress of the algorithm.

fritzo 6y ago
Agreed, blind testing is important for statistical correctness and code review is important to avoid adversarial backdoors like dieselgate.

WaitWaitWha 6y ago

Not in my experience. Most labs of various types supposed to get certification, but these certifications primarily about chain of custody, operation protocols, record keeping, and such. It has little to do with the veracity of their conclusions.