34 ms·

Accused murderer wins right to check source code of DNA testing kit

981 points by anfilt 6y ago

ABeeSea 6y ago
> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. So it’s definitely riddled with bugs. And I can’t imagine that much matlab code following rigorous software engineering practices.
- bryanrasmussen 6y ago
  and surprisingly all of the code is of equal importance so you really need to review each line sequentially! Instead of finding stuff that you think is most likely to relate to what you're trying to figure out and debug from there. Wow I would like to see this marvel of engineering myself!
  - Scandiravian 6y ago
    Kind of makes me wonder: if the argument is that the code is too complex to review and understand, does that mean the company is not doing code reviews themselves?
    - michaelmior 6y ago
      If the code was developed in less than 8.5 years, it seems like the answer must be yes based on their previous claims.
      - gambiting 6y ago
        Well I mean if something requires 8 man years of work you can do it in a year with 8 people. Not always, but usually.
        agravier 6y ago
        On the contrary, such linear scaling would be quite exceptional. I'm speaking from experience but you don't need to trust me; I invite you read any book on software engineering management, starting from The Mythical Man-Month by Brooks.
        bryanrasmussen 6y ago
        the mythical man month was first published in 1975, I think the typical applications programmers work on today have changed significantly since then and encompass many different disciplines (to be thought professional) - so many disciplines that one developer is likely to be the master of all. It is true that there is a communication overhead to adding more people so it will not scale linearly, but if a single developer has taken 8 years to build something in our era it seems likely that having 8 people might get it done say 1 and a half to two years.
        markwkw 6y ago
        I would claim that if something requires 8 man years, that it will most definitely take more than a year to develop with 8 people. Communication takes time, coordination takes time, there is an incremental cost to each news person added to a team. From experience, perhaps with 2-3 people who happen to gel well together you may get close to proportional scaling of output, but with 8 it’s really unlikely in the real world.
        watwut 6y ago
        I would find it extremely unlikely to work that way.
    - bryanrasmussen 6y ago
      this has actually only happened to me a couple times but it has happened - someone tells me Bryan, go look at the code X did in Y, figure out if we refactor. X would then tell me - that code is really complicated is full of algorithms! I go and look at the code realize that for what it is trying to do can be cut down from 10 pages of printed code to less than 1 and it was incredibly simple what actually needed to be done. In short when someone tells me the stuff is too complicated because too clever and advanced I tend to disbelieve them. that said I have of course written my too complicated stuff lots of times, but if asked I don't say it was because I'm clever. names anonymized so as to not accidentally hurt anyone's feelings. on edit: actually one time the code was clever but not especially difficult, they just used the algorithms line because they didn't want anyone messing with their stuff.
      - Scandiravian 6y ago
        I think there's a bias towards judging things to be "clever" if they're hard to understand It's a cliche to have a "what idiot wrote this" outburst, then realise it's your own code, because most of us have written our fair share of "clever" code My boss explicitly stated that he doesn't want to see any "clever" or "smart" code in our product - write code based on simple fundamentals, benchmark before deciding to optimise, and be respectful in your reviews I like my boss a lot
    - watwut 6y ago
      You can do code review on every single commit while not being able to make overall analysis of the massive codebase.
      - Scandiravian 6y ago
        I completely agree, but if the reviewer isn't able to (with some amount of accuracy) predict the impact the committed code will have on overall behaviour, then there's very limited value on doing the review in the first place
        watwut 6y ago
        In any larger project, the reviewer are not able to predict impact from reading commit. More importantly, typical reviewer have only small partia area where he has good idea about which commit is bad idea. He however does not understand whole codebase. Knowing what the whole does and knowing what my module does are two different things.
        Scandiravian 6y ago
        I again completely agree with you Looking back at my reply, I think I should have added a bit of background to clarify my comment My master's degree is in bioinformatics and I worked in the biotech industry until about a year ago. I mainly worked as a consultant for top 20 pharma companies, but also did work on different in-house projects and in academia From my experience in the industry, I find it very unlikely that the software mentioned in the article is structured in a modular way. I've yet to see good software practices outside one or two academic projects. Most pharma companies still use copying and renaming folders as version control. Naturally I'm sceptical of any code coming from the biotech industry On top of that, it's written in MATLAB. I have only ever seen this used by statisticians and university researchers, never by software engineers I'm therefore willing to bet, that when the reviewers open the source code, they'll find unstructured mess of spaghetti code, that has never been refactored, reviewed or tested So yes - I agree in all your points, but I find it unlikely that they're being applied to this particular project
- syshum 6y ago
  I also cant image anyone thinking that would be a winning legal argument... "This software is too complex to look at so just trust us" Really... that is what they went with...
  - avs733 6y ago
    unfortunately, the technical illiteracy of much of the US judiciary has made this effective.
  - delecti 6y ago
    Yeah, that's one of those "defenses" that feels like a confession.
- Scandiravian 6y ago
  My exact thoughts. This sounds like a classic example of launching a prototype created by domain specialists (biostatisticians and bioinformaticians) as production software and skipping on the expensive stuff, like sound development practices
  - gww 6y ago
    There is so little emphasis in production software development in bioinformatics and biostatistics. Despite a lot of groups open sourcing their code it's is nearly unusable and not reproducible due to hard coding, ignoring edge cases, and dumping the majority of the code on a single giant R or python function.
    - Scandiravian 6y ago
      That's my experience as well. My master's is in bioinformatics and I worked for several years in biotech. I got frustrated because my concerns that my team's development practices were causing issues on a regular basis, were ignored. I was continuously able to predict what issues we would run into, but no-one seemed to care - I even had a manager tell me, that it was good that our software was buggy, since the client would continue paying us to fix it I've since left the biotech industry. There's a limit to how many times I want to run my head against that particular wall
    - natechols 6y ago
      It's a real problem, and I've been struggling with it for two decades, but even so I am legitimately impressed (and not in a good way) if they have 170,000 lines of Matlab code in their production software. That takes a really special combination of productivity and cluelessness, even for academic specialists. Regardless of the facts of this particular case, it should be absolutely horrifying that anyone's freedom is left up to a gigantic pile of unaudited Matlab code. (That said I am almost certain he added some zeros to the number, I have a hard time imagining what they're doing that could be that complex.)
      - Scandiravian 6y ago
- bitexploder 6y ago
  I have done a lot of source code review in my time. For security assessments. Our general rule of thumb was about 10k lines per week that we can really get deep on. 10 lines an hour would only be for the most dense code and critical path stuff. They will need a reviewer that knows the domain (DNA), but it’s perfectly reasonable to review that code on a weeks/months time scale, definitely not years.
- moron4hire 6y ago
  That's like an accountant, accused of embezalling, refusing to hand over the ledgers because "there are just too many records to go through". Yeah-no, that's kind of the whole point. We want to find out what you've been hiding in that wall of paper.
- partyboat1586 6y ago
  >170,000 lines of MATLAB code This is a deep problem. Many scientists don't understand software engineering and more and more need to write bigger and bigger programs. And most of the time they don't open source their code. Open source science.
onetimemanytime 6y ago
Makes perfect sense. You have the right to cross-examine the witness, especially when you're being accused of murder.
egberts 6y ago
Well, DNA matching is crapshoot and hazardous toward the innocents. We can still find unrelated folks with partial match by the virtue of segmentation. https://dna-explained.com/2017/01/19/concepts-segment-size-legitimate-and-false-matches/ https://dna-explained.com/2017/01/19/concepts-segment-size-l...
giantg2 6y ago
Now if this can become precedent for all evidence derived from automated systems. There have been some past battles about BAC machines in this regard.
Snoozus 6y ago
Why don't they just redo the analysis with a few other software packages?
- jonnycomputer 6y ago
  Who is "they"
  - unnouinceput 6y ago
    Probably "they" refer to prosecution. And yeah, why is prosecution going with that lab/software only? One reason is probably the DNA available for more testing is not enough / one testing could been done reliable with DNA taken from crime scene. Other reason might be behind the scenes incentives, which seems is the defense strategy here
    - jlgaddis 6y ago
      > why is prosecution going with that lab/software only? Presumably becquse that lab/software gave them the answer they were looking for. Why pay for even more lab/software testing when you've already got what you want (especially when it's a possibility that the additional testing will contradict the first)?
    - vaduz 6y ago
      Even if you had plenty of DNA material, going with multiple labs comes with a risk that one of them comes with a "no match" result - and that is something that is supposed to be disclosed to defense as potential exculpatory evidence - and is a nice source of reasonable doubt for a jury. It's something that is terribly inconvenient for the prosecution when someone has already been charged with the crime...
- MaxBarraclough 6y ago
  That would help with variance, but not with bias. Business incentives presumably reward all such companies for helping the police get convictions, so it doesn't seem a far-fetched concern.
motohagiography 6y ago
It's an important development. Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. They are as likely to lie as any other witness. Perhaps even moreso. "Those arguing on behalf of the defense cited past problems with other genetic testing software such as STRmix and FST (Forensic Statistical Tool). Defense expert witnesses Mats Heimdahl and Jeanna Matthews, for example, said that STRmix had 13 coding errors that affected 60 criminal cases, errors not revealed until a source code review." "They also pointed out, as the appeals court ruling describes, how an FST source code review "uncovered that a 'secret function . . . was present in the software, tending to overestimate the likelihood of guilt.'" An analogous situation with an alcohol and drug testing lab caused a scandal in Canada and called into question 16,000 child protective services cases: https://en.wikipedia.org/wiki/Motherisk https://en.wikipedia.org/wiki/Motherisk , and then at the same hospital(!), a forensic pathologist was giving fake prosecution evidence, https://en.wikipedia.org/wiki/Charles_Smith_(pathologist) https://en.wikipedia.org/wiki/Charles_Smith_(pathologist) Maybe there's just something about Toronto and compromised processes, but defense challenges to the integrity of automated systems looks like a growth field.
- MaxBarraclough 6y ago
  > Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. I imagine they're also incentivised to favour a result indicating guilt.
  - tantalor 6y ago
    Why? The risk to their reputation & livelihood for a false positive should temper that.
    - JoeAltmaier 6y ago
      While there have been cases of technicians guilty of systemic fudging of data, that is a tiny exception to the vast majority of honest technicians with self-respect.
ChrisRR 6y ago
This is similar to a case a few years back where someone asked to see the source code of a breathalyser that had found them guilty. AFAIR, the breathalyser was incorrectly averaging the readings, giving disproportional weight to the first reading. I don't know if it was enough to rule in their favour, but I'm sure it called the data into question Edit: Looks like it was a Draeger breathalyser https://www.schneier.com/blog/archives/2009/05/software_proble.html https://www.schneier.com/blog/archives/2009/05/software_prob...
- dathinab 6y ago
  I wouldn't be surprised if "incorrectly averaging" and similar are very common software errors. The reasons are manifold, including: - Normalized values need to be averaged differently the absolute values. - Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount. Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it. I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.
- text70 6y ago
  A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.
  - sdflhasjd 6y ago
    > A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.
    - radu_floricica 6y ago
      And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.
Mashimo 6y ago
Yay, free code review?
marcodiego 6y ago
Wouldn't this set a precedent to eventually force any forensic software to become open source?
- jMyles 6y ago
  It's unfathomable that the state can introduce evidence of any variety that isn't open source. Secret policing and secrecy in prosecution have no place in decent society.
- gsich 6y ago
  Hopefully.
- vincnetas 6y ago
  Code inspections is not same as open source.
- Zenst 6y ago
  Not per-say, though it certainly affords it the same level of scrutiny. Which may well be a win as open source already has that. However, even open source has bugs that pass scrutiny as many CVE's can attest, so whilst a code review may find nothing wrong, that in itself could be used by a lawyer to create reasonable doubt, if the lawyer is good.
  - jlgaddis 6y ago
    > per-say https://en.wiktionary.org/wiki/per_se https://en.wiktionary.org/wiki/per_se
  - deleted 6y ago
    [deleted]
- schmorptron 6y ago
  That'd be ideal
- jimnotgym 6y ago
  I guess open-source implementations of various algorithms that had been demonstrated effective, if produced by a not-for-profit maybe, could be very disruptive to this 'industry'.
steerablesafe 6y ago
Are forensic labs often get blind tested? If there is a bias for guilty cases then it should turn out in those blind tests. Source code is a red herring here, there should be independent evaluations of forensic laboratories/methodologies/etc... regardless of software source code availability. Maybe these checks are already in place, I genuinely don't know.
- cmpb 6y ago
  >Source code is a red herring here If there were a way to ensure that the test suite applied to these forensic labs was all-encompassing w.r.t. the genetic variables at play, then maybe. But that sounds impossible. What if there's a coding error that causes the software to operate differently/incorrectly only for people with a certain (rare) genetic abnormality? For what it's worth, I'm totally unversed in genetics, though I have a great deal of experience writing software tests (and seeing them come up short in adequately modelling real-world data).
- scott00 6y ago
  I don't know the answer to your question, but blind testing is complementary to (not a substitute for) source code review. It's very common for software to work correctly a high percentage of the time, but fail on rare input data. If, say, the software works correctly 999,999 times out of a million, you're going to be very unlikely to discover that error by throwing random samples at it, especially if you need a physical process (ie, drawing blood) in order to generate a test case. On the other hand, once you have a known failing case (as you would if the defendant knows the result must be in error because he didn't commit the crime), it's often fairly straightforward to identify the error by reviewing the source and/or using a debugger to examine the progress of the algorithm.
  - fritzo 6y ago
    Agreed, blind testing is important for statistical correctness and code review is important to avoid adversarial backdoors like dieselgate.
- WaitWaitWha 6y ago
zyngaro 6y ago
This kind of software should be required by law to be open source.
- TheRealDunkirk 6y ago
  Yes, just like voting machines. But how do we confirm that the software which was vetted is the software that was actually _used_?
  - calciphus 6y ago
    Compile it, run it with the same inputs the prosecution claims was provided, and see if you get the same result. That's sorta the point of computers, right? Same inputs on the same program, same output?
    - antidocker 6y ago
      What if compiler hijacked? What if the OS itself hijacked? What if the hardware has built in backdoor?
    - curryst 6y ago
      > Compile it, run it with the same inputs the prosecution claims was provided, and see if you get the same result. That's sorta the point of computers, right? Same inputs on the same program, same output? One difficulty here is that the input is a real world effect. That means the answers are rarely exactly the same. The last time I looked, breathalyzers have a ~50% margin of error. So if you blow a 0.06, your BAC is really somewhere between 0.03 and 0.09. You could mod your breathalyzer to always assume that the reading was 25% higher than the reality without being noticed, as long as your breathalyzer doesn't return a result that's already 25+% higher during testing. Even if it does, your other results will be in range. They need to just stop using breathalyzers. They're inaccurate pieces of machinery, operated by people without any medical knowledge to speak of, who have a vested interest in the readings being high. The whole thing, from top to bottom, is just a long chain of poorly aligned incentives. Field sobriety tests are even worse. The fact that your license can be revoked for refusing the test under the above grounds is nothing short of an end-run around the 4th amendment by declaring driving a "privilege, not a right" despite the fact that it is extremely hard to live in most of the US and not be able to drive. How the fuck do I survive in rural Oklahoma/Georgia/Kentucky/Texas/etc, 40 miles from the nearest bus stop, without a car? And all without any due process. It's never impacted me, but it bothers me nonetheless. They need to force people to go in for a blood draw to establish BAC. It is an accurate test, administered by competent medical professionals, who have no vested interest in the outcome either way. Yes it does take longer. Yes, some people who were at 0.08 will be down to 0.07 by the time the test is administered. I still find that more comforting than the fact that some people at 0.06 will read at 0.09, and some people at 0.14 are going to read at 0.07. There's also the fact that a cop who doesn't calibrate their breathalyzer is merely going to get a slap on the wrist, and a medical professional who does the same is probably going to end up with a huge malpractice suit and possibly lose their license if it was egregious in some way.
spaetzleesser 6y ago
There is so much junk science going on in forensics that it would be great to require everything to be open sourced. Same for voting machines and anything police in general is using (predictive policing is pretty scary). There is way too much stuff hidden and can be challenged only if you have very deep pockets.
hirundo 6y ago
If an accused person has the right to see the source code that produced evidence against them, is it a violation of their rights for the source code to be obfuscated, or even just so spaghettified that not even an expert can understand it? I kinda think that should be a violation. But deciding whether a particular piece of code is so bad is so subjective that I'm not sure on how you'd make a legal standard out of it. Maybe start with "the linter found a ratio of warnings to lines > X%" or some such. Having a legal standard of code coherence/incoherence might help filter pull requests. "This PR cannot be merged to this project because it is configured to reject legally incoherent code." As code becomes more complex it may become more meaningful to have access to the test suite, and to challenge the evidence if the tests are inadequate to demonstrate the correct code behavior.
- londons_explore 6y ago
  You don't need a legal standard. You just need to put doubt into the minds of a jury. You can get an expert to stand up and say "I'm an expert in computers, and I couldn't understand how this DNA test works. I think it's likely there are mistakes in it that neither I nor the people that made it have discovered".
  - deleted 6y ago
    [deleted]
  - Spivak 6y ago
    “This system is an absolute mess that is impossible audit and so there is no way that the company using could find bugs in it either.”
- Jakobeha 6y ago
  In the case where code is used to convict or acquit someone, I think it should be a well-tested and established program: generally something with the software quality of Linux, or in this case, whatever DNA testing kit is being used by scientists in top-ranking universities. We could also use formal verification based on well-established axioms. For example, maybe we could "prove" that the DNA kit reports accurate results as long as the samples it's given are processed correctly.
known 6y ago
Sounds rational since conviction should be on irrefutable evidence;
zupreme 6y ago
What is unfortunate is that it took going to appeal to force the judge to allow the code review at all. Without, at minimum, an independent review (and preferably open source code) the software and lab processes being used constitute an inscrutable "black box" process within which any judgment can be made, for any conceivable reason, with life-changing effects for the defendant (and for the victims of a crime if, for example, a rapist or murderer is set free by a non-match decision). One could even say that unreviewable code here falls under the umbrella of "secret evidence", which much of the world already knows can be easily misused and/or misapplied at the whim of the court.
- criddell 6y ago
  In Canada you can't even get the breathalyzer maintenance records: https://www.canadianlawyermag.com/news/general/maintenance-records-of-breathalyzer-machines-are-subject-to-third-party-disclosure-regime-scc/275576 https://www.canadianlawyermag.com/news/general/maintenance-r...
- vmception 6y ago
  People sometimes ask me what my “number” is, like how much net worth or “money” I want, what would I do with it I say “I want to be able to afford appeals court where my rights matter” Infinite appeals court! Most people plea out, cant make bail, dont have counsel buddy buddy with the judge enough to get you bail, and lose the ability to keep good counsel for more and more motions and appeals I want that, there is almost no pride in American rights if you cant afford them. People tie their whole identity to a system they arent even part of
  - roywiggins 6y ago
    Appeals? Only ~3% of people charged even go to trial, the rest plead out. Just giving everyone a substantive right to trial would amount to a revolution.
    - vmception 6y ago
      Yeah. But just giving myself that right first, forever.
amelius 6y ago
How does this work? They only have to find 1 bug and the entire case is dropped / postponed?
- carlmr 6y ago
  I guess they have to show this bug changes the result. But if you look at e.g. the case against Toyota where they had expert witnesses from Carnegie Melon and NASA testifying that the code was a horrible mess, I think that was sufficient for a multi billion dollar fine. So I guess with the admission from the DNA company that their code is basically unreviewable, written in a language known for prototyping, if they get an expert witness to testify this it may be enough without showing how the software misbehaves exactly.
trynton 6y ago
In such a serious case, once the accused was identified by TrueAllele, shouldn't a further test be carried out by a human to verify the results.
chadlavi 6y ago
While this may lead to more transparent technology use in the justice system, a more realistic outcome is that everyone accused of any crime where a technology was used in the process of determining guilt (which could get quite picayune; did the prosecutor's office use Excel? Is that something the defendant could demand to check the source code on?) will use this as a way to slow-roll the process indefinitely.
- tehwebguy 6y ago
  Well sure, if they rely entirely on Excel to generate and spit out an expert witness testimony. If that ever happens it will be extremely important to understand what’s happening under the hood in Excel.
- bagacrap 6y ago
  Definitely if an excel macro determined my guilt based on a dataset, I would like to be able to read that macro.
  - abfan1127 6y ago
    or at least double check the math.
- Spivak 6y ago
  Yes? I mean the intimate details of how cell networks work have been the difference between guilty and not before when the only evidence is some black-box report generated by AT&T that seems more accurate than is actually guaranteed. This wouldn’t be necessary if the expert on the stand was a geneticist who ran the test. But when the expert literally is the software you can’t really cross-examine the company’s sales rep.
- Jolter 6y ago
  Since when do defendants have an interest in prolonged sentencing processes? For major crimes like this, they are in jail with restrictions. They would get better treatment once sentenced, in the prison system, wouldn’t they?
pvitz 6y ago
I have worked with MATLAB code with 20,000 lines of code. Only over the past years, OOP and unit-testing has become properly available and usable. My guess is that this 170,000 lines are written in the old procedural way (also for performance reasons) and are full of bugs, also thanks to the lack of supporting tools. Most likely, this grew out of a research prototype that just worked too well to be reimplemented in a proper production environment.
- fatnoah 6y ago
  I wonder if bug tracker and other reports would be part of discovery in this case.
  - pvitz 6y ago
    Equally interesting is in my opinion who should do the review. Mathworks' own consulting service is probably the best to do so, but I wonder if they would objectively work against one of their own customers.
    - fatnoah 6y ago
      Yeah, any technical expert in a trial concerns me. I was an alternate juror (meaning I had to sit through the trial but was not allowed to take part in any deliberations) in a trial that involved the testimony of a computer "expert". The expert's testimony was 100% true and appeared to definitely prove X to someone who knew nothing about the subject matter. It was things analogous to saying the system was secure because it had a security chip. There were 1,000,000 questions I wished had been asked.
- mattkrause 6y ago
  Matlab has had classes (both types!) for ages. The unit-testing stuff dates back to at least 2013, and there were toolboxes to do similar things even before that. The language certainly has some warts, but IMO, the bigger problem is that it's usually learned/used in contexts that focus on code quality: the goal is the resulting number or plot rather than the software that generates them.
johnnyfived 6y ago
How would a team of independent reviewers even go about examining source code of this scale? How can you possibly find bugs that aren't super obvious?
- pavel_lishin 6y ago
  > How would a team of independent reviewers even go about examining source code of this scale? How can you possibly find bugs that aren't super obvious? Something tells me that 170k of Matlab code is going to contain plenty of obvious bugs.
- twobitshifter 6y ago
  The defense team only needs to provide the jury with a reasonable doubt. They don’t have to prove that it’s entirely wrong, only the existence of mistakes that call into question the overall accuracy. The founder has already made their argument for them by claiming that the entire codebase is impossible to review. “since a complete review is not possible, we reviewed a sample of x files and found y errors in the code. It is obvious that the developers has not adhered to the strict software code quality review standards that are necessary when dealing with life and death situations.”
phire 6y ago
> The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. First, the defence doesn't necessarily have to evaluate all 170,000 lines. They just need to find one buggy line which could potentially overturn the result. Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process?
- Cthulhu_ 6y ago
  > Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? No, but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. But I think it's warranted. An independent software review, and a double blind assertion with the exact version of the software used in the conviction to test the accuracy and reliability of the application.
  - edgyquant 6y ago
    Any software used to convict people, especially on a such a serious crime, should be audited like the fed is. Twice yearly, once by a public firm and another by the government itself. It should have to pass both of these audits to be used.
    - contingencies 6y ago
      It should be audited by science.
      - edgyquant 6y ago
        Yes there should be teams comprised of people whose expertise is in biology (specifically whatever field is responsible for DNA matching) and other people who are programmers that review the code and make sure there’s no mistakes. That would do it and I’m surprised this isn’t already a thing.
aaron695 6y ago
This isn't a DNA testing kit as one would normally think - "TrueAllele uses a hierarchical Bayesian probability model that adds genotype alleles, accounts for artifacts, and determines variance to explain STR data and derive parameter values and their uncertainty. The computer employs Markov chain Monte Carlo (MCMC) statistical sampling to solve the Bayesian equations. The resulting joint posterior probability provides marginal distributions for contributor genotypes, mixture weights, and other explanatory variables." https://onlinelibrary.wiley.com/doi/full/10.1111/1556-4029.14204 https://onlinelibrary.wiley.com/doi/full/10.1111/1556-4029.1...
- Scandiravian 6y ago
  How is that different from a DNA testing kit "as one would normally think"?
hastradamus 6y ago
The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. This is hilarious. As if you need to read every damn line and you can’t skip blank lines? You can skip whole files that aren’t relevant. Weak excuse
- heavyset_go 6y ago
  "It could take too long :(" is an interesting excuse not to examine the evidence that could put someone behind bars for the rest of their life.
- jariel 6y ago
  The statement is odd, at the same time, it's not outrageous for him to make in the sense that - lines of code notwithstanding - the underlying science i.e. the application of the product is the thing in question. It's a pretty interesting case. At least the core nature of the algorithm should be made public if we're going to use it for public inquisition.
- sudosysgen 6y ago
  And of course, as soon as you find an issue (which you probably will given that it's 170k lines of MATLAB), you can stop then and there.
- astura 6y ago
  This is a stupid "argument", I've personally participated in an audit of a program that has 5-6x as many lines of code.
duluca 6y ago
Things are only going to get murkier when leveraging ML. All of this needs to become open source.
- fritzo 6y ago
  Agreed, and with ML we'll need both open source and open training data.
deleted 6y ago
[deleted]
- AS37 6y ago
  I'd like to encourage you, 5 minutes after you learn about someone's existence, not to wish for their death, execution, imprisonment, and /or disenfranchisement. The real world is too complicated to judge correctly in that time. As well, please remember that the comment guidelines exist at https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - deleted 6y ago
    [deleted]
    - deleted 6y ago
      [deleted]
  - dang 6y ago
    Thanks for pointing this out! We've deleted that comment by the commenter's request.
- retrac 6y ago
  There's a certain irony in advocating for capital punishment on a thread about the dangers of lying prosecutors, bogus forensics, and false convictions.
- deleted 6y ago
  [deleted]
- dang 6y ago
  We detached this subthread from https://news.ycombinator.com/item?id=26064707 https://news.ycombinator.com/item?id=26064707.
deleted 6y ago
[deleted]
fxleach 6y ago
> it would take eight and a half years to review at a rate of ten lines an hour Wow, the co-founder's argument to not disclose the source code to the defendant was that it's too many lines of code. Also... ten lines reviewed in an hour!?
- meowster 6y ago
  In the TV shows and books, opposing counsel always want to inundate the protagonist with truckloads of boxes full of printed papers when they could just handover a flashdrive instead. Are there any lawyers here who can speak to the accuracy of such portrayal? It would seem the next logical step would be for every other lawyer to say they shouldn't hand over discovery because it would take to long for the otherside when they bury them in paperwork.
  - _Wintermute 6y ago
    Not sure if it's the best source, but according to a "Youtube laywer reacts" video [0], it's a common tactic but you can complain about it to the judge and request documents in a more appropriate form. [0] https://youtu.be/spr5smxuO5E?t=1284 https://youtu.be/spr5smxuO5E?t=1284
fritzo 6y ago
Code review is not the way to validate statistical inferences. Even during internal review of statistical inference code, committers argue correctness through a combination of readable code, readable tests, and statistical validation on a combination of synthetic and real data. While I agree the TrueAllele source should be provided, the "zero-defects" bar is neither sufficient nor necessary for correctness of the inferences made.
- damagednoob 6y ago
  How much uncertainty in the system are you willing to live with if you were falsely accused of murder?
  - fritzo 6y ago
    That is a question for the judge or jury. The statistician's job is to provide a probability distribution, and to argue for the statistical correctness of that distribution based on similar inferences made on validation data.
    - damagednoob 6y ago
      In that case, at least with respect to the death penalty, "zero-defects" is precisely what the courts have moved towards.
- akyu 6y ago
  > Code review is not the way to validate statistical inferences. But it is a way to invalidate the inference, which is what the defense is interested in.
qwertox 6y ago
Here in Germany we have somewhat similar cases, but where the accusation is way less damaging than the case of this article, in which a false positive would have the drastic result of being labeled a murderer. The cases are related to new speeding cameras which work with laser, where the defendants are complaining that these new devices are black boxes, and that they demand access to the raw data which these devices process. The problem is that these devices discard the raw data after having processed it and come to a conclusion that the driver was or was not speeding. The devices in question are Traffistar S350 from Jenoptik and PoliScan SM1 from Vitronic. There were discussions about a required software update which retains all this data, but apparently the devices lack the storage capability to do so. The National Metrology Institute of Germany (Physikalisch-Technische Bundesanstalt (PTB)) responded to this, that they would not re-certify these devices with updated software because from their point of view they work "as specified".
- Kinrany 6y ago
  Surely there aren't so many speeders that storing a few seconds per violation would add up to a significant amount.
- nolok 6y ago
  Now all you need is the proper amount of collusion/corruption between the certifying agency and the manufacturer to have a magic box that does whatever the one paying the bills want. Might seem far fetched in a developped country, until you read about the Boeing/FAA thing happened.
  - josefx 6y ago
    As far as I could find the courts basically decided that there has to be a way to examine how a result was reached if there was any doubt about it. That the PTB still allowed their use didn't change that and if a case got to court the results could get thrown out. The PTB probably doesn't care because only a small percentage of speeding cases end up in court.
dannykwells 6y ago
"170,000 lines of MATLAB code" (used as software to determine if humans are spending their lives in jail). Well if that's not a line to make you feel awful on a Monday morning.
zyxzevn 6y ago
All source code for important decisions should be open and be verified by different parties. Just as that for voting machines.
LatteLazy 6y ago
We had a case in the uk with accounting software. Dozens of people were convicted over decades because the software couldn't be wrong. Until eventually someone actually checked. https://www.bbc.co.uk/news/business-54384427 https://www.bbc.co.uk/news/business-54384427
pnathan 6y ago
This is excellent! As a principle, we should always be able to properly critique the expert witnesses and analyses! Or, at least, hire an expert firm to do the critique. It at least restores a proper ability to challenge an assertion of guilt.
rland 6y ago
These companies are disgusting. They peddle black box "models," that essentially ride the good reputation of DNA as infallible (which it is most certainly not) to get convictions on dubious or no evidence. The way it works is that if there is a sample from a crime scene, they send it to these guys and they analyze it with their software to detect "statistical" DNA from the sample. These samples are the ones that are too crappy to actually make a definitive match -- they are a statistical match. So you say "I think Jim, Bob, and Alice were on scene," and it says "10% likelihood Jim DNA, 5% likelihood Bob DNA, 45% Alice DNA." Do you think it ever says "99% no DNA" in the sample? It's basically Theranos, except instead of wasting $50 on a shitty blood test you get life in prison. Ostensibly, it searches the entire DNA database for matches, and only returns a positive result if there's a positive match. But it's a statistical model, using inputs that are crappy at best (because if it was an actual DNA match, they would send it off to in house forensics who would be able to do PCR...) and which includes inputs from circumstantial evidence as priors. Like we believe Alice was at the scene therefore if you find any statistical likelihood that this is Alice's DNA boost that. They often run the model multiple times in a row, and use the result that the DA likes the most to enter into evidence. This is because the models return different results each time -- of course they'd say, iTs StAtiStIcaL, so they can do that... And the source code is completely impenetrable. They argue that it's a "trade secret" that jeopardizes their ability to make future profits, so it cannot be open-sourced. These guys could have a model that just says "what percentage should the thing read, Señor D.A.?" The entire product is a sham. And because it's 170k LOC, no one has the time or the qualifications (Judges/Attorneys reading source code? Yeah right!) to review it, even if it were open source. Pure quackery, and often times, decades-long sentences or life in prison for the defendant. These companies are pure filth worthy of the lowest revulsion. It's a wonder any convictions happen at all because of this stuff, but jurors have very inaccurate conceptions of forensic science, thanks to shit like CSI, Law and Order, etc. These companies happily play into that image and people really believe this stuff works.
- deleted 6y ago
  [deleted]
- jrm5100 6y ago
  Issues with source code access aside, your description is mostly wrong. These programs take a DNA profile as input- it's just that the DNA profile is mixed (i.e. from multiple people). It reporting no DNA would be nonsensical. Figuring out exactly how many people are in a mixture isn't quite nailed down statistically (last I knew of), but it's usually pretty clear for up to 4 or so people. Yes, you could run different models and get different probabilities. For example, the likelihood that the sample is a mixture of the suspect, the victim, and some unknown person vs victim and two unknown people compared to saying the victim isn't in the sample. However, the specification of those models is part of the trial process. And the output probabilities (at least when being used to determine guilt) are usually quite high, orders of magnitude higher than 90% or even 99.99%. My point is that the science behind these calculations is well developed- validation studies get published all the time. Whether or not the specific software has errors (or isn't coded exactly as modeled) is an entirely different matter, but it still isn't all that likely. All of these cases rely on expert witnesses anyway- it's not the prosecutor pressing some buttons and printing a report. There is far more concerning quackery that gets used in forensics- bite marks, hair matching, etc.
vlovich123 6y ago
I'm curious. I've always been frustrated by this "closed" business model in the legal system. I feel like the entire process & details should be detailed in the open (code, methodology, controls, etc). Of course the counterpoint is that it makes it easier to copy this business & undercut all the time & energy spent on building it (copying is easier). Is that the only reason? I feel like open kimono is a critically important concept for anything related to the legal system because of how any perversion removes its legitimacy. If it really is that prohibitive to run a profitable business in this space, is there open standards that can be enforced (e.g. "this is the core algorithm that is approved" & businesses must get regular audits to continue to be used & any failed audit causes a reexamining of any court cases you were involved in the past year?). That's less ideal because then who audits the auditors but maybe at least it's an acceptable middle ground from where we are? In general I've been extremely frustrated how regularly & consistently this entire industry keeps everything secretive & trust-based despite consistent examples of how insufficient trust is for this field & how devastating the results are when that trust is violated.
lowbloodsugar 6y ago
"The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour." Well then the probability that it works correctly is zero.
femto113 6y ago
Interesting that they're putting so much focus on the DNA piece, since the actual crime was witnessed by a police officer and there is circumstantial evidence tying the suspect to the weapon (it was found along the path they used to flee from the scene).
LockAndLol 6y ago
For the Europeans among us, I would like to remind you that there's an initiative to opensource all publicly funded code https://publiccode.eu/ https://publiccode.eu/ If you care, add your signature and vote for a party during the next elections that considers opensource at least somewhat important.
WaitWaitWha 6y ago
The tone of this thread is leaning to "forensics evil, government lackeys out to get the little guy". In the USA, all sides can call and rely on their own forensics. There is no government mandated & approved single-source-of-truth (with some minor exceptions). > Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. This statement is at minimum an extreme generalization. Forensic field is a very large field, with many government and private "forensicators". There are no "black box processes". The very word "forensic" is based on presenting in full view, front of the fact finders (jury & judges usually). How deep this gets dug into depends on the fact finders and attorneys. In all court cases the forensic examiner can be called to demonstrate with extreme nuance how they performed the procedures. Any tools used are can be requested to be examined, including software. Vendors that I have worked with all have experts on staff specifically to appear in court and detail the inner workings of their tools. Not only tools, processes, and environments, but the examiner can be drilled on their experience, education, degrees, previous cases, failures, etc. They do call it "voir dire" for a reason. This does not negate the fact that there are some bad apples, bad prosecutors, and bad judges.
- anomaloustho 6y ago
  > Any tools used are can be requested to be examined, including software. Out of curiosity, if this is a common occurrence that is willfully obeyed by all vendors and parties involved. Could you shed some light on why this vendor is resisting?
  - WaitWaitWha 6y ago
    No, it is not common but does happen. There are several cases on drug testing, DNA, bite mark, and software that comes to mind. Scientifically well established processes, tools, etc. are rarely called up. Bite mark was a big deal in the industry because it turns out, it is not so unique and the methodologies developed were weak. My personal opinion why the are resisting? They are resisting because they are bottom dwellers. Forensicators whom I associate with, tool vendors, and I understand that we have to share knowledge. I dare say, we are the most open source scientific knowledge industry. This is not because of some altruistic reason, but because tomorrow they can be called on to explain.
Thorentis 6y ago
> If TrueAllele is found wanting, presumably that will not affect the dozen individuals said to have been exonerated by the software. Buried right at the end, but an interesting thought. It would depend on the individual cases, but if there was a whole body of evidence vs. one DNA test, surely these cases would need to be retried?
- clarle 6y ago
  IANAL but if the individuals were already found not guilty, I think double jeopardy laws would apply. The prosecution wouldn't be able to try again unless they made an appeal that there's a substantially different crime.
brutal_chaos_ 6y ago
Disclaimer, I am FOR seeing the source in these cases. Say the defense finds the software in a very, very troubling state. Could that be telling enough to not trust the outcomes of the software? Assuming the DNA parts were correct, but the rest is junk, this is where the plaintiff would bring in their expert to counter the arguments? Perhaps ^ is just a weak argument. I don't know how well versed courts are in these matters. (My guess is NOT that well versed because of the CFAA of the 80's up to Aaron Schwartz, and more). My legal background is Law and Order, so I have no idea what I'm talking about, just curious.
kazinator 6y ago
> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour What an deplorably dishonest argument to present to laypeople. If there is some weakness in it so the accused has a way to plausibly deny the DNA test, it will take a consultant who has relevant competence something like week or two to find it. Finding a problem doesn't require looking at every line of 170,000.
ineedasername 6y ago
If ever there was something that should be fully transparent it is the mechanisms by which a person might be found guilty of a crime. The defendant shouldn't even have had to fight for this. It should be a fundamental cornerstone of criminal prosecutions.
- meowface 6y ago
  These cases also often come up with drug and alcohol detection tests, and as John Oliver points out in https://www.youtube.com/watch?v=1f2iawp0y5Y https://www.youtube.com/watch?v=1f2iawp0y5Y, software used to select jurors. All of these companies claim that their source code is valuable intellectual property and that disclosing it can hurt their business. Even if this were true, when you're providing something that can be a significant factor in someone being imprisoned or executed, when creating the business you should accept that you're providing a public service that needs to be publicly accountable. If it's not open source, at the very least there should be a requirement that software code and hardware designs must be provided on-demand to experts in court cases (with a non-disclosure clause to mitigate leaks and corporate espionage etc.).
  - alistairSH 6y ago
    ...disclosing it can hurt their business... And there's an entire body of law based around IP which they can use to protect their business, just like everybody else.
    - zymhan 6y ago
      Their business interests are also of minuscule importance compared to the impact to society these tools have.
  - bargl 6y ago
    Without jumping on the conspiracy bandwagon, I'd also like to see this applied to voting software. I know it's a hot topic, and I'm honestly not trying to get political. Software that is critical to our fundamental human rights, and is being used by our government should be open source, or at least audited by a group of people who sign Non-competes/NDA and can't go work for competitors, or with some other mechanism to protect IP that I can't think of.
rudylee 6y ago
Why can't just they run the DNA test again in 2 or 3 other different software ? If all of them come back with the same result then we can assume that the first software is not buggy right ?
- handoflixue 6y ago
  It could be that they're all copying the same Stack Exchange answer, or making the same basic mistake - If 50% of programmers do averages wrong, it's not ridiculous to claim that 3 in a row got it wrong. Tripling your costs and still having a 12.5% failure rate isn't great.
protomyth 6y ago
I do wonder if the next step is check the compiler / interpreter the code is running on. Is Matlab assumed to be error free in this situation? Will certified compilers and CPUs be required in the future? Looks like someone could end up reviving a modern version of the Viper https://en.wikipedia.org/wiki/VIPER_microprocessor https://en.wikipedia.org/wiki/VIPER_microprocessor
iamleppert 6y ago
Any company who has software that is used in such a way should have independent review of their source code. It’s amazing to me they didn’t think of this originally and have the necessary endorsements already available for viewing.
Abimelex 6y ago
I'm not familiar with genome sequencing, but why are they even relying only on one program? Wouldn't it be possible to check it with different programs to reduce the possibility of failure here? And just assuming they find a possible bug, would that mean all the sentences based on the programs result will be obsolete?
astura 6y ago
There should be no such thing as a "trade secret" when talking about evidence that is used to convict someone. Correct judgement here.
dwg 6y ago
> The maker of the software, Cybergenetics, has insisted in lower court proceedings that the program's source code is a trade secret. Should this be, in general, a disqualifying condition when something is to be used as evidence?
jxramos 6y ago
Aren't there lower hanging fruits to go after? Like are there not competitor or predicate devices to run the same DNA against and see what it says as far as a match goes?
kovac 6y ago
I don't think it's meaningful to let defendant review the code each trial. Such code reviews should have been done before it was ever allowed to be used in the legal system. On the other hand, there's always the more practical option of getting the dna testing done by a second, third or even a tenth different approved company if the defendants want to pursue this line of reasoning. From what I understand, dna testing is statistical which means even if they find some bug in the code there will have to be extended debates on whether or not a particular bug affected the credibility of this test result. This is just bullocks. Having said that, the excuse the software company is giving is a bit lame.
michaelmrose 6y ago
One would suppose that for a word of such import there would be a detailed analysis of the work product along with testing they had done to prove it actually worked. They ought to ask for access to THAT in addition to the source code. One would imagine that it doesn't come with 170k lines of documentation or analysis and it would be orders of magnitude easier to review than the source code alone.
anfilt 6y ago
Link to the ruling: https://www.eff.org/files/2021/02/03/decision_appdiv_232021.pdf https://www.eff.org/files/2021/02/03/decision_appdiv_232021....

981 points by anfilt 6y ago

> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. So it’s definitely riddled with bugs. And I can’t imagine that much matlab code following rigorous software engineering practices.

bryanrasmussen 6y ago
and surprisingly all of the code is of equal importance so you really need to review each line sequentially! Instead of finding stuff that you think is most likely to relate to what you're trying to figure out and debug from there. Wow I would like to see this marvel of engineering myself!
- Scandiravian 6y ago
  Kind of makes me wonder: if the argument is that the code is too complex to review and understand, does that mean the company is not doing code reviews themselves?
  - michaelmior 6y ago
    If the code was developed in less than 8.5 years, it seems like the answer must be yes based on their previous claims.
    - gambiting 6y ago
      Well I mean if something requires 8 man years of work you can do it in a year with 8 people. Not always, but usually.
      - agravier 6y ago
        On the contrary, such linear scaling would be quite exceptional. I'm speaking from experience but you don't need to trust me; I invite you read any book on software engineering management, starting from The Mythical Man-Month by Brooks.
        bryanrasmussen 6y ago
        the mythical man month was first published in 1975, I think the typical applications programmers work on today have changed significantly since then and encompass many different disciplines (to be thought professional) - so many disciplines that one developer is likely to be the master of all. It is true that there is a communication overhead to adding more people so it will not scale linearly, but if a single developer has taken 8 years to build something in our era it seems likely that having 8 people might get it done say 1 and a half to two years.
      - markwkw 6y ago
        I would claim that if something requires 8 man years, that it will most definitely take more than a year to develop with 8 people. Communication takes time, coordination takes time, there is an incremental cost to each news person added to a team. From experience, perhaps with 2-3 people who happen to gel well together you may get close to proportional scaling of output, but with 8 it’s really unlikely in the real world.
      - watwut 6y ago
        I would find it extremely unlikely to work that way.
  - bryanrasmussen 6y ago
    this has actually only happened to me a couple times but it has happened - someone tells me Bryan, go look at the code X did in Y, figure out if we refactor. X would then tell me - that code is really complicated is full of algorithms! I go and look at the code realize that for what it is trying to do can be cut down from 10 pages of printed code to less than 1 and it was incredibly simple what actually needed to be done. In short when someone tells me the stuff is too complicated because too clever and advanced I tend to disbelieve them. that said I have of course written my too complicated stuff lots of times, but if asked I don't say it was because I'm clever. names anonymized so as to not accidentally hurt anyone's feelings. on edit: actually one time the code was clever but not especially difficult, they just used the algorithms line because they didn't want anyone messing with their stuff.
    - Scandiravian 6y ago
      I think there's a bias towards judging things to be "clever" if they're hard to understand It's a cliche to have a "what idiot wrote this" outburst, then realise it's your own code, because most of us have written our fair share of "clever" code My boss explicitly stated that he doesn't want to see any "clever" or "smart" code in our product - write code based on simple fundamentals, benchmark before deciding to optimise, and be respectful in your reviews I like my boss a lot
  - watwut 6y ago
    You can do code review on every single commit while not being able to make overall analysis of the massive codebase.
    - Scandiravian 6y ago
      I completely agree, but if the reviewer isn't able to (with some amount of accuracy) predict the impact the committed code will have on overall behaviour, then there's very limited value on doing the review in the first place
      - watwut 6y ago
        In any larger project, the reviewer are not able to predict impact from reading commit. More importantly, typical reviewer have only small partia area where he has good idea about which commit is bad idea. He however does not understand whole codebase. Knowing what the whole does and knowing what my module does are two different things.
        Scandiravian 6y ago
        I again completely agree with you Looking back at my reply, I think I should have added a bit of background to clarify my comment My master's degree is in bioinformatics and I worked in the biotech industry until about a year ago. I mainly worked as a consultant for top 20 pharma companies, but also did work on different in-house projects and in academia From my experience in the industry, I find it very unlikely that the software mentioned in the article is structured in a modular way. I've yet to see good software practices outside one or two academic projects. Most pharma companies still use copying and renaming folders as version control. Naturally I'm sceptical of any code coming from the biotech industry On top of that, it's written in MATLAB. I have only ever seen this used by statisticians and university researchers, never by software engineers I'm therefore willing to bet, that when the reviewers open the source code, they'll find unstructured mess of spaghetti code, that has never been refactored, reviewed or tested So yes - I agree in all your points, but I find it unlikely that they're being applied to this particular project
syshum 6y ago
I also cant image anyone thinking that would be a winning legal argument... "This software is too complex to look at so just trust us" Really... that is what they went with...
- avs733 6y ago
  unfortunately, the technical illiteracy of much of the US judiciary has made this effective.
- delecti 6y ago
  Yeah, that's one of those "defenses" that feels like a confession.
Scandiravian 6y ago
My exact thoughts. This sounds like a classic example of launching a prototype created by domain specialists (biostatisticians and bioinformaticians) as production software and skipping on the expensive stuff, like sound development practices
- gww 6y ago
  There is so little emphasis in production software development in bioinformatics and biostatistics. Despite a lot of groups open sourcing their code it's is nearly unusable and not reproducible due to hard coding, ignoring edge cases, and dumping the majority of the code on a single giant R or python function.
  - Scandiravian 6y ago
    That's my experience as well. My master's is in bioinformatics and I worked for several years in biotech. I got frustrated because my concerns that my team's development practices were causing issues on a regular basis, were ignored. I was continuously able to predict what issues we would run into, but no-one seemed to care - I even had a manager tell me, that it was good that our software was buggy, since the client would continue paying us to fix it I've since left the biotech industry. There's a limit to how many times I want to run my head against that particular wall
  - natechols 6y ago
    It's a real problem, and I've been struggling with it for two decades, but even so I am legitimately impressed (and not in a good way) if they have 170,000 lines of Matlab code in their production software. That takes a really special combination of productivity and cluelessness, even for academic specialists. Regardless of the facts of this particular case, it should be absolutely horrifying that anyone's freedom is left up to a gigantic pile of unaudited Matlab code. (That said I am almost certain he added some zeros to the number, I have a hard time imagining what they're doing that could be that complex.)
    - Scandiravian 6y ago
bitexploder 6y ago
I have done a lot of source code review in my time. For security assessments. Our general rule of thumb was about 10k lines per week that we can really get deep on. 10 lines an hour would only be for the most dense code and critical path stuff. They will need a reviewer that knows the domain (DNA), but it’s perfectly reasonable to review that code on a weeks/months time scale, definitely not years.
moron4hire 6y ago
That's like an accountant, accused of embezalling, refusing to hand over the ledgers because "there are just too many records to go through". Yeah-no, that's kind of the whole point. We want to find out what you've been hiding in that wall of paper.
partyboat1586 6y ago
>170,000 lines of MATLAB code This is a deep problem. Many scientists don't understand software engineering and more and more need to write bigger and bigger programs. And most of the time they don't open source their code. Open source science.

onetimemanytime 6y ago

Makes perfect sense. You have the right to cross-examine the witness, especially when you're being accused of murder.

egberts 6y ago

Well, DNA matching is crapshoot and hazardous toward the innocents. We can still find unrelated folks with partial match by the virtue of segmentation. https://dna-explained.com/2017/01/19/concepts-segment-size-legitimate-and-false-matches/ https://dna-explained.com/2017/01/19/concepts-segment-size-l...

giantg2 6y ago

Now if this can become precedent for all evidence derived from automated systems. There have been some past battles about BAC machines in this regard.

Snoozus 6y ago

Why don't they just redo the analysis with a few other software packages?

jonnycomputer 6y ago
Who is "they"
- unnouinceput 6y ago
  Probably "they" refer to prosecution. And yeah, why is prosecution going with that lab/software only? One reason is probably the DNA available for more testing is not enough / one testing could been done reliable with DNA taken from crime scene. Other reason might be behind the scenes incentives, which seems is the defense strategy here
  - jlgaddis 6y ago
    > why is prosecution going with that lab/software only? Presumably becquse that lab/software gave them the answer they were looking for. Why pay for even more lab/software testing when you've already got what you want (especially when it's a possibility that the additional testing will contradict the first)?
  - vaduz 6y ago
    Even if you had plenty of DNA material, going with multiple labs comes with a risk that one of them comes with a "no match" result - and that is something that is supposed to be disclosed to defense as potential exculpatory evidence - and is a nice source of reasonable doubt for a jury. It's something that is terribly inconvenient for the prosecution when someone has already been charged with the crime...
MaxBarraclough 6y ago
That would help with variance, but not with bias. Business incentives presumably reward all such companies for helping the police get convictions, so it doesn't seem a far-fetched concern.

motohagiography 6y ago

It's an important development. Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. They are as likely to lie as any other witness. Perhaps even moreso. "Those arguing on behalf of the defense cited past problems with other genetic testing software such as STRmix and FST (Forensic Statistical Tool). Defense expert witnesses Mats Heimdahl and Jeanna Matthews, for example, said that STRmix had 13 coding errors that affected 60 criminal cases, errors not revealed until a source code review." "They also pointed out, as the appeals court ruling describes, how an FST source code review "uncovered that a 'secret function . . . was present in the software, tending to overestimate the likelihood of guilt.'" An analogous situation with an alcohol and drug testing lab caused a scandal in Canada and called into question 16,000 child protective services cases: https://en.wikipedia.org/wiki/Motherisk https://en.wikipedia.org/wiki/Motherisk , and then at the same hospital(!), a forensic pathologist was giving fake prosecution evidence, https://en.wikipedia.org/wiki/Charles_Smith_(pathologist) https://en.wikipedia.org/wiki/Charles_Smith_(pathologist) Maybe there's just something about Toronto and compromised processes, but defense challenges to the integrity of automated systems looks like a growth field.

MaxBarraclough 6y ago
> Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. I imagine they're also incentivised to favour a result indicating guilt.
- tantalor 6y ago
  Why? The risk to their reputation & livelihood for a false positive should temper that.
  - JoeAltmaier 6y ago
    While there have been cases of technicians guilty of systemic fudging of data, that is a tiny exception to the vast majority of honest technicians with self-respect.

ChrisRR 6y ago

This is similar to a case a few years back where someone asked to see the source code of a breathalyser that had found them guilty. AFAIR, the breathalyser was incorrectly averaging the readings, giving disproportional weight to the first reading. I don't know if it was enough to rule in their favour, but I'm sure it called the data into question Edit: Looks like it was a Draeger breathalyser https://www.schneier.com/blog/archives/2009/05/software_proble.html https://www.schneier.com/blog/archives/2009/05/software_prob...

dathinab 6y ago
I wouldn't be surprised if "incorrectly averaging" and similar are very common software errors. The reasons are manifold, including: - Normalized values need to be averaged differently the absolute values. - Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount. Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it. I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.
text70 6y ago
A tool designed to find people guilty is biased to find people guilty. As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.
- sdflhasjd 6y ago
  > A tool designed to find people guilty is biased to find people guilty. I don't see those particular issues make it biased, just inaccurate - it could go either way.
  - radu_floricica 6y ago
    And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry. Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.

Mashimo 6y ago

Yay, free code review?

marcodiego 6y ago

Wouldn't this set a precedent to eventually force any forensic software to become open source?

jMyles 6y ago
It's unfathomable that the state can introduce evidence of any variety that isn't open source. Secret policing and secrecy in prosecution have no place in decent society.
gsich 6y ago
Hopefully.
vincnetas 6y ago
Code inspections is not same as open source.
Zenst 6y ago
Not per-say, though it certainly affords it the same level of scrutiny. Which may well be a win as open source already has that. However, even open source has bugs that pass scrutiny as many CVE's can attest, so whilst a code review may find nothing wrong, that in itself could be used by a lawyer to create reasonable doubt, if the lawyer is good.
- jlgaddis 6y ago
  > per-say https://en.wiktionary.org/wiki/per_se https://en.wiktionary.org/wiki/per_se
- deleted 6y ago
  [deleted]
schmorptron 6y ago
That'd be ideal
jimnotgym 6y ago
I guess open-source implementations of various algorithms that had been demonstrated effective, if produced by a not-for-profit maybe, could be very disruptive to this 'industry'.

steerablesafe 6y ago

Are forensic labs often get blind tested? If there is a bias for guilty cases then it should turn out in those blind tests. Source code is a red herring here, there should be independent evaluations of forensic laboratories/methodologies/etc... regardless of software source code availability. Maybe these checks are already in place, I genuinely don't know.

cmpb 6y ago
>Source code is a red herring here If there were a way to ensure that the test suite applied to these forensic labs was all-encompassing w.r.t. the genetic variables at play, then maybe. But that sounds impossible. What if there's a coding error that causes the software to operate differently/incorrectly only for people with a certain (rare) genetic abnormality? For what it's worth, I'm totally unversed in genetics, though I have a great deal of experience writing software tests (and seeing them come up short in adequately modelling real-world data).
scott00 6y ago
I don't know the answer to your question, but blind testing is complementary to (not a substitute for) source code review. It's very common for software to work correctly a high percentage of the time, but fail on rare input data. If, say, the software works correctly 999,999 times out of a million, you're going to be very unlikely to discover that error by throwing random samples at it, especially if you need a physical process (ie, drawing blood) in order to generate a test case. On the other hand, once you have a known failing case (as you would if the defendant knows the result must be in error because he didn't commit the crime), it's often fairly straightforward to identify the error by reviewing the source and/or using a debugger to examine the progress of the algorithm.
- fritzo 6y ago
  Agreed, blind testing is important for statistical correctness and code review is important to avoid adversarial backdoors like dieselgate.
WaitWaitWha 6y ago

zyngaro 6y ago

This kind of software should be required by law to be open source.

TheRealDunkirk 6y ago
Yes, just like voting machines. But how do we confirm that the software which was vetted is the software that was actually _used_?
- calciphus 6y ago
  Compile it, run it with the same inputs the prosecution claims was provided, and see if you get the same result. That's sorta the point of computers, right? Same inputs on the same program, same output?
  - antidocker 6y ago
    What if compiler hijacked? What if the OS itself hijacked? What if the hardware has built in backdoor?
  - curryst 6y ago
    > Compile it, run it with the same inputs the prosecution claims was provided, and see if you get the same result. That's sorta the point of computers, right? Same inputs on the same program, same output? One difficulty here is that the input is a real world effect. That means the answers are rarely exactly the same. The last time I looked, breathalyzers have a ~50% margin of error. So if you blow a 0.06, your BAC is really somewhere between 0.03 and 0.09. You could mod your breathalyzer to always assume that the reading was 25% higher than the reality without being noticed, as long as your breathalyzer doesn't return a result that's already 25+% higher during testing. Even if it does, your other results will be in range. They need to just stop using breathalyzers. They're inaccurate pieces of machinery, operated by people without any medical knowledge to speak of, who have a vested interest in the readings being high. The whole thing, from top to bottom, is just a long chain of poorly aligned incentives. Field sobriety tests are even worse. The fact that your license can be revoked for refusing the test under the above grounds is nothing short of an end-run around the 4th amendment by declaring driving a "privilege, not a right" despite the fact that it is extremely hard to live in most of the US and not be able to drive. How the fuck do I survive in rural Oklahoma/Georgia/Kentucky/Texas/etc, 40 miles from the nearest bus stop, without a car? And all without any due process. It's never impacted me, but it bothers me nonetheless. They need to force people to go in for a blood draw to establish BAC. It is an accurate test, administered by competent medical professionals, who have no vested interest in the outcome either way. Yes it does take longer. Yes, some people who were at 0.08 will be down to 0.07 by the time the test is administered. I still find that more comforting than the fact that some people at 0.06 will read at 0.09, and some people at 0.14 are going to read at 0.07. There's also the fact that a cop who doesn't calibrate their breathalyzer is merely going to get a slap on the wrist, and a medical professional who does the same is probably going to end up with a huge malpractice suit and possibly lose their license if it was egregious in some way.

spaetzleesser 6y ago

There is so much junk science going on in forensics that it would be great to require everything to be open sourced. Same for voting machines and anything police in general is using (predictive policing is pretty scary). There is way too much stuff hidden and can be challenged only if you have very deep pockets.

hirundo 6y ago

If an accused person has the right to see the source code that produced evidence against them, is it a violation of their rights for the source code to be obfuscated, or even just so spaghettified that not even an expert can understand it? I kinda think that should be a violation. But deciding whether a particular piece of code is so bad is so subjective that I'm not sure on how you'd make a legal standard out of it. Maybe start with "the linter found a ratio of warnings to lines > X%" or some such. Having a legal standard of code coherence/incoherence might help filter pull requests. "This PR cannot be merged to this project because it is configured to reject legally incoherent code." As code becomes more complex it may become more meaningful to have access to the test suite, and to challenge the evidence if the tests are inadequate to demonstrate the correct code behavior.

londons_explore 6y ago
You don't need a legal standard. You just need to put doubt into the minds of a jury. You can get an expert to stand up and say "I'm an expert in computers, and I couldn't understand how this DNA test works. I think it's likely there are mistakes in it that neither I nor the people that made it have discovered".
- deleted 6y ago
  [deleted]
- Spivak 6y ago
  “This system is an absolute mess that is impossible audit and so there is no way that the company using could find bugs in it either.”
Jakobeha 6y ago
In the case where code is used to convict or acquit someone, I think it should be a well-tested and established program: generally something with the software quality of Linux, or in this case, whatever DNA testing kit is being used by scientists in top-ranking universities. We could also use formal verification based on well-established axioms. For example, maybe we could "prove" that the DNA kit reports accurate results as long as the samples it's given are processed correctly.

known 6y ago

Sounds rational since conviction should be on irrefutable evidence;

zupreme 6y ago

What is unfortunate is that it took going to appeal to force the judge to allow the code review at all. Without, at minimum, an independent review (and preferably open source code) the software and lab processes being used constitute an inscrutable "black box" process within which any judgment can be made, for any conceivable reason, with life-changing effects for the defendant (and for the victims of a crime if, for example, a rapist or murderer is set free by a non-match decision). One could even say that unreviewable code here falls under the umbrella of "secret evidence", which much of the world already knows can be easily misused and/or misapplied at the whim of the court.

criddell 6y ago
In Canada you can't even get the breathalyzer maintenance records: https://www.canadianlawyermag.com/news/general/maintenance-records-of-breathalyzer-machines-are-subject-to-third-party-disclosure-regime-scc/275576 https://www.canadianlawyermag.com/news/general/maintenance-r...
vmception 6y ago
People sometimes ask me what my “number” is, like how much net worth or “money” I want, what would I do with it I say “I want to be able to afford appeals court where my rights matter” Infinite appeals court! Most people plea out, cant make bail, dont have counsel buddy buddy with the judge enough to get you bail, and lose the ability to keep good counsel for more and more motions and appeals I want that, there is almost no pride in American rights if you cant afford them. People tie their whole identity to a system they arent even part of
- roywiggins 6y ago
  Appeals? Only ~3% of people charged even go to trial, the rest plead out. Just giving everyone a substantive right to trial would amount to a revolution.
  - vmception 6y ago
    Yeah. But just giving myself that right first, forever.

amelius 6y ago

How does this work? They only have to find 1 bug and the entire case is dropped / postponed?

carlmr 6y ago
I guess they have to show this bug changes the result. But if you look at e.g. the case against Toyota where they had expert witnesses from Carnegie Melon and NASA testifying that the code was a horrible mess, I think that was sufficient for a multi billion dollar fine. So I guess with the admission from the DNA company that their code is basically unreviewable, written in a language known for prototyping, if they get an expert witness to testify this it may be enough without showing how the software misbehaves exactly.

trynton 6y ago

In such a serious case, once the accused was identified by TrueAllele, shouldn't a further test be carried out by a human to verify the results.

chadlavi 6y ago

While this may lead to more transparent technology use in the justice system, a more realistic outcome is that everyone accused of any crime where a technology was used in the process of determining guilt (which could get quite picayune; did the prosecutor's office use Excel? Is that something the defendant could demand to check the source code on?) will use this as a way to slow-roll the process indefinitely.

tehwebguy 6y ago
Well sure, if they rely entirely on Excel to generate and spit out an expert witness testimony. If that ever happens it will be extremely important to understand what’s happening under the hood in Excel.
bagacrap 6y ago
Definitely if an excel macro determined my guilt based on a dataset, I would like to be able to read that macro.
- abfan1127 6y ago
  or at least double check the math.
Spivak 6y ago
Yes? I mean the intimate details of how cell networks work have been the difference between guilty and not before when the only evidence is some black-box report generated by AT&T that seems more accurate than is actually guaranteed. This wouldn’t be necessary if the expert on the stand was a geneticist who ran the test. But when the expert literally is the software you can’t really cross-examine the company’s sales rep.
Jolter 6y ago
Since when do defendants have an interest in prolonged sentencing processes? For major crimes like this, they are in jail with restrictions. They would get better treatment once sentenced, in the prison system, wouldn’t they?

pvitz 6y ago

I have worked with MATLAB code with 20,000 lines of code. Only over the past years, OOP and unit-testing has become properly available and usable. My guess is that this 170,000 lines are written in the old procedural way (also for performance reasons) and are full of bugs, also thanks to the lack of supporting tools. Most likely, this grew out of a research prototype that just worked too well to be reimplemented in a proper production environment.

fatnoah 6y ago
I wonder if bug tracker and other reports would be part of discovery in this case.
- pvitz 6y ago
  Equally interesting is in my opinion who should do the review. Mathworks' own consulting service is probably the best to do so, but I wonder if they would objectively work against one of their own customers.
  - fatnoah 6y ago
    Yeah, any technical expert in a trial concerns me. I was an alternate juror (meaning I had to sit through the trial but was not allowed to take part in any deliberations) in a trial that involved the testimony of a computer "expert". The expert's testimony was 100% true and appeared to definitely prove X to someone who knew nothing about the subject matter. It was things analogous to saying the system was secure because it had a security chip. There were 1,000,000 questions I wished had been asked.
mattkrause 6y ago
Matlab has had classes (both types!) for ages. The unit-testing stuff dates back to at least 2013, and there were toolboxes to do similar things even before that. The language certainly has some warts, but IMO, the bigger problem is that it's usually learned/used in contexts that focus on code quality: the goal is the resulting number or plot rather than the software that generates them.

johnnyfived 6y ago

How would a team of independent reviewers even go about examining source code of this scale? How can you possibly find bugs that aren't super obvious?

pavel_lishin 6y ago
> How would a team of independent reviewers even go about examining source code of this scale? How can you possibly find bugs that aren't super obvious? Something tells me that 170k of Matlab code is going to contain plenty of obvious bugs.
twobitshifter 6y ago
The defense team only needs to provide the jury with a reasonable doubt. They don’t have to prove that it’s entirely wrong, only the existence of mistakes that call into question the overall accuracy. The founder has already made their argument for them by claiming that the entire codebase is impossible to review. “since a complete review is not possible, we reviewed a sample of x files and found y errors in the code. It is obvious that the developers has not adhered to the strict software code quality review standards that are necessary when dealing with life and death situations.”

phire 6y ago

> The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. First, the defence doesn't necessarily have to evaluate all 170,000 lines. They just need to find one buggy line which could potentially overturn the result. Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process?

Cthulhu_ 6y ago
> Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? No, but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. But I think it's warranted. An independent software review, and a double blind assertion with the exact version of the software used in the conviction to test the accuracy and reliability of the application.
- edgyquant 6y ago
  Any software used to convict people, especially on a such a serious crime, should be audited like the fed is. Twice yearly, once by a public firm and another by the government itself. It should have to pass both of these audits to be used.
  - contingencies 6y ago
    It should be audited by science.
    - edgyquant 6y ago
      Yes there should be teams comprised of people whose expertise is in biology (specifically whatever field is responsible for DNA matching) and other people who are programmers that review the code and make sure there’s no mistakes. That would do it and I’m surprised this isn’t already a thing.

aaron695 6y ago

This isn't a DNA testing kit as one would normally think - "TrueAllele uses a hierarchical Bayesian probability model that adds genotype alleles, accounts for artifacts, and determines variance to explain STR data and derive parameter values and their uncertainty. The computer employs Markov chain Monte Carlo (MCMC) statistical sampling to solve the Bayesian equations. The resulting joint posterior probability provides marginal distributions for contributor genotypes, mixture weights, and other explanatory variables." https://onlinelibrary.wiley.com/doi/full/10.1111/1556-4029.14204 https://onlinelibrary.wiley.com/doi/full/10.1111/1556-4029.1...

Scandiravian 6y ago
How is that different from a DNA testing kit "as one would normally think"?

hastradamus 6y ago

The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. This is hilarious. As if you need to read every damn line and you can’t skip blank lines? You can skip whole files that aren’t relevant. Weak excuse

heavyset_go 6y ago
"It could take too long :(" is an interesting excuse not to examine the evidence that could put someone behind bars for the rest of their life.
jariel 6y ago
The statement is odd, at the same time, it's not outrageous for him to make in the sense that - lines of code notwithstanding - the underlying science i.e. the application of the product is the thing in question. It's a pretty interesting case. At least the core nature of the algorithm should be made public if we're going to use it for public inquisition.
sudosysgen 6y ago
And of course, as soon as you find an issue (which you probably will given that it's 170k lines of MATLAB), you can stop then and there.
astura 6y ago
This is a stupid "argument", I've personally participated in an audit of a program that has 5-6x as many lines of code.

duluca 6y ago

Things are only going to get murkier when leveraging ML. All of this needs to become open source.

fritzo 6y ago
Agreed, and with ML we'll need both open source and open training data.

deleted 6y ago

[deleted]

AS37 6y ago
I'd like to encourage you, 5 minutes after you learn about someone's existence, not to wish for their death, execution, imprisonment, and /or disenfranchisement. The real world is too complicated to judge correctly in that time. As well, please remember that the comment guidelines exist at https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deleted 6y ago
  [deleted]
  - deleted 6y ago
    [deleted]
- dang 6y ago
  Thanks for pointing this out! We've deleted that comment by the commenter's request.
retrac 6y ago
There's a certain irony in advocating for capital punishment on a thread about the dangers of lying prosecutors, bogus forensics, and false convictions.
deleted 6y ago
[deleted]
dang 6y ago
We detached this subthread from https://news.ycombinator.com/item?id=26064707 https://news.ycombinator.com/item?id=26064707.

deleted 6y ago

[deleted]

fxleach 6y ago

> it would take eight and a half years to review at a rate of ten lines an hour Wow, the co-founder's argument to not disclose the source code to the defendant was that it's too many lines of code. Also... ten lines reviewed in an hour!?

meowster 6y ago
In the TV shows and books, opposing counsel always want to inundate the protagonist with truckloads of boxes full of printed papers when they could just handover a flashdrive instead. Are there any lawyers here who can speak to the accuracy of such portrayal? It would seem the next logical step would be for every other lawyer to say they shouldn't hand over discovery because it would take to long for the otherside when they bury them in paperwork.
- _Wintermute 6y ago
  Not sure if it's the best source, but according to a "Youtube laywer reacts" video [0], it's a common tactic but you can complain about it to the judge and request documents in a more appropriate form. [0] https://youtu.be/spr5smxuO5E?t=1284 https://youtu.be/spr5smxuO5E?t=1284

fritzo 6y ago

Code review is not the way to validate statistical inferences. Even during internal review of statistical inference code, committers argue correctness through a combination of readable code, readable tests, and statistical validation on a combination of synthetic and real data. While I agree the TrueAllele source should be provided, the "zero-defects" bar is neither sufficient nor necessary for correctness of the inferences made.

damagednoob 6y ago
How much uncertainty in the system are you willing to live with if you were falsely accused of murder?
- fritzo 6y ago
  That is a question for the judge or jury. The statistician's job is to provide a probability distribution, and to argue for the statistical correctness of that distribution based on similar inferences made on validation data.
  - damagednoob 6y ago
    In that case, at least with respect to the death penalty, "zero-defects" is precisely what the courts have moved towards.
akyu 6y ago
> Code review is not the way to validate statistical inferences. But it is a way to invalidate the inference, which is what the defense is interested in.

qwertox 6y ago

Here in Germany we have somewhat similar cases, but where the accusation is way less damaging than the case of this article, in which a false positive would have the drastic result of being labeled a murderer. The cases are related to new speeding cameras which work with laser, where the defendants are complaining that these new devices are black boxes, and that they demand access to the raw data which these devices process. The problem is that these devices discard the raw data after having processed it and come to a conclusion that the driver was or was not speeding. The devices in question are Traffistar S350 from Jenoptik and PoliScan SM1 from Vitronic. There were discussions about a required software update which retains all this data, but apparently the devices lack the storage capability to do so. The National Metrology Institute of Germany (Physikalisch-Technische Bundesanstalt (PTB)) responded to this, that they would not re-certify these devices with updated software because from their point of view they work "as specified".

Kinrany 6y ago
Surely there aren't so many speeders that storing a few seconds per violation would add up to a significant amount.
nolok 6y ago
Now all you need is the proper amount of collusion/corruption between the certifying agency and the manufacturer to have a magic box that does whatever the one paying the bills want. Might seem far fetched in a developped country, until you read about the Boeing/FAA thing happened.
- josefx 6y ago
  As far as I could find the courts basically decided that there has to be a way to examine how a result was reached if there was any doubt about it. That the PTB still allowed their use didn't change that and if a case got to court the results could get thrown out. The PTB probably doesn't care because only a small percentage of speeding cases end up in court.

dannykwells 6y ago

"170,000 lines of MATLAB code" (used as software to determine if humans are spending their lives in jail). Well if that's not a line to make you feel awful on a Monday morning.

zyxzevn 6y ago

All source code for important decisions should be open and be verified by different parties. Just as that for voting machines.

LatteLazy 6y ago

We had a case in the uk with accounting software. Dozens of people were convicted over decades because the software couldn't be wrong. Until eventually someone actually checked. https://www.bbc.co.uk/news/business-54384427 https://www.bbc.co.uk/news/business-54384427

pnathan 6y ago

This is excellent! As a principle, we should always be able to properly critique the expert witnesses and analyses! Or, at least, hire an expert firm to do the critique. It at least restores a proper ability to challenge an assertion of guilt.

rland 6y ago

These companies are disgusting. They peddle black box "models," that essentially ride the good reputation of DNA as infallible (which it is most certainly not) to get convictions on dubious or no evidence. The way it works is that if there is a sample from a crime scene, they send it to these guys and they analyze it with their software to detect "statistical" DNA from the sample. These samples are the ones that are too crappy to actually make a definitive match -- they are a statistical match. So you say "I think Jim, Bob, and Alice were on scene," and it says "10% likelihood Jim DNA, 5% likelihood Bob DNA, 45% Alice DNA." Do you think it ever says "99% no DNA" in the sample? It's basically Theranos, except instead of wasting $50 on a shitty blood test you get life in prison. Ostensibly, it searches the entire DNA database for matches, and only returns a positive result if there's a positive match. But it's a statistical model, using inputs that are crappy at best (because if it was an actual DNA match, they would send it off to in house forensics who would be able to do PCR...) and which includes inputs from circumstantial evidence as priors. Like we believe Alice was at the scene therefore if you find any statistical likelihood that this is Alice's DNA boost that. They often run the model multiple times in a row, and use the result that the DA likes the most to enter into evidence. This is because the models return different results each time -- of course they'd say, iTs StAtiStIcaL, so they can do that... And the source code is completely impenetrable. They argue that it's a "trade secret" that jeopardizes their ability to make future profits, so it cannot be open-sourced. These guys could have a model that just says "what percentage should the thing read, Señor D.A.?" The entire product is a sham. And because it's 170k LOC, no one has the time or the qualifications (Judges/Attorneys reading source code? Yeah right!) to review it, even if it were open source. Pure quackery, and often times, decades-long sentences or life in prison for the defendant. These companies are pure filth worthy of the lowest revulsion. It's a wonder any convictions happen at all because of this stuff, but jurors have very inaccurate conceptions of forensic science, thanks to shit like CSI, Law and Order, etc. These companies happily play into that image and people really believe this stuff works.

deleted 6y ago
[deleted]
jrm5100 6y ago
Issues with source code access aside, your description is mostly wrong. These programs take a DNA profile as input- it's just that the DNA profile is mixed (i.e. from multiple people). It reporting no DNA would be nonsensical. Figuring out exactly how many people are in a mixture isn't quite nailed down statistically (last I knew of), but it's usually pretty clear for up to 4 or so people. Yes, you could run different models and get different probabilities. For example, the likelihood that the sample is a mixture of the suspect, the victim, and some unknown person vs victim and two unknown people compared to saying the victim isn't in the sample. However, the specification of those models is part of the trial process. And the output probabilities (at least when being used to determine guilt) are usually quite high, orders of magnitude higher than 90% or even 99.99%. My point is that the science behind these calculations is well developed- validation studies get published all the time. Whether or not the specific software has errors (or isn't coded exactly as modeled) is an entirely different matter, but it still isn't all that likely. All of these cases rely on expert witnesses anyway- it's not the prosecutor pressing some buttons and printing a report. There is far more concerning quackery that gets used in forensics- bite marks, hair matching, etc.

vlovich123 6y ago

I'm curious. I've always been frustrated by this "closed" business model in the legal system. I feel like the entire process & details should be detailed in the open (code, methodology, controls, etc). Of course the counterpoint is that it makes it easier to copy this business & undercut all the time & energy spent on building it (copying is easier). Is that the only reason? I feel like open kimono is a critically important concept for anything related to the legal system because of how any perversion removes its legitimacy. If it really is that prohibitive to run a profitable business in this space, is there open standards that can be enforced (e.g. "this is the core algorithm that is approved" & businesses must get regular audits to continue to be used & any failed audit causes a reexamining of any court cases you were involved in the past year?). That's less ideal because then who audits the auditors but maybe at least it's an acceptable middle ground from where we are? In general I've been extremely frustrated how regularly & consistently this entire industry keeps everything secretive & trust-based despite consistent examples of how insufficient trust is for this field & how devastating the results are when that trust is violated.

lowbloodsugar 6y ago

"The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour." Well then the probability that it works correctly is zero.

femto113 6y ago

Interesting that they're putting so much focus on the DNA piece, since the actual crime was witnessed by a police officer and there is circumstantial evidence tying the suspect to the weapon (it was found along the path they used to flee from the scene).

LockAndLol 6y ago

For the Europeans among us, I would like to remind you that there's an initiative to opensource all publicly funded code https://publiccode.eu/ https://publiccode.eu/ If you care, add your signature and vote for a party during the next elections that considers opensource at least somewhat important.

WaitWaitWha 6y ago

The tone of this thread is leaning to "forensics evil, government lackeys out to get the little guy". In the USA, all sides can call and rely on their own forensics. There is no government mandated & approved single-source-of-truth (with some minor exceptions). > Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. This statement is at minimum an extreme generalization. Forensic field is a very large field, with many government and private "forensicators". There are no "black box processes". The very word "forensic" is based on presenting in full view, front of the fact finders (jury & judges usually). How deep this gets dug into depends on the fact finders and attorneys. In all court cases the forensic examiner can be called to demonstrate with extreme nuance how they performed the procedures. Any tools used are can be requested to be examined, including software. Vendors that I have worked with all have experts on staff specifically to appear in court and detail the inner workings of their tools. Not only tools, processes, and environments, but the examiner can be drilled on their experience, education, degrees, previous cases, failures, etc. They do call it "voir dire" for a reason. This does not negate the fact that there are some bad apples, bad prosecutors, and bad judges.

anomaloustho 6y ago
> Any tools used are can be requested to be examined, including software. Out of curiosity, if this is a common occurrence that is willfully obeyed by all vendors and parties involved. Could you shed some light on why this vendor is resisting?
- WaitWaitWha 6y ago
  No, it is not common but does happen. There are several cases on drug testing, DNA, bite mark, and software that comes to mind. Scientifically well established processes, tools, etc. are rarely called up. Bite mark was a big deal in the industry because it turns out, it is not so unique and the methodologies developed were weak. My personal opinion why the are resisting? They are resisting because they are bottom dwellers. Forensicators whom I associate with, tool vendors, and I understand that we have to share knowledge. I dare say, we are the most open source scientific knowledge industry. This is not because of some altruistic reason, but because tomorrow they can be called on to explain.

Thorentis 6y ago

> If TrueAllele is found wanting, presumably that will not affect the dozen individuals said to have been exonerated by the software. Buried right at the end, but an interesting thought. It would depend on the individual cases, but if there was a whole body of evidence vs. one DNA test, surely these cases would need to be retried?

clarle 6y ago
IANAL but if the individuals were already found not guilty, I think double jeopardy laws would apply. The prosecution wouldn't be able to try again unless they made an appeal that there's a substantially different crime.

brutal_chaos_ 6y ago

Disclaimer, I am FOR seeing the source in these cases. Say the defense finds the software in a very, very troubling state. Could that be telling enough to not trust the outcomes of the software? Assuming the DNA parts were correct, but the rest is junk, this is where the plaintiff would bring in their expert to counter the arguments? Perhaps ^ is just a weak argument. I don't know how well versed courts are in these matters. (My guess is NOT that well versed because of the CFAA of the 80's up to Aaron Schwartz, and more). My legal background is Law and Order, so I have no idea what I'm talking about, just curious.

kazinator 6y ago

> Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour What an deplorably dishonest argument to present to laypeople. If there is some weakness in it so the accused has a way to plausibly deny the DNA test, it will take a consultant who has relevant competence something like week or two to find it. Finding a problem doesn't require looking at every line of 170,000.

ineedasername 6y ago

If ever there was something that should be fully transparent it is the mechanisms by which a person might be found guilty of a crime. The defendant shouldn't even have had to fight for this. It should be a fundamental cornerstone of criminal prosecutions.

meowface 6y ago
These cases also often come up with drug and alcohol detection tests, and as John Oliver points out in https://www.youtube.com/watch?v=1f2iawp0y5Y https://www.youtube.com/watch?v=1f2iawp0y5Y, software used to select jurors. All of these companies claim that their source code is valuable intellectual property and that disclosing it can hurt their business. Even if this were true, when you're providing something that can be a significant factor in someone being imprisoned or executed, when creating the business you should accept that you're providing a public service that needs to be publicly accountable. If it's not open source, at the very least there should be a requirement that software code and hardware designs must be provided on-demand to experts in court cases (with a non-disclosure clause to mitigate leaks and corporate espionage etc.).
- alistairSH 6y ago
  ...disclosing it can hurt their business... And there's an entire body of law based around IP which they can use to protect their business, just like everybody else.
  - zymhan 6y ago
    Their business interests are also of minuscule importance compared to the impact to society these tools have.
- bargl 6y ago
  Without jumping on the conspiracy bandwagon, I'd also like to see this applied to voting software. I know it's a hot topic, and I'm honestly not trying to get political. Software that is critical to our fundamental human rights, and is being used by our government should be open source, or at least audited by a group of people who sign Non-competes/NDA and can't go work for competitors, or with some other mechanism to protect IP that I can't think of.

rudylee 6y ago

Why can't just they run the DNA test again in 2 or 3 other different software ? If all of them come back with the same result then we can assume that the first software is not buggy right ?

handoflixue 6y ago
It could be that they're all copying the same Stack Exchange answer, or making the same basic mistake - If 50% of programmers do averages wrong, it's not ridiculous to claim that 3 in a row got it wrong. Tripling your costs and still having a 12.5% failure rate isn't great.

protomyth 6y ago

I do wonder if the next step is check the compiler / interpreter the code is running on. Is Matlab assumed to be error free in this situation? Will certified compilers and CPUs be required in the future? Looks like someone could end up reviving a modern version of the Viper https://en.wikipedia.org/wiki/VIPER_microprocessor https://en.wikipedia.org/wiki/VIPER_microprocessor

iamleppert 6y ago

Any company who has software that is used in such a way should have independent review of their source code. It’s amazing to me they didn’t think of this originally and have the necessary endorsements already available for viewing.

Abimelex 6y ago

I'm not familiar with genome sequencing, but why are they even relying only on one program? Wouldn't it be possible to check it with different programs to reduce the possibility of failure here? And just assuming they find a possible bug, would that mean all the sentences based on the programs result will be obsolete?

astura 6y ago

There should be no such thing as a "trade secret" when talking about evidence that is used to convict someone. Correct judgement here.

dwg 6y ago

> The maker of the software, Cybergenetics, has insisted in lower court proceedings that the program's source code is a trade secret. Should this be, in general, a disqualifying condition when something is to be used as evidence?

jxramos 6y ago

Aren't there lower hanging fruits to go after? Like are there not competitor or predicate devices to run the same DNA against and see what it says as far as a match goes?

kovac 6y ago

I don't think it's meaningful to let defendant review the code each trial. Such code reviews should have been done before it was ever allowed to be used in the legal system. On the other hand, there's always the more practical option of getting the dna testing done by a second, third or even a tenth different approved company if the defendants want to pursue this line of reasoning. From what I understand, dna testing is statistical which means even if they find some bug in the code there will have to be extended debates on whether or not a particular bug affected the credibility of this test result. This is just bullocks. Having said that, the excuse the software company is giving is a bit lame.

michaelmrose 6y ago

One would suppose that for a word of such import there would be a detailed analysis of the work product along with testing they had done to prove it actually worked. They ought to ask for access to THAT in addition to the source code. One would imagine that it doesn't come with 170k lines of documentation or analysis and it would be orders of magnitude easier to review than the source code alone.

anfilt 6y ago

Link to the ruling: https://www.eff.org/files/2021/02/03/decision_appdiv_232021.pdf https://www.eff.org/files/2021/02/03/decision_appdiv_232021....