16 ms·

> The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of M

by phire 6y ago

> The co-founder of the company, Mark Perlin, is said to have argued against source code analysis by claiming that the program, consisting of 170,000 lines of MATLAB code, is so dense it would take eight and a half years to review at a rate of ten lines an hour. First, the defence doesn't necessarily have to evaluate all 170,000 lines. They just need to find one buggy line which could potentially overturn the result. Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process?

Cthulhu_ 6y ago
> Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? No, but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. But I think it's warranted. An independent software review, and a double blind assertion with the exact version of the software used in the conviction to test the accuracy and reliability of the application.
- edgyquant 6y ago
  Any software used to convict people, especially on a such a serious crime, should be audited like the fed is. Twice yearly, once by a public firm and another by the government itself. It should have to pass both of these audits to be used.
  - contingencies 6y ago
    It should be audited by science.
    - edgyquant 6y ago
      Yes there should be teams comprised of people whose expertise is in biology (specifically whatever field is responsible for DNA matching) and other people who are programmers that review the code and make sure there’s no mistakes. That would do it and I’m surprised this isn’t already a thing.
      - bigiain 6y ago
        It should totally be available to be "red teamed", like the researchers who exploit fingerprint readers with gummy bear moulds made with laser prints of fingerprints, or who hold up photographs to face recognition systems. The government should send a half a dozen to DefCon/CCC and let attendees loose trying to fool them.
      - contingencies 6y ago
        Hardware and business process would also need review. Its no good having perfect code if you can insert the wrong sample. It's also no good having perfect code if a well-timed EMI burst or power level shift games the result.
        0134340 6y ago
        Definitely. There was even a documentary about it on Netflix about how crimelab technicians were pushed for results rather than accuracy. A couple major scandals involved intentional faking of tests and resulting in countless people going to prison and the judicial consequences it caused as well as the friction involved to seek justice. Like, oh no, all these people convicted for life on flimsy grounds is such a burden of our time.
        edgyquant 6y ago
        Eh that’s overkill. If can be proofed and guaranteed that it isn’t bugging out we don’t need to run through every building block this way.
        contingencies 6y ago
        I would have said that too before starting a hardware company. Now, I'm not so sure.
- bigiain 6y ago
  > but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. Sure. And the prosecution using the company claiming to have "totally reliable DNA evidence" should be totally on the hook for those costs (plus damages) when that analysis or expert witnesses show up "reasonable doubt" flaws in the software or the processes in which that software is used, including then risking retrials or mistrials of all other cases in which it was used. If the prosecutors want to play high stakes games with defendants lives and liberty using "evidence" from proprietary software or devices, they need to be held to the consequences of losing their stakes. [Edit: I wonder what the legal system would think of a CyberGenetics competitor funding the expert witness analysis of their software on behalf of the defence???]
abfan1127 6y ago
10 lines per hour? doesn't that seem painfully slow?
- flavius29663 6y ago
  have you ever seen MATLAB code?
  - ashtonkem 6y ago
    That’s actually a pretty good argument for banning such code from the criminal justice system. The idea that unreadable code is deciding who gets locked up is really worrying.
    - deathlight 6y ago
      This thread makes me sad because when I was taught and used matlab we had strong pressures to properly comment and document our code to make it legible (if only to our own future selves). It feels almost criminal to not do that in these circumstances.
    - cozzyd 6y ago
      and depending on which MATLAB version they run it in, it might have completely different results if they haven't tested carefully...
      - willis936 6y ago
        This is not my experience. Could you share an example?
        cozzyd 6y ago
        well, this script, for example, gave me all sorts of fits trying to run on the then-new version of MATLAB I tried it on: http://cpc.cs.qub.ac.uk/summaries/AEJS_v1_0.html http://cpc.cs.qub.ac.uk/summaries/AEJS_v1_0.html
breck 6y ago
Lol. If it would take 8.5 yrs to review, it's probably god awful, and should never ever ever be used to convict someone of such a crime. My prediction: this firm will probably try to get removed from the case, rather than open source their shitty code. Source: I've worked on MATLAB codebases for various genomics research projects in the past.
- nickodell 6y ago
  >My prediction: this firm will probably try to get removed from the case, rather than open source their shitty code. That isn't necessarily their choice. The prosecutors will make the decision about whether to withdraw the DNA evidence. They probably won't, given that they would need to give the defendant a new trial, which could lead to an accused murderer getting off. A bad look for any prosecutor. More to the point, if the firm withdraws from any case where their credibility is questioned, what does that say to law enforcement agencies who are thinking about using their software?
  - bdavisx 6y ago
    It would also give every person convicted using their software an incentive to open an appeal.
    - Spivak 6y ago
      I like how this is considered a bad thing. Like we can’t let this guy point out that he’s being convicted by an unauditable black box that suddenly isn’t worth using if it has to stand up to scrutiny because then everyone would want to. The horror. Like I’m actually kinda shocked this is the reality. I would have assumed that DNA evidence would have some blessed methodologies and tools/algorithms, with a strict definition of what constitutes a match or partial match specifically so this wouldn’t happen.
      - hluska 6y ago
        This is one of these scary areas where reality matches my teenaged experiences playing Shadowrun. I used to hope that the brutal dystopia we played through was just fun. Now I’m seeing that the present needs a word even more brutal than dystopia. :(
wdobbels 6y ago
If their argument is that their codebase is basically unreadable, then I see why they're scared someone might find some bugs here and there.
- carlmr 6y ago
  I'd get an expert witness in there to testify basically that. I also don't think you should code anything mission critical like this in Matlab. It's a decent language for prototyping, not for production.
  - ChrisLomont 6y ago
    The numerics in Matlab are far better than pretty much any developer can produce in production. This is why Matlab is used in production - it's vastly more reliable than people rebuilding the things it is good at by hand for bespoke solutions.
    - WmyEE0UsWAwC2i 6y ago
      True. But I believe this is a case where correctness and clarity are the paramount concerns. There should be a public reference implantation of these methods if they are going to be used in court.
      - generalizations 6y ago
        Seems like mathematical clarity is what Matlab is good at, at least compared to e.g. python and C.
        carlmr 6y ago
        Most industry Matlab I've seen is similar to numpy code, heavily vectorized to make it work fast, somewhat inscrutable for everything that's not linear algebra, and a lot of assumptions about perfect floating point precision. Couple that with a unit-testing unfriendly culture and you have a code disaster. Especially on 170k lines.
kspacewalk2 6y ago
>Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? "It's just gonna take so long, plus the code is a bit messy. We're gonna be doing all that work just because the rest of someone's life teeters on the results of the inquiry? Maan, that's a bummer."
kenjackson 6y ago
8 years is a long time. What he then wanted to code review Matlab, and then the compiler that Matlab used, then do some silicon verification... Six to nine months seems like enough to do a very good code review with some testing. There's a good chance that 75% of that Matlab code doesn't execute for his test.
- edgyquant 6y ago
  This is this persons life we’re talking about. No amount of time is too much and it isn’t hurting anyone to let him do a code review.
  - kenjackson 6y ago
    No. You give a reasonable amount of time. What if he asked for 100 years?
    - PurpleFoxy 6y ago
      We should flip the responsibility. How does the company providing the software prove it is not flawed.
    - edgyquant 6y ago
      As long as he’s actually reading the code, and still in jail unable to hurt anyone, let him have it. He’ll likely pass before that time is reached and if he’s just wasting time he’ll get bored having to spend hours a day looking at code that he doesn’t care to look at. I get what you’re saying but my point is him wasting time reading code isn’t hurting anyone and is a better use of an inmates time than sitting around.
      - kenjackson 6y ago
        If he’s in jail doing it then that’s fine.
        6y ago
maceurt 6y ago
It shouldn't take 10 lines an hour should it? I don't have experience reviewing professional code of this size, so please correct me if my assumption is wrong, but that number doesn't seem right.
- raldi 6y ago
  I'd say more like ten lines a second, if you're skimming it to look for something. And of course, the work can be parallelized.
  - lopatin 6y ago
    Is this satire?
    - bigiain 6y ago
      "With enough eyeballs, all bugs are shallow^h^h^h^h^h^h buried in the critical open source dependancy underpinning the entire internet maintained by that one guy who's holding down a day job and doing it in his spare time." -- with apologies to Linus (As always, XKCD beat me to this gag: https://xkcd.com/2347/ https://xkcd.com/2347/ )
    - raldi 6y ago
      I'm not sure what you mean. What numbers would you call realistic?
- novembermike 6y ago
  It really isn't. Most of the code is probably going to be uninteresting and you can do 10 lines a minute or more. Some of the code will be more relevant and might take a day for 10 lines. This would just be checking for accuracy though so you could probably just ignore a huge chunk of it.
- jjkaczor 6y ago
  As part of a quality control team, I personally went through over 1.2 million lines of working code (i.e. not including comments) over the span of about 8 months, M-F, 9am-5pm. So, yeah - this number is bunk.
deleted 6y ago
[deleted]
buzzerbetrayed 6y ago
Wait until you hear how long it would take at a rate of 2 lines per hour, as long as we're just throwing out random numbers.
- deleted 6y ago
  [deleted]
unishark 6y ago
A lot of bashing over a vague third-hand quote without a source. In the very next paragraph they say: > The company offered the defense access under tightly controlled conditions outlined in a non-disclosure agreement, which included accepting a $1m liability fine in the event code details leaked. But the defense team objected to the conditions, which they argued would hinder their evaluation and would deter any expert witness from participating. So it's a concern about IP protection for them.
- gowld 6y ago
  There is no legitimate need for IP protection against destroying innocent lives.
  - colejohnson66 6y ago
    “If the criminals know how our system works, they can work around it!” is probably what they’re thinking. i.e. security through obscurity.
  - contingencies 6y ago
    Indeed. In fact, if there were a globally agreeable case for open science and open source, to which all governments could contribute, DNA analysis for criminal attribution would have to be it, no? A side benefit would be that sending evidence to numerous international labs would greatly frustrate attempts at domestic law enforcement / lab corruption.
- breischl 6y ago
  Maybe, or maybe that's just the smokescreen they put up to prevent anyone looking at it. I notice that they didn't say "here's the results of our last independency audit and verification of correctness", which I think would be a fantastic counter-argument... if they had one.
- michaelmrose 6y ago
  Arguably the public analysis of the correctness of their methods ought to be done once in a fashion that is usable by all former and subsequent defendants. This is supported by their own claims that such an analysis will be extremely onerous. If its a multi million dollar affair like they claim its virtually impossible that every defendant will be able to fund such an affair. In fact in the case that a disastrous flaw is found it may be advantageous to simply drop the case and hope that past and future defendants wont be able to each afford to press the point.
sudosysgen 6y ago
10 lines an hour seems quite low.

by phire 6y ago

Cthulhu_ 6y ago

> Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? No, but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. But I think it's warranted. An independent software review, and a double blind assertion with the exact version of the software used in the conviction to test the accuracy and reliability of the application.

edgyquant 6y ago
Any software used to convict people, especially on a such a serious crime, should be audited like the fed is. Twice yearly, once by a public firm and another by the government itself. It should have to pass both of these audits to be used.
- contingencies 6y ago
  It should be audited by science.
  - edgyquant 6y ago
    Yes there should be teams comprised of people whose expertise is in biology (specifically whatever field is responsible for DNA matching) and other people who are programmers that review the code and make sure there’s no mistakes. That would do it and I’m surprised this isn’t already a thing.
    - bigiain 6y ago
      It should totally be available to be "red teamed", like the researchers who exploit fingerprint readers with gummy bear moulds made with laser prints of fingerprints, or who hold up photographs to face recognition systems. The government should send a half a dozen to DefCon/CCC and let attendees loose trying to fool them.
    - contingencies 6y ago
      Hardware and business process would also need review. Its no good having perfect code if you can insert the wrong sample. It's also no good having perfect code if a well-timed EMI burst or power level shift games the result.
      - 0134340 6y ago
        Definitely. There was even a documentary about it on Netflix about how crimelab technicians were pushed for results rather than accuracy. A couple major scandals involved intentional faking of tests and resulting in countless people going to prison and the judicial consequences it caused as well as the friction involved to seek justice. Like, oh no, all these people convicted for life on flimsy grounds is such a burden of our time.
      - edgyquant 6y ago
        Eh that’s overkill. If can be proofed and guaranteed that it isn’t bugging out we don’t need to run through every building block this way.
        contingencies 6y ago
        I would have said that too before starting a hardware company. Now, I'm not so sure.
bigiain 6y ago
> but the person that wants to have it analyzed will have to either spend the time themselves, or pay the expert witness for their time; it could be a costly affair. Sure. And the prosecution using the company claiming to have "totally reliable DNA evidence" should be totally on the hook for those costs (plus damages) when that analysis or expert witnesses show up "reasonable doubt" flaws in the software or the processes in which that software is used, including then risking retrials or mistrials of all other cases in which it was used. If the prosecutors want to play high stakes games with defendants lives and liberty using "evidence" from proprietary software or devices, they need to be held to the consequences of losing their stakes. [Edit: I wonder what the legal system would think of a CyberGenetics competitor funding the expert witness analysis of their software on behalf of the defence???]

abfan1127 6y ago

10 lines per hour? doesn't that seem painfully slow?

flavius29663 6y ago
have you ever seen MATLAB code?
- ashtonkem 6y ago
  That’s actually a pretty good argument for banning such code from the criminal justice system. The idea that unreadable code is deciding who gets locked up is really worrying.
  - deathlight 6y ago
    This thread makes me sad because when I was taught and used matlab we had strong pressures to properly comment and document our code to make it legible (if only to our own future selves). It feels almost criminal to not do that in these circumstances.
  - cozzyd 6y ago
    and depending on which MATLAB version they run it in, it might have completely different results if they haven't tested carefully...
    - willis936 6y ago
      This is not my experience. Could you share an example?
      - cozzyd 6y ago
        well, this script, for example, gave me all sorts of fits trying to run on the then-new version of MATLAB I tried it on: http://cpc.cs.qub.ac.uk/summaries/AEJS_v1_0.html http://cpc.cs.qub.ac.uk/summaries/AEJS_v1_0.html

breck 6y ago

Lol. If it would take 8.5 yrs to review, it's probably god awful, and should never ever ever be used to convict someone of such a crime. My prediction: this firm will probably try to get removed from the case, rather than open source their shitty code. Source: I've worked on MATLAB codebases for various genomics research projects in the past.

nickodell 6y ago
>My prediction: this firm will probably try to get removed from the case, rather than open source their shitty code. That isn't necessarily their choice. The prosecutors will make the decision about whether to withdraw the DNA evidence. They probably won't, given that they would need to give the defendant a new trial, which could lead to an accused murderer getting off. A bad look for any prosecutor. More to the point, if the firm withdraws from any case where their credibility is questioned, what does that say to law enforcement agencies who are thinking about using their software?
- bdavisx 6y ago
  It would also give every person convicted using their software an incentive to open an appeal.
  - Spivak 6y ago
    I like how this is considered a bad thing. Like we can’t let this guy point out that he’s being convicted by an unauditable black box that suddenly isn’t worth using if it has to stand up to scrutiny because then everyone would want to. The horror. Like I’m actually kinda shocked this is the reality. I would have assumed that DNA evidence would have some blessed methodologies and tools/algorithms, with a strict definition of what constitutes a match or partial match specifically so this wouldn’t happen.
    - hluska 6y ago
      This is one of these scary areas where reality matches my teenaged experiences playing Shadowrun. I used to hope that the brutal dystopia we played through was just fun. Now I’m seeing that the present needs a word even more brutal than dystopia. :(

wdobbels 6y ago

If their argument is that their codebase is basically unreadable, then I see why they're scared someone might find some bugs here and there.

carlmr 6y ago
I'd get an expert witness in there to testify basically that. I also don't think you should code anything mission critical like this in Matlab. It's a decent language for prototyping, not for production.
- ChrisLomont 6y ago
  The numerics in Matlab are far better than pretty much any developer can produce in production. This is why Matlab is used in production - it's vastly more reliable than people rebuilding the things it is good at by hand for bespoke solutions.
  - WmyEE0UsWAwC2i 6y ago
    True. But I believe this is a case where correctness and clarity are the paramount concerns. There should be a public reference implantation of these methods if they are going to be used in court.
    - generalizations 6y ago
      Seems like mathematical clarity is what Matlab is good at, at least compared to e.g. python and C.
      - carlmr 6y ago
        Most industry Matlab I've seen is similar to numpy code, heavily vectorized to make it work fast, somewhat inscrutable for everything that's not linear algebra, and a lot of assumptions about perfect floating point precision. Couple that with a unit-testing unfriendly culture and you have a code disaster. Especially on 170k lines.

kspacewalk2 6y ago

>Second, even if it did take a full 8 years, is that a good reason to deny the defendant due process? "It's just gonna take so long, plus the code is a bit messy. We're gonna be doing all that work just because the rest of someone's life teeters on the results of the inquiry? Maan, that's a bummer."

kenjackson 6y ago

8 years is a long time. What he then wanted to code review Matlab, and then the compiler that Matlab used, then do some silicon verification... Six to nine months seems like enough to do a very good code review with some testing. There's a good chance that 75% of that Matlab code doesn't execute for his test.

edgyquant 6y ago
This is this persons life we’re talking about. No amount of time is too much and it isn’t hurting anyone to let him do a code review.
- kenjackson 6y ago
  No. You give a reasonable amount of time. What if he asked for 100 years?
  - PurpleFoxy 6y ago
    We should flip the responsibility. How does the company providing the software prove it is not flawed.
  - edgyquant 6y ago
    As long as he’s actually reading the code, and still in jail unable to hurt anyone, let him have it. He’ll likely pass before that time is reached and if he’s just wasting time he’ll get bored having to spend hours a day looking at code that he doesn’t care to look at. I get what you’re saying but my point is him wasting time reading code isn’t hurting anyone and is a better use of an inmates time than sitting around.
    - kenjackson 6y ago
      If he’s in jail doing it then that’s fine.
      - 6y ago

maceurt 6y ago

It shouldn't take 10 lines an hour should it? I don't have experience reviewing professional code of this size, so please correct me if my assumption is wrong, but that number doesn't seem right.

raldi 6y ago
I'd say more like ten lines a second, if you're skimming it to look for something. And of course, the work can be parallelized.
- lopatin 6y ago
  Is this satire?
  - bigiain 6y ago
    "With enough eyeballs, all bugs are shallow^h^h^h^h^h^h buried in the critical open source dependancy underpinning the entire internet maintained by that one guy who's holding down a day job and doing it in his spare time." -- with apologies to Linus (As always, XKCD beat me to this gag: https://xkcd.com/2347/ https://xkcd.com/2347/ )
  - raldi 6y ago
    I'm not sure what you mean. What numbers would you call realistic?
novembermike 6y ago
It really isn't. Most of the code is probably going to be uninteresting and you can do 10 lines a minute or more. Some of the code will be more relevant and might take a day for 10 lines. This would just be checking for accuracy though so you could probably just ignore a huge chunk of it.
jjkaczor 6y ago
As part of a quality control team, I personally went through over 1.2 million lines of working code (i.e. not including comments) over the span of about 8 months, M-F, 9am-5pm. So, yeah - this number is bunk.

deleted 6y ago

[deleted]

buzzerbetrayed 6y ago

Wait until you hear how long it would take at a rate of 2 lines per hour, as long as we're just throwing out random numbers.

deleted 6y ago
[deleted]

unishark 6y ago

A lot of bashing over a vague third-hand quote without a source. In the very next paragraph they say: > The company offered the defense access under tightly controlled conditions outlined in a non-disclosure agreement, which included accepting a $1m liability fine in the event code details leaked. But the defense team objected to the conditions, which they argued would hinder their evaluation and would deter any expert witness from participating. So it's a concern about IP protection for them.

gowld 6y ago
There is no legitimate need for IP protection against destroying innocent lives.
- colejohnson66 6y ago
  “If the criminals know how our system works, they can work around it!” is probably what they’re thinking. i.e. security through obscurity.
- contingencies 6y ago
  Indeed. In fact, if there were a globally agreeable case for open science and open source, to which all governments could contribute, DNA analysis for criminal attribution would have to be it, no? A side benefit would be that sending evidence to numerous international labs would greatly frustrate attempts at domestic law enforcement / lab corruption.
breischl 6y ago
Maybe, or maybe that's just the smokescreen they put up to prevent anyone looking at it. I notice that they didn't say "here's the results of our last independency audit and verification of correctness", which I think would be a fantastic counter-argument... if they had one.
michaelmrose 6y ago
Arguably the public analysis of the correctness of their methods ought to be done once in a fashion that is usable by all former and subsequent defendants. This is supported by their own claims that such an analysis will be extremely onerous. If its a multi million dollar affair like they claim its virtually impossible that every defendant will be able to fund such an affair. In fact in the case that a disastrous flaw is found it may be advantageous to simply drop the case and hope that past and future defendants wont be able to each afford to press the point.

sudosysgen 6y ago

10 lines an hour seems quite low.