9 ms·
Convicted by Code: Defendants should be able to inspect code used in forensics
- 6stringmerc 11y agoOkay so this article is bumping up against the hysteria that I'd categorize as "semi-technology literate" yet makes some good points. Almost like talking about how dangerous it is to walk through a minefield and then stepping on one. There's a valid point in there somewhere. Copyright reform is one of my favorite subjects, and for a multitude of reasons. Should the prosecution be able to dump a case straight up without recourse because the "Stingray" gathering tool is too lovely to submit to review? Nope. Should breathalyzer code be held from review just because it's a product made by somebody? Nope. Should FOIA be stonewalled or pay-walled and inhibit the Constitutional freedom of the press? Nope! Innocent until proven guilty is a very, very important premise for the US legal system. It's backed up by both the Fourth and First amendments to the Constitution. Any justification to put them aside for "War on ____" might seem reasonable on the surface, until taking a closer look at multiple murder evidence that comes from within the borders more often than on a laptop of a Citizen who just so happens to be coming back from a foreign country and gets worked for passwords under duress or has to forfeit hardware without recourse. I dunno, maybe I sound like some kind of off-the-rocker dude by thinking about such things, but I love my country, I'm willing to sit down and think about this kind of stuff. It doesn't have to be extreme. Taking the small steps of talking with one another about what we really value is important in my opinion.
- jfoutz 11y agoSo... if i build and sell a "breathalyzer" to your local pd that simply randomly selects a number between .08 and .16, you'd be ok with that? Pretty much every other element of the system is biased toward conviction. Police have quotas. Prosecutors need convictions. Private prisons get more money for more convicts. It would be easy to dodge questions for a long, long time. Why would police administer the test to a sober person? In this one weird example a sober person registered as drunk. oh well, its still good for 99.9% of other cases. The right to confront your accuser is there for a reason. More and more, software is the accuser. We all, at HN, struggle to make our code correct. Step back, think a second, how do you QA a breathalyzer? How do you deal with variations in sensor packages? Yes, they probably do more good than harm, but are they "accurate"? how do you know that?
- furyg3 11y agoI think you guys agree: > Should breathalyzer code be held from review just because it's a product made by somebody? Nope.
- ubernostrum 11y agoBreathalyzers are known to be wildly inaccurate, to such an extent that in some jurisdictions DUI is just defined in terms of a breathalyzer result because they know it doesn't bear a reliable relationship to the actual BAC.
- Cthulhu_ 11y agoBut is it accurate enough to say "you're drunk driving"? IIRC at least on this side of the pond, if you're caught you either get fined, or you're taken to the station for a more accurate breath test or a blood test.
- ubernostrum 11y agoIn the US, you have the right to request a blood test, but the blood test is not required and the breathalyzer is considered sufficient evidence (again, since the offense is defined in terms of the breathalyzer).
- efaref 11y agoScams like this absolutely do happen, with potentially deadly consequences. http://www.bbc.co.uk/news/uk-29459896 http://www.bbc.co.uk/news/uk-29459896
- tptacek 11y agoThe presumption of innocence comes from English common law, not the Constitution; it in fact appears nowhere in the Constitution, let alone in the 1st or 4th Amendment.
- 6stringmerc 11y agoOkay, so I've conflated the direct inspiration for the framing of the Constitution with what it actually says, and tossed in a couple Amendments that are relevant practically speaking, and yet you still understood the jist of what I was saying. Potatoe.
- tptacek 11y agoNot really. We're probably closer than we are far apart, but I think the distinction is important; for instance, we usually discuss the presumption of innocence in terms of "innocent until proven guilty" and then base our opinions off the modern (indeed, message-board) interpretation of the word "proof". Mathematical certainty was not an objective of the framers, or of their predecessors. Which is relevant here, of course, because of the demand for access to source code to increase certainty. But: you'd be right if you assumed the "1st and 4th Amendment" thing was what actually motivated me to comment. I'm a message board nerd too!
- 6stringmerc 11y agoOkay, gotcha! Nicely done and a hat tip for getting the rough edges out of the way.
- geggam 11y agoThe constitution defines those rights granted to citizens from natural law that shall not be reduced by the govt. The constitution limits the government... citizens have #alltherights
- mikecb 11y agoRoman law even.
- finance-geek 11y agoI think things will become even worse now that criminal "scouting" and even vetting is being done via learning models. So you may not even find hard filters or conditionals...instead the errors (or stereotypes?) would be embedded deep inside some neural net. I'm not even sure how one would explain that one to a jury.
- TazeTSchnitzel 11y agoVia learning models? I wonder how long it takes before it just automatically selects any black poor person.
- DasIch 11y agoMachine learning doesn't make a difference between correlation and causation. No doubt such systems already help reproduce the inequalities present in the data they've been trained on.
- TazeTSchnitzel 11y agoYeah, that's what I'm thinking. GIGO basically: the US records it would be using would suggest poor, black, mentally-ill etc. people are most likely to be criminals. The software will pick up on that.
- TazeTSchnitzel 11y agoThe essential problem is that in such environments the process of doing a task must be open to inspection, but software exists as a loophole that circumvents making process public.
- downandout 11y agoThis defense attorney was creative for asking to examine the source code, but that isn't the only way to cast doubt on the accuracy of the software that DNA matched his client to the crime scene. He could simply obtain a copy of it and have an expert run tests to determine a false positive rate and also what types of scenarios cause the software to deliver false positives, then call that expert as a witness.
- struppi 11y ago...which could be much more expensive and sill not find the error you'd need to defend the client. You'd essentially be black box testing without even knowing what kind of errors you want to find. Or am I missing something?
- downandout 11y agoIt would be more expensive probably. However, under the current legal framework, this would be the only way to question the accuracy of the software.
- kabdib 11y agoYou can also subpoena a list of bugs that have been fixed, and procedures they use to test the software. "So, how many bugs are left in your software?" might be a good start, too . . .
- larrik 11y agoCan't they just have other independent labs run the sample too? If they don't have any sample left, well... tough for the prosecution.
- donkeyd 11y agoI once nearly lost a contest because of a faulty SQL query on their side. If I didn't get to see the query, I wouldn't have been able to defend my entry and would've lost. Losing this contest would've been trivial, but if I applied this to a trial, it would be horrible. The error was that a 'group by' was used to find the number of unique entries, even though they had leading spaces, that were part of their uniqueness. Group by doesn't take leading spaces into account, leading them to get a different result than me. I think that this could've happened to a lot of people, even forensic IT engineers.
- esnard 11y agoActually it isn't specific to GROUP BY, the problem comes from the fact that some DMBS (MySQL for example) which ignore leading and trailing whitespaces (and other stuff like capitalization) while comparing strings, leading them to count less unique strings that a stricter comparision would count.
- cm2187 11y agoI'm not sure I agree with that view. Independant testing by another lab should remove any doubt on the validity of a forensic, rather than forcing companies to open source their technology. And of course some form of certification/random tests that ensures that the company providing the forensic isn't a bunch of conmen.
- mc808 11y agoBut how can you verify that the two independent companies aren't both using the same faulty code? You could ask someone under oath, but how would they know whether an employee bought or stole code from the competitor, or they both made similar errors in implementing a published algorithm, etc? As mentioned regarding Volkswagen, black box testing doesn't necessarily cut it (especially if the flaws are statistical in nature or triggered in unusual circumstances, and more so if they are deliberate).
- dogma1138 11y agoHow can you currently verify that 2 labs aren't using the same faulty forensic process when it comes to other types of evidence? Most forensic evidence isn't 100% conclusive things like DNA and fingerprint identification aren't a 1 to 1 unique match they are usually in the range of 1 out of 100-400,000 match. This is sufficient for most cases because the likelihood of that evidence being "wrong" when you combine it with other factors like motive, eye-witness accounts and other non-physical evidence is very slim.
- mc808 11y agoI'm not sure but would hope that traditional forensic labs and processes are also open to scrutiny. E.g. if the defense suggests that some important mark on the body could have been introduced while it was being handled, it should be possible to show why that couldn't happen with the procedures that were followed, and that the procedures were followed.
- dogma1138 11y ago
- triggercut 11y agoThere are similar issues with this in Structural and Mechanical engineering. Engineers are expected to rely more and more on software to execute and document complex calculations to verify designs, but how can you be sure those underlying calculations/theorems/models are correctly implemented? Some packages are constantly patching particular edge cases that get sent to them from their users. Many issue announcements to warn of bugs that could cause an incorrect result. If a result from software led to a critical failure in a design, the onus is most likely still on the Engineer. I have seen cases where software is formally reviewed by independent verification bodies, much in the same way your ISO 9001 compliance is. I can't see why this wouldn't apply here. Have an independent party, who has signed an appropriate NDA, asses and certify that your product does what it says on the tin and audit it at regular periods.
- shabble 11y agoI wonder if it's possible/practical to model the mechanical properties using multiple different tools, and check for agreement, along the lines of N-version programming[1] for software. It might get tricky where you're doing extremely specialised stuff where only a single implementation exists, but I suppose you'd have to assign some time to validating your tools personally for your use. [1] https://en.wikipedia.org/wiki/N-version_programming https://en.wikipedia.org/wiki/N-version_programming
- irq-1 11y ago> Have an independent party, who has signed an appropriate NDA, asses and certify that your product does what it says on the tin and audit it at regular periods. This may work for safety standards or financial liability, but it doesn't work for the legal system, because "independent" parties are influenced by who pays them.
- triggercut 11y agoYes, that's always a factor. Larger IVBs will tend to err on the side of their reputation in the marketplace (which may also be a factor). One could envisage a (horrible) situation where both parties engage an IVB (a practice common in some areas of engineering) and then they both jointly engage another body to effectively vet their processes and pass that joint 50/50 cost into their fees back to you. Or the court appoints one and you agree to share the costs (probably wouldn't fly in the US).
- jhwhite 11y agoThis use to be a problem in Florida with drunk driving arrests. The company that makes the code for the breathalyzers wouldn't allow their code to be reviewed by defendants. There was finally a precedent set that defendants couldn't mount a viable defense without reviewing the code. So for a while people accused of a DUI could wind up getting off, under the right circumstances, by requesting the source code then getting refused by the company. The company finally allowed pieces of the code to be reviewed by the courts.
- joesmo 11y agoThis is what happens when you have companies profiting off the misery of others. The biggest reason for companies wanting to protect their source code in this case is that they already know their software is broken, like pretty much every other software, and they don't want to fix it. The arguments against losing money and such are total bullshit as courts have plenty of procedures for disclosing materials only to the relevant parties present, not to the public as a whole. These companies simply don't want to spend the money auditing and making sure their code runs correctly because the only consequence of that is wrongfully convicting someone they don't give a fuck about. I'd say, let them see the code and let the highest paid expert witness win. That is, after all, the American way.