4 ms·
Every prosecutor in possession computer evidence of a crime having to prove the software correct is a nonstarter. Assuming it's right, and proving it buggy, is
by kazinator 3y ago
Every prosecutor in possession computer evidence of a crime having to prove the software correct is a nonstarter.
Assuming it's right, and proving it buggy, is the right principle.
- zubairq 3y agoI’m quite interested to learn and understand the line of reasoning for this comment. Care to elaborate?
- jpc0 3y agoIf bugs were easy to find and solve there wouldn't be an entire highly paid industry to do it. Add to that malicious actors actively trying to work against systems (bank/post office/medical system etc) and it becomes pretty hard to prove that both the software and the data referenced by the software is in fact correct. Now think about the last time a critical piece of data was recorded on paper with a pen/analogue typewriter and both parties got a physical copy of it. Last time you made a cash withdrawal did you have to go to a teller at the bank, fill in paperwork, have two copies of that paperwork exist and then that paperwork was digitised or was the entire process digital beginning to end? Think about how you, if you were an expert witness, would prove that an transaction made on a banking website was not a computer fault. Do you show a code audit which meets some standards? Do you have the bank's CISO explain all of the security measures? INAL but I'm reasonably sure the defendant only has to prove that a system with the same security and audit trail has in fact been faulty and your evidence goes out the door or at least the validity of it decreases.
- zubairq 3y agoThanks for the detailed answer. I think however that the real reason that there is a big industry to find and solve bugs is because of decision made when building and designing the software in the first place. For example if software was built in a robust way with care taken to refactor code along the way then maintenance and bug fixing would probably be a lot easier and may not need an entire industry to support it
- jpc0 3y agoI would hope so but I don't think that is the case, there is a portion of bugs, including security vulnerabilities that are actually hardware related but has a workaround in software. That and no matter how hard you try there will always be logic errors in programming that will need to be fixed.
- zubairq 3y agoYes that is actually a very good point, I stand corrected
- kazinator 3y agoIf programs were easy to prove correct, there wouldn't be a testing and debugging industry. Bugs don't have to be root caused and solved in order to call into question the use of some computer situation as circumstantial evidence. The claimed buggy, externally-visible behavior just has to be reproduced. If there are third-party reports of it, that would help.
- jpc0 3y agoI would argue programs are impossible to prove correct, I would speculate it is related to the halting problem. That doesn't mean that in general the subsections of it related to any specific court case isn't correct. Unfortunately disproving something is much much harder than proving something. Proving that there was a bug in a program is hard but proving that there isn't is nearly impossible and that is likely why the law is the way it is.
- dboreham 3y ago> If programs were easy to prove correct This is just another turtle layer because such proof is only vs a "specification", which turns out to also be a program, written in another language.
- reuben364 3y agoTaking this to absurdity you can create a new obfuscated program for each case that presumes guilt based on whatever evidence is could be in principle be decided with a computer, maximizing the burden on the defendants. Maybe another principle would be that a program is assumed to function as proven in the past. The further a given program is shown prior to be correct, the more the burden of proof shifts to the defendant to show otherwise.
- checkyoursudo 3y agoThis is contrary to the principle of presumed innocence. This is presumed guilt. The burden should always be on the prosecution to prove guilt, and if that requires the prosecution to prove the correctness of hardware, models, software, or data, then that is what should be required. And if the prosecution cannot prove the legitimacy of evidence, then than evidence should be disallowed. The standard does not have to be beyond all doubt. The standard is fine at beyond reasonable doubt. If the prosecution can replicate consistent evidence for and against the defendant using the same computer systems with simulated data, then that is probably enough to clear reasonable doubt. But the burden of proof should absolutely always be on the party wishing to use "evidence" to take someone else's property, freedom, or life. Assuming it's right is assuming guilt. That is the wrong principle.
- kazinator 3y ago> This is contrary to the principle of presumed innocence. No it isn't. It just means that some situation involving information from computers is "circumstantial evidence". If programs have to be proved correct, that basically means data from computers could never be introduced as circumstantial evidence. For instance, guns are normally assumed not to go off unless someone pulls the trigger. If a member of the Crips guns down a Bloods member across the street, he could claim that his gun accidentally went off the moment he was inadvertently pointing a gun in the direction of the enemy gang member. If he could prove that true, then it could be ruled an accidental death. If the defense lawyer does not make that argument, it will be assumed that there was nothing wrong with the gun, and thus someone pulled the trigger.
- kazinator 3y ago> For instance, guns are normally assumed not to go off unless someone pulls the trigger. Well, I looked into this; it's not that clear-cut.