4 ms·

I heard that in the British system there’s a legal principle (or precedent?) that says that a computer system can be considered to do what its creators said it

I heard that in the British system there’s a legal principle (or precedent?) that says that a computer system can be considered to do what its creators said it does. I can’t find a citation for this though. Does HN?

IronWolve 3y ago
Theres talk to change the law, so that computer evidence isn't considered reliable by default. But its been talk for years, its only getting attention now. https://www.theguardian.com/uk-news/2024/jan/12/update-law-on-computer-evidence-to-avoid-horizon-repeat-ministers-urged https://www.theguardian.com/uk-news/2024/jan/12/update-law-o...
d1sxeyes 3y ago
It's not QUITE that directly, but it comes close. I think the Guardian article which was linked to in the sibling comment covers it quite well but basically: 1. Historically, mechanical tools are presumed to be working well. This makes things simpler. The example quoted by the Guardian is a good one: if someone wants to question the accuracy of a clock, it's on the person claiming the inaccuracy to prove their point. 2. In 1984, it became clear that computers are not just simple mechanical tools, and they were explicitly excluded from this assumption, by saying that computer evidence should be considered 'hearsay' (and therefore inadmissible) unless the prosecution can prove that the evidence is correct, either by a certificate from someone who can reasonably be expected to certify the correct functioning of that particular evidence, or by oral evidence. 3. This meant that anyone depending on the reliability of evidence from a computer (or piece of software, hardware, etc.) as part of their legal argument could be called upon to prove this, and the burden of proof lay with them (i.e.: as a defendant, I could require the prosecution to prove that the computer works as it is intended). 4. Following a review, it seems to be basically the conclusion that the requirements are inconsistent, unnecessarily onerous and time-consuming, and the way it was written was allowing criminals to get off on technicalities because the prosecution were not able to prove minor or irrelevant points about the functioning of the computer, and anyway other countries don't have any special rules about computers. You can read for yourself the recommendation here: https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.amazonaws.com/uploads/sites/30/2016/08/No.138-Criminal-Law-Evidence-in-Criminal-Proceedings-Hearsay-and-Related-Topics-A-Consultation-Paper-1.pdf https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (starting page 200 of the document, 215 of the PDF). 5. In 1999, the specific requirement for computer evidence to be treated as hearsay was removed. There's a subtle difference between what you said and the current state of affairs. It is still possible to challenge the accuracy of a computer system, but the burden of proof lies with the defence. It's not going to be good enough to say 'well I don't know what happened, it must be a computer glitch', and as a result, cause the prosecution to need to produce evidence that the terminal in the Post Office was working correctly, as well as all of the back end servers that may have been responsible in some part for producing the output. There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly, and the evidence be thrown out in case the prosecution are unable to do so. The problem with the Horizon convictions is that in many cases, the evidence produced by computers was the only evidence. Also, as the Post Office has its own prosecutors, they could chase and prosecute cases which would not normally have been tried by the CPS due to lack of evidence. It's also clear that the Post Office bullied and threatened not just the sub-postmasters, but also journalists, to keep quiet about the existence of evidence which might throw into question the correct functioning of the system. This whole debacle is not primarily caused by the principle that you're referring to. The presumption that computers function correctly has undoubtedly saved billions of pounds, hours, and allowed a huge number of successful, correct convictions, which otherwise might have resulted in not guilty verdicts due to clever litigation, rather than actual innocence.
- Etherlord87 3y ago
  > There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly, and the evidence be thrown out in case the prosecution are unable to do so. So a poor individual with no resources should find experts to analyze a software he doesn't even have access to (I know, discovery, but the practical limitations often make it near impossible to find the error within a given time, like having the access only in an isolated room without the internet or your own software, or having an access only to printed source code on paper), rather than the prosecutor (the state) or the company that definitely have the resources? Obviously almost no one would go to jail then. But this would be because of a very low coding standard. You pretty much can't prove the validity of software, because somewhere around 99,99% just isn't coded in a language and regime that would guarantee being bug-free. Check out the video by Veritasium "The Universe is Hostile to Computers" [1] - how much software, do you think, has internal error correction implemented to deal with a possible cosmic ray flipping a bit? The solution is to have extensive logs of data, stored by independent actors, with a lot of redundancy or otherwise error detection and correction. I find it horrifying that someone as well versed in law as you is fine with putting the burden of proof on the defendant, because it would be impractical to force programmers of serious software with serious consequences to actually employ serious standards of work... [1] https://www.youtube.com/watch?v=AaZ_RSt0KP8 https://www.youtube.com/watch?v=AaZ_RSt0KP8
  - d1sxeyes 3y ago
    > I find it horrifying that someone as well versed in law as you is fine with putting the burden of proof on the defendant, because it would be impractical to force programmers of serious software with serious consequences to actually employ serious standards of work... That’s not exactly my position. Basically my concern is that with the hearsay provision of PACE, a fraudster could conceivably require the prosecution to prove that Excel works from first principles, which is absurd. It’s also patently absurd to refuse to entertain the idea that a computer system may produce an incorrect response. But the law is an ass and you can only work with what you’re given.

IronWolve 3y ago

Theres talk to change the law, so that computer evidence isn't considered reliable by default. But its been talk for years, its only getting attention now. https://www.theguardian.com/uk-news/2024/jan/12/update-law-on-computer-evidence-to-avoid-horizon-repeat-ministers-urged https://www.theguardian.com/uk-news/2024/jan/12/update-law-o...

d1sxeyes 3y ago

It's not QUITE that directly, but it comes close. I think the Guardian article which was linked to in the sibling comment covers it quite well but basically: 1. Historically, mechanical tools are presumed to be working well. This makes things simpler. The example quoted by the Guardian is a good one: if someone wants to question the accuracy of a clock, it's on the person claiming the inaccuracy to prove their point. 2. In 1984, it became clear that computers are not just simple mechanical tools, and they were explicitly excluded from this assumption, by saying that computer evidence should be considered 'hearsay' (and therefore inadmissible) unless the prosecution can prove that the evidence is correct, either by a certificate from someone who can reasonably be expected to certify the correct functioning of that particular evidence, or by oral evidence. 3. This meant that anyone depending on the reliability of evidence from a computer (or piece of software, hardware, etc.) as part of their legal argument could be called upon to prove this, and the burden of proof lay with them (i.e.: as a defendant, I could require the prosecution to prove that the computer works as it is intended). 4. Following a review, it seems to be basically the conclusion that the requirements are inconsistent, unnecessarily onerous and time-consuming, and the way it was written was allowing criminals to get off on technicalities because the prosecution were not able to prove minor or irrelevant points about the functioning of the computer, and anyway other countries don't have any special rules about computers. You can read for yourself the recommendation here: https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.amazonaws.com/uploads/sites/30/2016/08/No.138-Criminal-Law-Evidence-in-Criminal-Proceedings-Hearsay-and-Related-Topics-A-Consultation-Paper-1.pdf https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (starting page 200 of the document, 215 of the PDF). 5. In 1999, the specific requirement for computer evidence to be treated as hearsay was removed. There's a subtle difference between what you said and the current state of affairs. It is still possible to challenge the accuracy of a computer system, but the burden of proof lies with the defence. It's not going to be good enough to say 'well I don't know what happened, it must be a computer glitch', and as a result, cause the prosecution to need to produce evidence that the terminal in the Post Office was working correctly, as well as all of the back end servers that may have been responsible in some part for producing the output. There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly, and the evidence be thrown out in case the prosecution are unable to do so. The problem with the Horizon convictions is that in many cases, the evidence produced by computers was the only evidence. Also, as the Post Office has its own prosecutors, they could chase and prosecute cases which would not normally have been tried by the CPS due to lack of evidence. It's also clear that the Post Office bullied and threatened not just the sub-postmasters, but also journalists, to keep quiet about the existence of evidence which might throw into question the correct functioning of the system. This whole debacle is not primarily caused by the principle that you're referring to. The presumption that computers function correctly has undoubtedly saved billions of pounds, hours, and allowed a huge number of successful, correct convictions, which otherwise might have resulted in not guilty verdicts due to clever litigation, rather than actual innocence.

Etherlord87 3y ago
> There's an extent to which I think this is reasonable. If I'm accused of fraud based on evidence recovered from a bank computer, it should not be the case that I can require the prosecution to prove that the bank's computers function correctly, and the evidence be thrown out in case the prosecution are unable to do so. So a poor individual with no resources should find experts to analyze a software he doesn't even have access to (I know, discovery, but the practical limitations often make it near impossible to find the error within a given time, like having the access only in an isolated room without the internet or your own software, or having an access only to printed source code on paper), rather than the prosecutor (the state) or the company that definitely have the resources? Obviously almost no one would go to jail then. But this would be because of a very low coding standard. You pretty much can't prove the validity of software, because somewhere around 99,99% just isn't coded in a language and regime that would guarantee being bug-free. Check out the video by Veritasium "The Universe is Hostile to Computers" [1] - how much software, do you think, has internal error correction implemented to deal with a possible cosmic ray flipping a bit? The solution is to have extensive logs of data, stored by independent actors, with a lot of redundancy or otherwise error detection and correction. I find it horrifying that someone as well versed in law as you is fine with putting the burden of proof on the defendant, because it would be impractical to force programmers of serious software with serious consequences to actually employ serious standards of work... [1] https://www.youtube.com/watch?v=AaZ_RSt0KP8 https://www.youtube.com/watch?v=AaZ_RSt0KP8
- d1sxeyes 3y ago
  > I find it horrifying that someone as well versed in law as you is fine with putting the burden of proof on the defendant, because it would be impractical to force programmers of serious software with serious consequences to actually employ serious standards of work... That’s not exactly my position. Basically my concern is that with the hearsay provision of PACE, a fraudster could conceivably require the prosecution to prove that Excel works from first principles, which is absurd. It’s also patently absurd to refuse to entertain the idea that a computer system may produce an incorrect response. But the law is an ass and you can only work with what you’re given.