4 ms·

The more I read of this the more I think it’s the legal system that’s really at fault here. Any developer that says “trust me, the software works” can’t be trus

The more I read of this the more I think it’s the legal system that’s really at fault here. Any developer that says “trust me, the software works” can’t be trusted. I feel like every user of software has experienced this fact. I feel like they should need third party code audits, field testing and generally a much higher burden of proof such to actually rule out software bugs sufficiently to jail someone.

datavirtue 2y ago
Instead, they got caught up in the post office judicial system, eliminating many basic rights and protections.
- chrisjj 2y ago
  They went through the regular judicial system. But there they apparently lost one major right. https://news.ycombinator.com/item?id=40808808 https://news.ycombinator.com/item?id=40808808
IshKebab 2y ago
Even worse, "trust me, the software I wrote works". I guess the problem is the defence didn't have an expert witness who had access to the source code and bug database (if they even had one) etc. to even counter that. Feels a lot like the nonsense forensics that has been in the news fairly recently.
tialaramex 2y ago
Historically the assumption was that a machine under question probably doesn't work, so the prosecution must show to a jury that it works or else they haven't met the burden, you're innocent because the machine doesn't work. This probably feels fine for a while, and then you discover you're repeatedly releasing people who are very obviously crooks who did exactly what they were accused of, but you couldn't prove to the satisfaction of a jury that the machine everybody knows works does in fact work. They need offer no evidence that it doesn't work, that's your problem, they walk free. So this assumption was reversed. But then you get Horizon. Did you steal £18,284.27 from the Post Office? Don't say "No" because the computer says you did, and unless you can prove why it's faulty, the Post Office can rely on that £18,284.27 because a computer said so and under law the assumption is that it's correct. Clearly neither of these simple assumptions is adequate. That's a problem. None of which justifies what was done to these people, that's over and above the technical legal problem of how to deliver a just outcome - nobody at POL or ICL/Fujitsu seems to have cared about a just outcome anyway.
- conductr 2y ago
  > This probably feels fine for a while, and then you discover you're repeatedly releasing people who are very obviously crooks who did exactly what they were accused of, but you couldn't prove to the satisfaction of a jury I’m personally ok with this. I’ll always err on the side of innocence. It’s just money and relatively small amounts. In the US we’d usually just terminate someone that we suspected of theft. We don’t try to sue them and may not even call the authorities at some of these amounts. We’d reevaluate our hiring processes and criteria if we noticed the problem was chronic and unexplained. But if we did seek out a court case, criminal or civil, the thing is this “expert witness” even on paper can’t be trusted. He built the software and should be considered biased towards saving face and any confidence he has in the software should be taken with a grain of salt. That’s without hearing him speak and purely on his resume. The prosecution would(hopefully) need to introduce other evidences. Like, follow the money type stuff. Unusual purchases and such. It would have to still paint a pretty clear picture to get put in jail I would hope. It seems this whole judicial body needs to be reevaluated to me if this type of injustice can be tolerated for years and dozens of people being sent to jail.
  - tialaramex 2y ago
    This assumption goes to everything, not just theft. Cell phone records show you were in the vicinity of the victim's house at the appropriate time, not actually at home watching Netflix? Unless they can prove the cell network is infallible you can expect to be able to rely on the claim you were home watching a show. Stopped a mile up the road from a hit-and-run, weaving drunkenly at excess speed? No worries they'll need to prove the speed camera can't be wrong, and prove that all the alcohol technology is entirely reliable and couldn't be mistaken. The guy who arrested you can say you weren't driving well, but chances are your lawyer can trick him into asserting you were drunk, which is inadmissible since machines aren't perfect, that means it's a mistrial, you walk. The problem isn't necessarily that this happens in every single case, but just that it feels so obviously unjust, just as Horizon does. The UK cut a hole in "Double Jeopardy" rules exactly because of such injustice - with double jeopardy you could do a crime, get off on a technicality and then tell anybody who asks that you did the crime, knowing you can't be prosecuted no matter what. Under modern rules while it won't be easy to prosecute you for a crime twice it's no longer impossible, which makes such boasting an obviously bad idea, you set yourself up by effectively confessing.

datavirtue 2y ago

Instead, they got caught up in the post office judicial system, eliminating many basic rights and protections.

chrisjj 2y ago
They went through the regular judicial system. But there they apparently lost one major right. https://news.ycombinator.com/item?id=40808808 https://news.ycombinator.com/item?id=40808808

IshKebab 2y ago

Even worse, "trust me, the software I wrote works". I guess the problem is the defence didn't have an expert witness who had access to the source code and bug database (if they even had one) etc. to even counter that. Feels a lot like the nonsense forensics that has been in the news fairly recently.

tialaramex 2y ago

Historically the assumption was that a machine under question probably doesn't work, so the prosecution must show to a jury that it works or else they haven't met the burden, you're innocent because the machine doesn't work. This probably feels fine for a while, and then you discover you're repeatedly releasing people who are very obviously crooks who did exactly what they were accused of, but you couldn't prove to the satisfaction of a jury that the machine everybody knows works does in fact work. They need offer no evidence that it doesn't work, that's your problem, they walk free. So this assumption was reversed. But then you get Horizon. Did you steal £18,284.27 from the Post Office? Don't say "No" because the computer says you did, and unless you can prove why it's faulty, the Post Office can rely on that £18,284.27 because a computer said so and under law the assumption is that it's correct. Clearly neither of these simple assumptions is adequate. That's a problem. None of which justifies what was done to these people, that's over and above the technical legal problem of how to deliver a just outcome - nobody at POL or ICL/Fujitsu seems to have cared about a just outcome anyway.

conductr 2y ago
> This probably feels fine for a while, and then you discover you're repeatedly releasing people who are very obviously crooks who did exactly what they were accused of, but you couldn't prove to the satisfaction of a jury I’m personally ok with this. I’ll always err on the side of innocence. It’s just money and relatively small amounts. In the US we’d usually just terminate someone that we suspected of theft. We don’t try to sue them and may not even call the authorities at some of these amounts. We’d reevaluate our hiring processes and criteria if we noticed the problem was chronic and unexplained. But if we did seek out a court case, criminal or civil, the thing is this “expert witness” even on paper can’t be trusted. He built the software and should be considered biased towards saving face and any confidence he has in the software should be taken with a grain of salt. That’s without hearing him speak and purely on his resume. The prosecution would(hopefully) need to introduce other evidences. Like, follow the money type stuff. Unusual purchases and such. It would have to still paint a pretty clear picture to get put in jail I would hope. It seems this whole judicial body needs to be reevaluated to me if this type of injustice can be tolerated for years and dozens of people being sent to jail.
- tialaramex 2y ago
  This assumption goes to everything, not just theft. Cell phone records show you were in the vicinity of the victim's house at the appropriate time, not actually at home watching Netflix? Unless they can prove the cell network is infallible you can expect to be able to rely on the claim you were home watching a show. Stopped a mile up the road from a hit-and-run, weaving drunkenly at excess speed? No worries they'll need to prove the speed camera can't be wrong, and prove that all the alcohol technology is entirely reliable and couldn't be mistaken. The guy who arrested you can say you weren't driving well, but chances are your lawyer can trick him into asserting you were drunk, which is inadmissible since machines aren't perfect, that means it's a mistrial, you walk. The problem isn't necessarily that this happens in every single case, but just that it feels so obviously unjust, just as Horizon does. The UK cut a hole in "Double Jeopardy" rules exactly because of such injustice - with double jeopardy you could do a crime, get off on a technicality and then tell anybody who asks that you did the crime, knowing you can't be prosecuted no matter what. Under modern rules while it won't be easy to prosecute you for a crime twice it's no longer impossible, which makes such boasting an obviously bad idea, you set yourself up by effectively confessing.