3 ms·

I get it, but I have to disagree on two ground. Firstly, in UK law there is an implicit assumption that computers are "reliable". This means that the burden of

I get it, but I have to disagree on two ground. Firstly, in UK law there is an implicit assumption that computers are "reliable". This means that the burden of proof is on the accused proving the software has bugs, rather than them being presumed innocent until proven guilty. That means in the Post Office example, people were convicted because they couldn't prove the software had bugs, they were just saying "you have to prove I'm guilty, because I'm not", and the court said "no, we don't, computers are reliable and that proves you are lying and therefore guilty". In the Ofsted case, given that schools can be put into special measures, teachers and headteachers fired, and in one case a headteacher killed herself because of the data collected and decision made on it, if those systems are not reliable they are not fit for purpose, because it turns the innocent into victims in terms of criminal, and civil employment law. There have been calls for this assumption to be changed [1], but right now: if you are innocent but a piece of software says you are guilty, in the UK the burden of proof falls to you, regardless of how awful the people, their processes and conduct are on the other side. Secondly, there's a deeper moral issue here. Engineers are arguably aiding and abetting. It is almost certainly the case that engineers working on Horizon and Ofsted systems did not tell all the truth all the time at the bequest of their managers and other colleagues throughout all this. "Just doing my job", is the excuse of the scoundrel. I do not buy it. If you're complicit with despicable people, you're complicit. End of. Yes, UK HMG has answers to give, culpability and responsibility, but let's not pretend that engineers built terrible systems, didn't have to face any consequences, and were enabled by arse covering exercises further up the chain. We can - and must - do better than this. [1] https://www.theguardian.com/uk-news/2024/jan/12/update-law-on-computer-evidence-to-avoid-horizon-repeat-ministers-urged https://www.theguardian.com/uk-news/2024/jan/12/update-law-o...

dazc 3y ago
'That means in the Post Office example, people were convicted because they couldn't prove the software had bugs, they were just saying "you have to prove I'm guilty, because I'm not", and the court said "no, we don't, computers are reliable and that proves you are lying and therefore guilty".' Not sure this is true, people were mainly convicted because they pleaded guilty to avoid going to jail. There may be instances where you are correct but I'm not aware of any? The general rule of being innocent unless proven guilty does, generally, hold. The issue with these cases is that many people convicted early on had no proper legal representation. Alan Bates, for instance, refused to accept the charges and was never convicted because no evidence of his guilt was ever presented. He stood his ground and was an outlier case.
BlueTemplar 3y ago
In this case, one of the main issues seems to be that assigning a grade of "unknown/uncertain" wasn't an option, regardless of what issues could have happened during the field work ? I'm also annoyed at how the Guardian is parroting the "made up" term, which is a terrible misrepresentation of "relying on your memory of the events". Speaking of decision-helping tools, there has been a French law for some years now : for any administrative use of them, the target citizen can request the algorithm to be explained to them in clear language. (I already have my popcorn ready for the inevitable clash with the fiscal inspection, which not only tends to operate as if laws didn't apply to them, but also I hear started using neural networks, which have basically no way to conform to this requirement.)
jevoten 3y ago
> "Just doing my job", is the excuse of the scoundrel. I do not buy it. If you're complicit with despicable people, you're complicit. End of. There's a world of difference between human fallibility, and knowingly doing evil as part of one's job. I won't call the bricklayer evil if someone builds a 10-story skyscraper [1] atop his humble garden shed foundation, that ends up collapsing and killing everyone inside. [1] To make the analogy perfectly clear - the skyscraper is not the software, but the persecution based on assuming the software is infallible.
LtWorf 3y ago
In uk they weren't allowed to inspect the software for a looooong time. And you can't prove anything without inspection.
CogitoCogito 3y ago
> Firstly, in UK law there is an implicit assumption that computers are "reliable". This means that the burden of proof is on the accused proving the software has bugs, rather than them being presumed innocent until proven guilty. This really is an incredibly stupid law. I could see assuming that computers are correctly operating as _instructed_ (e.g. the cpu doesn’t have bugs), but assuming that software is implemented correctly is just crazy. Everyone knows that software is buggy and updates are released all the time. Declaring that software is infallible is ridiculous.

dazc 3y ago

'That means in the Post Office example, people were convicted because they couldn't prove the software had bugs, they were just saying "you have to prove I'm guilty, because I'm not", and the court said "no, we don't, computers are reliable and that proves you are lying and therefore guilty".' Not sure this is true, people were mainly convicted because they pleaded guilty to avoid going to jail. There may be instances where you are correct but I'm not aware of any? The general rule of being innocent unless proven guilty does, generally, hold. The issue with these cases is that many people convicted early on had no proper legal representation. Alan Bates, for instance, refused to accept the charges and was never convicted because no evidence of his guilt was ever presented. He stood his ground and was an outlier case.

BlueTemplar 3y ago

In this case, one of the main issues seems to be that assigning a grade of "unknown/uncertain" wasn't an option, regardless of what issues could have happened during the field work ? I'm also annoyed at how the Guardian is parroting the "made up" term, which is a terrible misrepresentation of "relying on your memory of the events". Speaking of decision-helping tools, there has been a French law for some years now : for any administrative use of them, the target citizen can request the algorithm to be explained to them in clear language. (I already have my popcorn ready for the inevitable clash with the fiscal inspection, which not only tends to operate as if laws didn't apply to them, but also I hear started using neural networks, which have basically no way to conform to this requirement.)

jevoten 3y ago

> "Just doing my job", is the excuse of the scoundrel. I do not buy it. If you're complicit with despicable people, you're complicit. End of. There's a world of difference between human fallibility, and knowingly doing evil as part of one's job. I won't call the bricklayer evil if someone builds a 10-story skyscraper [1] atop his humble garden shed foundation, that ends up collapsing and killing everyone inside. [1] To make the analogy perfectly clear - the skyscraper is not the software, but the persecution based on assuming the software is infallible.

LtWorf 3y ago

In uk they weren't allowed to inspect the software for a looooong time. And you can't prove anything without inspection.

CogitoCogito 3y ago

> Firstly, in UK law there is an implicit assumption that computers are "reliable". This means that the burden of proof is on the accused proving the software has bugs, rather than them being presumed innocent until proven guilty. This really is an incredibly stupid law. I could see assuming that computers are correctly operating as _instructed_ (e.g. the cpu doesn’t have bugs), but assuming that software is implemented correctly is just crazy. Everyone knows that software is buggy and updates are released all the time. Declaring that software is infallible is ridiculous.