5 ms·

The failing is as much with the court as it is with Fujitsu. Why did they blindly accept Horizon’s data as evidence? What if the computer said the Queen stole a

by cedws 1y ago

The failing is as much with the court as it is with Fujitsu. Why did they blindly accept Horizon’s data as evidence? What if the computer said the Queen stole all the money and ran off to Barbados, would they have thrown her in jail? Why was the output of a black box, which may as well have been a notebook Fujitsu could have written anything they wanted into, treated as gospel?

rwmj 1y ago
The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary (and getting that evidence is basically impossible for the individuals being charged who were post office workers, not computer experts, and the source code was a trade secret). This might change, partly in response to this case: https://www.gov.uk/government/calls-for-evidence/use-of-evidence-generated-by-software-in-criminal-proceedings/use-of-evidence-generated-by-software-in-criminal-proceedings-call-for-evidence https://www.gov.uk/government/calls-for-evidence/use-of-evid... Quite interesting article about this: https://www.counselmagazine.co.uk/articles/the-presumption-that-computers-are-reliable https://www.counselmagazine.co.uk/articles/the-presumption-t...
- imtringued 1y ago
  The emperor has no clothes. Oxford is the worlds AI Safety research hub and yet they didn't think about campaigning to overturn a law which negates their entire reason for existing?
  - PaulRobinson 1y ago
    This happened a long time before the current resurgence in AI.
    - silon42 1y ago
      Imagine how much will "machine is right and can't be changed" happen with AI.
  - nightpool 1y ago
    Oxford is the world's what? If you believe that then I have a bridge to sell you.
  - jen20 1y ago
    Arguments made towards right-wing government (which the UK had for the past decade) from higher education are unlikely to be well received. Perhaps somewhat by Cameron, certainly not in the post-Brexit idiocracy of May, Johnson, Truss or Sunak.
- noisy_boy 1y ago
  > The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary That is just mind bogglingly stupid - who the hell are the idiots who wrote a law like that? Any of them wrote a line of code in their life?
  - whycome 1y ago
    Isn’t it a similar case in the USA where intoxication breath test computers are similarly obscured from scrutiny? People have argued that they have a right to “face their accuser” and see the source code only to have that request denied. So, black box.
    - BobaFloutist 1y ago
      Breathalyzers aren't typically considered sufficient evidence in of themselves to convict (or exonerate), iirc many PDs have a policy of treating a breathalyzer hit as probable cause more than anything and then either they throw you in the drunk tank if you don't demand a blood test to verify, or, if they want to actually prosecute you, they get a warrant for a blood test.
    - SoftTalker 1y ago
      AIUI breath test only establishes probable cause. If you fail a breath test you are taken for a blood draw. Breath test results are routinely challenged (sometimes successfully) by demanding records showing that the device has been tested and calibrated according to the required schedule.
      - worik 1y ago
        In my country (Aotearoa) the breath tests are "strict viability ", so proof You can demand a blood test, but you have to know. Most people do not know
  - arrowsmith 1y ago
    > who wrote a law That's not what "common law" means.
- mystraline 1y ago
  Governments should have access to all the source of code they buy licenses to (and provided at sale), as a precondition of selling to a government. When these sorts of things happen, the source can be subpoena'd with the relevant legal tool, and reviewed appropriately. Why governments don't do this is beyond me. It greatly limits liability of gov procurement, and puts the liability on the companies selling such goods.
  - varispeed 1y ago
    > Why governments don't do this is beyond me. Brown envelopes most likely and de facto non functioning SFO.
  - gnfargbl 1y ago
    Governments don't do get source code for the same reason as every other customer doesn't get source code: software vendors are incentivized to refuse the request. Why are the vendors so incentivized? Well, coming back to Fujitsu and the Post Office, the answer is that refusing to share the source was worth about a billion dollars: https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o
    - flir 1y ago
      Then they shouldn't get the contract. I hope lessons are learned, but I doubt it.
    - ChromaticPanic 1y ago
      This is why it's unethical for governments to use closed source software. Anything related to government functioning should be auditable.
  - daveoc64 1y ago
    Governments (certainly in the UK) aren't willing to pay enough to make this work for vendors. An escrow approach is quite common to protect the government in the event of a vendor going bankrupt or similar.
- bauble 1y ago
  Relevant XKCD: https://xkcd.com/2030/ https://xkcd.com/2030/
- cedws 1y ago
  I was not aware of this. Wow. I hope they're taking a hard look at past cases where they've done this.
  - masfuerte 1y ago
    No chance. The article concludes with the depressing statement that the government has no plans to reform the law, so the injustices will continue. They certainly won't be spending money on digging up old injustices.
- mbonnet 1y ago
  > There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary This is horrifying. I presume software is working incorrectly until proven otherwise.
blipvert 1y ago
Part of the answer is that the Post Office had (has?) special legal status in that it can prosecute cases by itself - no need to present a convincing case to the CPS like the police do. Many people were scared into pleading guilty just to avoid the upfront legal costs and the ruinous fines if contesting and found guilty (“the computer is always right”). Often the PO knew that they didn’t have much of a case but just used their special status to bully them into submission.
- foldr 1y ago
  This is a myth as far as I’ve been able to determine. The prosecutions were ordinary private prosecutions. The Post Office didn’t need any kind of special legal status in order to prosecute.
  - blipvert 1y ago
    Well, hey, far be it for me to tell you that you’re wrong, but the BBC says that you’re wrong as do numerous other sources. > The Post Office itself took many cases to court, prosecuting 700 people between 1999 and 2015. Another 283 cases were brought by other bodies, including the Crown Prosecution Service (CPS). https://www.bbc.co.uk/news/articles/c1wpp4w14pqo.amp https://www.bbc.co.uk/news/articles/c1wpp4w14pqo.amp
    - comp_throw7 1y ago
      https://en.wikipedia.org/wiki/Private_prosecution#United_Kingdom https://en.wikipedia.org/wiki/Private_prosecution#United_Kin...
    - foldr 1y ago
      That isn’t inconsistent with what I said. The Post Office prosecuted under the general right for private individuals or companies to bring a prosecution.

by cedws 1y ago

rwmj 1y ago

The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary (and getting that evidence is basically impossible for the individuals being charged who were post office workers, not computer experts, and the source code was a trade secret). This might change, partly in response to this case: https://www.gov.uk/government/calls-for-evidence/use-of-evidence-generated-by-software-in-criminal-proceedings/use-of-evidence-generated-by-software-in-criminal-proceedings-call-for-evidence https://www.gov.uk/government/calls-for-evidence/use-of-evid... Quite interesting article about this: https://www.counselmagazine.co.uk/articles/the-presumption-that-computers-are-reliable https://www.counselmagazine.co.uk/articles/the-presumption-t...

imtringued 1y ago
The emperor has no clothes. Oxford is the worlds AI Safety research hub and yet they didn't think about campaigning to overturn a law which negates their entire reason for existing?
- PaulRobinson 1y ago
  This happened a long time before the current resurgence in AI.
  - silon42 1y ago
    Imagine how much will "machine is right and can't be changed" happen with AI.
- nightpool 1y ago
  Oxford is the world's what? If you believe that then I have a bridge to sell you.
- jen20 1y ago
  Arguments made towards right-wing government (which the UK had for the past decade) from higher education are unlikely to be well received. Perhaps somewhat by Cameron, certainly not in the post-Brexit idiocracy of May, Johnson, Truss or Sunak.
noisy_boy 1y ago
> The actual answer to this is terrible. Courts had to trust the computer was correct. There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary That is just mind bogglingly stupid - who the hell are the idiots who wrote a law like that? Any of them wrote a line of code in their life?
- whycome 1y ago
  Isn’t it a similar case in the USA where intoxication breath test computers are similarly obscured from scrutiny? People have argued that they have a right to “face their accuser” and see the source code only to have that request denied. So, black box.
  - BobaFloutist 1y ago
    Breathalyzers aren't typically considered sufficient evidence in of themselves to convict (or exonerate), iirc many PDs have a policy of treating a breathalyzer hit as probable cause more than anything and then either they throw you in the drunk tank if you don't demand a blood test to verify, or, if they want to actually prosecute you, they get a warrant for a blood test.
  - SoftTalker 1y ago
    AIUI breath test only establishes probable cause. If you fail a breath test you are taken for a blood draw. Breath test results are routinely challenged (sometimes successfully) by demanding records showing that the device has been tested and calibrated according to the required schedule.
    - worik 1y ago
      In my country (Aotearoa) the breath tests are "strict viability ", so proof You can demand a blood test, but you have to know. Most people do not know
- arrowsmith 1y ago
  > who wrote a law That's not what "common law" means.
mystraline 1y ago
Governments should have access to all the source of code they buy licenses to (and provided at sale), as a precondition of selling to a government. When these sorts of things happen, the source can be subpoena'd with the relevant legal tool, and reviewed appropriately. Why governments don't do this is beyond me. It greatly limits liability of gov procurement, and puts the liability on the companies selling such goods.
- varispeed 1y ago
  > Why governments don't do this is beyond me. Brown envelopes most likely and de facto non functioning SFO.
- gnfargbl 1y ago
  Governments don't do get source code for the same reason as every other customer doesn't get source code: software vendors are incentivized to refuse the request. Why are the vendors so incentivized? Well, coming back to Fujitsu and the Post Office, the answer is that refusing to share the source was worth about a billion dollars: https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o https://www.bbc.co.uk/news/articles/cgm8lmz1xk1o
  - flir 1y ago
    Then they shouldn't get the contract. I hope lessons are learned, but I doubt it.
  - ChromaticPanic 1y ago
    This is why it's unethical for governments to use closed source software. Anything related to government functioning should be auditable.
- daveoc64 1y ago
  Governments (certainly in the UK) aren't willing to pay enough to make this work for vendors. An escrow approach is quite common to protect the government in the event of a vendor going bankrupt or similar.
bauble 1y ago
Relevant XKCD: https://xkcd.com/2030/ https://xkcd.com/2030/
cedws 1y ago
I was not aware of this. Wow. I hope they're taking a hard look at past cases where they've done this.
- masfuerte 1y ago
  No chance. The article concludes with the depressing statement that the government has no plans to reform the law, so the injustices will continue. They certainly won't be spending money on digging up old injustices.
mbonnet 1y ago
> There was a common law presumption that a computer was operating correctly unless there is evidence to the contrary This is horrifying. I presume software is working incorrectly until proven otherwise.

blipvert 1y ago

Part of the answer is that the Post Office had (has?) special legal status in that it can prosecute cases by itself - no need to present a convincing case to the CPS like the police do. Many people were scared into pleading guilty just to avoid the upfront legal costs and the ruinous fines if contesting and found guilty (“the computer is always right”). Often the PO knew that they didn’t have much of a case but just used their special status to bully them into submission.

foldr 1y ago
This is a myth as far as I’ve been able to determine. The prosecutions were ordinary private prosecutions. The Post Office didn’t need any kind of special legal status in order to prosecute.
- blipvert 1y ago
  Well, hey, far be it for me to tell you that you’re wrong, but the BBC says that you’re wrong as do numerous other sources. > The Post Office itself took many cases to court, prosecuting 700 people between 1999 and 2015. Another 283 cases were brought by other bodies, including the Crown Prosecution Service (CPS). https://www.bbc.co.uk/news/articles/c1wpp4w14pqo.amp https://www.bbc.co.uk/news/articles/c1wpp4w14pqo.amp
  - comp_throw7 1y ago
    https://en.wikipedia.org/wiki/Private_prosecution#United_Kingdom https://en.wikipedia.org/wiki/Private_prosecution#United_Kin...
  - foldr 1y ago
    That isn’t inconsistent with what I said. The Post Office prosecuted under the general right for private individuals or companies to bring a prosecution.