8 ms·

Proposed amendment to legal presumption about the reliability of computers

181 points by chrisjj 2y ago

dang 2y ago
Links related to the UK post office scandal. Others? Fujitsu bugs that sent innocent people to prison were known "from the start" - https://news.ycombinator.com/item?id=39059307 https://news.ycombinator.com/item?id=39059307 - Jan 2024 (270 comments) Fujitsu CEO Deposition – Post Office Horizon IT Inquiry - https://news.ycombinator.com/item?id=39059302 https://news.ycombinator.com/item?id=39059302 - Jan 2024 (1 comment) Fixing Horizon bugs would have been too costly, Post Office inquiry told - https://news.ycombinator.com/item?id=39039712 https://news.ycombinator.com/item?id=39039712 - Jan 2024 (59 comments) Fujitsu says it will pay compensation in UK Post Office scandal - https://news.ycombinator.com/item?id=39023695 https://news.ycombinator.com/item?id=39023695 - Jan 2024 (26 comments) How a software glitch at the UK Post Office ruined lives - https://news.ycombinator.com/item?id=39010070 https://news.ycombinator.com/item?id=39010070 - Jan 2024 (326 comments) Post Office Horizon scandal explained: Everything you need to know - https://news.ycombinator.com/item?id=38983144 https://news.ycombinator.com/item?id=38983144 - Jan 2024 (8 comments) A TV Show Forced Britain's Devastating Post Office Scandal into the Light - https://news.ycombinator.com/item?id=38951802 https://news.ycombinator.com/item?id=38951802 - Jan 2024 (168 comments) British Post Office Scandal - https://news.ycombinator.com/item?id=38937705 https://news.ycombinator.com/item?id=38937705 - Jan 2024 (149 comments) How the Post Office's Horizon system failed: a technical breakdown - https://news.ycombinator.com/item?id=38931792 https://news.ycombinator.com/item?id=38931792 - Jan 2024 (4 comments) Ex Post Office CEO hands back award after IT failures lead to false convictions - https://news.ycombinator.com/item?id=38930011 https://news.ycombinator.com/item?id=38930011 - Jan 2024 (127 comments) Post Office Horizon Enquiry – Fujitsu Report on Eposs PinICL Task Force (1998) - https://news.ycombinator.com/item?id=38926582 https://news.ycombinator.com/item?id=38926582 - Jan 2024 (1 comment) Fujitsu bosses knew about Post Office Horizon IT flaws, says insider (2021) - https://news.ycombinator.com/item?id=38890468 https://news.ycombinator.com/item?id=38890468 - Jan 2024 (8 comments) Mr Bates vs. the Post Office - https://news.ycombinator.com/item?id=38869011 https://news.ycombinator.com/item?id=38869011 - Jan 2024 (3 comments) What went wrong with Horizon: learning from the Post Office Trial - https://news.ycombinator.com/item?id=38867712 https://news.ycombinator.com/item?id=38867712 - Jan 2024 (19 comments) UK Post Office: 700 Horizon software scandal victims to receive £600k each - https://news.ycombinator.com/item?id=37561428 https://news.ycombinator.com/item?id=37561428 - Sept 2023 (40 comments) After 20 years, the Post Office scandal cover-up is happening in plain sight - https://news.ycombinator.com/item?id=36778486 https://news.ycombinator.com/item?id=36778486 - July 2023 (1 comment) The UK post office database scandal – “can't see the bug = user is a thief” - https://news.ycombinator.com/item?id=35837576 https://news.ycombinator.com/item?id=35837576 - May 2023 (2 comments) Hundreds of lives ruined by faulty UK Post Office computer system - https://news.ycombinator.com/item?id=35792896 https://news.ycombinator.com/item?id=35792896 - May 2023 (4 comments) Ex UK Post Office staff tell inquiry of stress of IT scandal - https://news.ycombinator.com/item?id=30394685 https://news.ycombinator.com/item?id=30394685 - Feb 2022 (2 comments) Post Office scandal: Public inquiry to examine wrongful convictions - https://news.ycombinator.com/item?id=30329668 https://news.ycombinator.com/item?id=30329668 - Feb 2022 (149 comments) Post Office scandal: 'I want someone else to be charged and jailed like I was' - https://news.ycombinator.com/item?id=30329510 https://news.ycombinator.com/item?id=30329510 - Feb 2022 (2 comments) Bad software sent postal workers to jail - https://news.ycombinator.com/item?id=26973583 https://news.ycombinator.com/item?id=26973583 - April 2021 (1 comment) Convicted Post Office workers have names cleared - https://news.ycombinator.com/item?id=26924882 https://news.ycombinator.com/item?id=26924882 - April 2021 (187 comments) UK court clears post office staff convicted due to ‘corrupt data’ - https://news.ycombinator.com/item?id=26913037 https://news.ycombinator.com/item?id=26913037 - April 2021 (284 comments) UK legal system assumes that computers don't have bugs - https://news.ycombinator.com/item?id=25518936 https://news.ycombinator.com/item?id=25518936 - Dec 2020 (24 comments) Post Office scandal: Postmasters celebrate victory against convictions - https://news.ycombinator.com/item?id=24661321 https://news.ycombinator.com/item?id=24661321 - Oct 2020 (2 comments) Bankruptcy, jail, ruined lives: inside the Post Office scandal - https://news.ycombinator.com/item?id=24440476 https://news.ycombinator.com/item?id=24440476 - Sept 2020 (1 comment) Postmasters were prosecuted using unreliable evidence - https://news.ycombinator.com/item?id=23454606 https://news.ycombinator.com/item?id=23454606 - June 2020 (2 comments) Faults in Post Office accounting system led to workers being convicted of theft - https://news.ycombinator.com/item?id=21795219 https://news.ycombinator.com/item?id=21795219 - Dec 2019 (104 comments) Post Office hires accountants to review sub-postmasters' computer claims - https://news.ycombinator.com/item?id=4143107 https://news.ycombinator.com/item?id=4143107 - June 2012 (1 comment)
deleted 2y ago
[deleted]
Macha 2y ago
For those not familiar with recent UK scandals, there has been one ongoing for a few years where there was a bug in a system designed by Fujitsu that was calculating the finances for post offices. This resulted in the system quoting wrong balances for post offices, and when that didn't match the actual cash in hand, resulted in prosecutions of the postmasters responsible for those offices which in turn led to convictions, dismissals, fines, imprisonments, and suicides over a 15 year period as the computer system was presumed correct. Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation. https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
- Latty 2y ago
  A strong recommendation for the excellent "Mr Bates vs The Post Office" to anyone who hasn't followed it, a short (four part) drama about it which is really worth a watch, and a good way to get a rough feel for what went on, and also managed to finally catalyse a real response by the government.
  - n4r9 2y ago
    Alan Bates sounds like the epitome of the British "stiff upper lip". Stubbornly standing against overwhelming adversity for years, for the good of his fellow society and his own integrity.
- bobnamob 2y ago
  > where there was a bug in a system designed by Fujitsu To clarify, there was not _one_ bug, but hundreds. https://en.wikipedia.org/wiki/British_Post_Office_scandal#Problems_with_the_software https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr... From my read of the information publicly available, there was a fundamental lack of distributed/transactional system understanding from the developers at Fujitsu. This[1] article recently posted here[2] reminded me of the Post Office Scandal. The lack of respect for established CS theory (transactions and distributed systems) and established accounting practices (double entry book keeping or even the idea of a ledger) is mind boggling. [1] https://news.alvaroduran.com/p/engineers-do-not-get-to-make-startup https://news.alvaroduran.com/p/engineers-do-not-get-to-make-... [2] https://news.ycombinator.com/item?id=42269227 https://news.ycombinator.com/item?id=42269227
greatgib 2y ago
That will be wonderful with AI: Sir, I have this conversation with chatgpt where the assistant tells me that you are guilty. Based on the fact that chatgpt is able to correctly count raspberries we can now legally consider that it is reliable and so that you are guilty! Game over.
- Etheryte 2y ago
  Can count raspberries, but not how many times the letter "r" appears in raspberries.
  - lionkor 2y ago
    It can once you point it out in the training data, hence we cannot know if it understands anything unless we can __prove__ that it wasn't in the training set (which we can't)
    - zarzavat 2y ago
      "Pray, Mr Babbage, if I ask an LLM a question that isn't in its training set, will it give the right answer?"
      - Terr_ 2y ago
        There's a little ambiguity in the original quote which I think requires a lot more attention with LLMs in the mix: Whether the "rightness" of an answer also depends on the correctness of the process used to reach it. I'd like to think that Babbage would have had the stricter interpretation.
    - int_19h 2y ago
      There are infinitely many combinations of things. It's not hard to come up with a bunch such that the probability of any one of them being in the training set is infinitesimal.
      - 2y ago
Devasta 2y ago
If there is one thing that should be crystal clear to everyone its that for some classes of work software should be like real engineering, requiring signoff by the senior engineer with personal liability and fines and jail for negligence. If the engineers dealing with the Citicorp center had dealt with the problem like software engineers, the fix would have been to update documentation in confluence to not expose the building to high winds and that would have been the end of it.
- jack_riminton 2y ago
  Whilst I agree in principle, would any Senior Engineer want to work on such a system that had their personal liability attached? I'd want 10's of millions in annual comp just for the risk
  - nottorp 2y ago
    The real problem is who could afford a system that a competent senior engineer would take personal liability for...
    - sfn42 2y ago
      The real problem is our entire industry is a giant clusterfuck. Do a 2-week bootcamp, congratulations now you're a software engineer. Every other discipline has education requirements, codified standards for how to do things etc.
      - nottorp 2y ago
        > codified standards for how to do things I don't know you but i bet that if you and me were locked up in a room together for a month we wouldn't be able to 100% agree on "codified standards for how to do things" :) Industry isn't mature enough for that and it's perhaps doubtful that it will ever be. See the halting problem.
        bobnamob 2y ago
jack_riminton 2y ago
The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law. Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%. Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.
- impossiblefork 2y ago
  I find it really strange to have presumptions in general. If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another. If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.
  - ndsipa_pomu 2y ago
    The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
  - beardyw 2y ago
    I imagine it stems from something like: the clock was correct at 1pm and at 3pm but was it correct at 2pm?
- Oarch 2y ago
  Maybe it was a presumption that was legally convenient... until it wasn't and reality struck?
  - 2y ago
gmerc 2y ago
Critical in the age of generative AI reliability.
- MrMcCall 2y ago
  The only thing reliable about them (and the humans that create them) is that they easily fool a bunch of fools. Of course, it does appear that they are incredibly reliable at being somewhat reliable, but that's not the kind of reliable that I'm looking for, especially when the environmental cost of generating them is so expensive, and doubly so as global heating ramps up.
kjrfghslkdjfl 2y ago
"Presumption about reliability of computers" Anyone else get the sense that these were laws written without any input whatsoever from people who actually have working experience in complex systems? I'm convinced that no moral software engineer would ever suggest a conviction on the presumption that the system is correct. At minimum you'd have to investigate.
- Robin_Message 2y ago
  The presumption was about mechanical instruments, from a simpler time. As computers expanded from being purely mechanical, it never got revised, which is why it is being revised now (three or four decades too late, I'll grant you, but legislation is a slow process.)
  - int_19h 2y ago
    Except it was revised once already to point out that computers aren't reliable. And then a decade later, at the peak of the dot-com boom, that revision was reverted after a review by committee.
deleted 2y ago
[deleted]
Kye 2y ago
Computers always do exactly as they're told. Unfortunately, there are so many people telling them to do so many things at so many layers of abstraction that the operator can't be sure of anything anymore. edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY FUNCTIONING COMPUTER," someone would still roll in to pick it apart. You can't please pedants and make a worthwhile point at the same time. Do circuits get weird? Can a stray cosmic ray flip a bit even in a system with ECC RAM? Sure. Does this meaningfully affect a bit about the hazards of abstractions on top of abstractions? No. The point is that abstractions amplify computer problems whether they're human error or ghosts in the machine.
- timthorn 2y ago
  They don't - random bit flips are expected over time. ECC memory is a thing because of this, but cosmic rays can strike the processors too.
  - Kye 2y ago
    I know I was doing a bit
    - bowsamic 2y ago
      I don't think jokes make sense when they are just entirely wrong though. Instead it just comes across as you being ignorant or perpetuating misinformation, rather than a joke
      - Kye 2y ago
        So far only two of about ten people took it in an overly literal way rather than understanding and going with the core of it so I think I'll keep aligning my bits toward them. Have a nice day.
        bowsamic 2y ago
        You’ll align your bits towards falsity and ineptitude? It is a big deal that computers don’t always do what they’re told in this case. STOP perpetuating lies
deleted 2y ago
[deleted]
jl6 2y ago
Seems to me that computers do generally perform their tasks reliably. (I say generally, not infallibly; cosmic rays and power failures and such). The issue is really about whether their human programmers have successfully programmed the computer to do the intended task, and about whether their human operators are successfully interpreting computer outputs. So this is surely about what constitutes an expert witness, and how reasonable it is to rely on the testimony of an expert witness, where computers are concerned.
- Ekaros 2y ago
  On can really consider how rare hardware issues leading to wrong computing are for your widely used computer chips. In general it seems they are very reliably correct and fixed in reasonably quick order. Same really cannot be said about software. But on computational level it is actually amazing when you think about it.
jbb67 2y ago
I understand why the presumption was there. If it's not then every single case of tax evasion people would claim that they had paid and the computer must have lost the payment. Every single traffic camera offense would be contested. Every time you didn't have a train ticket people would claim the computer lost it. And so on. And there would be months and months of delays if every time it had to be proved beyond a doubt that the equipment was working fully in that case out of all proportion to the risk of there actually being a problem. However clearly there does need to be a way to challenge the presumption too. Of course just assuming equipment is working is entirely unfair. After a brief look at the article it looks a reasonable weakening of the presumption which allows the court to consider and reject a challenge when they think fit according to the court rules (to be established) Pretty much like any other form of evidence.
thinkingemote 2y ago
For some background. The legal presumption dates from 1997. "Previously, section 69 of the Police and Criminal Evidence Act 1984 required anyone introducing computer-generated evidence to show the system was operating correctly and not being used improperly. The change followed the Law Commission’s 1997 review of the law on hearsay evidence."[1] from https://www.lawgazette.co.uk/law/it-experts-call-for-review-of-computer-is-always-right/5118414.article https://www.lawgazette.co.uk/law/it-experts-call-for-review-... [1] review https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.amazonaws.com/uploads/sites/30/2015/03/lc245_Legislating_the_Criminal_Code_Evidence_in_Criminal_Proceedings.pdf https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (page 197) --- Another interesting briefing note: "The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences": https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-computers-are-presumed-to-be-operating-correctly-unforeseen-and-unjust-consequences/ https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-... They suggest, when requested, that a party gets: - records of known errors and bugs in the system, their effect, and the actions taken in response - description of information security and other relevant standards and processes followed - reports of audits performed on the system and how it is managed, - evidence showing that reports of errors are managed properly and that changes to the system are properly controlled - evidence confirming that the search for documents was performed adequately, and was done so by a person with appropriate authority and knowledge, and - assurance that reasonable steps have been taken to establish that the evidence presented has not been tampered with.
Neil44 2y ago
When I worked for a drug company everything had to be documented and proceduralised. Because if you ask the FDA if you can sell your pills to the public they will ask how you know they're safe. And if you keep asking enough why's you get down to whether you can trust the output of computers. So every computer's build process and software stack has to be to documented and approved procedure so that you have a wad of paper to show them, when they ask that 'why'. It seems reasonable that if you're asking to send someone to jail based on the output of a computer you should be able to ask for reasonable proof that the output is correct.
Hilift 2y ago
Next up: What did Birmingham do with the £216 million for the 2020 Oracle Financials upgrade that will not be completed until 2026? For a city that has a 46% child poverty rate. Sometimes the technology isn't the problem, it's the people.
- onion2k 2y ago
  Sometimes the technology isn't the problem, it's the people. It's always a people problem.
linsomniac 2y ago
This reminds me of Kurt Vonnegut's book Player Piano. Imagine LLMs/AGI but implemented with vacuum tubes. It includes the line (from memory): We're sure of his sentencing, because we replaced all the vacuum tubes and ran the trial again and got the same result." https://en.wikipedia.org/wiki/Player_Piano_(novel) https://en.wikipedia.org/wiki/Player_Piano_(novel)
deskr 2y ago
This scandal is probably the biggest miscarriage of justice in Europe. The fundamental issue was perhaps "reliability of computers". But the biggest scandal is that a lot of people knew what was happening and were either silenced or told others to be silent.
- w14 2y ago
  Indeed, but of equal stature in my opinion is that they people on the receiving end of the miscarriage of justice are continuing to be abused by the so-called compensation process. During the inquiry, the statement was made, can't remember by whom, that value for taxpayer money had to be respected. I think its past that point, and I would like to know why the instigators of the miscarriage aren't at risk of losing their personal assets to (at least partly) cover the cost. Maybe we have to wait for the inquiry report for that, but in the meantime it is pretty clear that 'value for taxpayer money' means, at least to some degree, kicking the can down the road until as many of the claimants are dead as possible.
deleted 2y ago
[deleted]
gadders 2y ago
Does this apply to voting machines?
- M2Ys4U 2y ago
  The UK doesn't use any form of electronic voting. I don't think anywhere even uses electronic counting machines any more either.
deleted 2y ago
[deleted]
djoldman 2y ago
The linked piece by Stephen Mason sets out all the interesting context for the law portion (not the postal service fiasco). https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b42298ac587/section/0f113452-78ad-4c82-b28f-08cff1877036#fn517 https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b4...
lifeisstillgood 2y ago
For a long time I was pushing a campaign that every piece of software paid for with public funds should be made FOSS (unless national security etc). I struggle to work out why a post office point of sale should be vital to Britains security and we should have been able to see the code. On top of which I believe that making such code open means there will become a eco-system of ISPs who will be able to support, integrate and improve the software and provide local government users (ie postmasters) with worthwhile consultancy Under these circumstances it’s hard to see how this would have gone uncovered for so long. (Or rather, not uncovered, but unbelieved. The great tragedy of this affair is that us was known and reported on for years - but nothing happened. You know those films where the hero manages to get the proof to the newspaper / tv station and the film ends as the bad guys are bundled into police vans - yeah not so much.
bitwize 2y ago
If they don't call this the Computer Says No Act, I'd say there was a missed opportunity.
asdefghyk 2y ago
From the wikipedia page "...Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases......" Fujitsu and Post Office management both new of software faults. Either or both of these groups should have dug much deeper. The Post Office Management should have known OR hired Software EXPERTS to advise them on how to determine software reliability. Especially once it was evident a large number of people accused of theft. My Opinion is Post Office Management was negligent, particularly the CEO and other senior managers. ANd they should be charged with criminal negligence. CEO and senior managers are paid "big money" to deal with such companies, they should have dug "much deeper" to investigate, especially since their would have been large numbers who claimed innocence The software company Fujitsu, is also criminal negligence. Producing a financial system and not validating it. The defence lawyers ( maybe with software computer system speciality ) could have asked for test records, test data how the system was validated. That would have been my first question. OK , Fujitsu and Post office management claim system is reliable, WHERE IS THE EVIDENCE OF SYSTEM RELIABILITY. Where are the exact test records and test data to support this claim. Where are the system design documents. Where is the company's bug / issue tracking, reports. How exactly is the software tested? It seems the problem could have been detected at several levels , but for various reasons they all missed. I'm shocked that this issue went on for so long. (There must have good people somewhere , that said nothing, ) My opinion is shaped by my work experience as a software test analysist for many years ( a few decades ) , who has worked on software that does financial transactions, every release ( except ones where change was deemed minor ) of our software was examined by auditors from the government. These government auditors could do anything. ask for design documents, ask for test records. request tests be re run in their presence. That we run certain tests designed by the auditors. Interview any staff about the change. These software audits could be from a few hours to several days , depending on the change. Our company maintained a complete test system of the distributed system for this testing.
patrakov 2y ago
I have read the proposed amendment, and it still looks woefully insufficient. It changes the rules from "software is presumed to be reliable" to "software previously certified by any government authority is presumed to be reliable." This applies even if the certification body did not, in fact, do any significant stress-testing that would, e.g., uncover the races, or was not aware of the specific use case that triggers a bug. In other words: "the Court will presume correct operation of the computer system and will ignore the fact that the other party challenges this; the amendment references certifications coming from a government body at any time in the past as a valid basis to ignore such challenges."
- Terr_ 2y ago
  Now if only there were a little bug in whatever certified software handles judges' wages...
h4ck_th3_pl4n3t 2y ago
Boi are they gonna wake up in a cyber apocalypse. Why do people that don't know shit about technological topics have the power of legislation to write laws about science? Why can a court decide/confirm something like that without any technological experts involved?

181 points by chrisjj 2y ago

dang 2y ago

Links related to the UK post office scandal. Others? Fujitsu bugs that sent innocent people to prison were known "from the start" - https://news.ycombinator.com/item?id=39059307 https://news.ycombinator.com/item?id=39059307 - Jan 2024 (270 comments) Fujitsu CEO Deposition – Post Office Horizon IT Inquiry - https://news.ycombinator.com/item?id=39059302 https://news.ycombinator.com/item?id=39059302 - Jan 2024 (1 comment) Fixing Horizon bugs would have been too costly, Post Office inquiry told - https://news.ycombinator.com/item?id=39039712 https://news.ycombinator.com/item?id=39039712 - Jan 2024 (59 comments) Fujitsu says it will pay compensation in UK Post Office scandal - https://news.ycombinator.com/item?id=39023695 https://news.ycombinator.com/item?id=39023695 - Jan 2024 (26 comments) How a software glitch at the UK Post Office ruined lives - https://news.ycombinator.com/item?id=39010070 https://news.ycombinator.com/item?id=39010070 - Jan 2024 (326 comments) Post Office Horizon scandal explained: Everything you need to know - https://news.ycombinator.com/item?id=38983144 https://news.ycombinator.com/item?id=38983144 - Jan 2024 (8 comments) A TV Show Forced Britain's Devastating Post Office Scandal into the Light - https://news.ycombinator.com/item?id=38951802 https://news.ycombinator.com/item?id=38951802 - Jan 2024 (168 comments) British Post Office Scandal - https://news.ycombinator.com/item?id=38937705 https://news.ycombinator.com/item?id=38937705 - Jan 2024 (149 comments) How the Post Office's Horizon system failed: a technical breakdown - https://news.ycombinator.com/item?id=38931792 https://news.ycombinator.com/item?id=38931792 - Jan 2024 (4 comments) Ex Post Office CEO hands back award after IT failures lead to false convictions - https://news.ycombinator.com/item?id=38930011 https://news.ycombinator.com/item?id=38930011 - Jan 2024 (127 comments) Post Office Horizon Enquiry – Fujitsu Report on Eposs PinICL Task Force (1998) - https://news.ycombinator.com/item?id=38926582 https://news.ycombinator.com/item?id=38926582 - Jan 2024 (1 comment) Fujitsu bosses knew about Post Office Horizon IT flaws, says insider (2021) - https://news.ycombinator.com/item?id=38890468 https://news.ycombinator.com/item?id=38890468 - Jan 2024 (8 comments) Mr Bates vs. the Post Office - https://news.ycombinator.com/item?id=38869011 https://news.ycombinator.com/item?id=38869011 - Jan 2024 (3 comments) What went wrong with Horizon: learning from the Post Office Trial - https://news.ycombinator.com/item?id=38867712 https://news.ycombinator.com/item?id=38867712 - Jan 2024 (19 comments) UK Post Office: 700 Horizon software scandal victims to receive £600k each - https://news.ycombinator.com/item?id=37561428 https://news.ycombinator.com/item?id=37561428 - Sept 2023 (40 comments) After 20 years, the Post Office scandal cover-up is happening in plain sight - https://news.ycombinator.com/item?id=36778486 https://news.ycombinator.com/item?id=36778486 - July 2023 (1 comment) The UK post office database scandal – “can't see the bug = user is a thief” - https://news.ycombinator.com/item?id=35837576 https://news.ycombinator.com/item?id=35837576 - May 2023 (2 comments) Hundreds of lives ruined by faulty UK Post Office computer system - https://news.ycombinator.com/item?id=35792896 https://news.ycombinator.com/item?id=35792896 - May 2023 (4 comments) Ex UK Post Office staff tell inquiry of stress of IT scandal - https://news.ycombinator.com/item?id=30394685 https://news.ycombinator.com/item?id=30394685 - Feb 2022 (2 comments) Post Office scandal: Public inquiry to examine wrongful convictions - https://news.ycombinator.com/item?id=30329668 https://news.ycombinator.com/item?id=30329668 - Feb 2022 (149 comments) Post Office scandal: 'I want someone else to be charged and jailed like I was' - https://news.ycombinator.com/item?id=30329510 https://news.ycombinator.com/item?id=30329510 - Feb 2022 (2 comments) Bad software sent postal workers to jail - https://news.ycombinator.com/item?id=26973583 https://news.ycombinator.com/item?id=26973583 - April 2021 (1 comment) Convicted Post Office workers have names cleared - https://news.ycombinator.com/item?id=26924882 https://news.ycombinator.com/item?id=26924882 - April 2021 (187 comments) UK court clears post office staff convicted due to ‘corrupt data’ - https://news.ycombinator.com/item?id=26913037 https://news.ycombinator.com/item?id=26913037 - April 2021 (284 comments) UK legal system assumes that computers don't have bugs - https://news.ycombinator.com/item?id=25518936 https://news.ycombinator.com/item?id=25518936 - Dec 2020 (24 comments) Post Office scandal: Postmasters celebrate victory against convictions - https://news.ycombinator.com/item?id=24661321 https://news.ycombinator.com/item?id=24661321 - Oct 2020 (2 comments) Bankruptcy, jail, ruined lives: inside the Post Office scandal - https://news.ycombinator.com/item?id=24440476 https://news.ycombinator.com/item?id=24440476 - Sept 2020 (1 comment) Postmasters were prosecuted using unreliable evidence - https://news.ycombinator.com/item?id=23454606 https://news.ycombinator.com/item?id=23454606 - June 2020 (2 comments) Faults in Post Office accounting system led to workers being convicted of theft - https://news.ycombinator.com/item?id=21795219 https://news.ycombinator.com/item?id=21795219 - Dec 2019 (104 comments) Post Office hires accountants to review sub-postmasters' computer claims - https://news.ycombinator.com/item?id=4143107 https://news.ycombinator.com/item?id=4143107 - June 2012 (1 comment)

deleted 2y ago

[deleted]

Macha 2y ago

For those not familiar with recent UK scandals, there has been one ongoing for a few years where there was a bug in a system designed by Fujitsu that was calculating the finances for post offices. This resulted in the system quoting wrong balances for post offices, and when that didn't match the actual cash in hand, resulted in prosecutions of the postmasters responsible for those offices which in turn led to convictions, dismissals, fines, imprisonments, and suicides over a 15 year period as the computer system was presumed correct. Anyway, it was later proven the computer system was incorrect but the government there dragged their heels on exoneration and compensation. https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal

Latty 2y ago
A strong recommendation for the excellent "Mr Bates vs The Post Office" to anyone who hasn't followed it, a short (four part) drama about it which is really worth a watch, and a good way to get a rough feel for what went on, and also managed to finally catalyse a real response by the government.
- n4r9 2y ago
  Alan Bates sounds like the epitome of the British "stiff upper lip". Stubbornly standing against overwhelming adversity for years, for the good of his fellow society and his own integrity.
bobnamob 2y ago
> where there was a bug in a system designed by Fujitsu To clarify, there was not _one_ bug, but hundreds. https://en.wikipedia.org/wiki/British_Post_Office_scandal#Problems_with_the_software https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr... From my read of the information publicly available, there was a fundamental lack of distributed/transactional system understanding from the developers at Fujitsu. This[1] article recently posted here[2] reminded me of the Post Office Scandal. The lack of respect for established CS theory (transactions and distributed systems) and established accounting practices (double entry book keeping or even the idea of a ledger) is mind boggling. [1] https://news.alvaroduran.com/p/engineers-do-not-get-to-make-startup https://news.alvaroduran.com/p/engineers-do-not-get-to-make-... [2] https://news.ycombinator.com/item?id=42269227 https://news.ycombinator.com/item?id=42269227

greatgib 2y ago

That will be wonderful with AI: Sir, I have this conversation with chatgpt where the assistant tells me that you are guilty. Based on the fact that chatgpt is able to correctly count raspberries we can now legally consider that it is reliable and so that you are guilty! Game over.

Etheryte 2y ago
Can count raspberries, but not how many times the letter "r" appears in raspberries.
- lionkor 2y ago
  It can once you point it out in the training data, hence we cannot know if it understands anything unless we can __prove__ that it wasn't in the training set (which we can't)
  - zarzavat 2y ago
    "Pray, Mr Babbage, if I ask an LLM a question that isn't in its training set, will it give the right answer?"
    - Terr_ 2y ago
      There's a little ambiguity in the original quote which I think requires a lot more attention with LLMs in the mix: Whether the "rightness" of an answer also depends on the correctness of the process used to reach it. I'd like to think that Babbage would have had the stricter interpretation.
  - int_19h 2y ago
    There are infinitely many combinations of things. It's not hard to come up with a bunch such that the probability of any one of them being in the training set is infinitesimal.
    - 2y ago

Devasta 2y ago

If there is one thing that should be crystal clear to everyone its that for some classes of work software should be like real engineering, requiring signoff by the senior engineer with personal liability and fines and jail for negligence. If the engineers dealing with the Citicorp center had dealt with the problem like software engineers, the fix would have been to update documentation in confluence to not expose the building to high winds and that would have been the end of it.

jack_riminton 2y ago
Whilst I agree in principle, would any Senior Engineer want to work on such a system that had their personal liability attached? I'd want 10's of millions in annual comp just for the risk
- nottorp 2y ago
  The real problem is who could afford a system that a competent senior engineer would take personal liability for...
  - sfn42 2y ago
    The real problem is our entire industry is a giant clusterfuck. Do a 2-week bootcamp, congratulations now you're a software engineer. Every other discipline has education requirements, codified standards for how to do things etc.
    - nottorp 2y ago
      > codified standards for how to do things I don't know you but i bet that if you and me were locked up in a room together for a month we wouldn't be able to 100% agree on "codified standards for how to do things" :) Industry isn't mature enough for that and it's perhaps doubtful that it will ever be. See the halting problem.
      - bobnamob 2y ago

jack_riminton 2y ago

The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law. Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%. Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.

impossiblefork 2y ago
I find it really strange to have presumptions in general. If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another. If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.
- ndsipa_pomu 2y ago
  The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
- beardyw 2y ago
  I imagine it stems from something like: the clock was correct at 1pm and at 3pm but was it correct at 2pm?
Oarch 2y ago
Maybe it was a presumption that was legally convenient... until it wasn't and reality struck?
- 2y ago

gmerc 2y ago

Critical in the age of generative AI reliability.

MrMcCall 2y ago
The only thing reliable about them (and the humans that create them) is that they easily fool a bunch of fools. Of course, it does appear that they are incredibly reliable at being somewhat reliable, but that's not the kind of reliable that I'm looking for, especially when the environmental cost of generating them is so expensive, and doubly so as global heating ramps up.

kjrfghslkdjfl 2y ago

"Presumption about reliability of computers" Anyone else get the sense that these were laws written without any input whatsoever from people who actually have working experience in complex systems? I'm convinced that no moral software engineer would ever suggest a conviction on the presumption that the system is correct. At minimum you'd have to investigate.

Robin_Message 2y ago
The presumption was about mechanical instruments, from a simpler time. As computers expanded from being purely mechanical, it never got revised, which is why it is being revised now (three or four decades too late, I'll grant you, but legislation is a slow process.)
- int_19h 2y ago
  Except it was revised once already to point out that computers aren't reliable. And then a decade later, at the peak of the dot-com boom, that revision was reverted after a review by committee.

deleted 2y ago

[deleted]

Kye 2y ago

Computers always do exactly as they're told. Unfortunately, there are so many people telling them to do so many things at so many layers of abstraction that the operator can't be sure of anything anymore. edit: The thing is, had I prefaced it with "ASSUMING A PROPERLY FUNCTIONING COMPUTER," someone would still roll in to pick it apart. You can't please pedants and make a worthwhile point at the same time. Do circuits get weird? Can a stray cosmic ray flip a bit even in a system with ECC RAM? Sure. Does this meaningfully affect a bit about the hazards of abstractions on top of abstractions? No. The point is that abstractions amplify computer problems whether they're human error or ghosts in the machine.

timthorn 2y ago
They don't - random bit flips are expected over time. ECC memory is a thing because of this, but cosmic rays can strike the processors too.
- Kye 2y ago
  I know I was doing a bit
  - bowsamic 2y ago
    I don't think jokes make sense when they are just entirely wrong though. Instead it just comes across as you being ignorant or perpetuating misinformation, rather than a joke
    - Kye 2y ago
      So far only two of about ten people took it in an overly literal way rather than understanding and going with the core of it so I think I'll keep aligning my bits toward them. Have a nice day.
      - bowsamic 2y ago
        You’ll align your bits towards falsity and ineptitude? It is a big deal that computers don’t always do what they’re told in this case. STOP perpetuating lies

deleted 2y ago

[deleted]

jl6 2y ago

Seems to me that computers do generally perform their tasks reliably. (I say generally, not infallibly; cosmic rays and power failures and such). The issue is really about whether their human programmers have successfully programmed the computer to do the intended task, and about whether their human operators are successfully interpreting computer outputs. So this is surely about what constitutes an expert witness, and how reasonable it is to rely on the testimony of an expert witness, where computers are concerned.

Ekaros 2y ago
On can really consider how rare hardware issues leading to wrong computing are for your widely used computer chips. In general it seems they are very reliably correct and fixed in reasonably quick order. Same really cannot be said about software. But on computational level it is actually amazing when you think about it.

jbb67 2y ago

I understand why the presumption was there. If it's not then every single case of tax evasion people would claim that they had paid and the computer must have lost the payment. Every single traffic camera offense would be contested. Every time you didn't have a train ticket people would claim the computer lost it. And so on. And there would be months and months of delays if every time it had to be proved beyond a doubt that the equipment was working fully in that case out of all proportion to the risk of there actually being a problem. However clearly there does need to be a way to challenge the presumption too. Of course just assuming equipment is working is entirely unfair. After a brief look at the article it looks a reasonable weakening of the presumption which allows the court to consider and reject a challenge when they think fit according to the court rules (to be established) Pretty much like any other form of evidence.

thinkingemote 2y ago

For some background. The legal presumption dates from 1997. "Previously, section 69 of the Police and Criminal Evidence Act 1984 required anyone introducing computer-generated evidence to show the system was operating correctly and not being used improperly. The change followed the Law Commission’s 1997 review of the law on hearsay evidence."[1] from https://www.lawgazette.co.uk/law/it-experts-call-for-review-of-computer-is-always-right/5118414.article https://www.lawgazette.co.uk/law/it-experts-call-for-review-... [1] review https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.amazonaws.com/uploads/sites/30/2015/03/lc245_Legislating_the_Criminal_Code_Evidence_in_Criminal_Proceedings.pdf https://cloud-platform-e218f50a4812967ba1215eaecede923f.s3.a... (page 197) --- Another interesting briefing note: "The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences": https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-computers-are-presumed-to-be-operating-correctly-unforeseen-and-unjust-consequences/ https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-... They suggest, when requested, that a party gets: - records of known errors and bugs in the system, their effect, and the actions taken in response - description of information security and other relevant standards and processes followed - reports of audits performed on the system and how it is managed, - evidence showing that reports of errors are managed properly and that changes to the system are properly controlled - evidence confirming that the search for documents was performed adequately, and was done so by a person with appropriate authority and knowledge, and - assurance that reasonable steps have been taken to establish that the evidence presented has not been tampered with.

Neil44 2y ago

When I worked for a drug company everything had to be documented and proceduralised. Because if you ask the FDA if you can sell your pills to the public they will ask how you know they're safe. And if you keep asking enough why's you get down to whether you can trust the output of computers. So every computer's build process and software stack has to be to documented and approved procedure so that you have a wad of paper to show them, when they ask that 'why'. It seems reasonable that if you're asking to send someone to jail based on the output of a computer you should be able to ask for reasonable proof that the output is correct.

Hilift 2y ago

Next up: What did Birmingham do with the £216 million for the 2020 Oracle Financials upgrade that will not be completed until 2026? For a city that has a 46% child poverty rate. Sometimes the technology isn't the problem, it's the people.

onion2k 2y ago
Sometimes the technology isn't the problem, it's the people. It's always a people problem.

linsomniac 2y ago

This reminds me of Kurt Vonnegut's book Player Piano. Imagine LLMs/AGI but implemented with vacuum tubes. It includes the line (from memory): We're sure of his sentencing, because we replaced all the vacuum tubes and ran the trial again and got the same result." https://en.wikipedia.org/wiki/Player_Piano_(novel) https://en.wikipedia.org/wiki/Player_Piano_(novel)

deskr 2y ago

This scandal is probably the biggest miscarriage of justice in Europe. The fundamental issue was perhaps "reliability of computers". But the biggest scandal is that a lot of people knew what was happening and were either silenced or told others to be silent.

w14 2y ago
Indeed, but of equal stature in my opinion is that they people on the receiving end of the miscarriage of justice are continuing to be abused by the so-called compensation process. During the inquiry, the statement was made, can't remember by whom, that value for taxpayer money had to be respected. I think its past that point, and I would like to know why the instigators of the miscarriage aren't at risk of losing their personal assets to (at least partly) cover the cost. Maybe we have to wait for the inquiry report for that, but in the meantime it is pretty clear that 'value for taxpayer money' means, at least to some degree, kicking the can down the road until as many of the claimants are dead as possible.

deleted 2y ago

[deleted]

gadders 2y ago

Does this apply to voting machines?

M2Ys4U 2y ago
The UK doesn't use any form of electronic voting. I don't think anywhere even uses electronic counting machines any more either.

deleted 2y ago

[deleted]

djoldman 2y ago

The linked piece by Stephen Mason sets out all the interesting context for the law portion (not the postal service fiasco). https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b42298ac587/section/0f113452-78ad-4c82-b28f-08cff1877036#fn517 https://read.uolpress.co.uk/read/3a219939-14d3-45c9-9fe0-9b4...

lifeisstillgood 2y ago

For a long time I was pushing a campaign that every piece of software paid for with public funds should be made FOSS (unless national security etc). I struggle to work out why a post office point of sale should be vital to Britains security and we should have been able to see the code. On top of which I believe that making such code open means there will become a eco-system of ISPs who will be able to support, integrate and improve the software and provide local government users (ie postmasters) with worthwhile consultancy Under these circumstances it’s hard to see how this would have gone uncovered for so long. (Or rather, not uncovered, but unbelieved. The great tragedy of this affair is that us was known and reported on for years - but nothing happened. You know those films where the hero manages to get the proof to the newspaper / tv station and the film ends as the bad guys are bundled into police vans - yeah not so much.

bitwize 2y ago

If they don't call this the Computer Says No Act, I'd say there was a missed opportunity.

asdefghyk 2y ago

From the wikipedia page "...Although many subpostmasters had reported problems with the new software, and Fujitsu was aware that Horizon contained software bugs as early as 1999, the Post Office insisted that Horizon was robust and failed to disclose knowledge of the faults in the system during criminal and civil cases......" Fujitsu and Post Office management both new of software faults. Either or both of these groups should have dug much deeper. The Post Office Management should have known OR hired Software EXPERTS to advise them on how to determine software reliability. Especially once it was evident a large number of people accused of theft. My Opinion is Post Office Management was negligent, particularly the CEO and other senior managers. ANd they should be charged with criminal negligence. CEO and senior managers are paid "big money" to deal with such companies, they should have dug "much deeper" to investigate, especially since their would have been large numbers who claimed innocence The software company Fujitsu, is also criminal negligence. Producing a financial system and not validating it. The defence lawyers ( maybe with software computer system speciality ) could have asked for test records, test data how the system was validated. That would have been my first question. OK , Fujitsu and Post office management claim system is reliable, WHERE IS THE EVIDENCE OF SYSTEM RELIABILITY. Where are the exact test records and test data to support this claim. Where are the system design documents. Where is the company's bug / issue tracking, reports. How exactly is the software tested? It seems the problem could have been detected at several levels , but for various reasons they all missed. I'm shocked that this issue went on for so long. (There must have good people somewhere , that said nothing, ) My opinion is shaped by my work experience as a software test analysist for many years ( a few decades ) , who has worked on software that does financial transactions, every release ( except ones where change was deemed minor ) of our software was examined by auditors from the government. These government auditors could do anything. ask for design documents, ask for test records. request tests be re run in their presence. That we run certain tests designed by the auditors. Interview any staff about the change. These software audits could be from a few hours to several days , depending on the change. Our company maintained a complete test system of the distributed system for this testing.

patrakov 2y ago

I have read the proposed amendment, and it still looks woefully insufficient. It changes the rules from "software is presumed to be reliable" to "software previously certified by any government authority is presumed to be reliable." This applies even if the certification body did not, in fact, do any significant stress-testing that would, e.g., uncover the races, or was not aware of the specific use case that triggers a bug. In other words: "the Court will presume correct operation of the computer system and will ignore the fact that the other party challenges this; the amendment references certifications coming from a government body at any time in the past as a valid basis to ignore such challenges."

Terr_ 2y ago
Now if only there were a little bug in whatever certified software handles judges' wages...

h4ck_th3_pl4n3t 2y ago

Boi are they gonna wake up in a cyber apocalypse. Why do people that don't know shit about technological topics have the power of legislation to write laws about science? Why can a court decide/confirm something like that without any technological experts involved?