10 ms·

The presumption that computers are working correctly

285 points by ColinWright 3y ago

nuc1e0n 3y ago
As a developer I never assume computer systems are working correctly. Quite the opposite in fact. If something isn't on the 'happy path' I assume something about the systems involved isn't working correctly and go check the facts for myself.
deleted 3y ago
[deleted]
chrisjj 3y ago
https://archive.ph/wip/AJl8i https://archive.ph/wip/AJl8i for permanance.
SoftTalker 3y ago
My usual presumption is that the computers are not working correctly, as I've seen way too many cases where they in fact are not working correctly. I don't even believe Google Maps when I'm driving, I always sanity check the route and double check street names/exit numbers when it tells me to turn. I've been burned too many times.
quickthrower2 3y ago
The title is maybe the opposite of clickbait. The article talks about the Post Office scandal. That is a big topic to take on!
topspin 3y ago
"Code is law." -Lawrence Lessig The UK apparently made that explicit. Doubtless others have as well: if we could forego the flag waving and whataboutery, That Would Be Great (tm).
- User23 3y ago
  Lawrence Lessig's greatest claim to fame is handing Disney and friends unlimited copyright before the Supreme Court. I'd hold up Judge Alsup if I wanted to give an example of a tech saavy legal practitioner.
  - morelisp 3y ago
    While it's increasingly clear that Lessig's legacy will be as (to borrow a phrase) the internet's own failson, he is far more famous today for CC than Eldred v. Ashcroft.
phire 3y ago
As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
- TedDoesntTalk 3y ago
  It does happen to Hardware. Here are two older examples: “The Intel Pentium series of CPUs had two well-known bugs discovered after it was brought to market, the FDIV bug affecting floating point division which resulted in a recall in 1994, and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted.” https://en.m.wikipedia.org/wiki/Hardware_bug https://en.m.wikipedia.org/wiki/Hardware_bug
  - thaumasiotes 3y ago
    > and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted You reminded me of the Things I Won't Work With piece that observed in passing that the behavior of dioxygen difluoride was surprisingly well described by its chemical formula. ( https://www.science.org/content/blog-post/things-i-won-t-work-dioxygen-difluoride https://www.science.org/content/blog-post/things-i-won-t-wor... )
  - deleted 3y ago
    [deleted]
- tivert 3y ago
  > As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. > But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to. That's not a distinction that's relevant to this article. By "computer" it means a system composed of hardware and software (and which is almost always what "computer" is used to mean): > The presumption that computers are presumed to be operating correctly, unless there is evidence to the contrary is what lawyers call “a presumption of evidence”. > This means that a court can be satisfied that a relevant fact can be established just by computer records, unless there is evidence that the computer is not working properly. > And so when the computer record shows, for instance, a financial shortfall by postmaster or postmistress, the court will accept that as evidence of an actual shortfall - unless the defendant can show that the computer was not operating correctly. > In short, when the computer record is the essence of a prosecution case: computer says guilty.
jfoutz 3y ago
How does that work? I'm not a lawyer. I'm a lazy programmer. So, does the defense get to subpoena the logs, and depose a programmer to explain why some random request returned a 502? Does the defense get, like, red team access? Although I'm a lazy programmer, I do think a lot about failure modes. If my freedom is on the line, I can probably think up a few (orders of magnitude) test cases. Happy to provide a shell script or python or golang binary to check. Heck, if it comes down to it I'll provide a stack of fortran cards. Do I get to see the source? That "few" is probably going to move a decimal place or 5. I get volume is not reasonable in the eyes of the court. Scale is how you(I) find problems, but whatever. Gimmie source, gimme access, I doubt I could beat Knuth, but with my freedom on the line I could give him a run for his money. How do CVE's count? If I find evidence of log4j do I auto win? This makes me think of the Jepsen tire fire talk. The system probably mostly right. But there's no way it's correct. What's a "fair" level of access to prove that?
- qingcharles 3y ago
  In the US at least it is hard to bring source code into the trial, even though you technically have a right. I remember one case where the FBI installed malware via a browser exploit on thousands of computers to get their IPs and one defendant tried to get the source code but the government managed to keep it secret.
  - Terr_ 3y ago
    That sounds like one of the "Playpen" cases, for which the EFF has a writeup on the hacking tool. [0] Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service. [0] https://www.eff.org/pages/playpen-cases-frequently-asked-questions https://www.eff.org/pages/playpen-cases-frequently-asked-que... [1] https://www.eff.org/deeplinks/2021/02/federal-court-agrees-prosecutors-cant-keep-forensic-evidence-secret-defendants https://www.eff.org/deeplinks/2021/02/federal-court-agrees-p...
    - Roark66 3y ago
      >Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service. I'm glad at least in this instance the defendant got access to the source code. He was being helped in pursuing this by EFF so I assume there were volunteers that helped with the analysis. But what if your case is lower profile? What if you can't afford to hire 10 programmers for 3 months to write test cases? Also I wonder how such a request would go down in a UK court.
genr8 3y ago
a 1984 law
- di4na 3y ago
  Nope, a 1999 one. Sadly.
suzzer99 3y ago
This is the first I've heard of this scandal. I'm curious what the major problems were with the software that caused the discrepancies. Dropped connections? Wikipedia didn't go into much detail with that. Ok I found this: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
- di4na 3y ago
  It is one of the biggest scandal of our time related to software. I ... want to recommend learning more about it but only do so with a strong stomach. Numerous people that rabbithole into this case end up running to a nearby bucket. It is... not pretty.
  - zelos 3y ago
    I agree, I find it hard to read about because it makes me so angry. Four people driven to suicide, hundreds of lives ruined. It took them so long to hold all the inquiries and decide on compensation that dozens have died before being acquitted or receiving compensation.
    - di4na 3y ago
      And that is only the one we know of. There are thousands of cases that are hidden in the bowels of the post office court system Oh and the compensation are uh... let say hard to get the money for.
      - andrewaylett 3y ago
        And I've yet to hear mention of the people who weren't prosecuted because they paid the excess out of their own pocket. From the stories I have read, it seems that many who were prosecuted had tried that but ran out of money. Many more must have successfully dealt with the "shortfall" and are surely due their own restitution now? Every transaction the system touched should be inspected, because I'm sure there were plenty of sub-postmasters who took the "shortfall" at face value and paid up without even thinking about questioning the system.
thesalsabear 3y ago
While this is a question of what the court should presume, should we ask what does a software developer presume or doesn’t: A software developer would presume proper working of software if it is being used widely. I would presume JavaSDK or pandas library to be working properly, inspite of the fact that pandas that 3.5K bugs file on GitHub issues. But if I am thinking about integration a less known library or using a less known software I am less confident about its working.
- nine_k 3y ago
  Given enough eyeballs, all bugs are shallow. As such, they are complained about by thousands of people, for years. Less ironically, one can expect widely used features of widely used software to have been tested in realistic conditions so whatever bugs there might be are at least known. But it takes an insane amount of testing to cover the phase space even of a moderately complex piece of software: look how large is the test suite of SQLite. Apparently the only working approach to less buggy software is to compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. This is the philosophy behind the original Unix. This is the philosophy behind functional programming. This is partly the philosophy behind actor systems (even though reasoning about async interactions is harder). I expect to see more and more of these in practice.
  - makeitdouble 3y ago
    Yes. Bugs are there, but either they're shallow, well known and worked around, or have low impact for the majority of people using them. The epitome of that was utf-8 support for a long time: there were a huge number of bugs, but alaphabet based languages would be mostly ok, and most devs would know it was a mess and work around that as much as they can. To me one should expect any software to be broken in numerous ways, and take the time to check what happens around the area that are critical to them. > compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. That looks awfully close to the micro-service approach as well. The approach can be good, but it still gives me pause.
zubairq 3y ago
After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.
- HPsquared 3y ago
  Occam's razor can be double-edged sometimes.
- SanderNL 3y ago
  It is too much work. It’s way too complex.
- bmitc 3y ago
  > I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
  - zubairq 3y ago
    That makes sense, yes we have trained ourselves to see things that don’t work as working , like frogs in boiling water!
    - psychoslave 3y ago
      While the metaphore is great, it's actually stand on a myth. That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits. https://en.m.wikipedia.org/wiki/Boiling_frog https://en.m.wikipedia.org/wiki/Boiling_frog
kazinator 3y ago
Every prosecutor in possession computer evidence of a crime having to prove the software correct is a nonstarter. Assuming it's right, and proving it buggy, is the right principle.
- zubairq 3y ago
  I’m quite interested to learn and understand the line of reasoning for this comment. Care to elaborate?
  - jpc0 3y ago
    If bugs were easy to find and solve there wouldn't be an entire highly paid industry to do it. Add to that malicious actors actively trying to work against systems (bank/post office/medical system etc) and it becomes pretty hard to prove that both the software and the data referenced by the software is in fact correct. Now think about the last time a critical piece of data was recorded on paper with a pen/analogue typewriter and both parties got a physical copy of it. Last time you made a cash withdrawal did you have to go to a teller at the bank, fill in paperwork, have two copies of that paperwork exist and then that paperwork was digitised or was the entire process digital beginning to end? Think about how you, if you were an expert witness, would prove that an transaction made on a banking website was not a computer fault. Do you show a code audit which meets some standards? Do you have the bank's CISO explain all of the security measures? INAL but I'm reasonably sure the defendant only has to prove that a system with the same security and audit trail has in fact been faulty and your evidence goes out the door or at least the validity of it decreases.
    - zubairq 3y ago
      Thanks for the detailed answer. I think however that the real reason that there is a big industry to find and solve bugs is because of decision made when building and designing the software in the first place. For example if software was built in a robust way with care taken to refactor code along the way then maintenance and bug fixing would probably be a lot easier and may not need an entire industry to support it
chaps 3y ago
Similar happens with shotspotter in the context of early stages of criminal court proceedings. https://law.stanford.edu/publications/soundthinkings-black-box-gunshot-detection-method-untested-and-unvetted-tech-flourishes-in-the-criminal-justice-system/ https://law.stanford.edu/publications/soundthinkings-black-b...
denton-scratch 3y ago
A big part of the problem with these prosecutions is that the prosecuting authority is the Post Office itself. Until this case came along, I had no idea that the Post Office had these powers. So effectively, the Post Office was acting as both complainant and prosecuting authority. "The Post Office Investigation Branch (IB) is the oldest recognised criminal investigations force in the world. For more than 335 years it has worked to detect offences against the post and prosecute the perpetrators of these crimes." https://www.postalmuseum.org/blog/the-post-office-investigation-branch/ https://www.postalmuseum.org/blog/the-post-office-investigat...
- masfuerte 3y ago
  Anyone can bring a prosecution in England. In theory it provides a way to hold the powerful to account but it's very expensive so almost all private prosecutions are actually by corporations.
  - denton-scratch 3y ago
    It's true that anyone can bring a prosecution. It's also true that the Crown Prosecution Service can take over a private prosecution, and then drop the case. Obviously, the CPS should have done this in the Post Office case. During the relevant period, the CPS was underresourced and overloaded (it still is now).
stevesimmons 3y ago
David Allen Green is a great commentator on law in the UK. During the Brexit period, he was a contributing editor to the Financial Times, with similarly insightful analyses of how Boris Johnson et al were misusing the government's legal powers. Anyway, David also posted this to his non-gated "The Law and Policy Blog" at [1]. [1] https://davidallengreen.com/2023/09/computer-says-guilty-an-introduction-to-the-evidential-presumption-that-computers-are-operating-correctly/ https://davidallengreen.com/2023/09/computer-says-guilty-an-... There are more comments there, including this one from me (written for his blog's legal/non-technical audience, not for the HN crowd!): "There has been a big shift in the practice of computer programming in the last 25 years. In the “old days”, software was delivered on floppy disks or tape, and pretty much had to be correct first time. Nowadays, everything is on the web, “agile” means the functionality gets delivered incrementally in 2-week sprints. It is regarded as ok to leave out the difficult cases or ship bugs, provided they don’t affect too many users too much of the time. In other words, the software is never really finished. With that in mind, it becomes simple to rebut the presumption of correctness: if the company’s tech organisation describes itself as “agile”, using “scrum” etc, its software – by design – contains bugs and omissions, both known and unknown. Fortunately, modern development practices means evidence for this will be in the dev tickets in GitHub, Azure DevOps or whatever other systems their dev teams use."
- bostik 3y ago
  Thank you, I have been looking for a resource like this, one that would come with an RSS feed. Now I have one less thing to hunt around for.
- _dain_ 3y ago
  I had a visceral negative reaction to the idea that computers should be presumed to be working correctly, but I was having trouble articulating in precise words what was wrong with it. I mean, obviously we get work done with use computers every day and they seem to work alright. A commenter at that link does a good job: >To treat the operation of an extensive software system as being a mechanical device is as gross a category error as treating an airliner as being merely a collection of nuts and bolts. In either case we can satisfy ourselves that the elementary parts are performing within specification without having gained any confidence that the complete assemblage will function properly under all conditions to which it will be reasonably exposed; indeed we won’t even be confident that we understand what it was exposed to. Even showing that it has passed a collection of tests isn’t adequate: one needs to see what unfixed issues are being registered against it and how they are being resolved (fixed? Swept under the carpet to keep the numbers below since arbitrary threshold?) The opposing side in case hinging upon accuracy of computer records should have the right to access the issue database, and absence of a well-run system or a large number of issues in it (or evidence that issues have been “scrubbed”) should be taken as a strong argument that the computer records are unreliable. >Much could be learned by the software industry from the culture of aviation safety, where even minor incidents are followed up by detailed investigations intentionally run in a non-judgemental way and the findings distilled into public reports and recommendations. Similarly, one of the best legal developments in recent years has been the increasing tendency to publish detailed judgements.
didntcheck 3y ago
The last time I looked into the Post Office scandal, the impression I got was that "we just had too much faith in the IT system!" was a convenient story, but the high-ups knew full well about the problems, and full well that they were persecuting innocent people, but did it anyway to cover up their own failings. If so, people need to see substantial jail terms for this. Perverting the course of justice is no misdemeanor. There's even allegations that Paula Vennells knowingly lied to Parliament
- dazzawazza 3y ago
  There is substantial evidence that the people in power in the Post Office Board dismissed the evidence from the sub post masters because they were, more often than not of Asian descent (Indian, Bangladeshi, Pakistani,...). So the Post Office Board had a mental block to joining the dots because they were racist. When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they doubled down because they lack any sense of morality. So while it is a sad technical tale it is also an indictment of British Board room culture. The whole sorry affair has been covered by Private Eye magazine for over ten years.
  - switch007 3y ago
    Private Eye is an absolute gem. It’s quite quirky and jokey but it’s the real deal if you want to know how this country is really run.
  - envsubst 3y ago
    > they doubled down because they lack any sense of morality. I don't believe you know what their internal mental states are.
    - atoav 3y ago
      Most internal mental states that have any impact on the actions of an individual are leaky. That is because they affect actual real world actions. So given a set of actions you can infer a likely set of mental states — this is also done in the court room. Can you always infer with 100% accuracy and with every colorful detail fleshed out? No. Do you typically need to? No.
ozim 3y ago
I can see why they changed it. I expect in last 3 decades amount of computer evidence grew up to amounts where there is no practical way to prove computer was operating correctly. If I present a digital photo as evidence it would be dismissed by defense lawyers to easily. Then basically any digital evidence would be dismissed the same way. I can see how it was easy to manage in the 80’s and not anymore in 00’s.
vjk800 3y ago
The presumption becomes problematic mostly when the operator of the computer is the same party who is being persecuted. 1) It is difficult or impossible for any outsider to inspect the workings of the computer, 2) the operator has every incentive to fake the outputs of the computer or hide the errors. Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs of the bittorrent swarm as an evidence in courts. Faking of this data would have been both very easy and pretty much impossible to detect by an outsider. Also they had a massive financial incentive to do so. The fact that this "evidence" was considered in the courts seemed outrageous to me.
- agentgumshoe 3y ago
  Ah the Prenda Law saga, such memories
taneq 3y ago
Having seen this in action, it’s fascinating how fast humans flip from “this weird new computer is automatically wrong” to “computer is wrong? Haha nice try.”
layer8 3y ago
This will also become interesting in the context of AI, because it’s unclear what it means for AI to “work correctly”.
momirlan 3y ago
Computers, as in hardware, work as designed. Are there bugs in the software run on them ? Then the title should be about "bugs" or logic errors, not "computers"
- brokenmachine 3y ago
  That's not even close to true. Hardware regularly has bugs ("errata"), as other commenters here have noted. Also hardware and software are completely codependent nowadays. Your trust in hardware is hopelessly misplaced.
civilized 3y ago
Shouldn't the defense attorney consider the possibility that computers are not working correctly? Is there... not a defense attorney?
jonjacky 3y ago
This sort of thing has been going on since ... well, forever. I was reminded of this post [1] by Phil Agre in the RISKS Digest from 1988: "I've noticed that responses to attempts to complain about uncorrected technical problems tend to take certain recurring forms ... What has happened? Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. So they make noise about it. The response? 1. Misplaced accusations of uncooperativeness. ... 2. Condescending lectures about how `incredibly complicated' the systems in question are ... " etc. See the post itself for much more: 1. http://catless.ncl.ac.uk/Risks/7.09.html#subj1 http://catless.ncl.ac.uk/Risks/7.09.html#subj1
dboreham 3y ago
The Windows task bar auto-hide feature has been broken (spontaneously decides to quit hiding, but doesn't adjust the desktop viewport accordingly, so the bottom of any app window is obscured by the errant taskbar), since its first beta in 1994.

285 points by ColinWright 3y ago

As a developer I never assume computer systems are working correctly. Quite the opposite in fact. If something isn't on the 'happy path' I assume something about the systems involved isn't working correctly and go check the facts for myself.

deleted 3y ago

[deleted]

chrisjj 3y ago

https://archive.ph/wip/AJl8i https://archive.ph/wip/AJl8i for permanance.

SoftTalker 3y ago

My usual presumption is that the computers are not working correctly, as I've seen way too many cases where they in fact are not working correctly. I don't even believe Google Maps when I'm driving, I always sanity check the route and double check street names/exit numbers when it tells me to turn. I've been burned too many times.

quickthrower2 3y ago

The title is maybe the opposite of clickbait. The article talks about the Post Office scandal. That is a big topic to take on!

topspin 3y ago

"Code is law." -Lawrence Lessig The UK apparently made that explicit. Doubtless others have as well: if we could forego the flag waving and whataboutery, That Would Be Great (tm).

User23 3y ago
Lawrence Lessig's greatest claim to fame is handing Disney and friends unlimited copyright before the Supreme Court. I'd hold up Judge Alsup if I wanted to give an example of a tech saavy legal practitioner.
- morelisp 3y ago
  While it's increasingly clear that Lessig's legacy will be as (to borrow a phrase) the internet's own failson, he is far more famous today for CC than Eldred v. Ashcroft.

phire 3y ago

As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.

TedDoesntTalk 3y ago
It does happen to Hardware. Here are two older examples: “The Intel Pentium series of CPUs had two well-known bugs discovered after it was brought to market, the FDIV bug affecting floating point division which resulted in a recall in 1994, and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted.” https://en.m.wikipedia.org/wiki/Hardware_bug https://en.m.wikipedia.org/wiki/Hardware_bug
- thaumasiotes 3y ago
  > and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted You reminded me of the Things I Won't Work With piece that observed in passing that the behavior of dioxygen difluoride was surprisingly well described by its chemical formula. ( https://www.science.org/content/blog-post/things-i-won-t-work-dioxygen-difluoride https://www.science.org/content/blog-post/things-i-won-t-wor... )
- deleted 3y ago
  [deleted]
tivert 3y ago
> As a programmer, it's my experience that the computer itself usually works correctly. It's almost never a hardware bug. > But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to. That's not a distinction that's relevant to this article. By "computer" it means a system composed of hardware and software (and which is almost always what "computer" is used to mean): > The presumption that computers are presumed to be operating correctly, unless there is evidence to the contrary is what lawyers call “a presumption of evidence”. > This means that a court can be satisfied that a relevant fact can be established just by computer records, unless there is evidence that the computer is not working properly. > And so when the computer record shows, for instance, a financial shortfall by postmaster or postmistress, the court will accept that as evidence of an actual shortfall - unless the defendant can show that the computer was not operating correctly. > In short, when the computer record is the essence of a prosecution case: computer says guilty.

jfoutz 3y ago

How does that work? I'm not a lawyer. I'm a lazy programmer. So, does the defense get to subpoena the logs, and depose a programmer to explain why some random request returned a 502? Does the defense get, like, red team access? Although I'm a lazy programmer, I do think a lot about failure modes. If my freedom is on the line, I can probably think up a few (orders of magnitude) test cases. Happy to provide a shell script or python or golang binary to check. Heck, if it comes down to it I'll provide a stack of fortran cards. Do I get to see the source? That "few" is probably going to move a decimal place or 5. I get volume is not reasonable in the eyes of the court. Scale is how you(I) find problems, but whatever. Gimmie source, gimme access, I doubt I could beat Knuth, but with my freedom on the line I could give him a run for his money. How do CVE's count? If I find evidence of log4j do I auto win? This makes me think of the Jepsen tire fire talk. The system probably mostly right. But there's no way it's correct. What's a "fair" level of access to prove that?

qingcharles 3y ago
In the US at least it is hard to bring source code into the trial, even though you technically have a right. I remember one case where the FBI installed malware via a browser exploit on thousands of computers to get their IPs and one defendant tried to get the source code but the government managed to keep it secret.
- Terr_ 3y ago
  That sounds like one of the "Playpen" cases, for which the EFF has a writeup on the hacking tool. [0] Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service. [0] https://www.eff.org/pages/playpen-cases-frequently-asked-questions https://www.eff.org/pages/playpen-cases-frequently-asked-que... [1] https://www.eff.org/deeplinks/2021/02/federal-court-agrees-prosecutors-cant-keep-forensic-evidence-secret-defendants https://www.eff.org/deeplinks/2021/02/federal-court-agrees-p...
  - Roark66 3y ago
    >Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service. I'm glad at least in this instance the defendant got access to the source code. He was being helped in pursuing this by EFF so I assume there were volunteers that helped with the analysis. But what if your case is lower profile? What if you can't afford to hire 10 programmers for 3 months to write test cases? Also I wonder how such a request would go down in a UK court.

genr8 3y ago

a 1984 law

di4na 3y ago
Nope, a 1999 one. Sadly.

suzzer99 3y ago

This is the first I've heard of this scandal. I'm curious what the major problems were with the software that caused the discrepancies. Dropped connections? Wikipedia didn't go into much detail with that. Ok I found this: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...

di4na 3y ago
It is one of the biggest scandal of our time related to software. I ... want to recommend learning more about it but only do so with a strong stomach. Numerous people that rabbithole into this case end up running to a nearby bucket. It is... not pretty.
- zelos 3y ago
  I agree, I find it hard to read about because it makes me so angry. Four people driven to suicide, hundreds of lives ruined. It took them so long to hold all the inquiries and decide on compensation that dozens have died before being acquitted or receiving compensation.
  - di4na 3y ago
    And that is only the one we know of. There are thousands of cases that are hidden in the bowels of the post office court system Oh and the compensation are uh... let say hard to get the money for.
    - andrewaylett 3y ago
      And I've yet to hear mention of the people who weren't prosecuted because they paid the excess out of their own pocket. From the stories I have read, it seems that many who were prosecuted had tried that but ran out of money. Many more must have successfully dealt with the "shortfall" and are surely due their own restitution now? Every transaction the system touched should be inspected, because I'm sure there were plenty of sub-postmasters who took the "shortfall" at face value and paid up without even thinking about questioning the system.

thesalsabear 3y ago

While this is a question of what the court should presume, should we ask what does a software developer presume or doesn’t: A software developer would presume proper working of software if it is being used widely. I would presume JavaSDK or pandas library to be working properly, inspite of the fact that pandas that 3.5K bugs file on GitHub issues. But if I am thinking about integration a less known library or using a less known software I am less confident about its working.

nine_k 3y ago
Given enough eyeballs, all bugs are shallow. As such, they are complained about by thousands of people, for years. Less ironically, one can expect widely used features of widely used software to have been tested in realistic conditions so whatever bugs there might be are at least known. But it takes an insane amount of testing to cover the phase space even of a moderately complex piece of software: look how large is the test suite of SQLite. Apparently the only working approach to less buggy software is to compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. This is the philosophy behind the original Unix. This is the philosophy behind functional programming. This is partly the philosophy behind actor systems (even though reasoning about async interactions is harder). I expect to see more and more of these in practice.
- makeitdouble 3y ago
  Yes. Bugs are there, but either they're shallow, well known and worked around, or have low impact for the majority of people using them. The epitome of that was utf-8 support for a long time: there were a huge number of bugs, but alaphabet based languages would be mostly ok, and most devs would know it was a mess and work around that as much as they can. To me one should expect any software to be broken in numerous ways, and take the time to check what happens around the area that are critical to them. > compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. That looks awfully close to the micro-service approach as well. The approach can be good, but it still gives me pause.

zubairq 3y ago

After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.

HPsquared 3y ago
Occam's razor can be double-edged sometimes.
SanderNL 3y ago
It is too much work. It’s way too complex.
bmitc 3y ago
> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
- zubairq 3y ago
  That makes sense, yes we have trained ourselves to see things that don’t work as working , like frogs in boiling water!
  - psychoslave 3y ago
    While the metaphore is great, it's actually stand on a myth. That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits. https://en.m.wikipedia.org/wiki/Boiling_frog https://en.m.wikipedia.org/wiki/Boiling_frog

kazinator 3y ago

Every prosecutor in possession computer evidence of a crime having to prove the software correct is a nonstarter. Assuming it's right, and proving it buggy, is the right principle.

zubairq 3y ago
I’m quite interested to learn and understand the line of reasoning for this comment. Care to elaborate?
- jpc0 3y ago
  If bugs were easy to find and solve there wouldn't be an entire highly paid industry to do it. Add to that malicious actors actively trying to work against systems (bank/post office/medical system etc) and it becomes pretty hard to prove that both the software and the data referenced by the software is in fact correct. Now think about the last time a critical piece of data was recorded on paper with a pen/analogue typewriter and both parties got a physical copy of it. Last time you made a cash withdrawal did you have to go to a teller at the bank, fill in paperwork, have two copies of that paperwork exist and then that paperwork was digitised or was the entire process digital beginning to end? Think about how you, if you were an expert witness, would prove that an transaction made on a banking website was not a computer fault. Do you show a code audit which meets some standards? Do you have the bank's CISO explain all of the security measures? INAL but I'm reasonably sure the defendant only has to prove that a system with the same security and audit trail has in fact been faulty and your evidence goes out the door or at least the validity of it decreases.
  - zubairq 3y ago
    Thanks for the detailed answer. I think however that the real reason that there is a big industry to find and solve bugs is because of decision made when building and designing the software in the first place. For example if software was built in a robust way with care taken to refactor code along the way then maintenance and bug fixing would probably be a lot easier and may not need an entire industry to support it

chaps 3y ago

Similar happens with shotspotter in the context of early stages of criminal court proceedings. https://law.stanford.edu/publications/soundthinkings-black-box-gunshot-detection-method-untested-and-unvetted-tech-flourishes-in-the-criminal-justice-system/ https://law.stanford.edu/publications/soundthinkings-black-b...

denton-scratch 3y ago

A big part of the problem with these prosecutions is that the prosecuting authority is the Post Office itself. Until this case came along, I had no idea that the Post Office had these powers. So effectively, the Post Office was acting as both complainant and prosecuting authority. "The Post Office Investigation Branch (IB) is the oldest recognised criminal investigations force in the world. For more than 335 years it has worked to detect offences against the post and prosecute the perpetrators of these crimes." https://www.postalmuseum.org/blog/the-post-office-investigation-branch/ https://www.postalmuseum.org/blog/the-post-office-investigat...

masfuerte 3y ago
Anyone can bring a prosecution in England. In theory it provides a way to hold the powerful to account but it's very expensive so almost all private prosecutions are actually by corporations.
- denton-scratch 3y ago
  It's true that anyone can bring a prosecution. It's also true that the Crown Prosecution Service can take over a private prosecution, and then drop the case. Obviously, the CPS should have done this in the Post Office case. During the relevant period, the CPS was underresourced and overloaded (it still is now).

stevesimmons 3y ago

David Allen Green is a great commentator on law in the UK. During the Brexit period, he was a contributing editor to the Financial Times, with similarly insightful analyses of how Boris Johnson et al were misusing the government's legal powers. Anyway, David also posted this to his non-gated "The Law and Policy Blog" at [1]. [1] https://davidallengreen.com/2023/09/computer-says-guilty-an-introduction-to-the-evidential-presumption-that-computers-are-operating-correctly/ https://davidallengreen.com/2023/09/computer-says-guilty-an-... There are more comments there, including this one from me (written for his blog's legal/non-technical audience, not for the HN crowd!): "There has been a big shift in the practice of computer programming in the last 25 years. In the “old days”, software was delivered on floppy disks or tape, and pretty much had to be correct first time. Nowadays, everything is on the web, “agile” means the functionality gets delivered incrementally in 2-week sprints. It is regarded as ok to leave out the difficult cases or ship bugs, provided they don’t affect too many users too much of the time. In other words, the software is never really finished. With that in mind, it becomes simple to rebut the presumption of correctness: if the company’s tech organisation describes itself as “agile”, using “scrum” etc, its software – by design – contains bugs and omissions, both known and unknown. Fortunately, modern development practices means evidence for this will be in the dev tickets in GitHub, Azure DevOps or whatever other systems their dev teams use."

bostik 3y ago
Thank you, I have been looking for a resource like this, one that would come with an RSS feed. Now I have one less thing to hunt around for.
_dain_ 3y ago
I had a visceral negative reaction to the idea that computers should be presumed to be working correctly, but I was having trouble articulating in precise words what was wrong with it. I mean, obviously we get work done with use computers every day and they seem to work alright. A commenter at that link does a good job: >To treat the operation of an extensive software system as being a mechanical device is as gross a category error as treating an airliner as being merely a collection of nuts and bolts. In either case we can satisfy ourselves that the elementary parts are performing within specification without having gained any confidence that the complete assemblage will function properly under all conditions to which it will be reasonably exposed; indeed we won’t even be confident that we understand what it was exposed to. Even showing that it has passed a collection of tests isn’t adequate: one needs to see what unfixed issues are being registered against it and how they are being resolved (fixed? Swept under the carpet to keep the numbers below since arbitrary threshold?) The opposing side in case hinging upon accuracy of computer records should have the right to access the issue database, and absence of a well-run system or a large number of issues in it (or evidence that issues have been “scrubbed”) should be taken as a strong argument that the computer records are unreliable. >Much could be learned by the software industry from the culture of aviation safety, where even minor incidents are followed up by detailed investigations intentionally run in a non-judgemental way and the findings distilled into public reports and recommendations. Similarly, one of the best legal developments in recent years has been the increasing tendency to publish detailed judgements.

didntcheck 3y ago

The last time I looked into the Post Office scandal, the impression I got was that "we just had too much faith in the IT system!" was a convenient story, but the high-ups knew full well about the problems, and full well that they were persecuting innocent people, but did it anyway to cover up their own failings. If so, people need to see substantial jail terms for this. Perverting the course of justice is no misdemeanor. There's even allegations that Paula Vennells knowingly lied to Parliament

dazzawazza 3y ago
There is substantial evidence that the people in power in the Post Office Board dismissed the evidence from the sub post masters because they were, more often than not of Asian descent (Indian, Bangladeshi, Pakistani,...). So the Post Office Board had a mental block to joining the dots because they were racist. When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they doubled down because they lack any sense of morality. So while it is a sad technical tale it is also an indictment of British Board room culture. The whole sorry affair has been covered by Private Eye magazine for over ten years.
- switch007 3y ago
  Private Eye is an absolute gem. It’s quite quirky and jokey but it’s the real deal if you want to know how this country is really run.
- envsubst 3y ago
  > they doubled down because they lack any sense of morality. I don't believe you know what their internal mental states are.
  - atoav 3y ago
    Most internal mental states that have any impact on the actions of an individual are leaky. That is because they affect actual real world actions. So given a set of actions you can infer a likely set of mental states — this is also done in the court room. Can you always infer with 100% accuracy and with every colorful detail fleshed out? No. Do you typically need to? No.

ozim 3y ago

I can see why they changed it. I expect in last 3 decades amount of computer evidence grew up to amounts where there is no practical way to prove computer was operating correctly. If I present a digital photo as evidence it would be dismissed by defense lawyers to easily. Then basically any digital evidence would be dismissed the same way. I can see how it was easy to manage in the 80’s and not anymore in 00’s.

vjk800 3y ago

The presumption becomes problematic mostly when the operator of the computer is the same party who is being persecuted. 1) It is difficult or impossible for any outsider to inspect the workings of the computer, 2) the operator has every incentive to fake the outputs of the computer or hide the errors. Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs of the bittorrent swarm as an evidence in courts. Faking of this data would have been both very easy and pretty much impossible to detect by an outsider. Also they had a massive financial incentive to do so. The fact that this "evidence" was considered in the courts seemed outrageous to me.

agentgumshoe 3y ago
Ah the Prenda Law saga, such memories

taneq 3y ago

Having seen this in action, it’s fascinating how fast humans flip from “this weird new computer is automatically wrong” to “computer is wrong? Haha nice try.”

layer8 3y ago

This will also become interesting in the context of AI, because it’s unclear what it means for AI to “work correctly”.

momirlan 3y ago

Computers, as in hardware, work as designed. Are there bugs in the software run on them ? Then the title should be about "bugs" or logic errors, not "computers"

brokenmachine 3y ago
That's not even close to true. Hardware regularly has bugs ("errata"), as other commenters here have noted. Also hardware and software are completely codependent nowadays. Your trust in hardware is hopelessly misplaced.

civilized 3y ago

Shouldn't the defense attorney consider the possibility that computers are not working correctly? Is there... not a defense attorney?

jonjacky 3y ago

This sort of thing has been going on since ... well, forever. I was reminded of this post [1] by Phil Agre in the RISKS Digest from 1988: "I've noticed that responses to attempts to complain about uncorrected technical problems tend to take certain recurring forms ... What has happened? Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. So they make noise about it. The response? 1. Misplaced accusations of uncooperativeness. ... 2. Condescending lectures about how `incredibly complicated' the systems in question are ... " etc. See the post itself for much more: 1. http://catless.ncl.ac.uk/Risks/7.09.html#subj1 http://catless.ncl.ac.uk/Risks/7.09.html#subj1

dboreham 3y ago

The Windows task bar auto-hide feature has been broken (spontaneously decides to quit hiding, but doesn't adjust the desktop viewport accordingly, so the bottom of any app window is obscured by the errant taskbar), since its first beta in 1994.