6 ms·

The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between

The fact that this misnomer of infallible computer systems was ever enshrined in law is pretty damning of the whole UK legal system and the relationship between technical people and law. Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%. Not least because it seems to go against the core concept of "innocent until proven guilty" that the whole legal system is meant to rest upon.

impossiblefork 2y ago
I find it really strange to have presumptions in general. If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another. If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.
- ndsipa_pomu 2y ago
  The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
- beardyw 2y ago
  I imagine it stems from something like: the clock was correct at 1pm and at 3pm but was it correct at 2pm?
Oarch 2y ago
Maybe it was a presumption that was legally convenient... until it wasn't and reality struck?
- jack_riminton 2y ago
  Yeah exactly, until the miscarriage of justice was too big to ignore
oarsinsync 2y ago
> The fact that this misnomer of infallible computer systems was ever enshrined in law Is this actually a fact, or a fact taken to it’s logical conclusion to presume a new “fact”? The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Instead, precedents got set over time by enterprising lawyers, but setting a precedent when it’s convenient is not the same thing as writing a law. When I see mechanical systems, I think of something like an abacus. I’ve never used one, but I suspect the abacus itself is infallible. It’s open, it’s transparent, it is easily auditable, and the same inputs will always produce the same outputs. There is no black box translation occurring, like occurs with computer software.
- exe34 2y ago
  > setting a precedent when it’s convenient is not the same thing as writing a law In the UK, they're pretty much the same thing. You need a new case or a statute law to overturn a precedent.
- graemep 2y ago
  I think historically the law was introduced to make it harder for people to contest things like speed camera based speeding tickets.
- disgruntledphd2 2y ago
  > The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Apparently the law was introduced along with speed cameras, as they were continually being challenged in court.
- M2Ys4U 2y ago
  >The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. It's worse than that. The law was fixed in 1984[0] and then the fix was intentionally reversed in 1999.[1] [0] https://www.legislation.gov.uk/ukpga/1984/60/section/69/1991-02-01 https://www.legislation.gov.uk/ukpga/1984/60/section/69/1991... [1] https://www.legislation.gov.uk/ukpga/1999/23/section/60 https://www.legislation.gov.uk/ukpga/1999/23/section/60
michaelt 2y ago
I know that computers aren't infallible. But I also know that traditionally, tills and bank accounts are pretty reliable. Sure, the store might charge you the wrong amount if the price label on the shelf and the PC don't match up, because by law the label on the shelf is the source of truth. But other than that? If the till says my purchases add up to £23.45, and after making the purchase my bank account has a balance of £345.67? I don't validate the arithmetic. How much money is in my bank account? Pretty much the amount of money the bank's computer says is in my bank account, modulo any funnyness like pending transactions and cheques that fail to clear. The bank doesn't keep a shoebox of cash in their vault with my name on it for us to reconcile against.
mattlondon 2y ago
I suspect that this originated from the era well before computers were a common part of life and intended to relate to purely mechanical things, and the statutes were just applied to computers when they started to appear in court cases. So e.g. a mechanical time clock or mechanical scales etc were probably the sort of thing that was the target of the original acts. The assumption is they are working correctly if they appear fine. This makes sense for basic mechanical things, and there is no point arguing that actually the scales that weighed how much the truck weighed were wrong/defective only that one time Defendant X used it and never again afterwards, and it was not in fact due to Defendant X being negligent that the bridge collapsed due to an overweight vehicle etc As we know, computers are a different level of complexity. Being wrong randomly for one off things is very possible.
- rswail 2y ago
  Sorry? There have been departments of Weights and Measures for literaly millenia. They have required certification of those sorts of items with regular inspections and anti-tampering seals. So yes, you can challenge evidence from mechanical scales, if they haven't been properly inspected and certified at the required intervals under the regulations.
  - mattlondon 2y ago
    Exactly that is my point - so for the example of some mechanical scales, if they have all been properly inspected and certified, then you can expect them to be reliable and accurate. So in a case, if there is evidence against someone that relies on the scales doing their job properly, and those scales have been inspected and certified, then you probably do not have a valid argument to say "ah yeah but the scales the defendant used might have not been working properly when they used them!". I.e. the accepted assumption is that the scales work correctly. Up until recently the same sort of assumption was given to computer systems from what I can tell. This is how we got into the Post Office scandal situation where people implicitly trusted that the computer was doing the right thing.
    - rswail 2y ago
Hilift 2y ago
This is particularly true of accounting systems. Anyone who has worked in a similar environment with thousands of users on a distributed ledger.
Dalewyn 2y ago
>Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%. I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable. I know the computers powering the stuff my life depends on are perfectly not accurate and reliable. Put another way: The microwave oven or coffee maker in my kitchen? Yeah, the 'pooters in them are working perfectly. The mainframes jackhammering away at the Automated Clearing House? My money will get through the banking system perfectly eventually some day. The jetliner or my car I'm about to get in? Dude, that thing better have dozens of computers acting in redundancy because that shit ain't working. I wonder if there's a law stating that the reliability of a computer is inverse to the value of the workload.
- bostik 2y ago
  > I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable. Consider yourself lucky that your use cases are all on the happy path. There are entire categories of bugs and inconsistencies where Excel's behaviour is known to be wrong, but which can't be fixed because the rest of the ecosystem depends on those same errors to manifest in the same ways. For example - formulas with cycles have an upper bound as to how many times they are allowed to cycle. If you happen to hit the ceiling before your values converge, you will be left with the values calculated on the last iteration.
rich_sasha 2y ago
I think that take is slightly missing the point. Notwithstanding the horror that is the Horizon fallout, the legal rule is much narrower than what is bandied around. It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. Or in other words, you need evidence of incorrectness if you're going to claim it. Now with the Horizon scandal, there was very clearly plenty of evidence. The extraordinary number of mismatched books. Some cases of physical records not matching electronic (these were the few guys that got away). The issue was that the Post Office investigators lied about the evidence, buried it with intimidation, legal threats and NDAs. The law may or may not be bad as it stands but AFAIU this is like blaming a "computer bug" for the Boeing 737 Max crashes. It wasn't - it was human willful errors executed by imperfect code.
- MichaelDickens 2y ago
  > It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. According to the standard legal practice of innocent-until-proven-guilty, you can, in fact, say that and expect the other side to prove otherwise. So this ruling violates one of the most fundamental principles of criminal law.
  - rich_sasha 2y ago
    Right - it unquestionably is a custom rule. But it's not as broad as it is made out, not is it the fundamental problem here. The Post Office knew the system is faulty, hid that evidence and lied in court under oath about it. As to lack of evidence... It's not so simple. A popular UK insurance hack is when people overtake you on a motorway and slam the brakes, to make a claim on your insurance. Unless you present evidence this was done to you, like dashcam footage, you are presumed to be responsible. You don't need the scammer to prove they didn't crash into you. I'm sure the US has similar mechanisms, where fault is presumed, like if a car hits a pedestrian. This is similar. Computer systems are presumed to be correct unless evidence is presented. Maybe it's a bad rule, but it's not the horrendous dystopian catastrophe it is declared to be everywhere.
  - xxpor 2y ago

impossiblefork 2y ago

I find it really strange to have presumptions in general. If we're in a court and there hasn't been a decision yet, we're there because we're dealing with some kind of complicated edge case where one person has a strong argument for one thing and another for another. If one then decides to bring some problem up-- whether with how evidence is being judged or anything else, there can be no justification for ignoring him. This is why I like free evidence evaluation in Swedish courts and the absence of rigid precedent. Every question must then actually be dealt with. We do have this kind of idiotic rulemaking in other parts of our legal system though, so we're not fully free from it.

ndsipa_pomu 2y ago
The problem is allocating resources if you want to deal with every challenge to evidence provided by a computerised system. I think it was initially a problem with people challenging speed camera results.
beardyw 2y ago
I imagine it stems from something like: the clock was correct at 1pm and at 3pm but was it correct at 2pm?

Oarch 2y ago

Maybe it was a presumption that was legally convenient... until it wasn't and reality struck?

jack_riminton 2y ago
Yeah exactly, until the miscarriage of justice was too big to ignore

oarsinsync 2y ago

> The fact that this misnomer of infallible computer systems was ever enshrined in law Is this actually a fact, or a fact taken to it’s logical conclusion to presume a new “fact”? The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Instead, precedents got set over time by enterprising lawyers, but setting a precedent when it’s convenient is not the same thing as writing a law. When I see mechanical systems, I think of something like an abacus. I’ve never used one, but I suspect the abacus itself is infallible. It’s open, it’s transparent, it is easily auditable, and the same inputs will always produce the same outputs. There is no black box translation occurring, like occurs with computer software.

exe34 2y ago
> setting a precedent when it’s convenient is not the same thing as writing a law In the UK, they're pretty much the same thing. You need a new case or a statute law to overturn a precedent.
graemep 2y ago
I think historically the law was introduced to make it harder for people to contest things like speed camera based speeding tickets.
disgruntledphd2 2y ago
> The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. Apparently the law was introduced along with speed cameras, as they were continually being challenged in court.
M2Ys4U 2y ago
>The article cites “mechanical systems” as being infallible, and reading that language, it reads to me as some archaic legislation that never got updated for computer software. It's worse than that. The law was fixed in 1984[0] and then the fix was intentionally reversed in 1999.[1] [0] https://www.legislation.gov.uk/ukpga/1984/60/section/69/1991-02-01 https://www.legislation.gov.uk/ukpga/1984/60/section/69/1991... [1] https://www.legislation.gov.uk/ukpga/1999/23/section/60 https://www.legislation.gov.uk/ukpga/1999/23/section/60

michaelt 2y ago

I know that computers aren't infallible. But I also know that traditionally, tills and bank accounts are pretty reliable. Sure, the store might charge you the wrong amount if the price label on the shelf and the PC don't match up, because by law the label on the shelf is the source of truth. But other than that? If the till says my purchases add up to £23.45, and after making the purchase my bank account has a balance of £345.67? I don't validate the arithmetic. How much money is in my bank account? Pretty much the amount of money the bank's computer says is in my bank account, modulo any funnyness like pending transactions and cheques that fail to clear. The bank doesn't keep a shoebox of cash in their vault with my name on it for us to reconcile against.

mattlondon 2y ago

I suspect that this originated from the era well before computers were a common part of life and intended to relate to purely mechanical things, and the statutes were just applied to computers when they started to appear in court cases. So e.g. a mechanical time clock or mechanical scales etc were probably the sort of thing that was the target of the original acts. The assumption is they are working correctly if they appear fine. This makes sense for basic mechanical things, and there is no point arguing that actually the scales that weighed how much the truck weighed were wrong/defective only that one time Defendant X used it and never again afterwards, and it was not in fact due to Defendant X being negligent that the bridge collapsed due to an overweight vehicle etc As we know, computers are a different level of complexity. Being wrong randomly for one off things is very possible.

rswail 2y ago
Sorry? There have been departments of Weights and Measures for literaly millenia. They have required certification of those sorts of items with regular inspections and anti-tampering seals. So yes, you can challenge evidence from mechanical scales, if they haven't been properly inspected and certified at the required intervals under the regulations.
- mattlondon 2y ago
  Exactly that is my point - so for the example of some mechanical scales, if they have all been properly inspected and certified, then you can expect them to be reliable and accurate. So in a case, if there is evidence against someone that relies on the scales doing their job properly, and those scales have been inspected and certified, then you probably do not have a valid argument to say "ah yeah but the scales the defendant used might have not been working properly when they used them!". I.e. the accepted assumption is that the scales work correctly. Up until recently the same sort of assumption was given to computer systems from what I can tell. This is how we got into the Post Office scandal situation where people implicitly trusted that the computer was doing the right thing.
  - rswail 2y ago

Hilift 2y ago

This is particularly true of accounting systems. Anyone who has worked in a similar environment with thousands of users on a distributed ledger.

Dalewyn 2y ago

>Every person who has ever programmed a computer or worked in any complex system knows they can't be relied upon 100%. I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable. I know the computers powering the stuff my life depends on are perfectly not accurate and reliable. Put another way: The microwave oven or coffee maker in my kitchen? Yeah, the 'pooters in them are working perfectly. The mainframes jackhammering away at the Automated Clearing House? My money will get through the banking system perfectly eventually some day. The jetliner or my car I'm about to get in? Dude, that thing better have dozens of computers acting in redundancy because that shit ain't working. I wonder if there's a law stating that the reliability of a computer is inverse to the value of the workload.

bostik 2y ago
> I know I can rest assured the Excel spreadsheet for my monthly and annual budget is perfectly accurate and reliable. Consider yourself lucky that your use cases are all on the happy path. There are entire categories of bugs and inconsistencies where Excel's behaviour is known to be wrong, but which can't be fixed because the rest of the ecosystem depends on those same errors to manifest in the same ways. For example - formulas with cycles have an upper bound as to how many times they are allowed to cycle. If you happen to hit the ceiling before your values converge, you will be left with the values calculated on the last iteration.

rich_sasha 2y ago

I think that take is slightly missing the point. Notwithstanding the horror that is the Horizon fallout, the legal rule is much narrower than what is bandied around. It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. Or in other words, you need evidence of incorrectness if you're going to claim it. Now with the Horizon scandal, there was very clearly plenty of evidence. The extraordinary number of mismatched books. Some cases of physical records not matching electronic (these were the few guys that got away). The issue was that the Post Office investigators lied about the evidence, buried it with intimidation, legal threats and NDAs. The law may or may not be bad as it stands but AFAIU this is like blaming a "computer bug" for the Boeing 737 Max crashes. It wasn't - it was human willful errors executed by imperfect code.

MichaelDickens 2y ago
> It merely says you can't just say "computer got it wrong" and expect the other side to prove otherwise. According to the standard legal practice of innocent-until-proven-guilty, you can, in fact, say that and expect the other side to prove otherwise. So this ruling violates one of the most fundamental principles of criminal law.
- rich_sasha 2y ago
  Right - it unquestionably is a custom rule. But it's not as broad as it is made out, not is it the fundamental problem here. The Post Office knew the system is faulty, hid that evidence and lied in court under oath about it. As to lack of evidence... It's not so simple. A popular UK insurance hack is when people overtake you on a motorway and slam the brakes, to make a claim on your insurance. Unless you present evidence this was done to you, like dashcam footage, you are presumed to be responsible. You don't need the scammer to prove they didn't crash into you. I'm sure the US has similar mechanisms, where fault is presumed, like if a car hits a pedestrian. This is similar. Computer systems are presumed to be correct unless evidence is presented. Maybe it's a bad rule, but it's not the horrendous dystopian catastrophe it is declared to be everywhere.
- xxpor 2y ago