7 ms·

UK legal system assumes that computers don't have bugs

41 points by idw 6y ago

Meph504 6y ago
How is this not reasonable? "in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers."
- simonh 6y ago
  Er, have you ever used a computer system, or written a computer program? Their position was that you don’t need to check the calculations, don’t need to show why it produced the results it did, you just assume it’s always correct. I work in the finance industry myself and have supported financial applications in banks and exchanges and the way these cases were handled is unbelievable. We’d never make assumptions about our systems like this, it was grossly negligent and incompetent and the courts need very clear guidance on how to do better.
  - phaemon 6y ago
    Yes, do you know how they work? My computer is performing billions of calculations and operating on billions of results. Of course I don't check every one and of course I assume they're correct absent evidence to the contrary. Frankly this is so obvious that I'd suggest you lay off the beer while posting to HN, but this time of year that's a daft suggestion, so I suggest you lay off the HN and wire into the beer. Merry Christmas
    - simonh 6y ago
      If you're curious how reliable your desktop computer software is, I suggest you read the patch notes for the last few system updates. The general assumption in dealing with computer systems should always be to assume that they contain bugs. There have been detailed studies in the defence and medical industries into flaws in critical computer systems, and bugs are endemic in complex computer systems. We should always be prepared to question the results we get, check, test and double check. In critical cases where there is a question about the validity of the results, they must be checked. In these cases they were not checked, they were assumed to be correct because the subpost-masters and mistresses in the case could not provide evidence that the software was wrong. But then, how on earth could they? Questioning the validity of results and working hard to validate test and audit results is an approach that is well supported by the evidence and has served me well in my professional career.
      - phaemon 6y ago
        Edit: or, of course, I could be a clueless twat who hadn't read the article properly and realized the context. Sorry, mea culpa. > check, test and double check. Clearly that's not what you want to do, because you want to assume the computer is wrong without evidence as you stated. Testing might provide some evidence. > could not provide evidence that the software was wrong. But then, how on earth could they? Well, I suppose they could "check, test and double check." Or is that unreasonable?
  - Meph504 6y ago
    The Previously the legal position was "In any proceedings, a statement in a document produced by a computer shall not be admissible as evidence of any fact stated therein unless it is shown. That at all material times the computer was operating properly, or if not, that any respect in which it was not operating properly or was out of operation was not such as to affect the production of the document or the accuracy of its contents." they have now changed the position to "in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers." I have no earthly idea, where the assumption about not checking calculations, or "don’t need to show why it produced the results it did." comes from or how you arrived at that conclusion. your logical seems as daft as asking if I've ever used a computer system, on a computer system.
- gamblor956 6y ago
  It's perfectly reasonable to assume software generally works, because it generally does, and if it didn't we couldn't run modern society on software like we do right now. It's the proper default presumption to make. But the evidence to the contrary could have been something as simple as showing patch notes addressing a bug, or online complaints about that bug, which could have affected the software in a way that gave rise to the circumstances of the lawsuit.
- LatteLazy 6y ago
  In one case? Nothing. But if you get 100s of cases and it appears that ever user of this software without failure is stealing despite no other evidence and all of them protesting their innocence, then you might wanna reconsider.
- DanBC 6y ago
  EDIT. Here's a useful radio programme that goes through what happened: https://www.bbc.co.uk/programmes/m000jf7j https://www.bbc.co.uk/programmes/m000jf7j Bob works as a postmaster running a small local post office. Bob must use software provided to him by The Post Office. That software is faulty. One of the errors is that it reports more money present in Bob's office than there actually is. Bob contacts his superiors who tell him that no-one else is having a problem (this is a lie, other people are having the same problem and they know this) and that the software is correct. They give Bob a choice: supply the missing funds or deal with the police. Bob is acting in good faith and he assumes this is a simple mistake that'll get sorted out, so he supplies the missing money from his own funds, and keeps telling his superiors to fix it. Eventually, Bob runs out of money (or, maybe, he says "get the police involved".) So, the police arrest Bob (because we have a low threshold for arrest over here) and interview Bob under caution. Police are told by Post Office that the software is reliable, and that Bob is lying. Police aggressively question Bob. It goes to court. Bob is convicted. Because this is what happened to hundreds of people. What's Bob supposed to do? Ask the Post Office to supply the source code so he can have a look for bugs?
  - Meph504 6y ago
    "(this is a lie, other people are having the same problem and they know this)" So this is fraud, and makes the validate nature of a program immaterial. If they are willing to lie about this, they can produce false logs that show their system is working. But if Bob, kept records, of his transaction, and could show an irregularity between the his records and theirs this could be used as evidence of a faulty system an warrant further investigation. But you listed another issue that the law wouldn't help, the police don't believe Bob, they don't investigate, The courts don't believe bob. I don't see how you think this different finding would change that. In almost any case, if the cops don't believe you, and the court doesn't believe you, you are going to get screwed.
    - voxic11 6y ago
      They don't even need to produce false records/logs if the courts believe that any computer system is working as intended without evidence that it isn't then they don't need to produce any records as the court will already accept their computers result as correct.
gjvnq 6y ago
Why not make a compromise? Assume that software has no bugs if there is a publically available machine-checkable proof of the program correctness.
- Meph504 6y ago
  Because they would put an insane unreasonable burden on everyone who has a computer. and basically make everyone but multi-billion companies have to assume their systems are at fault.
  - pacamara619 6y ago
    Do you see the irony in your reply or would you like me to explain it?
waspentalive 6y ago
Babbage supposedly said something like "I cannot rightly understand how this question came to be, a politician asked me once: "If one enters the wrong information into the engine, will it still provide a correct answer?". "

41 points by idw 6y ago

How is this not reasonable? "in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers."

simonh 6y ago
Er, have you ever used a computer system, or written a computer program? Their position was that you don’t need to check the calculations, don’t need to show why it produced the results it did, you just assume it’s always correct. I work in the finance industry myself and have supported financial applications in banks and exchanges and the way these cases were handled is unbelievable. We’d never make assumptions about our systems like this, it was grossly negligent and incompetent and the courts need very clear guidance on how to do better.
- phaemon 6y ago
  Yes, do you know how they work? My computer is performing billions of calculations and operating on billions of results. Of course I don't check every one and of course I assume they're correct absent evidence to the contrary. Frankly this is so obvious that I'd suggest you lay off the beer while posting to HN, but this time of year that's a daft suggestion, so I suggest you lay off the HN and wire into the beer. Merry Christmas
  - simonh 6y ago
    If you're curious how reliable your desktop computer software is, I suggest you read the patch notes for the last few system updates. The general assumption in dealing with computer systems should always be to assume that they contain bugs. There have been detailed studies in the defence and medical industries into flaws in critical computer systems, and bugs are endemic in complex computer systems. We should always be prepared to question the results we get, check, test and double check. In critical cases where there is a question about the validity of the results, they must be checked. In these cases they were not checked, they were assumed to be correct because the subpost-masters and mistresses in the case could not provide evidence that the software was wrong. But then, how on earth could they? Questioning the validity of results and working hard to validate test and audit results is an approach that is well supported by the evidence and has served me well in my professional career.
    - phaemon 6y ago
      Edit: or, of course, I could be a clueless twat who hadn't read the article properly and realized the context. Sorry, mea culpa. > check, test and double check. Clearly that's not what you want to do, because you want to assume the computer is wrong without evidence as you stated. Testing might provide some evidence. > could not provide evidence that the software was wrong. But then, how on earth could they? Well, I suppose they could "check, test and double check." Or is that unreasonable?
- Meph504 6y ago
  The Previously the legal position was "In any proceedings, a statement in a document produced by a computer shall not be admissible as evidence of any fact stated therein unless it is shown. That at all material times the computer was operating properly, or if not, that any respect in which it was not operating properly or was out of operation was not such as to affect the production of the document or the accuracy of its contents." they have now changed the position to "in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers." I have no earthly idea, where the assumption about not checking calculations, or "don’t need to show why it produced the results it did." comes from or how you arrived at that conclusion. your logical seems as daft as asking if I've ever used a computer system, on a computer system.
gamblor956 6y ago
It's perfectly reasonable to assume software generally works, because it generally does, and if it didn't we couldn't run modern society on software like we do right now. It's the proper default presumption to make. But the evidence to the contrary could have been something as simple as showing patch notes addressing a bug, or online complaints about that bug, which could have affected the software in a way that gave rise to the circumstances of the lawsuit.
LatteLazy 6y ago
In one case? Nothing. But if you get 100s of cases and it appears that ever user of this software without failure is stealing despite no other evidence and all of them protesting their innocence, then you might wanna reconsider.
DanBC 6y ago
EDIT. Here's a useful radio programme that goes through what happened: https://www.bbc.co.uk/programmes/m000jf7j https://www.bbc.co.uk/programmes/m000jf7j Bob works as a postmaster running a small local post office. Bob must use software provided to him by The Post Office. That software is faulty. One of the errors is that it reports more money present in Bob's office than there actually is. Bob contacts his superiors who tell him that no-one else is having a problem (this is a lie, other people are having the same problem and they know this) and that the software is correct. They give Bob a choice: supply the missing funds or deal with the police. Bob is acting in good faith and he assumes this is a simple mistake that'll get sorted out, so he supplies the missing money from his own funds, and keeps telling his superiors to fix it. Eventually, Bob runs out of money (or, maybe, he says "get the police involved".) So, the police arrest Bob (because we have a low threshold for arrest over here) and interview Bob under caution. Police are told by Post Office that the software is reliable, and that Bob is lying. Police aggressively question Bob. It goes to court. Bob is convicted. Because this is what happened to hundreds of people. What's Bob supposed to do? Ask the Post Office to supply the source code so he can have a look for bugs?
- Meph504 6y ago
  "(this is a lie, other people are having the same problem and they know this)" So this is fraud, and makes the validate nature of a program immaterial. If they are willing to lie about this, they can produce false logs that show their system is working. But if Bob, kept records, of his transaction, and could show an irregularity between the his records and theirs this could be used as evidence of a faulty system an warrant further investigation. But you listed another issue that the law wouldn't help, the police don't believe Bob, they don't investigate, The courts don't believe bob. I don't see how you think this different finding would change that. In almost any case, if the cops don't believe you, and the court doesn't believe you, you are going to get screwed.
  - voxic11 6y ago
    They don't even need to produce false records/logs if the courts believe that any computer system is working as intended without evidence that it isn't then they don't need to produce any records as the court will already accept their computers result as correct.

gjvnq 6y ago

Why not make a compromise? Assume that software has no bugs if there is a publically available machine-checkable proof of the program correctness.

Meph504 6y ago
Because they would put an insane unreasonable burden on everyone who has a computer. and basically make everyone but multi-billion companies have to assume their systems are at fault.
- pacamara619 6y ago
  Do you see the irony in your reply or would you like me to explain it?

waspentalive 6y ago

Babbage supposedly said something like "I cannot rightly understand how this question came to be, a politician asked me once: "If one enters the wrong information into the engine, will it still provide a correct answer?". "