4 ms·

Summary: the presumption that computers are always correct had lead the UK post office to believe that hundreds of post offices had shortfalls in their accounts

by mcv 4y ago

Summary: the presumption that computers are always correct had lead the UK post office to believe that hundreds of post offices had shortfalls in their accounts, leading to hundreds of postmasters getting accused of fraud, resulting in their prosecution and conviction. Fortunately, group action has lead to the error in the system being discovered and their convictions being overturned. It seems pretty dangerous to blindly assume that a computer system is always correct without thorough proof that it is indeed correct. But we already knew that.

rayiner 4y ago
So all criminal prosecutions involving computers should require the state to prove that the computer is correct?
- constantcrying 4y ago
  I don't think it is an either or question. But it seems very reasonable to not assume computers are always correct. In a situation in which a bug could have conceivably caused the evidence against the defendant the defandant, should be allowed to examine the computer system (read the source code). Without that I see no reason why it should be allowed as evidence against someone, when the evidence could have been the result of a flaw in the software. Of course "could" is pretty ambigous. Things you might look at are: Have this or other, similar systems suffered from a bug which could have caused this? How much/often is the claim out of line with what the defence claims. How serious and easy to notice would a bug have to be to cause this? Is there other evidence, besides the computer? And probably many more. Of course "the camera changed the face of the person to the defendants" is something which probabbly should be disregarded.
- mcv 4y ago
  A complete proof of correctness might be a bit much to ask, but slightly more burden of proof than "it's what the computer says" would be nice. Every piece of software of any meaningful size has bugs. First check if the discrepancy might be the result of a bug before you start sending people to prison.
- lesuorac 4y ago
  Why not? Think of it in the other direction, why should you be allowed to use evidence that has no documented reason of being accurate.
  - MichaelZuo 4y ago
    It's not possible to prove, without billions of dollars of investigation per device, that any complex consumer electronic device is operating 100% correctly. So wouldn't it practically invalidate any evidence that was touched by any device?
    - CogitoCogito 4y ago
      Why would it cost billions of dollars to satisfy the court of the accuracy of a program’s claims when it doesn’t cost billions of dollars to satisfy the court of the accuracy of an investigator’s claims?
      - MichaelZuo 4y ago
        This is presuming they expect 100% confidence. If they are fine with a lower bound of confidence, then that's already the current situation.
        lesuorac 4y ago
        Courts don't bother themselves with confidence intervals nor do they bother with 100% ("reasonable doubt" is not 0/100%). The problem is the current situation. Evidence is being used without actual proof that said evidence was collected "truthfully". An algorithm will generate a result but if you haven't proven/shown that the algorithm generates a correct result then any result from that algorithm shouldn't be usable in court. This problem comes up a lot with people who (un)cleverly use NULL for things (i.e. License Plates). Or if you live in the middle of America and somebody decides your address will be the default for geo-ip lookups if none is found. To stick with the article, immediately when the system was deployed the postmasters thought something was wrong. If the state had to show that the computer was correct the whole situation would've been avoided.
- dusted 4y ago
  "Oh, but the computer is doing the right more often than (9.4608e+16)-1 times out of 9.4608e+16 times!"* Yes.. but that's "less than" a single mistake per year, per computer. How many cores do your data go through per year ? * Totally made up, rough number of cpu ticks of a 3 ghz core in a year. Still, the point I'm trying to make is that with the mind-numbing number of actions carried out by another mind-numbing number of machines on a third absolutely mind-numbing number of bits every year, you will need a metric fuckton of zeroes after your decimal point, before it's "not going to put innocent people in jail".
  - cwillu 4y ago
    If it's metric, it's a fucktonne, not a fuckton. ‹ducks›
    - dusted 4y ago
      Interesting, didn't know that, thanks! :) In Danish we use the word "Ton" which is 1000 Kg, didn't realized that English has two words for it, didn't realize there's an emperial ton either :D
      - sirtomato 4y ago
        Yep, US customary ton is 2000lb
- _hcuq 4y ago
  Of course… we all know how buggy all computer programs are.
Throwawayaerlei 4y ago
Prior to that I read of bogus prosecutions based on ATM withdraws.

by mcv 4y ago

rayiner 4y ago

So all criminal prosecutions involving computers should require the state to prove that the computer is correct?

constantcrying 4y ago
I don't think it is an either or question. But it seems very reasonable to not assume computers are always correct. In a situation in which a bug could have conceivably caused the evidence against the defendant the defandant, should be allowed to examine the computer system (read the source code). Without that I see no reason why it should be allowed as evidence against someone, when the evidence could have been the result of a flaw in the software. Of course "could" is pretty ambigous. Things you might look at are: Have this or other, similar systems suffered from a bug which could have caused this? How much/often is the claim out of line with what the defence claims. How serious and easy to notice would a bug have to be to cause this? Is there other evidence, besides the computer? And probably many more. Of course "the camera changed the face of the person to the defendants" is something which probabbly should be disregarded.
mcv 4y ago
A complete proof of correctness might be a bit much to ask, but slightly more burden of proof than "it's what the computer says" would be nice. Every piece of software of any meaningful size has bugs. First check if the discrepancy might be the result of a bug before you start sending people to prison.
lesuorac 4y ago
Why not? Think of it in the other direction, why should you be allowed to use evidence that has no documented reason of being accurate.
- MichaelZuo 4y ago
  It's not possible to prove, without billions of dollars of investigation per device, that any complex consumer electronic device is operating 100% correctly. So wouldn't it practically invalidate any evidence that was touched by any device?
  - CogitoCogito 4y ago
    Why would it cost billions of dollars to satisfy the court of the accuracy of a program’s claims when it doesn’t cost billions of dollars to satisfy the court of the accuracy of an investigator’s claims?
    - MichaelZuo 4y ago
      This is presuming they expect 100% confidence. If they are fine with a lower bound of confidence, then that's already the current situation.
      - lesuorac 4y ago
        Courts don't bother themselves with confidence intervals nor do they bother with 100% ("reasonable doubt" is not 0/100%). The problem is the current situation. Evidence is being used without actual proof that said evidence was collected "truthfully". An algorithm will generate a result but if you haven't proven/shown that the algorithm generates a correct result then any result from that algorithm shouldn't be usable in court. This problem comes up a lot with people who (un)cleverly use NULL for things (i.e. License Plates). Or if you live in the middle of America and somebody decides your address will be the default for geo-ip lookups if none is found. To stick with the article, immediately when the system was deployed the postmasters thought something was wrong. If the state had to show that the computer was correct the whole situation would've been avoided.
dusted 4y ago
"Oh, but the computer is doing the right more often than (9.4608e+16)-1 times out of 9.4608e+16 times!"* Yes.. but that's "less than" a single mistake per year, per computer. How many cores do your data go through per year ? * Totally made up, rough number of cpu ticks of a 3 ghz core in a year. Still, the point I'm trying to make is that with the mind-numbing number of actions carried out by another mind-numbing number of machines on a third absolutely mind-numbing number of bits every year, you will need a metric fuckton of zeroes after your decimal point, before it's "not going to put innocent people in jail".
- cwillu 4y ago
  If it's metric, it's a fucktonne, not a fuckton. ‹ducks›
  - dusted 4y ago
    Interesting, didn't know that, thanks! :) In Danish we use the word "Ton" which is 1000 Kg, didn't realized that English has two words for it, didn't realize there's an emperial ton either :D
    - sirtomato 4y ago
      Yep, US customary ton is 2000lb
_hcuq 4y ago
Of course… we all know how buggy all computer programs are.

Throwawayaerlei 4y ago

Prior to that I read of bogus prosecutions based on ATM withdraws.