3 ms·

That's what the inquiry is meant to do. British inquiries take forever and rarely yield any kind of justice, however. tl;dr summary is that the root cause depe

That's what the inquiry is meant to do. British inquiries take forever and rarely yield any kind of justice, however. tl;dr summary is that the root cause depends on your perspective. If you have a CS background you might define the root cause as being the bugs. If you have a management background, you might conclude the root cause was bad/malign management by the board, the civil service and the C-suite. If you have a legal background you might conclude the root cause was the unusual prosecutorial powers, the default assumption courts make that computers aren't wrong and so on. As this is Hacker News let's roll with the first. There were many different bugs over many years and there was no single root cause. For example at least one case was caused by defective touch screen hardware that would go haywire when nobody was around to see it e.g. overnight, and start generating random clicks. Given enough random button pushes it was possible to ring up transactions that never happened, and thus never had money deposited in the till, creating phantom losses. In other cases there was loss of transactional integrity. Database writes went missing or got duplicated due to race conditions, not helped by the original Horizon architecture in which the connection to the central DB was asynchronous, due to the existence of post offices in very rural places without good internet connectivity. It's what would nowadays be called a "local-first distributed edge deployment" or something similar. Lots of ways for that to go wrong, and if you don't have exactly-once execution in an accounting system it will appear that cash was stolen because the records say a sale or refund was made, but it wasn't.

janice1999 2y ago
> summary is that the root cause depends on your perspective. This sounds like an exercise in obfuscating responsibility. Post Office management knew from 2003 Horizon was unreliable - they chose to bury the report. They lied in court about its reliability and manipulated expert statements to say the same. They also knowingly lied about remote altering of accounts and buried a report in 2017 on it. Let's not get lost in the weeds - the root cause of the injustice is Post Office management.
- tantalor 2y ago
  No I really meant from a technical perspective, what caused the computer system to be unreliable. Management surely did not intend the system to be unreliable. But I see your point, it follows the next question after "why is the system unreliable?" would be "why was the system not fixed?" and the answer is "management knowingly allowed it to be unreliable". But that's no longer a technical reason so it's less interesting.
  - janice1999 2y ago
    > No I really meant from a technical perspective, what caused the computer system to be unreliable. Replicating transactions across distributed systems is very very difficult. Horizon was clearly not designed to handle this based on the litany of errors users documented. Transactions could be silently dropped or duplicated etc. The system was also insecure and records could be silently remotely altered by Fujitsu staff.
    - mike_hearn 2y ago
      It was designed to handle it. The design just didn't work.
graemep 2y ago
True, but lets not get too Aristotelian about this. They are all worth looking at and all are true. They are also interlinked. Technical decisions would have been different if everyone put a high priority on correct operation. The high cost of the project ("it was the biggest non-military IT project in Europe.") created an incentive to over-up its failures. The post office does not have unusual powers to prosecute. What it does have are powers to investigate: https://gunnercooke.com/understanding-the-role-of-the-law-in-the-post-office-scandal-part-2-the-role-of-a-prosecutor/ https://gunnercooke.com/understanding-the-role-of-the-law-in... > the default assumption courts make that computers aren't wrong and so on. This is the most disturbing aspect of it to me. Nothing has been done to change this so it will happen again. It probably happens regularly on a smaller scale.
- masfuerte 2y ago
  > It probably happens regularly on a smaller scale. It does. Ross Anderson's Security Engineering describes how UK banks had customers prosecuted for fraud after criminals exploited bugs in chip-and-pin terminals to complete transactions without entering a PIN. The banks also successfully argued that their cryptographic transaction logs (which could definitively prove if a PIN was used) should not be admitted as evidence.
  - graemep 2y ago
    I knew about the banks attempt to prevent him publishing the flaws but I did not know they had prosecuted customers.
    - masfuerte 2y ago
      Section 10.4.3 has an older example where a customer was prosecuted for someone else's ATM fraud: https://www.cl.cam.ac.uk/~rja14/Papers/SEv2-c10.pdf https://www.cl.cam.ac.uk/~rja14/Papers/SEv2-c10.pdf I can't find the details about the chip-and-pin case. They might have been on Anderson's blog rather than in the book.
mavhc 2y ago
The sync functionality was a) written for the Horizon project, thus brand new, and b) not tested in the beta phase rollout
crimsontech 2y ago
I always wonder, as it is known these software issues caused phantom losses for postmasters, was there also a number of phantom gains? I would find it difficult to believe that these wouldn't have been reported by at least most of the postmasters, but I've never seen any mention of it.

janice1999 2y ago

> summary is that the root cause depends on your perspective. This sounds like an exercise in obfuscating responsibility. Post Office management knew from 2003 Horizon was unreliable - they chose to bury the report. They lied in court about its reliability and manipulated expert statements to say the same. They also knowingly lied about remote altering of accounts and buried a report in 2017 on it. Let's not get lost in the weeds - the root cause of the injustice is Post Office management.

tantalor 2y ago
No I really meant from a technical perspective, what caused the computer system to be unreliable. Management surely did not intend the system to be unreliable. But I see your point, it follows the next question after "why is the system unreliable?" would be "why was the system not fixed?" and the answer is "management knowingly allowed it to be unreliable". But that's no longer a technical reason so it's less interesting.
- janice1999 2y ago
  > No I really meant from a technical perspective, what caused the computer system to be unreliable. Replicating transactions across distributed systems is very very difficult. Horizon was clearly not designed to handle this based on the litany of errors users documented. Transactions could be silently dropped or duplicated etc. The system was also insecure and records could be silently remotely altered by Fujitsu staff.
  - mike_hearn 2y ago
    It was designed to handle it. The design just didn't work.

graemep 2y ago

True, but lets not get too Aristotelian about this. They are all worth looking at and all are true. They are also interlinked. Technical decisions would have been different if everyone put a high priority on correct operation. The high cost of the project ("it was the biggest non-military IT project in Europe.") created an incentive to over-up its failures. The post office does not have unusual powers to prosecute. What it does have are powers to investigate: https://gunnercooke.com/understanding-the-role-of-the-law-in-the-post-office-scandal-part-2-the-role-of-a-prosecutor/ https://gunnercooke.com/understanding-the-role-of-the-law-in... > the default assumption courts make that computers aren't wrong and so on. This is the most disturbing aspect of it to me. Nothing has been done to change this so it will happen again. It probably happens regularly on a smaller scale.

masfuerte 2y ago
> It probably happens regularly on a smaller scale. It does. Ross Anderson's Security Engineering describes how UK banks had customers prosecuted for fraud after criminals exploited bugs in chip-and-pin terminals to complete transactions without entering a PIN. The banks also successfully argued that their cryptographic transaction logs (which could definitively prove if a PIN was used) should not be admitted as evidence.
- graemep 2y ago
  I knew about the banks attempt to prevent him publishing the flaws but I did not know they had prosecuted customers.
  - masfuerte 2y ago
    Section 10.4.3 has an older example where a customer was prosecuted for someone else's ATM fraud: https://www.cl.cam.ac.uk/~rja14/Papers/SEv2-c10.pdf https://www.cl.cam.ac.uk/~rja14/Papers/SEv2-c10.pdf I can't find the details about the chip-and-pin case. They might have been on Anderson's blog rather than in the book.

mavhc 2y ago

The sync functionality was a) written for the Horizon project, thus brand new, and b) not tested in the beta phase rollout

crimsontech 2y ago

I always wonder, as it is known these software issues caused phantom losses for postmasters, was there also a number of phantom gains? I would find it difficult to believe that these wouldn't have been reported by at least most of the postmasters, but I've never seen any mention of it.