8 ms·

Is there any actual technical report of what's wrong with the Horizon system? This feels like a weirdly long Wikipedia article that only touches on legal points

Is there any actual technical report of what's wrong with the Horizon system? This feels like a weirdly long Wikipedia article that only touches on legal points and that somewhat lacks in real substance.

nelox 3y ago
The technical appendix to the judgement in Bates v Post Office: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-appendix-1.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... The judgement in question: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-judgment.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... [edit: added judgement proper]
hermitcrab 3y ago
There is an excellent Computerphile video discussing some of the issues in the Horizons software system at the heart of this scandal: https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10
hashtag-til 3y ago
I was curious for the same, and ended up reading this - https://em360tech.com/tech-article/post-office-scandal https://em360tech.com/tech-article/post-office-scandal It seems like the classical mega-corp system issues: "These flaws included rounding errors, data corruption, and issues with synchronization between local and central databases."
- anentropic 3y ago
  I was listening to a programme about this on Radio 4 last night it seems like, on top of the buggy software, Fujitsu had contractors logging in to Post Office systems to manually 'fix' accounting errors the buggy software had produced one of them described how while they were logged in to the POS system it would appear to be working to the real user but would actually be doing nothing because it was in some kind of remote admin mode, seems like the sort of remediation that's likely to make things worse sounded like a total shit show
  - hermitcrab 3y ago
    Apparently Fujitsu could log in and change the live data on a subpostmaster's system remotely and they frequently did this to make corrections. However it looked as if the subpostmaster had made the change. There was no proper audit log. However the subpostmasters were told remote access was impossible and any errors were 100% their fault. The wrong people went to prison. All the existing convictions must be considered unsafe.
ChrisKnott 3y ago
I felt exactly the same. I read some of the original Second Sight reports and I very much got the impression that the issues were quite broad and related to inadequate training, poor/confusing UI etc. There isn't, as far as I can tell, a single interesting "bug" that was causing incorrect figures to be saved in a database, which is the impression you get from skimming the coverage.
- hermitcrab 3y ago
  My father was interviewed for a potential job as trainer for the Horizons system many years ago. He was horrified by the attitude of the company running the training. Thankfully he didn't get the job.
- ndsipa_pomu 3y ago
  https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There isn't a full list of the bugs produced, but it does look like a lot of bugs directly involved incorrect accounting to be recorded.
Gupie 3y ago
I don't know of any actual technical reports but this is interesting, https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown#:~:text=What%20went%20wrong%3F,inquiry%20into%20the%20scandal%20heard https://www.theguardian.com/uk-news/2024/jan/09/how-the-post.... "One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for." https://clarotesting.wordpress.com/tag/dalmellingto-bug/ https://clarotesting.wordpress.com/tag/dalmellingto-bug/ "The Dalmellington Bug entailed a user repeatedly hitting a key when the system froze as she was trying to acknowledge receipt of a consignment of £8,000 in cash. Unknown to her each time she struck the key she accepted responsibility for a further £8,000. The bug created a discrepancy of £24,000 for which she was held responsible."
ndsipa_pomu 3y ago
There's some more detail here: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... It only mentions a few bugs out of the hundreds that were discovered (a full list hasn't been published), but it appears that the bugs weren't the only issue: > In 2015, for instance, the Post Office told the House of Commons inquiry: “There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or remove transaction data once it has been recorded in a branch’s accounts.” This was untrue, and the Post Office admitted as such four years later during a high court case. > In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems, the inquiry heard. It's truly a scandal that no-one in charge at the Post Office or Fujitsu has been prosecuted for providing false testimony that led to so much misery and death for the sub postmasters. The MET has finally started to investigate fraud and perjury charges which seems like they only bother investigating crimes that have been made into a popular four-part series.
- throwaway167 3y ago
  > In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems I worked in core banking systems for a bit. This is beyond basic, it demonstrates Fujitsu do not have controls in place to design compliant systems. Fujitsu should forever have their reputation tarnished and never let near financial systems. I don't often advocate selling a business to Tata, but this might be one of those times.
- makomk 3y ago
  The Met's had an investigation open over this since 2020, all that seems to have happened as a result of this popular four-part series is that they've confirmed they're looking into specific possible criminal offences by the Post Office or its employees that we didn't know they were looking in to before. It's not clear that this is something which started investigating because of the popular TV series, but most likely not and they were just pressured into revealing more about the existing investigation - the police here don't generally give running updates on criminal investigations to the media.
dkdbejwi383 3y ago
https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... states "a full list [of bugs] has never been produced". The article gives one example of a bug where the UI would freeze/get stuck, but still respond to key presses, so a user might think the system had crashed but press 'enter' repeatedly, which would silently create several records.
mattbee 3y ago
I've also been looking and there has been no information about the nature of the software fault(s) that were the source of the accounting discrepancies that were the trigger for these miscarriages of justice. [edit: yes there are, see above] But we can't blame this scandal on a software fault, or even an averagely terrible government IT contractor - there were some monsters in the Post Office who chose to take this as far as they did. (for dang) I'd change this link to the main source on this story since the mid-2000s - https://www.private-eye.co.uk/special-reports/justice-lost-in-the-post https://www.private-eye.co.uk/special-reports/justice-lost-i...
- ndsipa_pomu 3y ago
  https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There isn't a full list of the bugs produced, but it does look like a lot of bugs directly involved incorrect accounting to be recorded.
  - cge 3y ago
    I had assumed that bugs allowing such miscarriages of justice would require a completely unsound accounting method in the software. Unless, eg, the accounting simply didn't store a transaction ledger and software didn't store logs, and instead mostly just updated balances with no history, it was unclear to me how software bugs in an otherwise reasonably sound accounting method could result in accounting so wrong that it couldn't seen to be incorrect when investigated at the level of thoroughness one might expect for court proceedings. But these bugs suggest that at least some of the errors would likely have been obvious in a ledger, and the Post Office simply didn't care. That rather makes Fujitsu look not quite as bad here, and the Post Office look worse.
rwmj 3y ago
This was interesting: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with... Sounds like a rather complex distributed system which no one fully understood combined with the failure of the database to meet ACID requirements.
di4na 3y ago
There is work in progress in the high-level inquiry that should manage to produce most of it. Besides that, you can try to check the appeals court reports, which are not too bad.
gfv 3y ago
Yes, there is the technical appendix to the 2019 judgement. https://www.jfsa.org.uk/uploads/5/4/3/1/54312921/technical_appendix_final_2.doc https://www.jfsa.org.uk/uploads/5/4/3/1/54312921/technical_a...

nelox 3y ago

The technical appendix to the judgement in Bates v Post Office: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-appendix-1.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... The judgement in question: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-judgment.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... [edit: added judgement proper]

hermitcrab 3y ago

There is an excellent Computerphile video discussing some of the issues in the Horizons software system at the heart of this scandal: https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10

hashtag-til 3y ago

I was curious for the same, and ended up reading this - https://em360tech.com/tech-article/post-office-scandal https://em360tech.com/tech-article/post-office-scandal It seems like the classical mega-corp system issues: "These flaws included rounding errors, data corruption, and issues with synchronization between local and central databases."

anentropic 3y ago
I was listening to a programme about this on Radio 4 last night it seems like, on top of the buggy software, Fujitsu had contractors logging in to Post Office systems to manually 'fix' accounting errors the buggy software had produced one of them described how while they were logged in to the POS system it would appear to be working to the real user but would actually be doing nothing because it was in some kind of remote admin mode, seems like the sort of remediation that's likely to make things worse sounded like a total shit show
- hermitcrab 3y ago
  Apparently Fujitsu could log in and change the live data on a subpostmaster's system remotely and they frequently did this to make corrections. However it looked as if the subpostmaster had made the change. There was no proper audit log. However the subpostmasters were told remote access was impossible and any errors were 100% their fault. The wrong people went to prison. All the existing convictions must be considered unsafe.

ChrisKnott 3y ago

I felt exactly the same. I read some of the original Second Sight reports and I very much got the impression that the issues were quite broad and related to inadequate training, poor/confusing UI etc. There isn't, as far as I can tell, a single interesting "bug" that was causing incorrect figures to be saved in a database, which is the impression you get from skimming the coverage.

hermitcrab 3y ago
My father was interviewed for a potential job as trainer for the Horizons system many years ago. He was horrified by the attitude of the company running the training. Thankfully he didn't get the job.
ndsipa_pomu 3y ago
https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There isn't a full list of the bugs produced, but it does look like a lot of bugs directly involved incorrect accounting to be recorded.

Gupie 3y ago

I don't know of any actual technical reports but this is interesting, https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown#:~:text=What%20went%20wrong%3F,inquiry%20into%20the%20scandal%20heard https://www.theguardian.com/uk-news/2024/jan/09/how-the-post.... "One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for." https://clarotesting.wordpress.com/tag/dalmellingto-bug/ https://clarotesting.wordpress.com/tag/dalmellingto-bug/ "The Dalmellington Bug entailed a user repeatedly hitting a key when the system froze as she was trying to acknowledge receipt of a consignment of £8,000 in cash. Unknown to her each time she struck the key she accepted responsibility for a further £8,000. The bug created a discrepancy of £24,000 for which she was held responsible."

ndsipa_pomu 3y ago

There's some more detail here: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... It only mentions a few bugs out of the hundreds that were discovered (a full list hasn't been published), but it appears that the bugs weren't the only issue: > In 2015, for instance, the Post Office told the House of Commons inquiry: “There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or remove transaction data once it has been recorded in a branch’s accounts.” This was untrue, and the Post Office admitted as such four years later during a high court case. > In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems, the inquiry heard. It's truly a scandal that no-one in charge at the Post Office or Fujitsu has been prosecuted for providing false testimony that led to so much misery and death for the sub postmasters. The MET has finally started to investigate fraud and perjury charges which seems like they only bother investigating crimes that have been made into a popular four-part series.

throwaway167 3y ago
> In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems I worked in core banking systems for a bit. This is beyond basic, it demonstrates Fujitsu do not have controls in place to design compliant systems. Fujitsu should forever have their reputation tarnished and never let near financial systems. I don't often advocate selling a business to Tata, but this might be one of those times.
makomk 3y ago
The Met's had an investigation open over this since 2020, all that seems to have happened as a result of this popular four-part series is that they've confirmed they're looking into specific possible criminal offences by the Post Office or its employees that we didn't know they were looking in to before. It's not clear that this is something which started investigating because of the popular TV series, but most likely not and they were just pressured into revealing more about the existing investigation - the police here don't generally give running updates on criminal investigations to the media.

dkdbejwi383 3y ago

https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... states "a full list [of bugs] has never been produced". The article gives one example of a bug where the UI would freeze/get stuck, but still respond to key presses, so a user might think the system had crashed but press 'enter' repeatedly, which would silently create several records.

mattbee 3y ago

I've also been looking and there has been no information about the nature of the software fault(s) that were the source of the accounting discrepancies that were the trigger for these miscarriages of justice. [edit: yes there are, see above] But we can't blame this scandal on a software fault, or even an averagely terrible government IT contractor - there were some monsters in the Post Office who chose to take this as far as they did. (for dang) I'd change this link to the main source on this story since the mid-2000s - https://www.private-eye.co.uk/special-reports/justice-lost-in-the-post https://www.private-eye.co.uk/special-reports/justice-lost-i...

ndsipa_pomu 3y ago
https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There isn't a full list of the bugs produced, but it does look like a lot of bugs directly involved incorrect accounting to be recorded.
- cge 3y ago
  I had assumed that bugs allowing such miscarriages of justice would require a completely unsound accounting method in the software. Unless, eg, the accounting simply didn't store a transaction ledger and software didn't store logs, and instead mostly just updated balances with no history, it was unclear to me how software bugs in an otherwise reasonably sound accounting method could result in accounting so wrong that it couldn't seen to be incorrect when investigated at the level of thoroughness one might expect for court proceedings. But these bugs suggest that at least some of the errors would likely have been obvious in a ledger, and the Post Office simply didn't care. That rather makes Fujitsu look not quite as bad here, and the Post Office look worse.

rwmj 3y ago

This was interesting: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with... Sounds like a rather complex distributed system which no one fully understood combined with the failure of the database to meet ACID requirements.

di4na 3y ago

There is work in progress in the high-level inquiry that should manage to produce most of it. Besides that, you can try to check the appeals court reports, which are not too bad.

gfv 3y ago

Yes, there is the technical appendix to the 2019 judgement. https://www.jfsa.org.uk/uploads/5/4/3/1/54312921/technical_appendix_final_2.doc https://www.jfsa.org.uk/uploads/5/4/3/1/54312921/technical_a...