5 ms·

This whole horizon system story is fascinating for so many reasons. One thing I don't understand though is exactly what the bug was? For context, it led to (so

This whole horizon system story is fascinating for so many reasons. One thing I don't understand though is exactly what the bug was? For context, it led to (sometimes huge) discrepancies in what it thought should be in the till based on sales data from the day. I don't build systems anything like Horizon, so this is probably a massive over simplification, but surely tracking stock and cash in tills should be really simple? Does anyone know if there are public details on what exactly the technical bug was?

andy_ppp 3y ago
Haha, obviously you haven’t worked in any large enterprises before that aren’t software companies. Basically the business rolls dice and comes up with some random requirements and then tries to change those requirements constantly until you end up with software that looks so hideous nobody understands anything about it and all the chopping and changing leaves the code in a complete mess with lots of bugs. The problem is the people making these requests do not understand the costs, if they were building a house they wouldn’t dig up the foundations on a whim, but because software is so ephemeral it seems (to them at least) cost free.
InsomniacL 3y ago
https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-appendix-1.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... I have only skimmed this so far but it seems there were many bugs. Also this stood out to me. > 4. Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu. They may be performed in order to ‘balance’ a discrepancy. These do NOT require acceptance by SPMs in the same manner as TCs and TAs.
- tauchunfall 3y ago
  >Fujitsu inserted transactions They inserted transactions for corrections of discrepancies where you could not see they were authored by them; it was just assumed the postmaster did. They did not let postmasters know about this; probably an order by the post office. These corrections where incorrect, and caused discrepancies to increase; they mixed plus and minus sign quite often. They added lines of code to the system running on the counter in the local post office. They used the message store system like wild west without defining a data schema.
  - blitzar 3y ago
    > They inserted transactions for corrections of discrepancies where you could not see they were authored by them; it was just assumed the postmaster did. Then prosecuted the postmasters for inserting transactions, because "only the postmaster can insert transactions".
cmsefton 3y ago
There wasn't just one bug, there were many. One of them was due to a "missing payments node", but there were many software faults, including rounding errors, data corruption, and issues with synchronization between local and central databases. Two bugs are detailed here: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There weren't just bugs, either Fujitsu had the ability of modifying or adding transactional records remotely, but always denied they had the ability to do so, as did the Post Office.
tgv 3y ago
There's a computerphile video on youtube where a prof goes through A, C, I and D and gives an example where each of these went wrong. Professionally speaking, it's shocking.
- codeulike 3y ago
  Hey you're right, from two years ago, good catch https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10
tauchunfall 3y ago
>One thing I don't understand though is exactly what the bug was? It was not only the bugs but also the UX and the overall process that made things worse. The system did things that the postmaster was either unaware of, or did not intend to do. When there was a customer session and a session time-out occurred it was changed into an actual transaction without the postmaster intending it; like you have a shopping cart on an e-commerce site and it transforms the shopping cart automatically into a sale after timeout. There were screen freezes and when the postmaster pressed enter while the screen freeze happend, it multiplied the amount of money, without the screen being updated. Postmasters are responsible for discrepancies, and when they called the telephone support they were occasionally told "not to worry, it will sort itself out". There were corrections from the outside and postmasters sometimes waited for weeks and even months for these corrections to happen to resolve the discrepancies. Sometimes they never happened.
codeulike 3y ago
It was a bunch of different bugs We're trying to figure out the architecture over on this other thread - https://news.ycombinator.com/item?id=38954516 https://news.ycombinator.com/item?id=38954516 But basically it was based on a sortof replicated XML database that sent text files back and forth, so the basics of synchronisation and consensus were just not covered properly. e.g. if someone kept pressing a key when the screen was frozen, duplicate transactions got created. Or if they turned their machine off earlier than 5pm, end-of-day processes did not get run properly. Or if their node was trying to synchronise with the central node a bit later than normal and the central node was doing a 'reindexing' operation, their messages got lost. Lots of edge case synronisation bugs basically. edit: also here's a good computerphile video about the bugs https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10
pjc50 3y ago
> surely tracking stock and cash in tills should be really simple? You'd think so, except they wanted a custom system built at incredible expense.

andy_ppp 3y ago

Haha, obviously you haven’t worked in any large enterprises before that aren’t software companies. Basically the business rolls dice and comes up with some random requirements and then tries to change those requirements constantly until you end up with software that looks so hideous nobody understands anything about it and all the chopping and changing leaves the code in a complete mess with lots of bugs. The problem is the people making these requests do not understand the costs, if they were building a house they wouldn’t dig up the foundations on a whim, but because software is so ephemeral it seems (to them at least) cost free.

InsomniacL 3y ago

https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-post-office-appendix-1.pdf https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-... I have only skimmed this so far but it seems there were many bugs. Also this stood out to me. > 4. Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu. They may be performed in order to ‘balance’ a discrepancy. These do NOT require acceptance by SPMs in the same manner as TCs and TAs.

tauchunfall 3y ago
>Fujitsu inserted transactions They inserted transactions for corrections of discrepancies where you could not see they were authored by them; it was just assumed the postmaster did. They did not let postmasters know about this; probably an order by the post office. These corrections where incorrect, and caused discrepancies to increase; they mixed plus and minus sign quite often. They added lines of code to the system running on the counter in the local post office. They used the message store system like wild west without defining a data schema.
- blitzar 3y ago
  > They inserted transactions for corrections of discrepancies where you could not see they were authored by them; it was just assumed the postmaster did. Then prosecuted the postmasters for inserting transactions, because "only the postmaster can insert transactions".

cmsefton 3y ago

There wasn't just one bug, there were many. One of them was due to a "missing payments node", but there were many software faults, including rounding errors, data corruption, and issues with synchronization between local and central databases. Two bugs are detailed here: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... > One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for. > Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors. There weren't just bugs, either Fujitsu had the ability of modifying or adding transactional records remotely, but always denied they had the ability to do so, as did the Post Office.

tgv 3y ago

There's a computerphile video on youtube where a prof goes through A, C, I and D and gives an example where each of these went wrong. Professionally speaking, it's shocking.

codeulike 3y ago
Hey you're right, from two years ago, good catch https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10

tauchunfall 3y ago

>One thing I don't understand though is exactly what the bug was? It was not only the bugs but also the UX and the overall process that made things worse. The system did things that the postmaster was either unaware of, or did not intend to do. When there was a customer session and a session time-out occurred it was changed into an actual transaction without the postmaster intending it; like you have a shopping cart on an e-commerce site and it transforms the shopping cart automatically into a sale after timeout. There were screen freezes and when the postmaster pressed enter while the screen freeze happend, it multiplied the amount of money, without the screen being updated. Postmasters are responsible for discrepancies, and when they called the telephone support they were occasionally told "not to worry, it will sort itself out". There were corrections from the outside and postmasters sometimes waited for weeks and even months for these corrections to happen to resolve the discrepancies. Sometimes they never happened.

codeulike 3y ago

It was a bunch of different bugs We're trying to figure out the architecture over on this other thread - https://news.ycombinator.com/item?id=38954516 https://news.ycombinator.com/item?id=38954516 But basically it was based on a sortof replicated XML database that sent text files back and forth, so the basics of synchronisation and consensus were just not covered properly. e.g. if someone kept pressing a key when the screen was frozen, duplicate transactions got created. Or if they turned their machine off earlier than 5pm, end-of-day processes did not get run properly. Or if their node was trying to synchronise with the central node a bit later than normal and the central node was doing a 'reindexing' operation, their messages got lost. Lots of edge case synronisation bugs basically. edit: also here's a good computerphile video about the bugs https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10

pjc50 3y ago

> surely tracking stock and cash in tills should be really simple? You'd think so, except they wanted a custom system built at incredible expense.