5 ms·

It seems to me that the Horizon system is just a simple accounting/POS thing. I can't understand how it could be built so shoddily. It doesn't even have that ma

It seems to me that the Horizon system is just a simple accounting/POS thing. I can't understand how it could be built so shoddily. It doesn't even have that many clients: there are less than 12k post offices in the UK.

MattGaiser 3y ago
Governments often have a ton of overly complex and poorly documented rules. Add in bad incentives and people put into roles they are poorly suited and it can go to hell pretty quickly. Here is an anecdote from a past job. 1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably. 2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note. 3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed. 4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions. 5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back. 6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it. 6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it. 7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.
- rightbyte 3y ago
  At some point one should just give up and do it by hand. There is so much overautomization nowadays that in practice are time sinks and deadlocks the org. A sane system would be 'contract A type' and all other be done manually or something.
- InCityDreams 3y ago
  >3. 3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed. Er....shit in. Shit out?
  - stuartjohnson12 3y ago
    Not necessarily. More of a lost in translation. A great example of something hard to digitise is mechanisms for reward/punishment. Algorithmic approaches for criminal sentencing that work on a parameterised basis have been proposed over the years but always found wanting.
rwmj 3y ago
There was a quote from one of the insiders[1] which said there was a team of 8 developers, 2 knew what they were doing, a couple more were competent, and the other 4 were incompetent (on the Joel scale, in the idiot + gets things done quadrant). I can imagine that trying to write a reliable product in that environment was difficult to say the least. [1] https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...
KineticLensman 3y ago
It’s not ‘simple’ EPOS. Some of the things sold were local and held at the counter, e.g books of stamps, others were centralised virtual services (eg taxing your car). Even today the Post Office has a range of weird options not found elsewhere, e.g use of ‘postal orders’ that allow people to send money to recipients who don’t have bank accounts. All of these can be accessed at a post office counter. That said, it was certainly shoddy
- InCityDreams 3y ago
  >shoddy 'Shoddy', being a euphemism for 'beyond knowingly-being fucked-up, so much so that we'll cover it up oh fuck we're still alive this wasnt what we were told would happen'? I dunno, or maybe just me surmising.
  - jjgreen 3y ago
    The rule of English understatement is that the more understated, the more damning. So "a touch shoddy" would "stunningly beyond knowingly-being absolutely totally fucked-up ...", you get the picture.
cosmodisk 3y ago
I'm currently working on a 'simple' integration between an accounting and crm software. I'm already pulling my hair because of various exceptions, limitations and whatnot. I think Horizon was anything but simple.
- phpisthebest 3y ago
  Just allow import and export from and to Excel... Problem Solved <<I wish I was joking>>>
- CaptainZapp 3y ago
  > I think Horizon was anything but simple. Yes, you're probably right. But this is the part that really bugs me : "The inquiry heard that in 2008, a glitch in a system called CABSProcess, which automatically summarises a post office’s transactions at around 7pm daily, resulted in users working at the same time having balancing issues." This, for an accounting system is just fucking inexcusable! I worked on a huge accounting project for a financial institution processing 2M account bookings per day (at that time, it's a lot more now). A lot of effort was put into the audit log (every booking was written to an append only log) and into the reconciliation process. When it was 20 Rappen off there was digging until the inconsistency was found. That transactions are not completely isolated when that CABSProcess kicked in is just completely and uttelry inexcusable.
- FireBeyond 3y ago
  Yeah. I work as a PM in a company that builds integrations between ERP and "heavier" accounting systems. It is absolutely not simple, to the point where we are building an ETL-style engine to drive things. It's not "copy record X to Y".
robertlagrant 3y ago
Slightly different take: it was a state contract fulfilled by the company best able to impress state procurement. Quality is rarely on the agenda for this sort of procurement, because if it goes wrong they can blame the management who were overseeing the implementation. Thus it was unlikely to be great from early on.
mavhc 3y ago
Except there's 20,000 POS, sometimes multiple per shop, with dial up ISDN on POS machine 1 per shop, except a few rural systems that need satellites
- forinti 3y ago
  Maintenance of 20k POSs might be a pain, but writing the software is not that hard, especially if your budget is in the billion ballpark.
  - mavhc 3y ago
    Except the "coping with dial on demand" part of the software was brand new, not even finished for the pilot. Not sure a sometimes offline distributed database where data is temp stored on POS machines is such an easy thing to write either As usual: rushed, not well tested, assumed to be perfect

MattGaiser 3y ago

Governments often have a ton of overly complex and poorly documented rules. Add in bad incentives and people put into roles they are poorly suited and it can go to hell pretty quickly. Here is an anecdote from a past job. 1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably. 2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note. 3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed. 4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions. 5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back. 6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it. 6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it. 7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.

rightbyte 3y ago
At some point one should just give up and do it by hand. There is so much overautomization nowadays that in practice are time sinks and deadlocks the org. A sane system would be 'contract A type' and all other be done manually or something.
InCityDreams 3y ago
>3. 3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed. Er....shit in. Shit out?
- stuartjohnson12 3y ago
  Not necessarily. More of a lost in translation. A great example of something hard to digitise is mechanisms for reward/punishment. Algorithmic approaches for criminal sentencing that work on a parameterised basis have been proposed over the years but always found wanting.

rwmj 3y ago

There was a quote from one of the insiders[1] which said there was a team of 8 developers, 2 knew what they were doing, a couple more were competent, and the other 4 were incompetent (on the Joel scale, in the idiot + gets things done quadrant). I can imagine that trying to write a reliable product in that environment was difficult to say the least. [1] https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...

KineticLensman 3y ago

It’s not ‘simple’ EPOS. Some of the things sold were local and held at the counter, e.g books of stamps, others were centralised virtual services (eg taxing your car). Even today the Post Office has a range of weird options not found elsewhere, e.g use of ‘postal orders’ that allow people to send money to recipients who don’t have bank accounts. All of these can be accessed at a post office counter. That said, it was certainly shoddy

InCityDreams 3y ago
>shoddy 'Shoddy', being a euphemism for 'beyond knowingly-being fucked-up, so much so that we'll cover it up oh fuck we're still alive this wasnt what we were told would happen'? I dunno, or maybe just me surmising.
- jjgreen 3y ago
  The rule of English understatement is that the more understated, the more damning. So "a touch shoddy" would "stunningly beyond knowingly-being absolutely totally fucked-up ...", you get the picture.

cosmodisk 3y ago

I'm currently working on a 'simple' integration between an accounting and crm software. I'm already pulling my hair because of various exceptions, limitations and whatnot. I think Horizon was anything but simple.

phpisthebest 3y ago
Just allow import and export from and to Excel... Problem Solved <<I wish I was joking>>>
CaptainZapp 3y ago
> I think Horizon was anything but simple. Yes, you're probably right. But this is the part that really bugs me : "The inquiry heard that in 2008, a glitch in a system called CABSProcess, which automatically summarises a post office’s transactions at around 7pm daily, resulted in users working at the same time having balancing issues." This, for an accounting system is just fucking inexcusable! I worked on a huge accounting project for a financial institution processing 2M account bookings per day (at that time, it's a lot more now). A lot of effort was put into the audit log (every booking was written to an append only log) and into the reconciliation process. When it was 20 Rappen off there was digging until the inconsistency was found. That transactions are not completely isolated when that CABSProcess kicked in is just completely and uttelry inexcusable.
FireBeyond 3y ago
Yeah. I work as a PM in a company that builds integrations between ERP and "heavier" accounting systems. It is absolutely not simple, to the point where we are building an ETL-style engine to drive things. It's not "copy record X to Y".

robertlagrant 3y ago

Slightly different take: it was a state contract fulfilled by the company best able to impress state procurement. Quality is rarely on the agenda for this sort of procurement, because if it goes wrong they can blame the management who were overseeing the implementation. Thus it was unlikely to be great from early on.

mavhc 3y ago

Except there's 20,000 POS, sometimes multiple per shop, with dial up ISDN on POS machine 1 per shop, except a few rural systems that need satellites

forinti 3y ago
Maintenance of 20k POSs might be a pain, but writing the software is not that hard, especially if your budget is in the billion ballpark.
- mavhc 3y ago
  Except the "coping with dial on demand" part of the software was brand new, not even finished for the pilot. Not sure a sometimes offline distributed database where data is temp stored on POS machines is such an easy thing to write either As usual: rushed, not well tested, assumed to be perfect