5 ms·
Fixing Horizon bugs would have been too costly, Post Office inquiry told
- oxfordmale 3y agoThis classic short-termism is all too common in software engineering. The reputational damage and likely compensation will exceed by far the costs of identifying and fixing these bugs. However, the CTO at the time has long moved on, and likely got a decent bonus at the time by cutting software engineering resources.
- sparrowInHand 3y agoWhich is why such reputational fallout needs to taint not only the MBA careers involved, it has to burn everyone involved and really hurt. The schools who educated them, there future employers, such shenanigans with the lifes of others have to make you radioactive. Taking the money for the responsibility has to involved some damocles swoards hanging over the boards.
- cmsj 3y agoWe need laws that allow historical clawback of bonuses/rewards that later turn out to have been given in the context of significant failures.
- lupusreal 3y agoThat should only trigger when there are legal proceedings with injured/wronged innocents. Otherwise you'll have companies clawing back bonuses when poorly conceived products flop.
- DanHulton 3y agoI'm not drawing a connection between your two sentences there. Like, that still doesn't sound like a bad idea!
- lupusreal 3y agoSo you do great work and get a bonus, but then the marketing department botches it, or executives decide to price it in some inane way that consumers reject and now you owe your employer money? Dystopian.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- nasmorn 3y agoThere would have been to compensations if they just had not persecuted the people for a known software bug. Might still have been cheaper to wait for a new system
- graemep 3y agoThis is a general problem in business, not just software. Management are motivated by short term bonuses. The biggest shareholders are fund managers who get judged on their annual performance (In the UK there are annual performance tables published. I assume it is similar in other countries) so they have a short term outlook as well. Private equity have a slightly longer term view, but not much more so. No one has a stake in what happens a decade from now.
- DemocracyFTW2 3y agoHowever bad the software is and however bad the software-related decisions at Fujitsu and the Post Office were, nothing should distract from the fact that the real culprit is really with the Post Office leadership who destroyed the lives of 900 people, ridiculed them, forced over 40 into suicide and countless others into falsely claiming guilty to bargain when faced with a lawsuit they had no hope to win at the time. Equally guilty is every politician and bureaucrat above them who failed to raise their voice or take action, as the case may be. Instead, the former head of the Post Office has moved on to the next sinecure (at NHS this time) and was awarded a CBE. Thankfully a recent High Court decision has condemned those in charge in no uncertain terms. The damage of this decade-long scandal for all of society is tremendous.
- soco 3y agoIt all started with a simple business decision: do we fix costly bugs, or send to jail a few folks?
- rwmj 3y agoThe sending people to jail option also included demanding postmasters pay £1,000s to the Post Office for money that the Post Office (wrongly) alleged had been stolen. It could have been the profitable option.
- soco 3y agoI suppose then the high compensations and prizes for the initiators of such a grand business scam^Widea appear more than justified.
- freetanga 3y agoCue the McKinsey analyst that did “the cost of an overdose death vs revenue from pushing doctors to over-prescribe” slide for the Purdue family…
- nness 3y ago
- croes 3y agoSo we need higher fines to make bug fixing the cheaper solution
- pasc1878 3y agoNo people don't expect to be caught or in this case don't expect the issues to have such an effect.
- __alexs 3y agoAre all the details of this out in the open yet? Certainly there were bugs that made it seem like there was more money coming in than actually did, but it seems like also there were capabilities in the platform that would also allow editing records to make it seem like there was actually less money coming in, which would enable skimming money out of the system almost (?) invisibly.
- thinkingemote 3y agoSome details are there, see the inquiry website. From what I gather the "covert" remote access could only add records rather than edit records. Like putting something on a message stack. However you could add a record to edit a balance I imagine. Where "records" are "messages". I don't think you could actually get money out that way only remotely. Also a record of the record would be made, you couldn't erase remote access activities on the stack. I also don't think this capability was ever suggested anywhere as being used in conjunction with a postmaster to skim money out. Maybe its possible in theory?
- forinti 3y agoIt seems to me that the Horizon system is just a simple accounting/POS thing. I can't understand how it could be built so shoddily. It doesn't even have that many clients: there are less than 12k post offices in the UK.
- MattGaiser 3y agoGovernments often have a ton of overly complex and poorly documented rules. Add in bad incentives and people put into roles they are poorly suited and it can go to hell pretty quickly. Here is an anecdote from a past job. 1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably. 2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note. 3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed. 4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions. 5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back. 6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it. 6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it. 7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.
- deleted 3y ago[deleted]
- makomk 3y agoIf I'm reading the article correctly, the headline claim is about a bug that was discovered in the original Horizon system right as it was about to be replaced with a new system, Horizon Online, at which point the bug fix would be completely useless. Other things like bugs in audit reports that continued to matter and affected the prosecution of subpostmasters were officially priorities, they just did a terrible job of handling them.
- jonatron 3y agoEven now, just by observing the use of systems at the Post Office, you can tell it's still a bad system. You can see the staff constantly mashing a button on their non-standard keyboard.
- thinkingemote 3y agoThe software company was also responsible for collecting data and for witness statements about the data for prosecutions not just for code. There were bugs that the company knew about which affected both the collection of data presented and the collecting of data. The conversations about costs related to these bugs in some instances it seems. Additionally the witness statements were written possibly with bias in them. Right now on YouTube there are Fujitsu staff being asked questions by the enquiry on this topic. Worth also saying that some bugs were discovered in a product which was due to be replaced by a new system so it was considered low priority to fix. However it impacted cases of several years before it was discovered. I think maybe that 2 Fujitsu employees are either investigation or being currently prosecuted by the met police, so there are criminal proceedings being under taken right now in relation (I guess) to these bugs in the older version of the code. There's not much info about this currently for obvious legal reasons.
- thinkingemote 3y agoedit: The police investigation was about the newer horizon not the older one so might not be related to those bugs (but maybe other ones!)
- tempaway123751 3y agoMost 'enterprisey' systems of the era were just as shoddy as Horizon. The problem was that the post office were in a position to aggressively prosecute users and decided to do so (and then the whole chain of cover ups and dissembling for years after as widely documented). All in all its a lesson in just how easy it is to cover your arse as an executive in a large company, and get away with pretty much anything. Only the most tenacious work by many people over decades has any chance of holding you accountable.
- greatgib 3y agoIf the CEO of the Post Office or Fujitsu would end in jail for a life sentence, trust me, you will never hear anymore a case where "it would hve been too costly". But now, in the end, the companies themselves will just get a financial fine at some point, not even matching what it would have costed them to do the change. And so no real consequence, no change...
- joefife 3y agoYup - and this is where we're at with car safety too. Car marque CEOs are free to say "Yeah, it'll cost hundreds of millions to fix, it's cheaper to pay out per death for a rare event". We're all worn down by this and, in essence, we operate on blood money.
- anotherhue 3y agoA*B < C
- moi2388 3y agoThis is what I don’t understand. A company violates a law, makes 10 million in profit from breaking said law, and gets a 1 million fine. Surely it ought to be: pay back all 10 million, PLUS compensation to the people from who you illegally got money, PLUS a fine, PLUS holding the actual people making the decision (because surprise, a company is just actual real people doing stuff) at the very least financially accountable.
- abhibeckert 3y agoFujitsu's didn't prosecute anyone or send anybody to jail. All they did was write software that had bugs in it. If you look at the version history of any software, you'll find it's full of bug fixes. Writing software without bugs is literally impossible. It's incredibly complex and written by "actual real people" and that means there will be mistakes. If those mistakes lead to millions in liability, then you're going to have to start paying me those millions upfront, before I write a single line of code. It's the same with a CEO - if people are criminally liable for mistakes made by people working under them... then nobody's ever going to accept a job where they delegate work to other people. The solution to this issue is not to attack people who made a mistake. It's to put systems in place where you check for mistakes. I guarantee you the jury and courts that found these people guilty were presented with "facts" such as "nobody else can remotely access the software" which, it turns out, were total bullshit. The people who presented those facts in court are the ones to blame for this. They clearly either lied or made assumptions. Both of those are serious crimes. You don't say anything in court unless you actually know it yourself. If someone else tells you something is true, you don't repeat what they said. You bring those people in as an expert witness to talk to the judge/jury directly, under oath, and you tell them not to make any assumptions either. There should have been millions spent on a full code audit done checking for bugs in the software before allowing it to be admitted as evidence in court.
- cmsj 3y ago"too costly"... remind me how much was taken as profit or handed out as bonuses in relation to Horizon?
- _fat_santa 3y agoMy burning question about this: What was the 'state of mind' of the higher-ups at the post office when they started prosecutions and then continued prosecutions into the hundreds? It's clear that the higher-ups at the post office knew these issues were caused by software rather than the postmasters, they must have known as early as 1998 or 1999. Maybe at first they must have thought that these were one-off issues and that they could just prosecute one or two postmasters, sweep the whole thing under the rug and forget about it. But that doesn't explain how they continued prosecutions into the hundreds years down the road. Like what did they think the endgame here was? Just prosecute indefinitely to cover up for the system?
- traceroute66 3y ago> What was the 'state of mind' of the higher-ups at the post office I'm not defending the post office, but I suspect you have to look at it in context. My understanding is that during that period of time (1999–2002) the Post Office's accounts took two large hits: - £570m write-off of the failed Horizon project - £1bn losses from the so-called renewal plan So perhaps the higher-ups regarded it as form of revenue management, claw-back of losses from the postmasters.
- simonh 3y agoI don't understand the state of mind of the government elected in 2011. The whole system was set up under the previous government, it would have been costly and difficult to come clean and do the right thing, but they could have blamed the entire thing on Labour and made political hay over it. Instead Ed Davey doubled down on the lying and persecuting ordinary postmasters, over an issue that until then he had no personal accountability for. This is what democracy is for, to provide the opportunity for a reset, to change course, and a mechanism for accountability of those who made the mistakes. Instead he allowed himself to be co-opted by the system.
- stefan_ 3y agoThe prosecutors were getting paid for convictions, of course: https://www.telegraph.co.uk/news/2024/01/10/post-office-executives-paid-bonuses-postmasters-conviction/ https://www.telegraph.co.uk/news/2024/01/10/post-office-exec...