8 ms·

Here some more information about what really went down, technically, behind the scenes: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-hor

Here some more information about what really went down, technically, behind the scenes: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post-offices-horizon-system-failed-a-technical-breakdown https://www.theguardian.com/uk-news/2024/jan/09/how-the-post... "One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”." (Just in case somebody says I am putting blame on developers) Obviously, the responsibility is firmly on management. People making code bugs should not be held responsible for other people going to prison for it.

onei 3y ago
From that article (and a few others) > In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems, the inquiry heard. This has always bothered me. Sure, it's possible to build APIs that audit access completely. But I can easily write code that circumvents those APIs. Code isn't like a building where the walls are impenetrable and the doors the only possible access points - we can redecorate without ever touching the door. Building in an unaudited backdoor for operators seems bad, but if you can edit the source code the backdoors are infinite.
- fbdab103 3y ago
  There should be application level auditing and database level. The people with access to managing the database level auditing should be extremely limited.
  - onetimeuse92304 3y ago
    Listen. We all know what should have been done. They were not able to do the first thing about running a transaction (ensure that one side of the transaction isn't executed multiple times). What you are saying is an obvious thing and yet it probably is well beyond the maturity of the team that was working on it.
  - willvarfar 3y ago
    Interestingly, it seems they may have built their own master-master xml-based database. It's easy to guess that they didn't add an audit feature etc.
    - mavhc 3y ago
      They were using Riposte from https://www.eschergroup.com/riposte-platform/ https://www.eschergroup.com/riposte-platform/ and Oracle. They were using dial up ISDN lines to send the data back, but Riposte didn't support that, or scale to 20k terminals, so that was all new code In general they had a distributed database that couldn't do ACID https://www.postofficetrial.com/2019/12/fisking-horizon-trial-1-meaning-of.html https://www.postofficetrial.com/2019/12/fisking-horizon-tria... https://www.computerweekly.com/news/252496560/Fujitsu-bosses-knew-about-Post-Office-Horizon-IT-flaws-says-insider https://www.computerweekly.com/news/252496560/Fujitsu-bosses... https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
  - robaato 3y ago
    Accounting 101 use journal entries to correct mistakes. Dont edit original records... Have a transaction log...
josephg 3y ago
> Obviously, the responsibility is firmly on management. People making code bugs should not be held responsible for other people going to prison for it. This is a controversial opinion but I disagree, at least to a point. Managers don’t really know what we do. The only people who really understand the engineering trade offs involved are engineers. When lives are on the line as a result of our work, we shouldn’t be insulated from the consequences of our choices. That’s not good for society and ultimately not good for us. We change the world with our work. It’s healthy to understand and own the consequences of that. The law agrees in parts. The principle of tort law is that everyone is responsible for foreseeable harm caused to your “neighbours”. Your degree of responsibility - and in turn liability - scales with how much expertise you have in the domain. An expert should have been able to foresee the harm more than a novice. The senior engineers on the team should have done better. I believe they are at fault. (IANAL, this is not legal advice, yadda yadda)
- LunaSea 3y ago
  > Managers don’t really know what we do That looks a lot like incompetent management.
  - josephg 3y ago
    Management aren’t trained engineers. They shouldn’t have to be. We’re the experts in the room. That’s literally what we’re hired for. We should act like it and stop trying to pass blame to other people.
    - jack_riminton 3y ago
      Whilst I agree in some respects, the biggest gulf to me is between companies like Stripe who successfully manage a large chunk of the words commerce (led by a brilliant engineer-CEO) and the 'IT Projects' that seem to plague the public sector here in the UK. My point is that particularly in the UK we have this culture that the Geeks should just do their job and let us Business Types take care of the rest. Countries like Germany have a much higher respect of technical people and qualifications e.g. it's very common for CEO's to have PhDs
stavros 3y ago
> People making code bugs should not be held responsible for other people going to prison for it. Why not? If I'm the single developer and seller of this app, should I not be held responsible? What if there's also a QA person? Two of each? Should the person selling or marketing the app be held responsible instead, even if they aren't technical? Why are the developers who didn't care enough to double-check their code free of responsibility?
- tpm 3y ago
  > If I'm the single developer and seller of this app In that case you are also the responsible manager or product owner. > Should the person selling or marketing the app be held responsible instead, even if they aren't technical? Of course. The person who takes the customers money is responsible for delivering the result and any warranty. > Why are the developers who didn't care enough to double-check their code free of responsibility? They are not the product owners. They don't decide what is the correct way the product works. Maybe they created the bugs because they implemented the specification exactly as written?
  - stavros 3y ago
    > They don't decide what is the correct way the product works. Of course we are, we're the ones writing it. > Maybe they created the bugs because they implemented the specification exactly as written? If your argument is "maybe they were told to write the bug in", I don't know what to tell you. If I were told to write a life-destroying bug into the software I worked on, I'd quit, because I don't want that on my conscience.
- watwut 3y ago
  In your hypothetical situation, I would blame the justice and banking system. It should not be so vulnerable or eager to believe an app made by one person, a self made "expert" on something, new theory etc. Like, you as a single seller are also responsible for making false claims. But, the justice itself should be more robust then that.
deleted 3y ago
[deleted]
mike_hearn 3y ago
> there were probably three or four who just weren’t up to it and weren’t capable of producing professional code See the other discussion on HN front page about coding tests in interviewing. Horizon was a child of the 90s. The software industry has changed a lot since then. Back in those days only Microsoft was routinely requiring programmers to code during the interview, so hiring was nearly random. Software teams often looked like that, with a tiny number of people who could write working code covering for many more who just couldn't at all. The Daily WTF dates from this time. It's full of stories like the Brillant Paula Bean: https://thedailywtf.com/articles/The_Brillant_Paula_Bean https://thedailywtf.com/articles/The_Brillant_Paula_Bean You don't hear stories like that much anymore. The industry settled on testing concrete skills before hiring, and that washed out a lot of the people who previously managed to get hired into projects despite not being able to code properly. Whether Fujitsu does it or not now, no clue. But that situation wasn't unusual back then.
- _fat_santa 3y ago
  I would say another child of the 90's is management / users being too trusting of technology which IMO is the real reason for this mess. These days, everyone knows that software can make mistakes and one shouldn't rely on a single system / data point when it comes to critical decisions (ie. sending someone to jail). I don't think that was so much the case in the 90's, attitudes seemed to have been that these sorts of systems don't make mistakes, and therefore can always be trusted. I look at this as the 90's version of "The Titanic is an unsinkable ship" EDIT: I would add that you can rely on systems / single data points for some decisions. How I see it is trust in a single point of data goes down as the criticality of the decision goes up. If someone sends you a message to meet them for lunch downstairs, that decision has low criticality and therefor you can rely on a single data point / application to make the decision. However for more critical decisions (ie. should we send this person to jail for stealing money), trust in any system should be low by default and a consensus from multiple data sources is required.
- commandlinefan 3y ago
  > that situation wasn't unusual Um, I'm working as a professional computer programmer today, in 2024, and I can assure that programming continues to be replete with incompetent programmers who are not capable of producing professional code.

onei 3y ago

From that article (and a few others) > In fact, staff at Fujitsu, which made and operated the Horizon system, were capable of remotely accessing branch accounts, and had “unrestricted and unaudited” access to those systems, the inquiry heard. This has always bothered me. Sure, it's possible to build APIs that audit access completely. But I can easily write code that circumvents those APIs. Code isn't like a building where the walls are impenetrable and the doors the only possible access points - we can redecorate without ever touching the door. Building in an unaudited backdoor for operators seems bad, but if you can edit the source code the backdoors are infinite.

fbdab103 3y ago
There should be application level auditing and database level. The people with access to managing the database level auditing should be extremely limited.
- onetimeuse92304 3y ago
  Listen. We all know what should have been done. They were not able to do the first thing about running a transaction (ensure that one side of the transaction isn't executed multiple times). What you are saying is an obvious thing and yet it probably is well beyond the maturity of the team that was working on it.
- willvarfar 3y ago
  Interestingly, it seems they may have built their own master-master xml-based database. It's easy to guess that they didn't add an audit feature etc.
  - mavhc 3y ago
    They were using Riposte from https://www.eschergroup.com/riposte-platform/ https://www.eschergroup.com/riposte-platform/ and Oracle. They were using dial up ISDN lines to send the data back, but Riposte didn't support that, or scale to 20k terminals, so that was all new code In general they had a distributed database that couldn't do ACID https://www.postofficetrial.com/2019/12/fisking-horizon-trial-1-meaning-of.html https://www.postofficetrial.com/2019/12/fisking-horizon-tria... https://www.computerweekly.com/news/252496560/Fujitsu-bosses-knew-about-Post-Office-Horizon-IT-flaws-says-insider https://www.computerweekly.com/news/252496560/Fujitsu-bosses... https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
- robaato 3y ago
  Accounting 101 use journal entries to correct mistakes. Dont edit original records... Have a transaction log...

josephg 3y ago

> Obviously, the responsibility is firmly on management. People making code bugs should not be held responsible for other people going to prison for it. This is a controversial opinion but I disagree, at least to a point. Managers don’t really know what we do. The only people who really understand the engineering trade offs involved are engineers. When lives are on the line as a result of our work, we shouldn’t be insulated from the consequences of our choices. That’s not good for society and ultimately not good for us. We change the world with our work. It’s healthy to understand and own the consequences of that. The law agrees in parts. The principle of tort law is that everyone is responsible for foreseeable harm caused to your “neighbours”. Your degree of responsibility - and in turn liability - scales with how much expertise you have in the domain. An expert should have been able to foresee the harm more than a novice. The senior engineers on the team should have done better. I believe they are at fault. (IANAL, this is not legal advice, yadda yadda)

LunaSea 3y ago
> Managers don’t really know what we do That looks a lot like incompetent management.
- josephg 3y ago
  Management aren’t trained engineers. They shouldn’t have to be. We’re the experts in the room. That’s literally what we’re hired for. We should act like it and stop trying to pass blame to other people.
  - jack_riminton 3y ago
    Whilst I agree in some respects, the biggest gulf to me is between companies like Stripe who successfully manage a large chunk of the words commerce (led by a brilliant engineer-CEO) and the 'IT Projects' that seem to plague the public sector here in the UK. My point is that particularly in the UK we have this culture that the Geeks should just do their job and let us Business Types take care of the rest. Countries like Germany have a much higher respect of technical people and qualifications e.g. it's very common for CEO's to have PhDs

stavros 3y ago

> People making code bugs should not be held responsible for other people going to prison for it. Why not? If I'm the single developer and seller of this app, should I not be held responsible? What if there's also a QA person? Two of each? Should the person selling or marketing the app be held responsible instead, even if they aren't technical? Why are the developers who didn't care enough to double-check their code free of responsibility?

tpm 3y ago
> If I'm the single developer and seller of this app In that case you are also the responsible manager or product owner. > Should the person selling or marketing the app be held responsible instead, even if they aren't technical? Of course. The person who takes the customers money is responsible for delivering the result and any warranty. > Why are the developers who didn't care enough to double-check their code free of responsibility? They are not the product owners. They don't decide what is the correct way the product works. Maybe they created the bugs because they implemented the specification exactly as written?
- stavros 3y ago
  > They don't decide what is the correct way the product works. Of course we are, we're the ones writing it. > Maybe they created the bugs because they implemented the specification exactly as written? If your argument is "maybe they were told to write the bug in", I don't know what to tell you. If I were told to write a life-destroying bug into the software I worked on, I'd quit, because I don't want that on my conscience.
watwut 3y ago
In your hypothetical situation, I would blame the justice and banking system. It should not be so vulnerable or eager to believe an app made by one person, a self made "expert" on something, new theory etc. Like, you as a single seller are also responsible for making false claims. But, the justice itself should be more robust then that.

deleted 3y ago

[deleted]

mike_hearn 3y ago

> there were probably three or four who just weren’t up to it and weren’t capable of producing professional code See the other discussion on HN front page about coding tests in interviewing. Horizon was a child of the 90s. The software industry has changed a lot since then. Back in those days only Microsoft was routinely requiring programmers to code during the interview, so hiring was nearly random. Software teams often looked like that, with a tiny number of people who could write working code covering for many more who just couldn't at all. The Daily WTF dates from this time. It's full of stories like the Brillant Paula Bean: https://thedailywtf.com/articles/The_Brillant_Paula_Bean https://thedailywtf.com/articles/The_Brillant_Paula_Bean You don't hear stories like that much anymore. The industry settled on testing concrete skills before hiring, and that washed out a lot of the people who previously managed to get hired into projects despite not being able to code properly. Whether Fujitsu does it or not now, no clue. But that situation wasn't unusual back then.

_fat_santa 3y ago
I would say another child of the 90's is management / users being too trusting of technology which IMO is the real reason for this mess. These days, everyone knows that software can make mistakes and one shouldn't rely on a single system / data point when it comes to critical decisions (ie. sending someone to jail). I don't think that was so much the case in the 90's, attitudes seemed to have been that these sorts of systems don't make mistakes, and therefore can always be trusted. I look at this as the 90's version of "The Titanic is an unsinkable ship" EDIT: I would add that you can rely on systems / single data points for some decisions. How I see it is trust in a single point of data goes down as the criticality of the decision goes up. If someone sends you a message to meet them for lunch downstairs, that decision has low criticality and therefor you can rely on a single data point / application to make the decision. However for more critical decisions (ie. should we send this person to jail for stealing money), trust in any system should be low by default and a consensus from multiple data sources is required.
commandlinefan 3y ago
> that situation wasn't unusual Um, I'm working as a professional computer programmer today, in 2024, and I can assure that programming continues to be replete with incompetent programmers who are not capable of producing professional code.