4 ms·

This is not a story of a failure in software engineering. Shit happens, no system is 100% reliable. The failure lies in how management handled it (pretend there

by cedws 2y ago

This is not a story of a failure in software engineering. Shit happens, no system is 100% reliable. The failure lies in how management handled it (pretend there’s no issue, don’t launch an internal investigation, blame others). No doubt management also oversaw the development of the system and rushed it to production. A healthy culture should accept failure as inevitable and learn from it when it occurs. It should also listen to the people who know best: the engineers who built the thing. You know, like the aerospace industry. The damage that morons in suits do in pursuit of their bonus cannot be overstated.

nuc1e0n 2y ago
There's also the issue where public software engineering related projects are too broad in scope and poorly defined. Rather than making small projects that are useful and then expanding upon them the trend is to specify massive nationwide databases and then just throw money at them, often using 'Big bang' deployments. These are so called because you deploy them and then there's a large explosion ;)
bobnamob 2y ago
I'm not sure I entirely agree in this circumstance. Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy. https://en.wikipedia.org/wiki/British_Post_Office_scandal#Problems_with_the_software https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr... Transaction idempotency is such a basic property for a financial system that I struggle to believe that Horizon was tested in any meaningful way. "the engineers who built the thing" (Gareth Jenkins) are also under investigation for perjury. [1] Horizon is still in use, in its buggy state, with replacement scheduled for 2030....
- cedws 2y ago
  >Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy. The developers were just doing their job. It's management's responsibility to construct a functioning system of checks and balances and understand the limitations of their systems, both of which they failed to do. If it weren't for their hubris the fundamental issues with Horizon could have come to light much earlier. Let's also not forget that the reason executives are compensated well is for them to take accountability in situations like this.
  - acdha 2y ago
    I think you’re trying to fit this into an exclusive OR when it really is AND. Unless the developers strongly warned management about the lack of idempotency and were ordered to continue anyway, they share the blame for those bugs even if they were not part of the conspiracy to cover them up (this also goes for anyone involved in those hot fixes trying to patch up errors without telling the customer). More broadly, your idea that this is solely a management problem is how we end in situations where developers are being told to unquestioningly code some design exactly as given, which never works. You don’t get professional judgement if you don’t accept responsibility, too.
danaris 2y ago
It can be more than one thing at a time. But I would also say that that kind of toxic management is absolutely a part of "software engineering culture". How many horror stories do people on here have of managers who care nothing about the quality of the product, only meeting the deadline so they can get their bonus? "Software engineering culture" is way, way more than just "how write good code." It includes how we work, how we manage/are managed, how we advocate for ourselves, or fail to do so, and much more. It certainly includes the very common resistance to unions among programmers, and assuming this was caused by management pushing a known-bad product out the door, a strong union would have (at least potentially) been able to stand up to such demands.
- gorft 2y ago
  Programmers who get paid asstons of money don't want to admit that these are also fundamentally business failures, partially because if they were fixed, they probably wouldn't get six figures for styling HTML buttons anymore.
marcosdumay 2y ago
> The failure lies in how management handled it The thing literally went into a criminal court as evidence, and was "presumed correct" in a way that overloaded any technical or reasonable discordance. The largest failure here was from the judges and lawyers. The software failure isn't even relevant.
gorft 2y ago
I'm so tired of seeing "no system is 100% reliable" being used to excuse the most fundamental failures in this industry. Nobody's asking for 100% perfection, we're asking for the people writing the software to do X to actually do X correctly in the first place. How about, I don't know, 95% correctness? Is that too onerous?

by cedws 2y ago

nuc1e0n 2y ago

There's also the issue where public software engineering related projects are too broad in scope and poorly defined. Rather than making small projects that are useful and then expanding upon them the trend is to specify massive nationwide databases and then just throw money at them, often using 'Big bang' deployments. These are so called because you deploy them and then there's a large explosion ;)

bobnamob 2y ago

I'm not sure I entirely agree in this circumstance. Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy. https://en.wikipedia.org/wiki/British_Post_Office_scandal#Problems_with_the_software https://en.wikipedia.org/wiki/British_Post_Office_scandal#Pr... Transaction idempotency is such a basic property for a financial system that I struggle to believe that Horizon was tested in any meaningful way. "the engineers who built the thing" (Gareth Jenkins) are also under investigation for perjury. [1] Horizon is still in use, in its buggy state, with replacement scheduled for 2030....

cedws 2y ago
>Normally I'd give the developers the benefit of the doubt. But the sheer number of issues, and how fundamental some of them ~were~ are[1] leave me little room for sympathy. The developers were just doing their job. It's management's responsibility to construct a functioning system of checks and balances and understand the limitations of their systems, both of which they failed to do. If it weren't for their hubris the fundamental issues with Horizon could have come to light much earlier. Let's also not forget that the reason executives are compensated well is for them to take accountability in situations like this.
- acdha 2y ago
  I think you’re trying to fit this into an exclusive OR when it really is AND. Unless the developers strongly warned management about the lack of idempotency and were ordered to continue anyway, they share the blame for those bugs even if they were not part of the conspiracy to cover them up (this also goes for anyone involved in those hot fixes trying to patch up errors without telling the customer). More broadly, your idea that this is solely a management problem is how we end in situations where developers are being told to unquestioningly code some design exactly as given, which never works. You don’t get professional judgement if you don’t accept responsibility, too.

danaris 2y ago

It can be more than one thing at a time. But I would also say that that kind of toxic management is absolutely a part of "software engineering culture". How many horror stories do people on here have of managers who care nothing about the quality of the product, only meeting the deadline so they can get their bonus? "Software engineering culture" is way, way more than just "how write good code." It includes how we work, how we manage/are managed, how we advocate for ourselves, or fail to do so, and much more. It certainly includes the very common resistance to unions among programmers, and assuming this was caused by management pushing a known-bad product out the door, a strong union would have (at least potentially) been able to stand up to such demands.

gorft 2y ago
Programmers who get paid asstons of money don't want to admit that these are also fundamentally business failures, partially because if they were fixed, they probably wouldn't get six figures for styling HTML buttons anymore.

marcosdumay 2y ago

> The failure lies in how management handled it The thing literally went into a criminal court as evidence, and was "presumed correct" in a way that overloaded any technical or reasonable discordance. The largest failure here was from the judges and lawyers. The software failure isn't even relevant.

gorft 2y ago

I'm so tired of seeing "no system is 100% reliable" being used to excuse the most fundamental failures in this industry. Nobody's asking for 100% perfection, we're asking for the people writing the software to do X to actually do X correctly in the first place. How about, I don't know, 95% correctness? Is that too onerous?