14 ms·

What Went Wrong?

405 points by headalgorithm 5y ago

foobiekr 5y ago
Part of my job is to help the executives that I report to understand why things went wrong from the security perspective in our business unit. These are purely internal discussions, not even investigations. There are no penalties, but really, for things as egregious as hard coded passwords. As will become clear in a moment, the fact that my executives care is quite unusual. Culturally the result is coverups and lies. Engineers lie, managers lie, test people lie, directors lie, senior directors lie, vice president lie, external interesting teams are negotiated into minimizing certain critical failures, and so on. Managers don’t want to hear it so that they can’t be accused of lying, vice presidents don’t wanna know, SVP’s just want green squares on the cross-BU PowerPoint. This is internal discussion of revenue impacting incidents. Do you know what executives do care about? Revenue. Lost deals. If the people who care about money, including the account teams, don’t care about security and severe quality issues enough to be honest enough to get to improvement, how could an external board accomplish anything for those very few incidents that actually become publicly visible? This isn’t like the NTSB; I spent my life reading NTSB accident reports. They have actual real authority, there are potentially issues that might impact someone more than being caught distorting things.
- izacus 5y ago
  I also wonder if "blameless postmortem" culture perhaps actively works against preventing these kind of incidents. It doesn't seem that anyone in IT is ever responsible for damage they cause. But yes, lying, "not seeing" and covering documentation is pretty much standard corporate behaviour I've seen around plenty of companies as well.
  - foobiekr 5y ago
    I no longer believe in blameless post mortem as a general rule. I have, through experience, come to believe that the contexts where blameless post mortems work are the contexts where literally anything works because they are organizations that have high hiring bars and high expectations. My current employer is not one of them; we are a mountain of mediocrity and all blameless post mortems do is act as an excuse to avoid raising the bar.
    - jolux 5y ago
      The principle of blameless postmortems is not supposed to absolve anyone of the responsibility to change anything, it’s supposed to foreground that serious failures are organizational failures first and foremost, because it’s the organization that has an obligation not to fail, not individuals, who fail all the time as a rule.
      - tialaramex 5y ago
        Exactly this. I spend a bunch of time reading accident reports from agencies like RAIB and MAIB, but my real jobs have been closer to the Web PKI and thus m.d.s.policy Back in 2015, Symantec's CA issued some certificates that shouldn't have existed, including for names owned by Google. What's wrong there? Well, a blameless postmortem would probably tell you that your processes and procedures are bad, you are creating bogus certificates to "test" a real CA whose certificates are actually trusted in the real world. Need better processes, training, oversight to ensure things improve. What did Symantec do when they were caught? They fired the low-level employees who conducted the tests and wrote a blog post "A Tough Day as Leaders" which blamed the fired employees for getting it wrong. Some leadership (the blog post of course no longer exists although I assume it's archived somewhere). Less than two years later, Symantec is back in trouble again because an RA they've worked with has been issuing certificates, using their CA infrastructure (and thus, from our point of view they were issuing these certificates even if they unaccountably believed this isn't their fault) that should not exist. This time Symantec's bosses blame not only low-level employees, but also auditors, bosses at the Korean RA, and anybody else they can think of... except themselves. This is a gross failure of leadership. Once upon a time a US President said "The buck stops here", but Donald Trump was very clear, "The buck stops with everybody" and "I take no responsibility" and it seems Symantec's leadership were made in that image. They quit the CA business rather than do what it would take to fix the problem. If you conduct a "postmortem" after an incident, then "Nobody was to blame and nothing needs to change" is almost certainly just as much the wrong outcome as "It's Jane's fault, fire her". I mentioned I read MAIB reports. One MAIB report sticks out, after many years, in the following way: Unlike every other MAIB report I've read, this one has No recommendations. Someone died, and yet there is nothing to recommend. Why not? Well, the cause is very simple, two men on a fishing boat took a lot of heroin, and their boat crashed, it sank and one died. No need to recommend that you shouldn't take heroin while operating a fishing boat since heroin is an illegal drug already and operating a vessel under the influence of drugs or alcohol is already a crime too. If your next "blameless portmortem" doesn't have any recommendations, ask yourself, was what happened already a crime? Are the people involved dead or in prison and so either way beyond the value of recommending a different course of action next time? No? Then we need to recommend how to actually avoid it happening again.
  - nanis 5y ago
    In my negative experiences, "blameless" turned in to "nobody did anything wrong" which, of course, undermines the whole point of finding out what actually happened so we can see if there is a thing we can do to reduce the likelihood of it happening again. Sometimes, the root cause is indeed someone with the privilege but not the good sense ignoring warning signs. If we can't identify that problem, then we can't improve our odds for the next time.
    - joshuamorton 5y ago
      A valid blameless answer then is "remove the privilege" and yes, despite whatever objections you'll raise, this is possible. Difficult, but possible. Like, even in the case of the extreme example of someone deciding to intentionally harm the company. You fire them, but then what, how do you prevent the next person to go rogue from causing equivalent harm?
      - tonyedgecombe 5y ago
        This is what slows big companies down. They accrete so many of these rules and restrictions that people can't do their work.
        rocqua 5y ago
        I believe more of these incidents should conclude: "this is better to accept as the cost of doing business than to try to spend money to fix".
        concordDance 5y ago
        Or people start ignoring the rules in order to do their work. Which has it's own problems (particularly as rules normally aren't divided into "important" and "unimportant".
        jbuhbjlnjbn 5y ago
        This is the thorn in my foot. I refuse to ignore company rules, and also comment on gross negligence, which more often then not means I am the quarreler, not the good engineer in the eyes of coworkers and bosses.
- slyall 5y ago
  I think you are overestimating the importance of "revenue impacting incidents" to company employees. If the company makes a couple of million extra or less this year it doesn't effect the majority of workers. Their bonus isn't going up or down etc. And remember this incident has already happened. By contrast if a report comes out blaming the loss on a worker, department or division then that could have major consequences. No matter how "blameless" it is, come next round of bonuses, promotions or layoffs everybody knows it'll be factored into the decisions. So people don't have an incentive to make themselves look bad and unlike with the NTSB there is no legal powers or fear of causing deaths behind the investigation.
  - laurent92 5y ago
    I understand, but it sounds like we are digging ourselves into the same hole as USSR workers who were not incentivized to deliver working products, when we do that. It’s a civilizational peril. How do we solve cooperation at large scale? Is the only way to watch large companies accumulate bored employees and constantly recreate “the small guy”, the startup, which will finally make things right, until they become too big to be incentivized?
    - WalterBright 5y ago
      That's exactly how it works, and why big companies do not grow until they consume the entire world. They become complacent and unable to change, and a startup eventually takes their business away.
      - saalweachter 5y ago
        It's funny that people expect management to function better because it's employee-owners of a capitalist corporation, instead of feudal lords or members of a communist central committee.
        laurent92 5y ago
        The feudal owners in a startup situation are the marketplace owners who organize all startups to compete. The day we lose is the day all of those marketplaces themselves are cannot be replaced.
- Aeolun 5y ago
  I care about the issues (as an angineer), but my experience with raising any security issues is that it results in a lot of pain for me personally. You report something to the ‘security’ team, and suddenly you’re responsible for doing all the work, as well as the prime suspect in an investigation ‘why didn’t you fix this before, since when did we know about this?!’ I’m absolutely incentiviced to just let any issue lie until it is discovered, because then it actually is the security teams problem.
  - arch-ninja 5y ago
    I was expelled from a university for reporting security flaws, the solution we all seek is the simplest one and for administrators it's easiest to hurt the people making noise which commonly results in the noise going away. fight-or-flight response at it's finest. Edit: reminder that for the "common folk" these "security issues" are not a 5-minute fix, they are fundamentally different realities which require every machine on the network to be re-checked before they can be used again. There is a clear communication failure between the ones who want security and the ones who want "security".
    - wkavey 5y ago
      Would like to hear the story behind this one.
      - Game_Ender 5y ago
        It’s common, a friend almost got expelled for reporting a flaw in the universities ID card system. That friend did not brake anything, they did not sneak into any protected spaces. Just discovered and validated a flaw and then reported it.
      - jacobolus 5y ago
        A high school friend (2 decades ago) told a teacher that the system keeping track of student grades was vulnerable to attack. The teacher asked the student to demonstrate by attacking the system and adjusting one of his test scores down by 1 point. My friend obliged, and the teacher reported the vulnerability to the administration. An administrator threatened my friend with expulsion, but when he proposed to go public with his story in response, they decided they wouldn't expel him. The resolution was "please don't tell anyone", and the vulnerability was never fixed.
- jjtheblunt 5y ago
  Do bureacracies essentially provide complexity in which dysfunction readily hides?
- oceanghost 5y ago
  As an engineer I found if I didn't lie and pretend to support the false reality of the managers above me, I wouldn't have that job long. Once we had a very large effort, creating a new hardware and software platform for the companies products. The CEO demanded it be done in 9 months, our estimate was accurate almost to the day-- 18 months. Politically it was easier for the PMs to agree with 9 months and then have 9 one-month delays it took, instead of getting fired for insisting on a good faith estimate. What do you think the security in that product was like? Security directly conflicts with the business model of cutting every corner possible and loading engineers as much as possible. A real security program would have to start with reasonable hours and goals.
- hollerith 5y ago
  I got curious about what legal authority the NTSB has. Here is what I found: >like a cop, the NTSB can secure an accident scene and keep others away. It can examine the aircraft wreckage. It can have aircraft parts tested to help determine why the accident happened. It can even subpoena evidence. . . . If there is a lawsuit concerning the crash, the NTSB will not get involved. Not only is the NTSB’s report of the accident’s probable cause inadmissible, but the NTSB investigators are prohibited by law from testifying in court, even if they are served with a subpoena. Source: https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-limited-role-after-an-aircraft-accident/ https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-li... >The NTSB may issue a subpoena, enforceable in Federal District Court . . . For purposes of the Health Insurance Portability and Accountability Act of 1996 (HIPAA), Public Law 104-191, and the regulations promulgated by the Department of Health and Human Services, 45 CFR 164.501 et seq., the NTSB is a “public health authority” to which protected health information may be disclosed by a HIPAA “covered entity” without the prior written authorization of the subject of the records. Source: https://www.law.cornell.edu/cfr/text/49/831.59 https://www.law.cornell.edu/cfr/text/49/831.59
- cratermoon 5y ago
  > There are no penalties, but really, for things as egregious as hard coded passwords Is this sentence missing a word or phrase?
- ganafagol 5y ago
  Let's not conflate security and safety here. The boards hailed by the article are all about investigating safety failures, and so is the advocated invetigation board. Security is a different beast. It's sub-par in many airplanes/power plants/... too.
- pomian 5y ago
  That's very disappointing. I've always brought up everyone up to think that of all the professions, you can always trust an engineer. They deal in facts. They don't lie. Perhaps it's as simple that in the real world, engineers can't lie. Because the world wouldn't work is they did.
- indymike 5y ago
  > Do you know what executives do care about? Revenue. Lost deals. This is often why engineering needs aren't covered. Things are presented as risks and expense, instead of in terms of revenue. A $10,000 expense actually wipes out $10,000 in net profit, so you need to generate revenue sufficient to create $10,000 in net profit. Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. Finally, there is how risk frames what you present. If you come at me with "this might happen" the other side of the coin is "this might not happen", and most managers will avoid the certainty of expense for the possibility of an expense. If you are working with a CEO, valuation is where it's at. Try to understand the swing in company valuation based on profit. Present to the CEO like this: "X is highly likely to happen within Y months, and it will impact $Z in net profits, leading to a change in company valuation of up to $N." Make sure $N is enough to matter. You just took 98% of the arguments against taking action off the table.
  - an_opabinia 5y ago
    > Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. I don’t think you are interpreting things correctly here. This is both counterintuitive and wrong.
    - indymike 5y ago
      I'm pretty sure I'm not wrong. Source: I own three companies. The $100k and $2.8M were just examples (very thin net profit). It's equally likely you could have $100k in expense and need $200-$400K in sales to offset it if you are in a very high margin business. Regardless, what matters is that the unexpected expense will reduce profit, every time, and by framing risk in terms of profit you will help your CEO make better decisions.
      - an_opabinia 5y ago
        You’re not “framing the risk in terms of profit.” You’re talking about a set of equations in your head, that only you know, that come up with weird results like “it takes way more revenue to ‘offset’ an expense” because you’re holding “net” or “gross” or whatever margin constant. Which is really confusing and surprising and not how anyone I’ve ever met talks about this stuff, and they’d be equally puzzled and I don’t think would experience an illuminating or aha moment or whatever. Personally I found it a little interesting, I get what you are going for, but it’s so strange and not really useful or true.
MrStonedOne 5y ago
In washington state, the state superior court ruled the police department was not liable for the impound fee paid by somebody who had their car impounded for 90* days for driving on what the computer reported was a suspended license, because they are exempt from mistakes from trusting their own computer system. This was the second time the department had wrongfully impounded his car and they made no attempt to fix the mistake from the first time, this didn't impact the ruling. Its gonna get much worse before it get any better.
- spaetzleesser 5y ago
  It will get much worse I think. More and more companies are hiding behind algorithms and other computer systems while cutting support staff. If you are wronged you have nobody to talk to and they make no effort to correct the situation. the only recourse is a lawsuit which is way too expensive for most people. And even when they are caught the fines are usually only nominal. I think we are building up the ultimate faceless bureaucracies.
  - Aeolun 5y ago
    If people implicitly trust the system (or at least don’t get blamed for it’s failures) it makes it much easier for hackers though.
  - swiley 5y ago
    At that point you might as well just hook everything up to etherium; the judge is redundant.
  - AlbertCory 5y ago
    You're certainly right that lots of very large companies go to every conceivable length to keep you from ever finding a human being, even for an online chat, forget a live phone conversation. Their "contact us" page is nothing but an FAQ. Is legislation or regulation the answer? That would be unfortunate. The government rarely makes things better, but to be honest, those accident investigation boards probably DO prevent management from sweeping things under the rug. Some kind of consumer ratings for Quality of Service will have to spring up. You can now find ratings of airlines for their on-time record and likelihood of losing your baggage. We need something similar for web companies.
dgb23 5y ago
Large amounts of money spent on government systems that never ship is a tragedy, but software projects like these tend to have a lot of open questions. We understand software development often as a discovery process (evolving requirements), especially if they are large or disruptive. So one critical output of any such project has to be knowledge that can be built upon, as in open, clearly specified and written papers. This should be done regardless of whether it failed or didn't fail.
- Aeolun 5y ago
  I can tell you now that nobody including the current engineers, can work with the requirement documents that are being generated over the course of our multi year system transformation project. Of course, it’s still an improvement over the legacy system which has none.
ithkuil 5y ago
Well written article
hamilyon2 5y ago
The industry fails to listen to lessons written in "Mythical man month" - 50 years from now. Half of a century ago. Of course some reports on why systems are being designed and coded poorly won't change anything. We know why, we just ignored the knowledge to the point of absurdity.
- torgard 5y ago
  Companies could be held liable for gross misconduct. Although GDPR is not exactly a shining example of IT regulation, I think it's a good example of liability. Companies get fined for breaking GDPR. Governmental projects should have similar requirements in place, and companies and people should be held accountable for breaking them.
  - a1369209993 5y ago
    > Companies get fined for breaking GDPR. Well, no, they don't, or least when they do it's only pittance fines. Google, Facebook, et al are not bankrupt.
    - torgard 5y ago
      Here's a website tracking GDPR enforcement: https://www.enforcementtracker.com/ https://www.enforcementtracker.com/ According to that website, one of Google's 10 fines is the highest fine ever, at €50 million. Hilariously, in a dystopian kind of way, they were fined €28 last year. Not as in millions, but twenty-eight euros. Facebook only appears on the list once, with a paltry €51,000. But in any case, the point of GDPR is not to bankrupt companies.
      - CogitoCogito 5y ago
        > But in any case, the point of GDPR is not to bankrupt companies. It _should_ be. Well okay that's a bit hyperbolic, but what I mean by that is that the point should be to _change_ behavior and if the fines necessary or the resulting change in business model leads to bankruptcy then that should be totally fine. Companies who cannot operate legally shouldn't continue operating.
verytrivial 5y ago
I agree with nearly everything in this artictle but the following question stumped me: when exactly would a software disaster investigation board be employed? Plane goes down, train goes off rails or passes signal at danger, easy. But at what exact what point did the UK postmaster system "fail" enough for an investigation?
- andersource 5y ago
  I would say at latest when people convicted because of it had their names cleared - https://www.bbc.com/news/business-56859357 https://www.bbc.com/news/business-56859357
- monkeydreams 5y ago
  > But at what exact what point did the UK postmaster system "fail" enough for an investigation? The moment it came to light that postmasters were being improperly convicted? The moment it came to light that some improperly convicted postmasters committed suicide?
- duckhelmet 5y ago
  > at what exact what point did the UK postmaster system "fail" enough for an investigation? There never was in independent investigation of the IT system. "Justice Lost In The Post" https://www.private-eye.co.uk/special-reports/justice-lost-in-the-post https://www.private-eye.co.uk/special-reports/justice-lost-i...
- chrischapman 5y ago
  This is all pretty new in the world. It took many years for aircraft investigations to get real professional at it. How about we start with a simple algorithm like this: 1. Check if software is involved in the incident. 2. If it is, carry out an internal review to evaluate the possibility that the software may be a cause of the incident. 3. If it is, hand over to an independent review board. 4. Involve lawyers if and only if, the independent review board recommends it. That was an off-the-cuff attempt at a suggestion. There is probably a much better way. Feel free to suggest alternatives. I have no objection to lawyers getting involved. I just object to them suppressing the facts to protect their clients and preventing an independent investigation. It's impossible to cover up that an aircraft has crashed. Lawyers involved in an aircraft accident on behalf of the manufacturer (or operator) are basically doing 'damage limitation'. Lawyers involved in a software incident on behalf of the manufacturer (or operator) are basically doing 'evidence suppression'. They should stick to 'damage limitation'.
nixpulvis 5y ago
I would gladly work for a prolific IRB.
deleted 5y ago
[deleted]
ashton314 5y ago
> Personal information is the helium of IT systems—it leaks out of every crack or imperfection faster than seems possible. Might as well call it the hydrogen of IT systems—get too much of it concentrated in one place, and all it takes is one little spark for it all to go up in flames. Boom!
- alisonkisk 5y ago
  It doesn't destroy anything, it just leaks.
openthc 5y ago
In Washington State we have a system to track cannabis, the enforcement officers are supposed to be able to get reports from this system. The system is super buggy and also doesn't have meaningful reports. So there is a secondary system for officers to export to Excel documents. In one of the trainings they've been instructed to look for anomalies -- not real analisys, not even a pivot table. One thing they find is "negative quantities" -- but how can that be? (hint: it's bugs in the tracking software). Then enforcement shows up at the cannabis business to audit these negative numbers (or demand the business try to correct the data (which they cannot due to bugs)). So, crappy software gets law enforcement officers to basically review data "anomalies" created by bugs by visiting a business. The second most expensive method for data sanatization I can imagine. It's a poor use of their time and disruptive to the business. The system in WA is so buggy that the agency has opted to freeze the software rather than try to fix the issues. The future of government software is bleak -- so long as they keep using closed source packages from low-cost bidders.
- laurent92 5y ago
  Why isn’t all software created for the government required to be open-source? Would that really drive the costs up, if the providers don’t have the choice?
  - openthc 5y ago
    The vendor claimed that if the code was out it would be a security risk. The agency claims the vendor needs to protect their intellectual property rights. We have (some) visibility into other things our taxes pay for -- the software should absolutely be one -- expecially the regulatory compliance ones that drive enforcement action. Edit: also, they were breached anyway shortly after launch (2018) and then an email went around offerting to sell the code and data from their entire system.
    - laurent92 5y ago
      And it is true: If their code were out, it would be painfully obvious that it is full of vulnerabilities. Security by obscurity! I know that because I’m myself afraid of making my old app open-source… I wish I had done a bug bounty from day #1. Bug bounties are a killer tool. I wish some lawyers had made a license like “Not open-source but here’s the source for vulnerability research.”
ldarby 5y ago
It's known what went wrong, computerphile has a video with some details: https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10 but it doesn't address any of the judicial and cultural fails, that's what needs to be fixed. Software bugs are a fact of life, people know this, except the judges in this case apparently.
- HarryHirsch 5y ago
  Bugs are a fact of life because of sloppy practices. The experience from SQLite is instructive, after a testsuite had been written, matters improved immensely. Why was the testsuite written? Because it was in the list of requirements from the client, aerospace standards demand that every possible branch is covered by a test. We choose to write bad software.
- II2II 5y ago
  One could argue that faults in the engineering and construction are also a fact of life, yet that doesn't mean we excuse them and it doesn't mean that assume that a failure is due to those faults. Investigations are performed in order to ascertain the truth. I think the authors comparison to the historical development of trains is appropriate. Investigating IT failures wasn't as important 50 years ago because IT infrastructure was not as critical. Investigating IT failures today is critical because the functioning of society depends upon it.
  - ldarby 5y ago
    I don't think you (or the sibling comment) got my point. No where am I "excusing" bugs. Civil engineering bugs are indeed a fact of life too, just check the god damn news. What I'm saying, is that I think where PHK says "nobody sat down and documented precisely what went wrong", I think he is just wrong. The guy in the Computerphile video, Steven Murdoch wrote this article: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with..., which has further details of the bugs, e.g. "There is a window of time between a user printing and cutting-off a report. If another user was to perform a transaction during that window, that transaction may not show on the report." which is from the long and very detailed judgment: https://www.benthamsgaze.org/wp-content/uploads/2021/07/Bates-and-Ors-v-the-Post-Office-Ltd-2019-EWHC-3408-QB.pdf https://www.benthamsgaze.org/wp-content/uploads/2021/07/Bate... The problem that needs investigating here is the miscarriage of justice, apparently the post office was aware of the bugs at the same time as prosecuting people for falling victim to the bugs, in order to save face about their IT decision.
Scoundreller 5y ago
> In 2017 the motor of an airplane exploded over the southern part of the Greenland icecap. Part of the engine landed on the ice while the plane continued to the first suitable airport way up north in Canada. eh, Happy Valley-Goose Bay isn't that far north as far as Canada goes. 53 degrees north. The actual droppings in Greendland were around 61 degrees N. Nuuk would have been ~60% closer, but not a chance it could handle an A380.
ChrisMarshallNY 5y ago
I really enjoyed this. Like most things, it's a matter of scale. If a train derails, we call in the NTSB, but they don't investigate car crashes. The issue that I see, is that the software industry seems to be absolutely obsessed with scale. Small applications are actively sneered at. Go big, or go home. So that means that every accident is a train wreck.
- PhantomGremlin 5y ago
  If a train derails, we call in the NTSB, but they don't investigate car crashes. They most certainly do investigate car crashes. They just don't have the resources to investigate very many of them. https://www.cnbc.com/2021/05/10/ntsb-releases-preliminary-report-on-fatal-tesla-crash-in-spring-texas.html https://www.cnbc.com/2021/05/10/ntsb-releases-preliminary-re...
Scoundreller 5y ago
Would also like to point out the fantastic videos created by the US Chemical Safety Board: https://www.youtube.com/user/USCSB https://www.youtube.com/user/USCSB
overgard 5y ago
So I wonder, in this situation what changes where the accident is averted? It's fine to demand accountability but what specifically is being monitored? Because I don't think there's a real answer yet. They can say IT needs to do a better job. Obviously. But in what dimension? Theres a lot of "you screwed up!" but very little "heres the fix"
Aeolun 5y ago
I’m baffed that nobody though that there might be an issue with the system when it suddenly turned out that 700 postmasters were disappearing funds. That’s way too high to be accidental.
ncmncm 5y ago
> $Millions, and in some cases $billions, in tax money pour into projects that almost invariably run late, over budget, fail to deliver In many, perhaps most, such cases, projects running late and over budget are performing exactly as intended by their sponsors. All too often, the nominal purpose for a project is just cover for a totally legal conduit from the public purse to politically-selected private pockets. Thus, in the US, we get the F-35, the SLS, the California bullet train. Sometimes we get something out the other end, many years late. (The delay is to keep the gravy train running longer.) Sometimes, nothing. In New York, we did end up with a 2nd Avenue Subway extension. There actually are F-35 planes at air bases, some of which can actually fly. They are stacking an SLS in Florida as I write this. They probably will launch at least the one, maybe a second, at $2B each.
- alisonkisk 5y ago
  > run late, over budget, fail to deliver An estimate (in time or money) is a wish your heart makes. only fail to deliver is a real problem here, due to lack of agile practices.
whoisthemachine 5y ago
> Compared to the 100 million euros Denmark spent on a new IT system for the police, a project that never delivered anything? Governments need "sunken-cost fallacy" triggers that automatically halt projects when they pass thresholds, even when many worry about the effect of the halting. It's like a scene from a movie - "halt my project when I go over 100M EUR, even if I'm begging you not to".
- concordDance 5y ago
  Given almost every single government project overruns, what could your threshold possibly be?
  - Kinrany 5y ago
    2x the initial cost seems reasonable. If the estimate was this wrong, it's useless and the project needs to be reestimated.
    - brigandish 5y ago
      The problem is that many initial estimates will be low ball offers to try and win a contract. Perhaps combine the sunken-cost-limit with delayed payments, otherwise companies might low ball offers knowing they can never finish but still don't care because they'll be paid. Better oversight is clearly needed too. There won't be one tool that'll do the job on its own.
  - whoisthemachine 5y ago
    The funny thing is, many projects (and I should have omitted government-run, any organization can suffer from this problem) have reasonable cost estimates when they begin. Why not start with those? Perhaps this rule would incentivize more realistic estimates?
    - cloudfifty 5y ago
      The contractors have all the incentives to extract as much government money as they possibly can during the entire process.
jhgorrell 5y ago
Believe this is the report referred to. I enjoyed the read. https://www.bea.aero/uploads/tx_elyextendttnews/F-HPJE_TECHNICAL_REPORT.pdf https://www.bea.aero/uploads/tx_elyextendttnews/F-HPJE_TECHN...
m0d0nne11 5y ago
How about a headline that isn't such irritating, content-free click-bait?
- yesenadam 5y ago
  I looked, and a fair proportion of your comments on HN are very similar complaints—arguably more irritating and content-free than what you complain against. I complained about an opaque headline once, and dang told me about how making the reader work a little is part of the HN philosophy. Pretty sensible when you learn about it: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20%22work%20a%20little%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
bitwize 5y ago
"We really need an IT accident investigation board to help us understand what causes wrong computer results that may harm innocent people." "...Computer says no."
learc83 5y ago
Software engineering hasn’t had our Quebec Bridge collapse yet. It will take an enormous visible disaster that does more than cost a company a few tens of millions, or exposes a few hundred million social security numbers before we start holding companies and engineers liable for security flaws.
- swiley 5y ago
  We've flown planes into the ground, killed people with Xrays, rigged elections, shut down oil delivery to sections of the east cost of the US. If anything we have bridge collapses constantly and special news anchors that report on which routes you should take today as if it were just like any other traffic incident.
  - cratermoon 5y ago
    Don't forget crashing the first Ariane 5 on its test flight[1], incorrect floating point math[2] and losing $440 million in 30 minutes[3]. 1 https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Presentation_of_Inquiry_Board_report https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Pre... 2 https://www.cs.earlham.edu/~dusko/cs63/fdiv.html https://www.cs.earlham.edu/~dusko/cs63/fdiv.html 3 https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... The Error of Our Ways: https://www.youtube.com/watch?v=3YaI6lhn78g https://www.youtube.com/watch?v=3YaI6lhn78g
- Goety 5y ago
  Ah yes the equifax hack https://www.nytimes.com/2017/09/07/business/equifax-cyberattack.html https://www.nytimes.com/2017/09/07/business/equifax-cyberatt... OPM Hack https://en.wikipedia.org/wiki/Office_of_Personnel_Management_data_breach https://en.wikipedia.org/wiki/Office_of_Personnel_Management... The only way to create software engineering reform you envision is to force a paradigm shift. The only thing that could instigate it would be like skynet with many dead. Though, the unknown damage from OPM could be quite severe. There might be 'security standards' but neither the government nor private sector can guarantee anything to be safe. The civilization built on IT networks will need to be rewowrked.
emmelaich 5y ago
Two technical things that make it more difficult. 1. lack of flight recorder (black box). 2. computer systems are brittle, complex and ever-changing
ridaj 5y ago
This is very sensible. One challenge is to encourage cooperation. One interesting aspect of strategy is to prevent findings to be used in court. For example, findings of probable cause by the US NTSB cannot, by law, be admitted as evidence to a trial. That helps to make it more about preventing reoccurrence rather than finding culprits.
aryehof 5y ago
At issue is that software continues to approach an opaque “ball of mud” as size and complexity increase. Silver bullets of the day like “microservices”, event-driven design, and functional programming do nothing to improve that. The prevalent analysis method of use-case driven procedural transactional scripts resulting in controllers or services [1] (typically just transforming a database) is problematic. Yet developers are taught or can use nothing else from computing’s rich history currently. It would seem that any attempt to improve how we model “external” [2] complex systems in code has ceased in favor of pursuing the low hanging fruit of technical detail. Why are we surprised that any software outside of computing and the data sciences, together with computing infrastructure, are challenged? At issue is that outsiders have no idea of this industry incompetence, and incorrectly rely on software as being accurate, complete and foolproof. ——- [1] Martin Fowler’s “controller-entity style” - P of EAA [2] Where expertise in the complex problem domain lies outside of the development team.
helsinkiandrew 5y ago
For more background on the UK Post office scandal, there was a recent computerphile youtube video https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10 And Private Eye report https://www.private-eye.co.uk/pictures/special_reports/justice-lost-in-the-post.pdf https://www.private-eye.co.uk/pictures/special_reports/justi... In the post office case in my view the problem wasn't that the IT system was faulty it was that Post Office management continued to prosecute long after (more than a decade) any competent person would have known it was faulty - these people should be prosecuted.
NHQ 5y ago
This happened to Capt. Kirk in an episode of Star Trek. The legal move is demand to face the accuser. The accuser is the computer. https://en.wikipedia.org/wiki/Court_Martial_(Star_Trek:_The_Original_Series) https://en.wikipedia.org/wiki/Court_Martial_(Star_Trek:_The_...
DangitBobby 5y ago
I agree with the article, though this doesn't quite sit right with me: > And no, it is not "self-incrimination" unless you did something criminal. People (in the US) are allowed to remain silent to prevent self-incrimination. While arresting someone, why don't we just say "talking right now can't be self-incriminating if you did nothing criminal?" These are in the same family as "why do you need privacy if you have nothing to hide?" and I just don't think it was well thought out.

405 points by headalgorithm 5y ago

Part of my job is to help the executives that I report to understand why things went wrong from the security perspective in our business unit. These are purely internal discussions, not even investigations. There are no penalties, but really, for things as egregious as hard coded passwords. As will become clear in a moment, the fact that my executives care is quite unusual. Culturally the result is coverups and lies. Engineers lie, managers lie, test people lie, directors lie, senior directors lie, vice president lie, external interesting teams are negotiated into minimizing certain critical failures, and so on. Managers don’t want to hear it so that they can’t be accused of lying, vice presidents don’t wanna know, SVP’s just want green squares on the cross-BU PowerPoint. This is internal discussion of revenue impacting incidents. Do you know what executives do care about? Revenue. Lost deals. If the people who care about money, including the account teams, don’t care about security and severe quality issues enough to be honest enough to get to improvement, how could an external board accomplish anything for those very few incidents that actually become publicly visible? This isn’t like the NTSB; I spent my life reading NTSB accident reports. They have actual real authority, there are potentially issues that might impact someone more than being caught distorting things.

izacus 5y ago
I also wonder if "blameless postmortem" culture perhaps actively works against preventing these kind of incidents. It doesn't seem that anyone in IT is ever responsible for damage they cause. But yes, lying, "not seeing" and covering documentation is pretty much standard corporate behaviour I've seen around plenty of companies as well.
- foobiekr 5y ago
  I no longer believe in blameless post mortem as a general rule. I have, through experience, come to believe that the contexts where blameless post mortems work are the contexts where literally anything works because they are organizations that have high hiring bars and high expectations. My current employer is not one of them; we are a mountain of mediocrity and all blameless post mortems do is act as an excuse to avoid raising the bar.
  - jolux 5y ago
    The principle of blameless postmortems is not supposed to absolve anyone of the responsibility to change anything, it’s supposed to foreground that serious failures are organizational failures first and foremost, because it’s the organization that has an obligation not to fail, not individuals, who fail all the time as a rule.
    - tialaramex 5y ago
      Exactly this. I spend a bunch of time reading accident reports from agencies like RAIB and MAIB, but my real jobs have been closer to the Web PKI and thus m.d.s.policy Back in 2015, Symantec's CA issued some certificates that shouldn't have existed, including for names owned by Google. What's wrong there? Well, a blameless postmortem would probably tell you that your processes and procedures are bad, you are creating bogus certificates to "test" a real CA whose certificates are actually trusted in the real world. Need better processes, training, oversight to ensure things improve. What did Symantec do when they were caught? They fired the low-level employees who conducted the tests and wrote a blog post "A Tough Day as Leaders" which blamed the fired employees for getting it wrong. Some leadership (the blog post of course no longer exists although I assume it's archived somewhere). Less than two years later, Symantec is back in trouble again because an RA they've worked with has been issuing certificates, using their CA infrastructure (and thus, from our point of view they were issuing these certificates even if they unaccountably believed this isn't their fault) that should not exist. This time Symantec's bosses blame not only low-level employees, but also auditors, bosses at the Korean RA, and anybody else they can think of... except themselves. This is a gross failure of leadership. Once upon a time a US President said "The buck stops here", but Donald Trump was very clear, "The buck stops with everybody" and "I take no responsibility" and it seems Symantec's leadership were made in that image. They quit the CA business rather than do what it would take to fix the problem. If you conduct a "postmortem" after an incident, then "Nobody was to blame and nothing needs to change" is almost certainly just as much the wrong outcome as "It's Jane's fault, fire her". I mentioned I read MAIB reports. One MAIB report sticks out, after many years, in the following way: Unlike every other MAIB report I've read, this one has No recommendations. Someone died, and yet there is nothing to recommend. Why not? Well, the cause is very simple, two men on a fishing boat took a lot of heroin, and their boat crashed, it sank and one died. No need to recommend that you shouldn't take heroin while operating a fishing boat since heroin is an illegal drug already and operating a vessel under the influence of drugs or alcohol is already a crime too. If your next "blameless portmortem" doesn't have any recommendations, ask yourself, was what happened already a crime? Are the people involved dead or in prison and so either way beyond the value of recommending a different course of action next time? No? Then we need to recommend how to actually avoid it happening again.
- nanis 5y ago
  In my negative experiences, "blameless" turned in to "nobody did anything wrong" which, of course, undermines the whole point of finding out what actually happened so we can see if there is a thing we can do to reduce the likelihood of it happening again. Sometimes, the root cause is indeed someone with the privilege but not the good sense ignoring warning signs. If we can't identify that problem, then we can't improve our odds for the next time.
  - joshuamorton 5y ago
    A valid blameless answer then is "remove the privilege" and yes, despite whatever objections you'll raise, this is possible. Difficult, but possible. Like, even in the case of the extreme example of someone deciding to intentionally harm the company. You fire them, but then what, how do you prevent the next person to go rogue from causing equivalent harm?
    - tonyedgecombe 5y ago
      This is what slows big companies down. They accrete so many of these rules and restrictions that people can't do their work.
      - rocqua 5y ago
        I believe more of these incidents should conclude: "this is better to accept as the cost of doing business than to try to spend money to fix".
      - concordDance 5y ago
        Or people start ignoring the rules in order to do their work. Which has it's own problems (particularly as rules normally aren't divided into "important" and "unimportant".
        jbuhbjlnjbn 5y ago
        This is the thorn in my foot. I refuse to ignore company rules, and also comment on gross negligence, which more often then not means I am the quarreler, not the good engineer in the eyes of coworkers and bosses.
slyall 5y ago
I think you are overestimating the importance of "revenue impacting incidents" to company employees. If the company makes a couple of million extra or less this year it doesn't effect the majority of workers. Their bonus isn't going up or down etc. And remember this incident has already happened. By contrast if a report comes out blaming the loss on a worker, department or division then that could have major consequences. No matter how "blameless" it is, come next round of bonuses, promotions or layoffs everybody knows it'll be factored into the decisions. So people don't have an incentive to make themselves look bad and unlike with the NTSB there is no legal powers or fear of causing deaths behind the investigation.
- laurent92 5y ago
  I understand, but it sounds like we are digging ourselves into the same hole as USSR workers who were not incentivized to deliver working products, when we do that. It’s a civilizational peril. How do we solve cooperation at large scale? Is the only way to watch large companies accumulate bored employees and constantly recreate “the small guy”, the startup, which will finally make things right, until they become too big to be incentivized?
  - WalterBright 5y ago
    That's exactly how it works, and why big companies do not grow until they consume the entire world. They become complacent and unable to change, and a startup eventually takes their business away.
    - saalweachter 5y ago
      It's funny that people expect management to function better because it's employee-owners of a capitalist corporation, instead of feudal lords or members of a communist central committee.
      - laurent92 5y ago
        The feudal owners in a startup situation are the marketplace owners who organize all startups to compete. The day we lose is the day all of those marketplaces themselves are cannot be replaced.
Aeolun 5y ago
I care about the issues (as an angineer), but my experience with raising any security issues is that it results in a lot of pain for me personally. You report something to the ‘security’ team, and suddenly you’re responsible for doing all the work, as well as the prime suspect in an investigation ‘why didn’t you fix this before, since when did we know about this?!’ I’m absolutely incentiviced to just let any issue lie until it is discovered, because then it actually is the security teams problem.
- arch-ninja 5y ago
  I was expelled from a university for reporting security flaws, the solution we all seek is the simplest one and for administrators it's easiest to hurt the people making noise which commonly results in the noise going away. fight-or-flight response at it's finest. Edit: reminder that for the "common folk" these "security issues" are not a 5-minute fix, they are fundamentally different realities which require every machine on the network to be re-checked before they can be used again. There is a clear communication failure between the ones who want security and the ones who want "security".
  - wkavey 5y ago
    Would like to hear the story behind this one.
    - Game_Ender 5y ago
      It’s common, a friend almost got expelled for reporting a flaw in the universities ID card system. That friend did not brake anything, they did not sneak into any protected spaces. Just discovered and validated a flaw and then reported it.
    - jacobolus 5y ago
      A high school friend (2 decades ago) told a teacher that the system keeping track of student grades was vulnerable to attack. The teacher asked the student to demonstrate by attacking the system and adjusting one of his test scores down by 1 point. My friend obliged, and the teacher reported the vulnerability to the administration. An administrator threatened my friend with expulsion, but when he proposed to go public with his story in response, they decided they wouldn't expel him. The resolution was "please don't tell anyone", and the vulnerability was never fixed.
jjtheblunt 5y ago
Do bureacracies essentially provide complexity in which dysfunction readily hides?
oceanghost 5y ago
As an engineer I found if I didn't lie and pretend to support the false reality of the managers above me, I wouldn't have that job long. Once we had a very large effort, creating a new hardware and software platform for the companies products. The CEO demanded it be done in 9 months, our estimate was accurate almost to the day-- 18 months. Politically it was easier for the PMs to agree with 9 months and then have 9 one-month delays it took, instead of getting fired for insisting on a good faith estimate. What do you think the security in that product was like? Security directly conflicts with the business model of cutting every corner possible and loading engineers as much as possible. A real security program would have to start with reasonable hours and goals.
hollerith 5y ago
I got curious about what legal authority the NTSB has. Here is what I found: >like a cop, the NTSB can secure an accident scene and keep others away. It can examine the aircraft wreckage. It can have aircraft parts tested to help determine why the accident happened. It can even subpoena evidence. . . . If there is a lawsuit concerning the crash, the NTSB will not get involved. Not only is the NTSB’s report of the accident’s probable cause inadmissible, but the NTSB investigators are prohibited by law from testifying in court, even if they are served with a subpoena. Source: https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-limited-role-after-an-aircraft-accident/ https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-li... >The NTSB may issue a subpoena, enforceable in Federal District Court . . . For purposes of the Health Insurance Portability and Accountability Act of 1996 (HIPAA), Public Law 104-191, and the regulations promulgated by the Department of Health and Human Services, 45 CFR 164.501 et seq., the NTSB is a “public health authority” to which protected health information may be disclosed by a HIPAA “covered entity” without the prior written authorization of the subject of the records. Source: https://www.law.cornell.edu/cfr/text/49/831.59 https://www.law.cornell.edu/cfr/text/49/831.59
cratermoon 5y ago
> There are no penalties, but really, for things as egregious as hard coded passwords Is this sentence missing a word or phrase?
ganafagol 5y ago
Let's not conflate security and safety here. The boards hailed by the article are all about investigating safety failures, and so is the advocated invetigation board. Security is a different beast. It's sub-par in many airplanes/power plants/... too.
pomian 5y ago
That's very disappointing. I've always brought up everyone up to think that of all the professions, you can always trust an engineer. They deal in facts. They don't lie. Perhaps it's as simple that in the real world, engineers can't lie. Because the world wouldn't work is they did.
indymike 5y ago
> Do you know what executives do care about? Revenue. Lost deals. This is often why engineering needs aren't covered. Things are presented as risks and expense, instead of in terms of revenue. A $10,000 expense actually wipes out $10,000 in net profit, so you need to generate revenue sufficient to create $10,000 in net profit. Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. Finally, there is how risk frames what you present. If you come at me with "this might happen" the other side of the coin is "this might not happen", and most managers will avoid the certainty of expense for the possibility of an expense. If you are working with a CEO, valuation is where it's at. Try to understand the swing in company valuation based on profit. Present to the CEO like this: "X is highly likely to happen within Y months, and it will impact $Z in net profits, leading to a change in company valuation of up to $N." Make sure $N is enough to matter. You just took 98% of the arguments against taking action off the table.
- an_opabinia 5y ago
  > Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. I don’t think you are interpreting things correctly here. This is both counterintuitive and wrong.
  - indymike 5y ago
    I'm pretty sure I'm not wrong. Source: I own three companies. The $100k and $2.8M were just examples (very thin net profit). It's equally likely you could have $100k in expense and need $200-$400K in sales to offset it if you are in a very high margin business. Regardless, what matters is that the unexpected expense will reduce profit, every time, and by framing risk in terms of profit you will help your CEO make better decisions.
    - an_opabinia 5y ago
      You’re not “framing the risk in terms of profit.” You’re talking about a set of equations in your head, that only you know, that come up with weird results like “it takes way more revenue to ‘offset’ an expense” because you’re holding “net” or “gross” or whatever margin constant. Which is really confusing and surprising and not how anyone I’ve ever met talks about this stuff, and they’d be equally puzzled and I don’t think would experience an illuminating or aha moment or whatever. Personally I found it a little interesting, I get what you are going for, but it’s so strange and not really useful or true.

MrStonedOne 5y ago

In washington state, the state superior court ruled the police department was not liable for the impound fee paid by somebody who had their car impounded for 90* days for driving on what the computer reported was a suspended license, because they are exempt from mistakes from trusting their own computer system. This was the second time the department had wrongfully impounded his car and they made no attempt to fix the mistake from the first time, this didn't impact the ruling. Its gonna get much worse before it get any better.

spaetzleesser 5y ago
It will get much worse I think. More and more companies are hiding behind algorithms and other computer systems while cutting support staff. If you are wronged you have nobody to talk to and they make no effort to correct the situation. the only recourse is a lawsuit which is way too expensive for most people. And even when they are caught the fines are usually only nominal. I think we are building up the ultimate faceless bureaucracies.
- Aeolun 5y ago
  If people implicitly trust the system (or at least don’t get blamed for it’s failures) it makes it much easier for hackers though.
- swiley 5y ago
  At that point you might as well just hook everything up to etherium; the judge is redundant.
- AlbertCory 5y ago
  You're certainly right that lots of very large companies go to every conceivable length to keep you from ever finding a human being, even for an online chat, forget a live phone conversation. Their "contact us" page is nothing but an FAQ. Is legislation or regulation the answer? That would be unfortunate. The government rarely makes things better, but to be honest, those accident investigation boards probably DO prevent management from sweeping things under the rug. Some kind of consumer ratings for Quality of Service will have to spring up. You can now find ratings of airlines for their on-time record and likelihood of losing your baggage. We need something similar for web companies.

dgb23 5y ago

Large amounts of money spent on government systems that never ship is a tragedy, but software projects like these tend to have a lot of open questions. We understand software development often as a discovery process (evolving requirements), especially if they are large or disruptive. So one critical output of any such project has to be knowledge that can be built upon, as in open, clearly specified and written papers. This should be done regardless of whether it failed or didn't fail.

Aeolun 5y ago
I can tell you now that nobody including the current engineers, can work with the requirement documents that are being generated over the course of our multi year system transformation project. Of course, it’s still an improvement over the legacy system which has none.

ithkuil 5y ago

Well written article

hamilyon2 5y ago

The industry fails to listen to lessons written in "Mythical man month" - 50 years from now. Half of a century ago. Of course some reports on why systems are being designed and coded poorly won't change anything. We know why, we just ignored the knowledge to the point of absurdity.

torgard 5y ago
Companies could be held liable for gross misconduct. Although GDPR is not exactly a shining example of IT regulation, I think it's a good example of liability. Companies get fined for breaking GDPR. Governmental projects should have similar requirements in place, and companies and people should be held accountable for breaking them.
- a1369209993 5y ago
  > Companies get fined for breaking GDPR. Well, no, they don't, or least when they do it's only pittance fines. Google, Facebook, et al are not bankrupt.
  - torgard 5y ago
    Here's a website tracking GDPR enforcement: https://www.enforcementtracker.com/ https://www.enforcementtracker.com/ According to that website, one of Google's 10 fines is the highest fine ever, at €50 million. Hilariously, in a dystopian kind of way, they were fined €28 last year. Not as in millions, but twenty-eight euros. Facebook only appears on the list once, with a paltry €51,000. But in any case, the point of GDPR is not to bankrupt companies.
    - CogitoCogito 5y ago
      > But in any case, the point of GDPR is not to bankrupt companies. It _should_ be. Well okay that's a bit hyperbolic, but what I mean by that is that the point should be to _change_ behavior and if the fines necessary or the resulting change in business model leads to bankruptcy then that should be totally fine. Companies who cannot operate legally shouldn't continue operating.

verytrivial 5y ago

I agree with nearly everything in this artictle but the following question stumped me: when exactly would a software disaster investigation board be employed? Plane goes down, train goes off rails or passes signal at danger, easy. But at what exact what point did the UK postmaster system "fail" enough for an investigation?

andersource 5y ago
I would say at latest when people convicted because of it had their names cleared - https://www.bbc.com/news/business-56859357 https://www.bbc.com/news/business-56859357
monkeydreams 5y ago
> But at what exact what point did the UK postmaster system "fail" enough for an investigation? The moment it came to light that postmasters were being improperly convicted? The moment it came to light that some improperly convicted postmasters committed suicide?
duckhelmet 5y ago
> at what exact what point did the UK postmaster system "fail" enough for an investigation? There never was in independent investigation of the IT system. "Justice Lost In The Post" https://www.private-eye.co.uk/special-reports/justice-lost-in-the-post https://www.private-eye.co.uk/special-reports/justice-lost-i...
chrischapman 5y ago
This is all pretty new in the world. It took many years for aircraft investigations to get real professional at it. How about we start with a simple algorithm like this: 1. Check if software is involved in the incident. 2. If it is, carry out an internal review to evaluate the possibility that the software may be a cause of the incident. 3. If it is, hand over to an independent review board. 4. Involve lawyers if and only if, the independent review board recommends it. That was an off-the-cuff attempt at a suggestion. There is probably a much better way. Feel free to suggest alternatives. I have no objection to lawyers getting involved. I just object to them suppressing the facts to protect their clients and preventing an independent investigation. It's impossible to cover up that an aircraft has crashed. Lawyers involved in an aircraft accident on behalf of the manufacturer (or operator) are basically doing 'damage limitation'. Lawyers involved in a software incident on behalf of the manufacturer (or operator) are basically doing 'evidence suppression'. They should stick to 'damage limitation'.

nixpulvis 5y ago

I would gladly work for a prolific IRB.

deleted 5y ago

[deleted]

ashton314 5y ago

> Personal information is the helium of IT systems—it leaks out of every crack or imperfection faster than seems possible. Might as well call it the hydrogen of IT systems—get too much of it concentrated in one place, and all it takes is one little spark for it all to go up in flames. Boom!

alisonkisk 5y ago
It doesn't destroy anything, it just leaks.

openthc 5y ago

In Washington State we have a system to track cannabis, the enforcement officers are supposed to be able to get reports from this system. The system is super buggy and also doesn't have meaningful reports. So there is a secondary system for officers to export to Excel documents. In one of the trainings they've been instructed to look for anomalies -- not real analisys, not even a pivot table. One thing they find is "negative quantities" -- but how can that be? (hint: it's bugs in the tracking software). Then enforcement shows up at the cannabis business to audit these negative numbers (or demand the business try to correct the data (which they cannot due to bugs)). So, crappy software gets law enforcement officers to basically review data "anomalies" created by bugs by visiting a business. The second most expensive method for data sanatization I can imagine. It's a poor use of their time and disruptive to the business. The system in WA is so buggy that the agency has opted to freeze the software rather than try to fix the issues. The future of government software is bleak -- so long as they keep using closed source packages from low-cost bidders.

laurent92 5y ago
Why isn’t all software created for the government required to be open-source? Would that really drive the costs up, if the providers don’t have the choice?
- openthc 5y ago
  The vendor claimed that if the code was out it would be a security risk. The agency claims the vendor needs to protect their intellectual property rights. We have (some) visibility into other things our taxes pay for -- the software should absolutely be one -- expecially the regulatory compliance ones that drive enforcement action. Edit: also, they were breached anyway shortly after launch (2018) and then an email went around offerting to sell the code and data from their entire system.
  - laurent92 5y ago
    And it is true: If their code were out, it would be painfully obvious that it is full of vulnerabilities. Security by obscurity! I know that because I’m myself afraid of making my old app open-source… I wish I had done a bug bounty from day #1. Bug bounties are a killer tool. I wish some lawyers had made a license like “Not open-source but here’s the source for vulnerability research.”

ldarby 5y ago

It's known what went wrong, computerphile has a video with some details: https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10 but it doesn't address any of the judicial and cultural fails, that's what needs to be fixed. Software bugs are a fact of life, people know this, except the judges in this case apparently.

HarryHirsch 5y ago
Bugs are a fact of life because of sloppy practices. The experience from SQLite is instructive, after a testsuite had been written, matters improved immensely. Why was the testsuite written? Because it was in the list of requirements from the client, aerospace standards demand that every possible branch is covered by a test. We choose to write bad software.
II2II 5y ago
One could argue that faults in the engineering and construction are also a fact of life, yet that doesn't mean we excuse them and it doesn't mean that assume that a failure is due to those faults. Investigations are performed in order to ascertain the truth. I think the authors comparison to the historical development of trains is appropriate. Investigating IT failures wasn't as important 50 years ago because IT infrastructure was not as critical. Investigating IT failures today is critical because the functioning of society depends upon it.
- ldarby 5y ago
  I don't think you (or the sibling comment) got my point. No where am I "excusing" bugs. Civil engineering bugs are indeed a fact of life too, just check the god damn news. What I'm saying, is that I think where PHK says "nobody sat down and documented precisely what went wrong", I think he is just wrong. The guy in the Computerphile video, Steven Murdoch wrote this article: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with-horizon-learning-from-the-post-office-trial/ https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with..., which has further details of the bugs, e.g. "There is a window of time between a user printing and cutting-off a report. If another user was to perform a transaction during that window, that transaction may not show on the report." which is from the long and very detailed judgment: https://www.benthamsgaze.org/wp-content/uploads/2021/07/Bates-and-Ors-v-the-Post-Office-Ltd-2019-EWHC-3408-QB.pdf https://www.benthamsgaze.org/wp-content/uploads/2021/07/Bate... The problem that needs investigating here is the miscarriage of justice, apparently the post office was aware of the bugs at the same time as prosecuting people for falling victim to the bugs, in order to save face about their IT decision.

Scoundreller 5y ago

> In 2017 the motor of an airplane exploded over the southern part of the Greenland icecap. Part of the engine landed on the ice while the plane continued to the first suitable airport way up north in Canada. eh, Happy Valley-Goose Bay isn't that far north as far as Canada goes. 53 degrees north. The actual droppings in Greendland were around 61 degrees N. Nuuk would have been ~60% closer, but not a chance it could handle an A380.

ChrisMarshallNY 5y ago

I really enjoyed this. Like most things, it's a matter of scale. If a train derails, we call in the NTSB, but they don't investigate car crashes. The issue that I see, is that the software industry seems to be absolutely obsessed with scale. Small applications are actively sneered at. Go big, or go home. So that means that every accident is a train wreck.

PhantomGremlin 5y ago
If a train derails, we call in the NTSB, but they don't investigate car crashes. They most certainly do investigate car crashes. They just don't have the resources to investigate very many of them. https://www.cnbc.com/2021/05/10/ntsb-releases-preliminary-report-on-fatal-tesla-crash-in-spring-texas.html https://www.cnbc.com/2021/05/10/ntsb-releases-preliminary-re...

Scoundreller 5y ago

Would also like to point out the fantastic videos created by the US Chemical Safety Board: https://www.youtube.com/user/USCSB https://www.youtube.com/user/USCSB

overgard 5y ago

So I wonder, in this situation what changes where the accident is averted? It's fine to demand accountability but what specifically is being monitored? Because I don't think there's a real answer yet. They can say IT needs to do a better job. Obviously. But in what dimension? Theres a lot of "you screwed up!" but very little "heres the fix"

Aeolun 5y ago

I’m baffed that nobody though that there might be an issue with the system when it suddenly turned out that 700 postmasters were disappearing funds. That’s way too high to be accidental.

ncmncm 5y ago

> $Millions, and in some cases $billions, in tax money pour into projects that almost invariably run late, over budget, fail to deliver In many, perhaps most, such cases, projects running late and over budget are performing exactly as intended by their sponsors. All too often, the nominal purpose for a project is just cover for a totally legal conduit from the public purse to politically-selected private pockets. Thus, in the US, we get the F-35, the SLS, the California bullet train. Sometimes we get something out the other end, many years late. (The delay is to keep the gravy train running longer.) Sometimes, nothing. In New York, we did end up with a 2nd Avenue Subway extension. There actually are F-35 planes at air bases, some of which can actually fly. They are stacking an SLS in Florida as I write this. They probably will launch at least the one, maybe a second, at $2B each.

alisonkisk 5y ago
> run late, over budget, fail to deliver An estimate (in time or money) is a wish your heart makes. only fail to deliver is a real problem here, due to lack of agile practices.

whoisthemachine 5y ago

> Compared to the 100 million euros Denmark spent on a new IT system for the police, a project that never delivered anything? Governments need "sunken-cost fallacy" triggers that automatically halt projects when they pass thresholds, even when many worry about the effect of the halting. It's like a scene from a movie - "halt my project when I go over 100M EUR, even if I'm begging you not to".

concordDance 5y ago
Given almost every single government project overruns, what could your threshold possibly be?
- Kinrany 5y ago
  2x the initial cost seems reasonable. If the estimate was this wrong, it's useless and the project needs to be reestimated.
  - brigandish 5y ago
    The problem is that many initial estimates will be low ball offers to try and win a contract. Perhaps combine the sunken-cost-limit with delayed payments, otherwise companies might low ball offers knowing they can never finish but still don't care because they'll be paid. Better oversight is clearly needed too. There won't be one tool that'll do the job on its own.
- whoisthemachine 5y ago
  The funny thing is, many projects (and I should have omitted government-run, any organization can suffer from this problem) have reasonable cost estimates when they begin. Why not start with those? Perhaps this rule would incentivize more realistic estimates?
  - cloudfifty 5y ago
    The contractors have all the incentives to extract as much government money as they possibly can during the entire process.

jhgorrell 5y ago

Believe this is the report referred to. I enjoyed the read. https://www.bea.aero/uploads/tx_elyextendttnews/F-HPJE_TECHNICAL_REPORT.pdf https://www.bea.aero/uploads/tx_elyextendttnews/F-HPJE_TECHN...

m0d0nne11 5y ago

How about a headline that isn't such irritating, content-free click-bait?

yesenadam 5y ago
I looked, and a fair proportion of your comments on HN are very similar complaints—arguably more irritating and content-free than what you complain against. I complained about an opaque headline once, and dang told me about how making the reader work a little is part of the HN philosophy. Pretty sensible when you learn about it: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20%22work%20a%20little%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

bitwize 5y ago

"We really need an IT accident investigation board to help us understand what causes wrong computer results that may harm innocent people." "...Computer says no."

learc83 5y ago

Software engineering hasn’t had our Quebec Bridge collapse yet. It will take an enormous visible disaster that does more than cost a company a few tens of millions, or exposes a few hundred million social security numbers before we start holding companies and engineers liable for security flaws.

swiley 5y ago
We've flown planes into the ground, killed people with Xrays, rigged elections, shut down oil delivery to sections of the east cost of the US. If anything we have bridge collapses constantly and special news anchors that report on which routes you should take today as if it were just like any other traffic incident.
- cratermoon 5y ago
  Don't forget crashing the first Ariane 5 on its test flight[1], incorrect floating point math[2] and losing $440 million in 30 minutes[3]. 1 https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Presentation_of_Inquiry_Board_report https://www.esa.int/Newsroom/Press_Releases/Ariane_501_-_Pre... 2 https://www.cs.earlham.edu/~dusko/cs63/fdiv.html https://www.cs.earlham.edu/~dusko/cs63/fdiv.html 3 https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-... The Error of Our Ways: https://www.youtube.com/watch?v=3YaI6lhn78g https://www.youtube.com/watch?v=3YaI6lhn78g
Goety 5y ago
Ah yes the equifax hack https://www.nytimes.com/2017/09/07/business/equifax-cyberattack.html https://www.nytimes.com/2017/09/07/business/equifax-cyberatt... OPM Hack https://en.wikipedia.org/wiki/Office_of_Personnel_Management_data_breach https://en.wikipedia.org/wiki/Office_of_Personnel_Management... The only way to create software engineering reform you envision is to force a paradigm shift. The only thing that could instigate it would be like skynet with many dead. Though, the unknown damage from OPM could be quite severe. There might be 'security standards' but neither the government nor private sector can guarantee anything to be safe. The civilization built on IT networks will need to be rewowrked.

emmelaich 5y ago

Two technical things that make it more difficult. 1. lack of flight recorder (black box). 2. computer systems are brittle, complex and ever-changing

ridaj 5y ago

This is very sensible. One challenge is to encourage cooperation. One interesting aspect of strategy is to prevent findings to be used in court. For example, findings of probable cause by the US NTSB cannot, by law, be admitted as evidence to a trial. That helps to make it more about preventing reoccurrence rather than finding culprits.

aryehof 5y ago

At issue is that software continues to approach an opaque “ball of mud” as size and complexity increase. Silver bullets of the day like “microservices”, event-driven design, and functional programming do nothing to improve that. The prevalent analysis method of use-case driven procedural transactional scripts resulting in controllers or services [1] (typically just transforming a database) is problematic. Yet developers are taught or can use nothing else from computing’s rich history currently. It would seem that any attempt to improve how we model “external” [2] complex systems in code has ceased in favor of pursuing the low hanging fruit of technical detail. Why are we surprised that any software outside of computing and the data sciences, together with computing infrastructure, are challenged? At issue is that outsiders have no idea of this industry incompetence, and incorrectly rely on software as being accurate, complete and foolproof. ——- [1] Martin Fowler’s “controller-entity style” - P of EAA [2] Where expertise in the complex problem domain lies outside of the development team.

helsinkiandrew 5y ago

For more background on the UK Post office scandal, there was a recent computerphile youtube video https://www.youtube.com/watch?v=hBJm9ZYqL10 https://www.youtube.com/watch?v=hBJm9ZYqL10 And Private Eye report https://www.private-eye.co.uk/pictures/special_reports/justice-lost-in-the-post.pdf https://www.private-eye.co.uk/pictures/special_reports/justi... In the post office case in my view the problem wasn't that the IT system was faulty it was that Post Office management continued to prosecute long after (more than a decade) any competent person would have known it was faulty - these people should be prosecuted.

NHQ 5y ago

This happened to Capt. Kirk in an episode of Star Trek. The legal move is demand to face the accuser. The accuser is the computer. https://en.wikipedia.org/wiki/Court_Martial_(Star_Trek:_The_Original_Series) https://en.wikipedia.org/wiki/Court_Martial_(Star_Trek:_The_...

DangitBobby 5y ago

I agree with the article, though this doesn't quite sit right with me: > And no, it is not "self-incrimination" unless you did something criminal. People (in the US) are allowed to remain silent to prevent self-incrimination. While arresting someone, why don't we just say "talking right now can't be self-incriminating if you did nothing criminal?" These are in the same family as "why do you need privacy if you have nothing to hide?" and I just don't think it was well thought out.