9 ms·

FAA says computer failure was caused by contractors who introduced data errors

114 points by r721 4y ago

r721 4y ago
>Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
- cratermoon 4y ago
  I've worked with my share of utter fools in software development. Never attribute to malice that which can be explained by stupidity. People are remarkably failure-prone in the absence of controls and guardrails. Sadly, I suspect the individuals will get the rap instead of whatever crap process made it possible for the changes to pass muster and get into production without being caught and corrected.
  - lampshades 4y ago
    I’m in complete agreement with you. Could you imagine 2 contractors from somewhere like Accenture, who are already underpaid and overworked, not making a mistake?
    - greggarious 4y ago
      No, I cannot picture someone from ACN failing to make a mistake. Malicious changes to the IT systems were done outside the FAA by messing directly with the elections of officials who run federal agencies like the FAA. Americans always look for a movie plot when the very real threats to “the homeland” looking to abuse their access are operating in the open and have been for many years… and they’re not low level software engineers.
      - goodSteveramos 4y ago
        Couldn’t agree more. Its totally insane that backwards middle eastern billionaires are allowed to just buy up American tech and media companies. At least this one has a sense of humor. https://twitter.com/Alwaleed_Talal/status/1585975226567110656 https://twitter.com/Alwaleed_Talal/status/158597522656711065... @Alwaleed_Talal Dear friend "Chief Twit" @elonmusk Together all the way @Twitter
        readonthegoapp 4y ago
        i don't know that dude in particular, but 'backwards middle eastern' sounds a bit racist. for instance, maybe that person has said something kind of evil, imo -- like maybe that an entire country should be ceded to some other country -- but even then i wouldn't want to overgeneralize -- i def don't like being associated with the most famous/rich americans when i go abroad, or interact online. but i would agree generally that non-us investors buying up us politicians, pr firms, and tech and media companies is something we/us should be careful about.
        goodSteveramos 4y ago
        >i def don't like being associated with the most famous/rich americans On some level Americans are responsible for what our government and economic leaders do. We cant identify as and benefit from being Americans and then chide other groups for being racist when they complain about what our leaders do in our name. The reverse is equally true
        mistrial9 4y ago
        you have it completely backwards and I can show it the people that pushed, bragged and elbowed their way into executive management at a massive, eternally funded bureacracy are experts at disregarding the viewpoint, input and most of all directions from others The first thing to do in such a position is to reward an inner loyal circle by increasing their benefits, authority and lessen the actual work, via contractors. This thread shows anecdotal evidence that setups involve far more than half the working headcount as contractors. The contractors do real work, take real blame, and are routinely pushed around since that is what people really do to each other. At what stage of any of this do "we" take responsibility and make change?!? it is fortified income protection, not responsive systems, described here..
  - hedora 4y ago
    I suggest reading the Simple Sabotage Field Manual; the individual and organizational behaviors you are describing are indistinguishable from those of a skilled saboteur: https://archive.org/download/SimpleSabotageFieldManualStrategicServicesProvisional/SimpleSabotageFieldManualStrategicServicesProvisional.pdf https://archive.org/download/SimpleSabotageFieldManualStrate...
    - r00fus 4y ago
      Ah yes - gray's law [1] applies here: "Any sufficiently advanced incompetence is indistinguishable from malice." Fact remains that a system wherein this is possible is a system that has been exploited. Marshaling the forces to fix the system is the preferred outcome, unless you want the system to remain exploited (ie, you benefit from the situation). [1] http://wikidumper.blogspot.com/2007/07/greys-law.html http://wikidumper.blogspot.com/2007/07/greys-law.html
      - cratermoon 4y ago
        I think the comment you're replying to is trying to say something about the management being compromised, deliberately hiring incompetent people and being lax about controls, and wanting the failures to happen for some unspecified conspiratorial reason. That merely bumps the question up another level, it's a question of malice or incompetence. Conspiracy theories tend to end up invoking some shadowy malicious cabal at the highest levels, above even leaders of nation-states, to explain just plain old human nature.
        goodSteveramos 4y ago
        When a society is run by people who care only for their own small wealthy in-group and nothing for the common good all kinds of incompetence is bound to ensue. It doesn't take a tin foil hat to see how that has happened to America.
        robocat 4y ago
        > When a society is run by people I tend to find that idea is adjacent to conspiracy thinking. If society is being “run” by anybody then either (a) they could do a much better job of it for themselves, or (b) they are very good at hiding. “Powerful” people seem to often have the same problems as the rest of us: they seem to be on the receiving end of systemic issues they don’t understand as much as any of us, they often work long hours, they still get tragic family issues and chronic health issues.
deleted 4y ago
[deleted]
eyear 4y ago
which contract company?
- abudabi123 4y ago
  what is the pay grade of the two contractors?
readonthegoapp 4y ago
This is 100% one of these situations A note is taped to the side of some filing cabinet that says Remember to add a % with requsite spaces after the code you enter into the NOTAM file BEFORE saving file.
ryandrake 4y ago
What about the software developers? A data entry problem or file corruption should not lead to a production outage! Robust software should gracefully reject bad input, show or log an error, and continue on. Those contractors should be able to feed emoji or line noise into the system and not cause an outage. This is why we fuzz test. If the take-away from this is “contractors should follow procedure and not make mistakes when entering data” then the FAA has not fixed anything.
- tyingq 4y ago
  The FAA had earlier referred to it as a "damaged database file". Which I suspect translates to something like a index file + data file pair, or a raw file for some kind of database. It sounds like the contractors accidentally overwrote something like that. So, yes, the system should have thrown some kind of "invalid magic number" or "unable to read header" error. Though that would have probably only helped a little. They were still slow to recover even after they understood the issue. Meaning they are missing still other things, like procedures to start over using fresh data. They had, for example, lots of partners that already had all the "known good data before the issue", but no way to take that and run.
- deleted 4y ago
  [deleted]
qbasic_forever 4y ago
The failure was in a technical organization architecting a solution that allowed a simple human error in data entry to take down the entire system. This is classic system design and process failure. Bad input should have been caught before it got pushed to prod, before it got pushed to backup databases, etc. There was clearly no testing, no change process, no backup verification, etc. People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.
- sokoloff 4y ago
  I agree that the ultimate root fault isn’t the recent data entry. > At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. That part doesn’t match my experience at all.
  - toomuchtodo 4y ago
    Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on. Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
    - marcosdumay 4y ago
      > Engineer closest to the fault gets scapegoated and fired That happens. But not every time, and even "usually" requires some statistics collecting to be sustained. Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
      - 4y ago
hedora 4y ago
People are blaming the system designers a lot in this thread, but: > The file or files were altered in spite of rules that prohibit those kind of changes on a live system. Can anyone here argue that, given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?
- ben_w 4y ago
  > given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system? I'm reminded of the punchline to an old joke: "Doctor, it hurts when I do this!" "Don't do that."
- chabons 4y ago
  This is a fair point. This press release was not aimed at software engineers, and "rules that prohibit those kind of changes" is vague. It could mean anything from a sign on the wall that says "please don't!" to a robust access control and deployment pipeline that they intentionally circumvented.
  - goodSteveramos 4y ago
    If the policy was violated then the policy was violated. If it was properly communicated and taken seriously by management (and it may not have been) then the person who violated it is at fault along with management for choosing the wrong person for the job. Given that this is a PR release, I suspect management’s diligence in the selection and training of the contractors is not being critically examined.
    - semicolon_storm 4y ago
      Good policies account for people not following the policy. Since this is aviation related, let's take "no smoking on a plane" as an example. There are "No smoking" signs everywhere and it's announced over the speakers several times prior to takeoff. Yet there's still an ashtray in the lavatory in case somebody disregards every warning so that they're not throwing a lit cigarette into a bin filled with paper products.
mumblemumble 4y ago
This is a great example of the general principle that what often gets dismissed as operator error is better recognized as design flaw. When SpaceShipTwo broke up mid-flight, the FAA did not place all the blame on the pilot who accidentally engaged the feathering mechanism during ascent. They thoroughly dragged the aircraft's designers for producing a design that made it so easy to do such a thing in the first place. They weren't willing to accept, "We said not to do it in the manual," as an appropriate safeguard in that case. Why are they doing it here? If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better.
- CoastalCoder 4y ago
  I agree with most of your points. But I think the term "engineer" means more to you than it does most practitioners, unfortunately.
  - mumblemumble 4y ago
    Job title inflation is a hell of a drug.
- vsareto 4y ago
  >If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better. There's a limit to individual accountability. You need professional accreditation which brings training, standards, and legal resources.
  - wyager 4y ago
    Software is still developing at a pace that burdensome regulatory red tape would represent a completely unacceptable destruction of value. Defaulting to red tape as the mechanism to improve the quality of software is also very short-sighted. Software engineering has the absolutely unique property among all engineering disciplines that we are working with abstract systems which are subject to formal verification. We can get arbitrarily high levels of assurance, with zero need whatsoever for some stodgy accreditation body. This is not happening much at the moment because money is (artificially) cheap, so it makes sense to hack together some crap and get it out the door as quickly as possible. There are solutions to this; slapping regulations on software is not the best one.
raydiatian 4y ago
Grew up around the FAA. I will say this of the organization: they are extremely thorough about root cause analysis and red tape, so I doubt this ever happens again.
- goodSteveramos 4y ago
  The FAA is the least diverse federal agency. https://www.cnsnews.com/commentary/hans-bader/faa-endangers-public-safety-through-illegal-racial-diversity-obsession https://www.cnsnews.com/commentary/hans-bader/faa-endangers-...
dang 4y ago
Related: A corrupt file led to the FAA ground stoppage – also found in backup system - https://news.ycombinator.com/item?id=34348388 https://news.ycombinator.com/item?id=34348388 - Jan 2023 (388 comments) US Halts Flights Nationwide After Key FAA System Goes Down - https://news.ycombinator.com/item?id=34337807 https://news.ycombinator.com/item?id=34337807 - Jan 2023 (36 comments) FAA NOTAM System Outage - https://news.ycombinator.com/item?id=34337158 https://news.ycombinator.com/item?id=34337158 - Jan 2023 (214 comments)
tristor 4y ago
There’s several comments in this thread pointing out that given a certain level of access, intentionally bypassing policy is possible in any organization. What I don’t think is being given due consideration is how often the people with this access are contractors, and worse often foreign nationals, which even in civilian orgs like the FAA has serious national security implications. The West has an addiction to foreign IT contractors. It’s a many hundreds of billions yearly industry globally, and is only possible because the largest and most critical institutions in our society refuse to invest in people to keep this knowledge in house and grow their own trained workforce.
- mistrial9 4y ago
  > The West has an addiction to foreign IT contractors LOL - let me show you Dubai
tarotuser 4y ago
It's always easier to blame contractors than to determine why and how. And Snowden talked about in his book that roughly 95% of the fed gov is contractors. Reason is that getting money is easy, but getting OMB approval for more hires is ridiculous. These people had a public trust or a clearance. They've worked their way in the fed gov. I wouldn't immediately question malice or otherwise. You don't just throw away a career with sabotage. Also having worked in a similar environment, I can guarantee you that although there may be a test env, it is nowhere near similar to prod. I've seen it myself with different firewall rules, different GPOs, different software hardening packages. And I also complained that there should be no distinctions between stage and prod - insomuch that DNS and SSL should be the only real differences. I'm also acutely aware if my project has a similar incident, that I too would be scapegoated and fired. It's part of the course of this job.
- codeslave13 4y ago
  I worked there for 10 years. What is this test env you speak of? /s We had dev and prod. No version control and no testing requirements. Was a while ago for sure (i used svn locally for sanity). But from what i hear not a lot has changed. This was not flight related software. They did have better tools and procedures but not by leaps and bounds
  - tarotuser 4y ago
    Oh, we always have a test environment :) Sometimes its also called prod. Ive had the displeasure of having another team write the Cloudformation scripts, and they wrote them in such a way that none could be redeployed, or changed. And since they're domain joined windows machines, effectively un-redeployable. My first start, which died in situ, was to do the work to get stage up and running, and then clone stage into prod and changing the DNS and naming bits. Since this was a no-go, ended up piecemealing everything. It still stings too. We're nearly done and ready to go live as well. Ive been pushing for live since November. But when a single person speaks out, "delays get delayed by meetings to discuss in meetings to have meetings about delays" (note: I'm not making this up)... I shouldn't complain too much. A good 4h a workday is filled with gaming and stuff, since the enforced delays and "hurry up and waits" require me to literally do nothing. But when I do work, I'm going at the 5-10x speed I'm accustomed to going. But when it takes weeks to open up SMB from internal machines to other internal machines... there's not a lot I can do to fix broken process. And, I'm just a lousy contractor out of hundreds of thousands.
photochemsyn 4y ago
There's not enough information to be sure, but it sounds kind of like an SQL injection type of issue. Which would be very surprising as that's been known about for many years and all systems should have some kind of input validation setup in place to prevent it. I suppose such any such setup could be overridden by someone with the appropriate level of access however.
awinter-py 4y ago
do we know the format of the file in question? are we talking json trailing comma, xml closing tag type of bug?
- MPSimmons 4y ago
  This system was built decades before either of those technologies. It could literally have been a lower case letter that didn't encode into whatever archaic language they use.
MPlus88 4y ago
Time to point fingers. Who are the contractors? Anybody know who they are?
zzzeek 4y ago
OH I see they edited the file that reads "DONT EDIT THIS FILE". A cronjob that relies upon the letter X appearing at character position 450 then failed.
semicolon_storm 4y ago
Even if you ignore the design issues that let contractors corrupt the database, it's still a massive failure of system design & policy because they clearly didn't actually have a backup system. > When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. Their entire backup plan was to fail over to an identical secondary system that was supposedly redundant. But, it was clearly sharing the same database system as the primary, thus they didn't actually have a fully redundant system. They had 1 system with some parts of it having a secondary, i.e. not an actual DR / failover plan.
gojomo 4y ago
Crypto-skeptics downvoted my previous suggestion (https://news.ycombinator.com/item?id=34349576 https://news.ycombinator.com/item?id=34349576) that NOTAM's sort of simple textual bulletin log – perhaps not even 150MB of data per year! – could be implemented as a distributed-consensus database. But blockchains are specifically designed to keep on appending, no matter the mistakes of individuals, or even motivated sabotage attempts of sophisticated attackers. Essentially, blockchains have a giant, full-coverage set of unit-tests in their shared & public consensus code – reviewed and running across many systems. And many of their designs have proven uptimes in adversarial environments for years. And the task gets even easier when there's a small fixed set of established authorities with privileged keys. Protecting those keys is also, of course, a hard problem. But compromised credentials is probably already a fatal threat for the legacy system(s) that were brought down by "contractors who introduced data errors". So a hypothetical modern crypto-consensus solution's failure modes remain a subset of those of the legacy system.
- 12baad4db82 4y ago
  I don't think it helps to characterise this as a simple textual bulletin log. I am by no means an expert, however to me this system appears to include: a globally agreed upon data format, a transmission protocol hardware distributed across the planet in ground stations and air vehicles, and 70 years of legacy. And all of that is enshrined in legislation across many countries. And while there is a modernisation effort in progress, the nature of this system means that things will move slowly. In 2023 anyone can suggest a better data format and distribution system if they start from scratch. Good luck getting the world to use it and good luck getting anyone to care.
  - gojomo 4y ago
    You're right that the main challeges are political & bureaucratic inertia. That shouldn't stop us from honestly & plainly describing its core, & the industry's state-of-the-art for similar systems that don't have the legacy system's encumbrances. The "globally agreed-upon data format" is essentially text blobs with a standard set of abbreviations, that assist search/filtering by flight-paths & planned waypoints. It seems to be a mere 150MB a year. The inherently "move slowly" modernization effort can only be helped by a clearer outline of what's now possible. This is too important to prematurely dumb-down brainstorming ideas, based on low-expectations for the adoption process! And a modern architecture for distributed updates & reliability could trivially mimic the old system's pilot-visible outputs, to remain familiar to tenured personnel.
cesaref 4y ago
`When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. ` That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything. One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
cesaref 4y ago
> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything. One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
- smilliken 4y ago
  To create a complete backup, one must first create a snapshot of the state of the universe. Then to be sure it worked, one must restore that backup and run regression tests.
PreInternet01 4y ago
OK, so I know nothing about NOTAMs and even less about the FAA infrastructure. But: data errors introduced by a bad disk? In 2022? Storing your data on a hardware-backed (or, ZFS-backed, if you must) RAID6 (or equivalent) partition has been bog-standard for the past decade or so, and a partition is either online (and OK) or degraded (but still OK), or offline, at which point 'restoring the latest backup' (or just failing over to a replica) should bring you back to an OK state. Then: apparently, restoring the offline partition caused bad data to be ingested, over and over, crashing the system again and again. Even if we pretend that solutions like SQLite have not been available for the past decade or so, 'skip past the head of the corrupted data file' has been best-practice since I last saw abominations requiring such hacks (say: Novell MHS), and sorry to repeat myself, over a decade ago? TL;DR: please pay me US$ 200M, and I'll make sure all your problems are at least not of the kind solved a long time ago?
- mmastrac 4y ago
  I'm going to disagree that "skip past the head of the corrupted data file" is something that should be done in a safety-critical environment. What happens if the head of the data file contains the location of a hazard that 100 planes need to be cautious of?
  - PreInternet01 4y ago
    Sure, "skipped past ## corrupted octets at offset ####, offending contents saved as `/mnt/log/data/yyyyMMdd/seq.json` and discarded" should absolutely be logged and trigger the highest alert level. It should not, however, impact the availability of the system.
krisoft 4y ago
FAA specifically should know better than blaming individuals. Whatever happened was a systemic issue because in a well designed system individuals simply should not be able to bring this much chaos. Neither accidentally nor maliciously. Is there anyone there who heard about just culture, and root cause analysis? I can’t even believe they would publish this irresponsible stuff.
metaphor 4y ago
I wonder if it's just a coincidence that the FAA started accepting public applications today[1] for 12 general engineer vacancies (of the program management office type) in the DC locality. [1] https://www.usajobs.gov/job/700094300 https://www.usajobs.gov/job/700094300
animitronix 4y ago
Lol that's what you get for using contractors

114 points by r721 4y ago

r721 4y ago

>Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.

cratermoon 4y ago
I've worked with my share of utter fools in software development. Never attribute to malice that which can be explained by stupidity. People are remarkably failure-prone in the absence of controls and guardrails. Sadly, I suspect the individuals will get the rap instead of whatever crap process made it possible for the changes to pass muster and get into production without being caught and corrected.
- lampshades 4y ago
  I’m in complete agreement with you. Could you imagine 2 contractors from somewhere like Accenture, who are already underpaid and overworked, not making a mistake?
  - greggarious 4y ago
    No, I cannot picture someone from ACN failing to make a mistake. Malicious changes to the IT systems were done outside the FAA by messing directly with the elections of officials who run federal agencies like the FAA. Americans always look for a movie plot when the very real threats to “the homeland” looking to abuse their access are operating in the open and have been for many years… and they’re not low level software engineers.
    - goodSteveramos 4y ago
      Couldn’t agree more. Its totally insane that backwards middle eastern billionaires are allowed to just buy up American tech and media companies. At least this one has a sense of humor. https://twitter.com/Alwaleed_Talal/status/1585975226567110656 https://twitter.com/Alwaleed_Talal/status/158597522656711065... @Alwaleed_Talal Dear friend "Chief Twit" @elonmusk Together all the way @Twitter
      - readonthegoapp 4y ago
        i don't know that dude in particular, but 'backwards middle eastern' sounds a bit racist. for instance, maybe that person has said something kind of evil, imo -- like maybe that an entire country should be ceded to some other country -- but even then i wouldn't want to overgeneralize -- i def don't like being associated with the most famous/rich americans when i go abroad, or interact online. but i would agree generally that non-us investors buying up us politicians, pr firms, and tech and media companies is something we/us should be careful about.
        goodSteveramos 4y ago
        >i def don't like being associated with the most famous/rich americans On some level Americans are responsible for what our government and economic leaders do. We cant identify as and benefit from being Americans and then chide other groups for being racist when they complain about what our leaders do in our name. The reverse is equally true
        mistrial9 4y ago
        you have it completely backwards and I can show it the people that pushed, bragged and elbowed their way into executive management at a massive, eternally funded bureacracy are experts at disregarding the viewpoint, input and most of all directions from others The first thing to do in such a position is to reward an inner loyal circle by increasing their benefits, authority and lessen the actual work, via contractors. This thread shows anecdotal evidence that setups involve far more than half the working headcount as contractors. The contractors do real work, take real blame, and are routinely pushed around since that is what people really do to each other. At what stage of any of this do "we" take responsibility and make change?!? it is fortified income protection, not responsive systems, described here..
- hedora 4y ago
  I suggest reading the Simple Sabotage Field Manual; the individual and organizational behaviors you are describing are indistinguishable from those of a skilled saboteur: https://archive.org/download/SimpleSabotageFieldManualStrategicServicesProvisional/SimpleSabotageFieldManualStrategicServicesProvisional.pdf https://archive.org/download/SimpleSabotageFieldManualStrate...
  - r00fus 4y ago
    Ah yes - gray's law [1] applies here: "Any sufficiently advanced incompetence is indistinguishable from malice." Fact remains that a system wherein this is possible is a system that has been exploited. Marshaling the forces to fix the system is the preferred outcome, unless you want the system to remain exploited (ie, you benefit from the situation). [1] http://wikidumper.blogspot.com/2007/07/greys-law.html http://wikidumper.blogspot.com/2007/07/greys-law.html
    - cratermoon 4y ago
      I think the comment you're replying to is trying to say something about the management being compromised, deliberately hiring incompetent people and being lax about controls, and wanting the failures to happen for some unspecified conspiratorial reason. That merely bumps the question up another level, it's a question of malice or incompetence. Conspiracy theories tend to end up invoking some shadowy malicious cabal at the highest levels, above even leaders of nation-states, to explain just plain old human nature.
      - goodSteveramos 4y ago
        When a society is run by people who care only for their own small wealthy in-group and nothing for the common good all kinds of incompetence is bound to ensue. It doesn't take a tin foil hat to see how that has happened to America.
        robocat 4y ago
        > When a society is run by people I tend to find that idea is adjacent to conspiracy thinking. If society is being “run” by anybody then either (a) they could do a much better job of it for themselves, or (b) they are very good at hiding. “Powerful” people seem to often have the same problems as the rest of us: they seem to be on the receiving end of systemic issues they don’t understand as much as any of us, they often work long hours, they still get tragic family issues and chronic health issues.

deleted 4y ago

[deleted]

eyear 4y ago

which contract company?

abudabi123 4y ago
what is the pay grade of the two contractors?

readonthegoapp 4y ago

This is 100% one of these situations A note is taped to the side of some filing cabinet that says Remember to add a % with requsite spaces after the code you enter into the NOTAM file BEFORE saving file.

ryandrake 4y ago

What about the software developers? A data entry problem or file corruption should not lead to a production outage! Robust software should gracefully reject bad input, show or log an error, and continue on. Those contractors should be able to feed emoji or line noise into the system and not cause an outage. This is why we fuzz test. If the take-away from this is “contractors should follow procedure and not make mistakes when entering data” then the FAA has not fixed anything.

tyingq 4y ago
The FAA had earlier referred to it as a "damaged database file". Which I suspect translates to something like a index file + data file pair, or a raw file for some kind of database. It sounds like the contractors accidentally overwrote something like that. So, yes, the system should have thrown some kind of "invalid magic number" or "unable to read header" error. Though that would have probably only helped a little. They were still slow to recover even after they understood the issue. Meaning they are missing still other things, like procedures to start over using fresh data. They had, for example, lots of partners that already had all the "known good data before the issue", but no way to take that and run.
deleted 4y ago
[deleted]

qbasic_forever 4y ago

The failure was in a technical organization architecting a solution that allowed a simple human error in data entry to take down the entire system. This is classic system design and process failure. Bad input should have been caught before it got pushed to prod, before it got pushed to backup databases, etc. There was clearly no testing, no change process, no backup verification, etc. People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.

sokoloff 4y ago
I agree that the ultimate root fault isn’t the recent data entry. > At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. That part doesn’t match my experience at all.
- toomuchtodo 4y ago
  Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on. Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
  - marcosdumay 4y ago
    > Engineer closest to the fault gets scapegoated and fired That happens. But not every time, and even "usually" requires some statistics collecting to be sustained. Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
    - 4y ago

hedora 4y ago

People are blaming the system designers a lot in this thread, but: > The file or files were altered in spite of rules that prohibit those kind of changes on a live system. Can anyone here argue that, given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?

ben_w 4y ago
> given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system? I'm reminded of the punchline to an old joke: "Doctor, it hurts when I do this!" "Don't do that."
chabons 4y ago
This is a fair point. This press release was not aimed at software engineers, and "rules that prohibit those kind of changes" is vague. It could mean anything from a sign on the wall that says "please don't!" to a robust access control and deployment pipeline that they intentionally circumvented.
- goodSteveramos 4y ago
  If the policy was violated then the policy was violated. If it was properly communicated and taken seriously by management (and it may not have been) then the person who violated it is at fault along with management for choosing the wrong person for the job. Given that this is a PR release, I suspect management’s diligence in the selection and training of the contractors is not being critically examined.
  - semicolon_storm 4y ago
    Good policies account for people not following the policy. Since this is aviation related, let's take "no smoking on a plane" as an example. There are "No smoking" signs everywhere and it's announced over the speakers several times prior to takeoff. Yet there's still an ashtray in the lavatory in case somebody disregards every warning so that they're not throwing a lit cigarette into a bin filled with paper products.

mumblemumble 4y ago

This is a great example of the general principle that what often gets dismissed as operator error is better recognized as design flaw. When SpaceShipTwo broke up mid-flight, the FAA did not place all the blame on the pilot who accidentally engaged the feathering mechanism during ascent. They thoroughly dragged the aircraft's designers for producing a design that made it so easy to do such a thing in the first place. They weren't willing to accept, "We said not to do it in the manual," as an appropriate safeguard in that case. Why are they doing it here? If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better.

CoastalCoder 4y ago
I agree with most of your points. But I think the term "engineer" means more to you than it does most practitioners, unfortunately.
- mumblemumble 4y ago
  Job title inflation is a hell of a drug.
vsareto 4y ago
>If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better. There's a limit to individual accountability. You need professional accreditation which brings training, standards, and legal resources.
- wyager 4y ago
  Software is still developing at a pace that burdensome regulatory red tape would represent a completely unacceptable destruction of value. Defaulting to red tape as the mechanism to improve the quality of software is also very short-sighted. Software engineering has the absolutely unique property among all engineering disciplines that we are working with abstract systems which are subject to formal verification. We can get arbitrarily high levels of assurance, with zero need whatsoever for some stodgy accreditation body. This is not happening much at the moment because money is (artificially) cheap, so it makes sense to hack together some crap and get it out the door as quickly as possible. There are solutions to this; slapping regulations on software is not the best one.

raydiatian 4y ago

Grew up around the FAA. I will say this of the organization: they are extremely thorough about root cause analysis and red tape, so I doubt this ever happens again.

goodSteveramos 4y ago
The FAA is the least diverse federal agency. https://www.cnsnews.com/commentary/hans-bader/faa-endangers-public-safety-through-illegal-racial-diversity-obsession https://www.cnsnews.com/commentary/hans-bader/faa-endangers-...

dang 4y ago

Related: A corrupt file led to the FAA ground stoppage – also found in backup system - https://news.ycombinator.com/item?id=34348388 https://news.ycombinator.com/item?id=34348388 - Jan 2023 (388 comments) US Halts Flights Nationwide After Key FAA System Goes Down - https://news.ycombinator.com/item?id=34337807 https://news.ycombinator.com/item?id=34337807 - Jan 2023 (36 comments) FAA NOTAM System Outage - https://news.ycombinator.com/item?id=34337158 https://news.ycombinator.com/item?id=34337158 - Jan 2023 (214 comments)

tristor 4y ago

There’s several comments in this thread pointing out that given a certain level of access, intentionally bypassing policy is possible in any organization. What I don’t think is being given due consideration is how often the people with this access are contractors, and worse often foreign nationals, which even in civilian orgs like the FAA has serious national security implications. The West has an addiction to foreign IT contractors. It’s a many hundreds of billions yearly industry globally, and is only possible because the largest and most critical institutions in our society refuse to invest in people to keep this knowledge in house and grow their own trained workforce.

mistrial9 4y ago
> The West has an addiction to foreign IT contractors LOL - let me show you Dubai

tarotuser 4y ago

It's always easier to blame contractors than to determine why and how. And Snowden talked about in his book that roughly 95% of the fed gov is contractors. Reason is that getting money is easy, but getting OMB approval for more hires is ridiculous. These people had a public trust or a clearance. They've worked their way in the fed gov. I wouldn't immediately question malice or otherwise. You don't just throw away a career with sabotage. Also having worked in a similar environment, I can guarantee you that although there may be a test env, it is nowhere near similar to prod. I've seen it myself with different firewall rules, different GPOs, different software hardening packages. And I also complained that there should be no distinctions between stage and prod - insomuch that DNS and SSL should be the only real differences. I'm also acutely aware if my project has a similar incident, that I too would be scapegoated and fired. It's part of the course of this job.

codeslave13 4y ago
I worked there for 10 years. What is this test env you speak of? /s We had dev and prod. No version control and no testing requirements. Was a while ago for sure (i used svn locally for sanity). But from what i hear not a lot has changed. This was not flight related software. They did have better tools and procedures but not by leaps and bounds
- tarotuser 4y ago
  Oh, we always have a test environment :) Sometimes its also called prod. Ive had the displeasure of having another team write the Cloudformation scripts, and they wrote them in such a way that none could be redeployed, or changed. And since they're domain joined windows machines, effectively un-redeployable. My first start, which died in situ, was to do the work to get stage up and running, and then clone stage into prod and changing the DNS and naming bits. Since this was a no-go, ended up piecemealing everything. It still stings too. We're nearly done and ready to go live as well. Ive been pushing for live since November. But when a single person speaks out, "delays get delayed by meetings to discuss in meetings to have meetings about delays" (note: I'm not making this up)... I shouldn't complain too much. A good 4h a workday is filled with gaming and stuff, since the enforced delays and "hurry up and waits" require me to literally do nothing. But when I do work, I'm going at the 5-10x speed I'm accustomed to going. But when it takes weeks to open up SMB from internal machines to other internal machines... there's not a lot I can do to fix broken process. And, I'm just a lousy contractor out of hundreds of thousands.

photochemsyn 4y ago

There's not enough information to be sure, but it sounds kind of like an SQL injection type of issue. Which would be very surprising as that's been known about for many years and all systems should have some kind of input validation setup in place to prevent it. I suppose such any such setup could be overridden by someone with the appropriate level of access however.

awinter-py 4y ago

do we know the format of the file in question? are we talking json trailing comma, xml closing tag type of bug?

MPSimmons 4y ago
This system was built decades before either of those technologies. It could literally have been a lower case letter that didn't encode into whatever archaic language they use.

MPlus88 4y ago

Time to point fingers. Who are the contractors? Anybody know who they are?

zzzeek 4y ago

OH I see they edited the file that reads "DONT EDIT THIS FILE". A cronjob that relies upon the letter X appearing at character position 450 then failed.

semicolon_storm 4y ago

Even if you ignore the design issues that let contractors corrupt the database, it's still a massive failure of system design & policy because they clearly didn't actually have a backup system. > When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. Their entire backup plan was to fail over to an identical secondary system that was supposedly redundant. But, it was clearly sharing the same database system as the primary, thus they didn't actually have a fully redundant system. They had 1 system with some parts of it having a secondary, i.e. not an actual DR / failover plan.

gojomo 4y ago

Crypto-skeptics downvoted my previous suggestion (https://news.ycombinator.com/item?id=34349576 https://news.ycombinator.com/item?id=34349576) that NOTAM's sort of simple textual bulletin log – perhaps not even 150MB of data per year! – could be implemented as a distributed-consensus database. But blockchains are specifically designed to keep on appending, no matter the mistakes of individuals, or even motivated sabotage attempts of sophisticated attackers. Essentially, blockchains have a giant, full-coverage set of unit-tests in their shared & public consensus code – reviewed and running across many systems. And many of their designs have proven uptimes in adversarial environments for years. And the task gets even easier when there's a small fixed set of established authorities with privileged keys. Protecting those keys is also, of course, a hard problem. But compromised credentials is probably already a fatal threat for the legacy system(s) that were brought down by "contractors who introduced data errors". So a hypothetical modern crypto-consensus solution's failure modes remain a subset of those of the legacy system.

12baad4db82 4y ago
I don't think it helps to characterise this as a simple textual bulletin log. I am by no means an expert, however to me this system appears to include: a globally agreed upon data format, a transmission protocol hardware distributed across the planet in ground stations and air vehicles, and 70 years of legacy. And all of that is enshrined in legislation across many countries. And while there is a modernisation effort in progress, the nature of this system means that things will move slowly. In 2023 anyone can suggest a better data format and distribution system if they start from scratch. Good luck getting the world to use it and good luck getting anyone to care.
- gojomo 4y ago
  You're right that the main challeges are political & bureaucratic inertia. That shouldn't stop us from honestly & plainly describing its core, & the industry's state-of-the-art for similar systems that don't have the legacy system's encumbrances. The "globally agreed-upon data format" is essentially text blobs with a standard set of abbreviations, that assist search/filtering by flight-paths & planned waypoints. It seems to be a mere 150MB a year. The inherently "move slowly" modernization effort can only be helped by a clearer outline of what's now possible. This is too important to prematurely dumb-down brainstorming ideas, based on low-expectations for the adoption process! And a modern architecture for distributed updates & reliability could trivially mimic the old system's pilot-visible outputs, to remain familiar to tenured personnel.

cesaref 4y ago

`When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. ` That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything. One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...

cesaref 4y ago

> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said. That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything. One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...

smilliken 4y ago
To create a complete backup, one must first create a snapshot of the state of the universe. Then to be sure it worked, one must restore that backup and run regression tests.

PreInternet01 4y ago

OK, so I know nothing about NOTAMs and even less about the FAA infrastructure. But: data errors introduced by a bad disk? In 2022? Storing your data on a hardware-backed (or, ZFS-backed, if you must) RAID6 (or equivalent) partition has been bog-standard for the past decade or so, and a partition is either online (and OK) or degraded (but still OK), or offline, at which point 'restoring the latest backup' (or just failing over to a replica) should bring you back to an OK state. Then: apparently, restoring the offline partition caused bad data to be ingested, over and over, crashing the system again and again. Even if we pretend that solutions like SQLite have not been available for the past decade or so, 'skip past the head of the corrupted data file' has been best-practice since I last saw abominations requiring such hacks (say: Novell MHS), and sorry to repeat myself, over a decade ago? TL;DR: please pay me US$ 200M, and I'll make sure all your problems are at least not of the kind solved a long time ago?

mmastrac 4y ago
I'm going to disagree that "skip past the head of the corrupted data file" is something that should be done in a safety-critical environment. What happens if the head of the data file contains the location of a hazard that 100 planes need to be cautious of?
- PreInternet01 4y ago
  Sure, "skipped past ## corrupted octets at offset ####, offending contents saved as `/mnt/log/data/yyyyMMdd/seq.json` and discarded" should absolutely be logged and trigger the highest alert level. It should not, however, impact the availability of the system.

krisoft 4y ago

FAA specifically should know better than blaming individuals. Whatever happened was a systemic issue because in a well designed system individuals simply should not be able to bring this much chaos. Neither accidentally nor maliciously. Is there anyone there who heard about just culture, and root cause analysis? I can’t even believe they would publish this irresponsible stuff.

metaphor 4y ago

I wonder if it's just a coincidence that the FAA started accepting public applications today[1] for 12 general engineer vacancies (of the program management office type) in the DC locality. [1] https://www.usajobs.gov/job/700094300 https://www.usajobs.gov/job/700094300

animitronix 4y ago

Lol that's what you get for using contractors