21 ms·
A corrupt file led to the FAA ground stoppage – also found in backup system
- uejfiweun 4y agoI wonder if the corruption could have been caused by a cosmic ray bit flip. It would be funny to think that one single tiny subatomic particle knocked down our entire flight system.
- GalenErso 4y agoCosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.
- bick_nyers 4y agoAliens using cosmic ray generators to flip specific bits and set off nukes.
- tsujamin 4y agoReminds me of the Sophons in The Three-Body Problem
- asdff 4y agoI really should have milked the "cosmic ray bitflip ate my homework" excuse in hindsight
- jupp0r 4y agoOne more reason to use ZFS.
- Helmut10001 4y agoSurprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.
- jupp0r 4y agoNot a lot of people are familiar with how well ZFS is engineered compared to most other file systems and how life changingly different it is.
- shemnon42 4y agoCosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem. https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship... File this under "early Google's infrastructure was a low grade cosmic ray detector."
- defrost 4y agoYou can expect 250 or so cosmic ray events per second in a 42 litre sodium iodide crystal pack at 100m above sea level. Source: 10 years airbourne geophysics, radiometric calibrations. Addendum: In-flight upset 154 km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 [1] was a probable (but uncertain) example of cosmic ray events causing multiple spikes in one of three air data inertial reference units (ADIRUs) that also went on to cause a failure mode of the "best of three" reporting system leading to a pitch down in which [2] > 110 of the 303 passengers and nine of the 12 crew members were injured; 12 of the occupants were seriously injured and another 39 received hospital medical treatment. HOWEVER .. despite 250 events per second in a 42 litre volume, it took 128 million hours of unit operation to see a failure mode. It's a lot of billiard balls going through a lot of space and a high bar for "something bad" ( just the right bit flip ) to happen. [1] https://www.atsb.gov.au/sites/default/files/media/3532398/ao2008070.pdf https://www.atsb.gov.au/sites/default/files/media/3532398/ao... [2] https://www.atsb.gov.au/publications/investigation_reports/2008/aair/ao-2008-070 https://www.atsb.gov.au/publications/investigation_reports/2... ( Final Report TAB )
- nix23 4y ago>They're so rare, They are not rare, just because you don't detect them (consumer hardware) means nothing. >>study done by IBM in the 1990’s that referenced 1 cosmic ray bit flip per 256MB of memory per month https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-what-flips-your-bit/ https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
- lioeters 4y agoOne cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery. > Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to sea level. The rate of upsets in aircraft may be more than 300 times the sea level upset rate. https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
- nix23 4y ago>One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. Yes thats why you have max altitude for normal servers/hardware and it's also more disastrous on modern HW -> smaller transistors.
- shadowgovt 4y agoThe reason we all now wear our seatbelts through the entire flight is there was a cross-Pacific flight that had a sudden altitude drop where several people were injured from rebounding off the ceiling. When the news reported on it, they reported the cause was believed to have been an undetected massive downdraft that shoved the plane down. I lost the thread on that story before the investigation concluded the actual cause was autopilot error. It is, in some ways, more comforting to me to know that the issue wasn't novel atmospheric phenomena, but instead relatively-mundane cosmic radiation flipping one packet of data from the sensors to the avionics that the avionics lacked sufficient redundancy to detect or discard. As a result, the autopilot believed the plane had suddenly pitched 90 degrees and drastically corrected to escape stall.
- numbsafari 4y agoMy guess is some input system was updated and now allows Unicode, but the backend systems do not. Somebody inadvertently put a poop emoji in a text field.
- rawgabbit 4y agoI wish to know what kind of system NOTAM runs on. An IBM 360 with tape backup?
- w-ll 4y agozOS for sure
- tyingq 4y agoYou can see references to JMS, Solace, and Weblogic in various docs, like: https://www.faa.gov/air_traffic/technology/swim/users_forum/media/SWIM%20Users%20Forum%2026%20Briefing.pdf https://www.faa.gov/air_traffic/technology/swim/users_forum/... https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%20Summit%20-%20Session%201%20-%20Connecting%20to%20NOTAM%20Data.pdf https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%... So probably Linux, though perhaps something else.
- winboot 4y ago> In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system -- a significant decision, because the reboot can take about 90 minutes, according to the source. 90 minutes; this certainly appears to be an advertisement about Windows OS.
- wrycoder 4y agoForced upgrade on reboot, my personal favorite.
- mindslight 4y agoOr just a legacy UNIX with a lot of disks. I've seen PH-UX take over an hour to boot due to that.
- mindslight 4y ago(Just to be clear I meant specifically just enumerating SCSI LUNs, not fsck or anything like that)
- linker3000 4y agoI worked for an organization in the 1990s that ran a legacy IBM mainframe (later replaced with an AS/400). After an unplanned power outage, the disk confidence test and rebuild took over 1/2 day.
- ComodoHacker 4y agoOr just waiting for thorough AV scan on boot.
- daniel-s 4y agoNow if their ATC software ran on Emacs lisp they wouldn't have had any problem.
- rubicks 4y agoI understood that reference.
- ananonymoususer 4y agoI didn't, so I looked it up. Interesting: https://news.ycombinator.com/item?id=33524683 https://news.ycombinator.com/item?id=33524683
- teruakohatu 4y agoI wonder what the cost of the reboot was to the FAA itself. Were a legion of contractors pulled out of bed to moniter the 90 minute reboot?
- dang 4y agoRecent and related: FAA NOTAM System Outage - https://news.ycombinator.com/item?id=34337158 https://news.ycombinator.com/item?id=34337158 - Jan 2023 (206 comments) US Halts Flights Nationwide After Key FAA System Goes Down - https://news.ycombinator.com/item?id=34337807 https://news.ycombinator.com/item?id=34337807 - Jan 2023 (36 comments)
- toddm 4y agoOn a positive note, you could have been on Amtrak: https://www.nytimes.com/2023/01/11/us/amtrak-delayed-va-fl.html https://www.nytimes.com/2023/01/11/us/amtrak-delayed-va-fl.h...
- timcavel 4y ago[dead]
- maxbond 4y agoI will always elect to be stuck on an Amtrak than stuck on a plane or in an airport. I'll take a book to the observation car and read next to the Mennonites doing their knitting and love every minute of it, no worries from me.
- robswc 4y agoWhat is stopping you from doing that in an airport?
- maxbond 4y agoI suppose nothing, I'll just be miserable while I'm doing it because I hate airports. A personal preference.
- themitigating 4y agoWhat is your preference based on?
- reidjs 4y agoProbably he’s been to… any airport. Maybe they’re bearable if you can get into a lounge
- maxbond 4y agoMore or less. The chairs are hostile to stop you from getting too comfortable. I've had 6 hour layovers before and had to sleep in those terrible chairs under fluorescent lights. On the last flight I took there was barely any good near my gate, there were like 4 bars instead. You can't leave your luggage anywhere, so you're dragging it all over the place. You get jammed like sardines into the plane, and you can't stretch your legs. You can't really relax because you have to stay abreast of announcements, in case they change your gate or cancel your flight or what have you. Everyone is on edge. Et cetera. On the train there's more room, you can move around, and the atmosphere is more relaxed. Some people like planes. I saw an HN comment where someone said they took lots of "flights to nowhere" over COVID and used the plane as their office. I don't understand that at all. But to each their own. I will say the views are unbeatable.
- mgsouth 4y agoQuote from FAA: "Our preliminary work has traced the outage to a damaged database file." UK news source The Independent is reporting that Nav Canada's NOTAM system also suffered an issue.[1] Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. edit: Better reporting of Canada's issue from Canada's CBC (and frankly, better reporting about the US, too). [2] "In Canada, pilots were still able to read NOTAMs, but there was an outage that meant new notices couldn't be entered into the system, NAV Canada said on social media." "NAV Canada said it did not believe the outage was related to the one in the U.S., but it said it was investigating." [1] https://www.independent.co.uk/news/world/americas/canada-flights-system-outage-grounded-b2260500.html https://www.independent.co.uk/news/world/americas/canada-fli... HN thread https://news.ycombinator.com/item?id=34347520 https://news.ycombinator.com/item?id=34347520 [2] https://www.cbc.ca/news/us-air-travel-chaos-notam-outage-1.6710734 https://www.cbc.ca/news/us-air-travel-chaos-notam-outage-1.6...
- markdown 4y ago> traced the outage to a damaged database file Lil' Bobby Tables strikes again!
- noman-land 4y agoMy first thought as well.
- deleted 4y ago[deleted]
- asveikau 4y ago> "Our preliminary work has traced the outage to a damaged database file." > Speculation: corrupting input, either international or North American? E.G. UTF-8, SQL escape, CSV quoting. I read it as filesystem corruption from a bad disk, coupled with redundancy that doesn't actually work.
- tigerBL00D 4y agoWhat kind of database are they using, I wonder, to end up with such a spectacular failure?
- noobermin 4y agoIs this the first time this system has failed so spectacularly? While the failure is notable, it does speak to the uptime of this system.
- thrdbndndn 4y agoI agree. Not exactly that we need to praise them or anything, but I think sarcastic tone in some replies ("if only they did something BASIC") aren't needed.
- kilotaras 4y agoIt's a different tradeoff. Systems that fails and *executes recovery path* from time to time tend to be more resilient to catastrophic failures. Ideally you would want the combination of both: system doesn't usually crash/fail, but you force it to do so regularly. See https://en.wikipedia.org/wiki/Chaos_engineering https://en.wikipedia.org/wiki/Chaos_engineering
- ilyt 4y agoUptime is kinda moot measure. If it had uptime of 2 days but only went down for a minute at a time that would not be a problem
- eddsh1994 4y agoI find it quite weird multiple things failed on the same day, such as the UK post office system unable to deliver abroad
- m463 4y agoI suspect this is just statistics, like "it is likely that two people will have the same birthday in a roomful of people"
- kqr 4y agoThe other classical example being the large number of Prussian soldiers killed by horse kicks in the late 19th century.
- alexchamberlain 4y agoWould this be an example of the Poisson distribution? (Perhaps exaggerated by post holiday upgrades?)
- deleted 4y ago[deleted]
- jakeogh 4y agoAccident or not, there are zero situations where a op/drill or real apt hack or would be officially ack'ed. Plausible deniability is a requirement of many centralized control systems.
- gojomo 4y agoAFAICT, this NOTAM system is a nationwide bulletin-board, using some cryptic standard abbreviations (to save space as if they were paying 1990s SMS), usually filtered by locale/coordinates/path, so pilots have the latest news hat might affect their flight plans. Has anyone seen exactly how many NOTAM messages are generated per day, and how long of a look-back is required? From 50k feet, it looks like something that could be replaced with a cryptographically-authenticated massively-replicated virtual data structure that's oblivious about, and robust against all sorts of failure in, the exact systems, languages, update-paths, etc used to keep it in sync or implement any one user's view. All that pilots need to know is: "I have a full local copy, signed by the right update-authorities, as of roughly-now." From a glance, NOTAM's uptime this year looks worse than Ethereum, but a bit better than Solana or Binance Smart Chain.
- walrus01 4y agocryptic abbreviations in the NOTAM system reminds me of telcos using CLLI codes for unique geographical locations of physical telecom infrastructure sites - a legacy of 1960s mainframe stuff where the number of characters in a database field for text entry was extremely constrained. https://en.wikipedia.org/wiki/CLLI_code https://en.wikipedia.org/wiki/CLLI_code
- jquery 4y ago> cryptographically-authenticated massively-replicated virtual data structure that's oblivious about, and robust against all sorts of failure in, the exact systems, languages, update-paths, etc used to keep it in sync or implement any one user's view. All that pilots need to know is: "I have a full local copy, signed by the right update-authorities, as of roughly-now." You've solved one problem and created several more problems.
- gojomo 4y agoCrypto-authenticated update & replication works really, really well nowadays. Lots of open-source support, extensive testing, proven track records even in adversarial deployments. Especially for a simple log-like system, with a limited number of permissioned authorities.
- kgeist 4y agoThere was a similar situation in Russia just a year ago (a federal aviation agency not having access to its online system): https://mentourpilot.com/breaking-russian-aviation-authority-suffers-cyberattack/ https://mentourpilot.com/breaking-russian-aviation-authority... >Due to temporary lack of access to Internet and malfunction of the electronic document flow system of Rosaviatsia the Federal Agency for Air Transport is switching to paper version. >“The document flow procedure is being determined by the current records management instructions. >“Information exchange will be carried out via AFTN channel (for urgent short message) and postal mail. >“Please make this information available to all Civil Aviation Organizations.”
- mylidlpony 4y agoI saw some sources claiming that was a hack, but not sure how credible that claim was.
- catminou 4y agoI recall years ago in the mid 2000s a moderately sized US ISP that sold internet access in office buildings to mostly businesses had a rather interesting outage. They had switches (Extreme switches - known for their purple color aka Barney the dinosaur switches) in the basement or many buildings and backhauled to a colo to the rest of their network. They were doing some mass network upgrade during the early hours maintenance window and devices weren't coming back. I don't think they had OOB access to their boxes and it requires someone going out and recovering them after hours of downtime and getting techs there. The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades.
- jdhendrickson 4y agoI had a boss purchase those switches without consulting me, he liked the color. They were pure garbage. Thanks for reminding me of that, I'll have to ping a former co-worker and have a laugh.
- bpye 4y ago> The culprit? They downloaded the OS image using FTP and forgot to set binary mode. Various other network kit vendors I recall would do some level of validation (and anyone downloading should be checking the hash). RCA sent to customers was vague and just said it was a corrupted image / failed upgrades. I've always been paranoid about this sort of thing when applying BIOS or firmware updates, despite those almost always having checksum validation, but I guess my caution is not unwarranted.
- zwilliamson 4y agoI bet a “corrupt file” means somebody accidentally dropped a table
- deleted 4y ago[deleted]
- mike_hock 4y agoTower, this is flight AA91347, please be advised'; DROP TABLE "notams"; --
- ComodoHacker 4y agoTFA clearly says "database". They dropped the whole database.
- dmix 4y agoEarlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was intended to prevent outages because it allows for seamless switching between the two channels without impacting air traffic control should a problem occur in the active channel. The Agency believed this redundancy would make it unlikely that a problem in one channel could migrate to the other channel. In addition, to provide additional backup capabilities during ERAM’s implementation, FAA planned to temporarily maintain EBUS, its pre-existing backup system, before phasing it out completely beginning in 2015, to rely solely on ERAM’s dual channels. > However, problems experienced during and since ERAM’s implementation have shown that the system remains susceptible to dual channel failures. As a result, FAA decided to maintain EBUS much longer than intended because air traffic controllers currently rely on EBUS for backup. However, with FAA’s ongoing and planned upgrades to ERAM, which will span the next 7 years, EBUS will soon become incompatible with the new hardware. As such, FAA plans to begin phasing out EBUS in April 2019, leaving ERAM without a backup system to supplement the system’s redundant dual channels. A $2 billion dollar Lockheed Martin system where a memory overflow takes down BOTH the main system and the backup system. Surely that's not a signal of fragility.
- jrockway 4y agoWhat's funny about this is that the FAA obviously knows what can go wrong with "yeah we have two of them" as they wrote ETOPS regulations to avoid some of the common pitfalls or amateur mistakes. They then failed to apply that to their software. Obviously at a big government agency, the same person is not writing both aviation regulations and software procurement contracts, but the institutional knowledge is there. Nobody thought to be as paranoid about software as they are about planes flying over the ocean, but honestly, paranoia is good if you want reliability.
- dmix 4y agoThis seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you keep them in sync the distinction comes down to hardware and lower-end systematic issues. But most orgs can't realistically have two distinct software systems. How do you create proper isolation or failure mechanisms between them? I'm guessing this sort of thing is what you mean by their experience with ETOPS.
- shemnon42 4y agoWas the file in ASCII when it should have been in EBCDIC?
- defrost 4y agoThat's just silly. Obviously it was due to details being entered using the now standard International foot when the software expected the now obsolete US survey foot [1] .. the discrepency across a transcontinental flight was large enough for a pilot to fall through. [1] https://www.nist.gov/news-events/news/2023/01/new-years-eve-2023-marked-retirement-us-survey-foot https://www.nist.gov/news-events/news/2023/01/new-years-eve-...
- soneil 4y agoMy money would have been on a byte-order mark
- chx 4y ago90 minutes boot time... I have a feeling the summarized age of the hard drives in that system exceeds the age of the United States itself (which is 247 years). When fsck was introduced in 4BSD in 1980, it checked every filesystem on boot because it had no better idea. If this thing is, say, forty years old then that's exactly the right age for this...
- nix23 4y agoMaybe you haven't seen big servers, but some blade-servers need >15 minutes just for the hardware check, same with big switches and san's.
- jonplackett 4y agoIf stuff this bad can happen by accident, we’d better hope there’s never a serious nation state cyber attack on this kind of system.
- pharmakom 4y agoI was recently rewatching some old Taleb talks about fragility. Our software systems are extremely fragile. I wonder if there is any way we can make them anti-fragile? What would this look like?
- f1refly 4y agoTDD helps a lot against fragility imo. The main issue is explaining why your project takes so much longer as your tests force you to work out every edge case.
- cbrogrammer 4y agoTDD helps close to 0 % with fragility and only exists to help development
- danuker 4y agoHow would you apply TDD to a backup system? By writing backup and restore tests? What if one test fails unreproducibly?
- ilyt 4y agoWell, you should test your backups. But TDD is entirely unrelated to the problem aside from including "test" in the name
- ilyt 4y agoTDD not really. Having tests help, writing them first is irrelevant
- icedchai 4y agoIt doesn't. All those unit tests don't guarantee you have a working system, especially when everything is mocked out (which is often the case.) End-to-end and integration tests are much more helpful. But even then, they won't look at operational concerns like backup and recovery.
- karlicoss 4y ago
- KronisLV 4y agoIn the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not corrupted (e.g. display/playback fails). If you have compression as a part of the backup process, then the compressed archive should also be validated, that it's not corrupt and that the data can also be extracted. Only then would checksums of the backup be actually useful, once you'd know that you don't just have a bunch of mostly useless binary data on hand. Sadly, the tooling just isn't there (e.g. CLI utilities for validating everything) and there are far more file formats than most people want to concern themselves with.
- dolmen 4y agoWell I would say that you must ensure that using the backup will solve the problem before choosing that solution. This implies that you must take time to analyze the source of the problem first.
- marginalia_nu 4y agoSeems like it would be prohibitively time-consuming. For a moderately large data-set (of say a few hundred Tb), by the time the backup of your data is vetted and validated, it's months old and effectively useless.
- ilyt 4y agoYet you still need to do it, even if it is every few months, else you will eventually hit the snag. If you don't have an option to restore the data your backup might as well not exist.
- capableweb 4y ago> In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". I've been trying to toot this horn at every place I've consulted/worked at, with the most common response from software engineers being: "We're using the AWS backup system of the database, of course it works, no need to test it" People have way too much trust in systems overall, especially ones they have no inside information about. And people also seems to default to "Of course it works" without testing it, instead of "How would I know it works if I haven't tested it?" or even "I don't know".
- BuckyBeaver 4y ago"...central database for all NOTAMs (Notice to Air Missions) nationwide" No. NOTAM is "notice to airmen."
- twblalock 4y agoNope. https://www.faa.gov/about/initiatives/notam/what_is_a_notam https://www.faa.gov/about/initiatives/notam/what_is_a_notam
- mango7283 4y agoI'm trying to imagine how many times this conversation is repeated across the internet and want to find the most entertaining exchange!
- javawizard 4y agoNote that this was changed about a year ago. https://www.faa.gov/documentLibrary/media/Order/7930.2S_Chg_2_dtd_12-2-21.pdf https://www.faa.gov/documentLibrary/media/Order/7930.2S_Chg_...
- BuckyBeaver 4y agoThis is the "explanation:" "These changes include modifying the acronym NOTAM from Notice to Airmen to the more applicable term Notice to Air Missions" This is not explanatory, and sounds like more pandering.
- vtail 4y agoYou are both right: https://www.faa.gov/documentLibrary/media/Order/7930.2S_Chg_2_dtd_12-2-21.pdf https://www.faa.gov/documentLibrary/media/Order/7930.2S_Chg_...
- mango7283 4y agoLooking at the response in the rest of the comment chain I wonder if it's such that he knows but does not accept the change.
- bob1029 4y agoIt seems like every year or so we have some gigantic technology meltdown in this industry. Imagine something like this happening at the NYSE, CME, et. at. Or, simply think about the last time you heard about a nationwide credit/debit card outage... Why can't we have our national infrastructure systems running at least as reliably as the Amex network? These systems are all information clearinghouses at the end of the day. If we have matching engines that flawlessly process millions of trades per second every day and mainframes that provide resilient source of truth, I think we could consider the same for a life safety critical system as well.
- capableweb 4y agoAll of those systems have outages all the time, mostly isolated incidents when it comes to NYSE, CME and similar (here is the outage page for NYSE for example: https://www.nyse.com/market-status/history https://www.nyse.com/market-status/history). Visa, Mastercard and others have nationwide outages from time to time too, they're in no way invincible like you seem to imply. Latest Visa outage I can remember must have been around 2018/2019 sometime, but probably it happened later than that too, except I didn't notice it then.
- silvestrov 4y agoHow many people do you thing will get fired due to the grounding of airplanes? versus: how many people would get fired if NYSE went down due to "we didn't think we needed backups". incentives matter. In political systems the main goal is to be able to point the finger at somebody else, not to ensure things run well.
- NikolaNovak 4y ago>> Or, simply think about the last time you heard about a nationwide credit/debit card outage... Last year in Canada when Rogers had a meltdown. E.g. https://dailyhive.com/vancouver/everything-impacted-rogers-outage https://dailyhive.com/vancouver/everything-impacted-rogers-o... Bell goes down more rarely but also can take large swathes of nation out.
- __MatrixMan__ 4y agoIt sounds like the kind of problem that blockchains were invented to prevent. Important systems should not blindly replicate data, they should validate it.
- gte525u 4y agoThis is the opposite of a blockchain in that there are centralized, trusted authorities (FAA, EASA) distributing this info.
- __MatrixMan__ 4y agoOk so it wouldn't be a public blockchain. The point is that when you replicate data you reject it if it doesn't conform to some ruleset. That way data that would break the system fails to replicate, and the scope of the problem is limited to whoever (or whatever) entered the bad data.
- zacharytelschow 4y agoData validation is a basic expectation in many technology stacks. I don't understand why blockchain would be uniquely positioned to resolve this problem.
- jve 4y agoFrom the article and comments I don't understand - are they talking about backup (copy of data) or backup system (nuke your primary system, failover to backup system and just keep working) ? I think it is a SYSTEM > It has a backup, which officials switched to when problems with the main system emerged, according to the source. > Officials ultimately found a corrupt file in the main NOTAM system, the source told CNN. A corrupt file was also found in the backup system. Perhaps the backup system is just using data from main system which currently is compatible but won't be in the near future? Still there should be data copy somewhere, right (with corrupt data...)?
- bob1029 4y agoI think you might be right. I am starting to wonder if this "backup" is an online log replica of the production system. Failover doesn't work out in the situation where you are replicating trash. A "reboot" from an actual backup/snapshot would be required if you ate a bad log stream.
- Eddy_Viscosity2 4y agoI am so tried of all this corruption in the government.
- deleted 4y ago[deleted]
- bambax 4y ago> NOTAMs (Notice to Air Missions) OT, but didn't NOTAM use to mean Notice to Airmen? It's still that on the ICAO website: https://www.icao.int/safety/istars/pages/notams.aspx https://www.icao.int/safety/istars/pages/notams.aspx Did it change, or is this a CNN initiative? (And how does one notify a mission?)
- JCM9 4y agoThe original definition was deemed to be not inclusive by DEI so address that they kept the acronym but changed the meaning.
- ThinkingGuy 4y agoThe FAA's website has the updated nomenclature: https://www.faa.gov/about/initiatives/notam/what_is_a_notam https://www.faa.gov/about/initiatives/notam/what_is_a_notam
- _ea1k 4y agoThe US changed it. AFAIK, the icao definition has not changed, though I guess it could in the future. Given that the US often does these kinds of things first, I've often wondered if it is a US-centric way of thinking?
- noisy_boy 4y agoMakes one think if they should have hourly snapshots combined with integrity checks of the snapshots (and the main database, assuming it is feasible to run that for an online database). They will know within an hour if a snapshot is corrupt + they can restore from the last known good snapshot and not lose too much data and be up relatively quickly. Obviously like most software, there is a bunch of "it depends" but would be interesting to know more about the system.
- VLM 4y ago"The source said the NOTAM system is an example of aging infrastructure due for an overhaul." The concept of "lint" a file and the concept of verifying a backup are truly ancient and coincidentally are also completely absent from the description of the problem. People that felt no need to do either 20 years ago are certainly not going to start doing it today, especially when the inevitable system failure in the distant future results in yet another lucrative replacement contract in the distant future for system 3.0.
- Someone1234 4y agoI think you're misunderstanding what the "backup" was in this context (in particular as the article is unhelpful by calling it a backup file, which isn't accurate). In this case it is two PRODUCTION systems running concurrently. The primary and the secondary (article calls the "backup"). Primary went down due to corruption, but the identical secondary system couldn't be switched to because the corruption also occurred there.
- VWWHFSfQ 4y agoSo the backup system failed
- bryan_w 4y agoNo, it's the RAID vs Backup distinction. If you have 2 disks in Raid 1, you have a system that can survive one drive dying (high availability), but you also need a backup system (offline). It sounds like their high availability system failed but they probably were able to restore from backup (offline)
- phaedrus 4y agoPerhaps they should have called it a "tandem" system or hot-standby.
- huslage 4y agoYou all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be minimized in any way. This is not some government corruption thing. There are hundreds of thousands of lives at stake every day. I, for one, appreciate the work that the FAA does to keep us safe in the air and appreciate that this was handled appropriately. Everything breaks. It's just a matter of how and when and what we do with it when it does. The FAA handled this outage appropriately and in a timely manner. I feel for the engineers who had to work on this incident.
- greggarious 4y agoIf they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.
- Cthulhu_ 4y agoMaybe the better question to ask is how bad would it be if they *weren't* so risk-averse.
- jjk166 4y agoImagine how bad it would be if Kim Jong Un *weren't* so pro-american. It's a ridiculous hypothetical. Everything can always be worse, that is not a testament to an entity's quality.
- r3trohack3r 4y ago> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.
- PaulWaldman 4y ago>In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system – a significant decision, because the reboot can take about 90 minutes, according to the source. How often is the system rebooted? I haven't heard of it happening before and some quick searching didn't find any historical examples. Is it a scheduled event and no flights have departure times while this maintenance is taking place? I've found that issues tend to manifest on production systems with infrequent restarts.
- bink 4y agoIt sounds more like there are a pair of systems where one operates as the primary and the other as a backup, and these can swap in case of problems with the primary. That's fairly typical in critical infrastructure. I've done testing on these types of systems in the past (carefully) and the owners will often let you test against the system that's currently the "backup". So they probably restarted the "primary" after performing a fail-over.
- throw7 4y agohttps://www.rawstory.com/faa-outage-cause/ https://www.rawstory.com/faa-outage-cause/ "An engineer 'replaced one file with another,' the official said, not realizing the mistake was being made Tuesday.
- NelsonMinar 4y agoMaybe someone tried to NOTAM with an emoji.
- pmarreck 4y agoShoulda used ZFS... /shrug
- xbar 4y agoTo be fair, that system has pretty good uptime.
- exabrial 4y agoBunch of Node.js programmers lecturing the FAA on reliability is hilarious. Your framework is out of date, go use whatever the latest js framework published today is.
- czbond 4y agoI'm sorry you can't fly today because the NOTAM system sent NaN alerts.
- exabrial 4y agoNot An airplaNe
- eurasiantiger 4y agoWhat’s even more hilarious is expert professionals being openly derisive and spreading stereotypes about others based on programming language preferences. Do you hate JavaScript programmers? Do you also agree with your friends how of people of different color are all the same? Do you feel the need to call out other people’s sexual orientation, at least behind their back when only your friends are hearing? Some of your friends might be JavaScript programmers, you know.
- czbond 4y agoI don't believe it. A system that important would have significant scrutiny on its components even by a legacy organization. Maybe it was a corrupted HACKED filed, I'd believe that. After all, China isn't too happy with our games about Taiwan.
- prirun 4y agoI wonder if all of their computer systems have ECC RAM. Kinda doubt it.