15 ms·

FAA NOTAM System Outage

422 points by chucksmash 4y ago

steponlego 4y ago
They're resuming operations despite "still searching for a cause." Comforting.
- ceejayoz 4y ago
  You've never rebooted a computer to get it working again, without immediately knowing why it crashed?
  - virusuy 4y ago
    I made that my career
  - steponlego 4y ago
    Not since I stopped using Microsoft Windows.
jerpint 4y ago
Looks like it’s been fixed: https://twitter.com/BBCBreaking/status/1613174866047340545 https://twitter.com/BBCBreaking/status/1613174866047340545
sandworm101 4y ago
I've got a trip to take in a couple weeks. I have a choice: 4 hours of driving followed by two regional flights. Or just forget flying and drive 16 hours point to point. Between today's issues and all the headaches at YVR over the holidays, driving 16 hours seems more comfortable and less stressful than rolling the dice with regional airline travel. I would rather be stuck driving my car for a day than spend a day sleeping on the floor of an airport lobby.
- outericky 4y ago
  Depending on the length of the stay and whether you need a car, I'd do the road trip in a heartbeat.
  - mgkimsal 4y ago
    Whenever I opt for 'roadtrip', almost inevitably, about 4-5 hours in, I wish I were flying. When I fly, I don't usually say "oh, I wish I'd driven" (unless there's a delay or problem). As I've driven more, I need to allot myself more stops. If a trip is over 10 hours, I'll probably split it in 2 days, and stay overnight some place. 20 years ago, I would have 'pressed on', but... it's usually not worth it any more. But some of the 'cost savings' of driving go away when you factor in hotels.
- toast0 4y ago
  How many people in the car and how many drivers? What do you expect for the weather? Anywhere you want to stop somewhere in the middle? Probably less effort and time to fly, but a lot of waiting without agency.
- lost_tourist 4y ago
  99% chance this won't happen to your flight.
  - sandworm101 4y ago
    It happened last time on a nearly identical itinerary. First flight was late. Missed connection. Landed at second airport too late for the bus (military to base where my car was parked). Had to spend night in hotel waiting for bus the following day. Time before that, I arrived on time but my bag did not. Had to wait at airport for six hours for bag. Missed bus. Hotel. Bus the following day.
- buffington 4y ago
  I used to need to be in a city every other week that was only a 1.5 hour flight. I decided to see if driving might get me there in comparable time if I compared when I closed my front door to when I opened the door of my destination. I was surprised to find that the time it took as almost the same. That's because flying requires: * start drive to airport (45 minutes) * park and get on a shuttle (15 minutes) * get to gate (30 minutes) * wait (60 minutes because I'm early by an hour) * fly (1:45) * taxi to gate (5-45 minutes) * walk to light rail stop (10 minutes) * ride light rail too stop (45 minutes) * walk to destination (15 minutes) Door to door flying: about 6 hours Driving: * start driving (6 hours) * fuel stop (15 minutes) * park (15 minutes) * arrive If I didn't obey speed limits, I could cut an hour or more off that 6 hours. That said, I still preferred flying. Hard to take a nap or space out while driving for 6 hours.
ineedasername 4y ago
I'm surprised there's not more activity on this here on HN. I guess info is limited, so I found what I could from ATCSCC notices for today [1]. These are all the notices but portions only w/ redundant bits removed. See link in citation if you want full messages. Times here appear to be GMT: 004 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 00:47 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT OUTAGE ISSUES FOR THE UNITED STATES NOTAM SYSTEM. 013 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 04:18 THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. SINCE THEN NO NEW NOTAMS OR AMENDMENTS HAVE BEEN PROCESSED. TECHNICIANS ARE CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR RESTORATION OF SERVICE AT THIS TIME. 014 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 04:19 HE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT OUTAGE ISSUES... 015 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 07:42 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT... 016 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 07:44 THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. TECHNICIANS ARE CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR RESTORATION OF SERVICE AT THIS TIME. THERE IS CURRENTLY A HOTLINE IN EFFECT WHICH HAS NAIMES/FAA FACILITIES/STAKEHOLDERS IN ATTENDANCE. 020 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 10:58 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED... 030 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 12:37 THE NOTAM OUTAGE HOTLINE HAS BEEN TERMINATED. In short, the terminology being used so far is "equipment failure". [1] 1/11/2023 link to the day's messages w/ index numbering for each one per my above quotes" https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATCSCC&AdvisoryCategory=All&dates=Wednesday%2C+01-11-2023&AirFlow=AirFlow&Ctop=Ctop&Gstop=Gstop&Gdelay=Gdelay&Route=Route&Other=Other https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATC...
- zitterbewegung 4y ago
  Probably because it’s not about a package manager , a social network, Cloud Service and a chat application.
  - eternalban 4y ago
    [heads-up: 'r/He\ is/I\ am' in your personal landing page]
eigenform 4y ago
Space weather?
adolph 4y ago
Where are the blockchain advocates? This seems like a perfect use case for a distributed ledger.
TecoAndJix 4y ago
In case you were like me and did not know what a NOTAM is: https://www.faa.gov/about/initiatives/notam/what_is_a_notam https://www.faa.gov/about/initiatives/notam/what_is_a_notam
dreamcompiler 4y ago
TIL that NOTAM now stands for "Notice to Air Missions." Back when I was flying it was "Notice to Airmen." Wondered when they would get around to fixing that. Guess they finally did.
- wstuartcl 4y ago
  Both are still in use. For instance during your pilots tests either is considered an equivalent answer.
  - NelsonMinar 4y ago
    that's the test that makes you a Temporary Airman.
- pookha 4y ago
  They JUST changed it. Made a point of signifying to the masses about it last month: https://generalaviationnews.com/2021/12/07/notams-are-now-notices-to-air-mission/ https://generalaviationnews.com/2021/12/07/notams-are-now-no... What this tells me is a group of government busy-workers are playing around with complex systems that can dramatically impact millions of people.
anigbrowl 4y ago
Wondering if this is just standard aging infrastructure embarrassment or could involve a cyberattack? Cause if I were a state actor with a beef this is exactly the sort of soft-but-expensive injury I'd want to inflict: extremely disruptive, but also hard to assess or explain to the public.
TechBro8615 4y ago
[flagged]
- tjohns 4y ago
  Most glitches are localized, or there's backup systems, or you can fall back to manual human processes. If pilots/dispatchers can't get current NOTAMs, the aircraft can't safely depart - especially if they're flying IFR. There's no way around it. (One could argue about whether it's legal for general aviation traffic to depart VFR - the regs say you need "all available information", and the information isn't available. But that's arguing semantics.) There was a NOTAM outage about a year ago where Leidos' (the Flight Service contractor in the lower 48) computer system failed and prevented entering new notices, but as a backup they were able to call Alaska FSS on the phone. Since Alaska's FSS is still run by the federal government and used a different computer system, their system was still online. But in this case the outage is on the FAA's (upstream) side.
- themitigating 4y ago
  Evidence?
  - TechBro8615 4y ago
    Do you have any evidence of a technical glitch? Or evidence that there has never been such a technical glitch since 2001?
    - smlacy 4y ago
      Do you? Your arguments are classic whataboutism, and conspiracy/fearmongering. Read up on the Burden of Proof. If you make outlandish claims, they have to come with substantial evidence.
      - TechBro8615 4y ago
        I think it's an outlandish claim that a technical glitch happened today that caused such a degree of widespread outage that the FAA had to take a step it hadn't taken since September 11, 2001.
sidcool 4y ago
What is the root cause?
allen-munsch 4y ago
I was wondering if this might be related to elastic's "several customer facing deployments deleted across multiple regions" issue from earlier today? The timing seemed to line up a bit. https://status.elastic.co/ https://status.elastic.co/
throwaway290 4y ago
Could it have been a nuclear option used by a federal agency to hold someone's possible escape? Perhaps they were not sure which plane, so they had to hold all. Or it was just the most immediate route
zrail 4y ago
As of 06:24 US Eastern the appears to no longer be true. There is a NOTAM outage but no full ground stop is mentioned, just a few specific due to weather. Edit: obviously a developing situation, see responses to this comment for current advisories.
- emn13 4y ago
  https://www.reuters.com/business/aerospace-defense/us-faa-says-flight-personnel-alert-system-not-processing-updates-after-outage-2023-01-11/ https://www.reuters.com/business/aerospace-defense/us-faa-sa... Clearly still developing; we'll see.
- callahad 4y ago
  I don't know how these things work, but the most recent advisory, 027, seems to imply that United Airlines has a full ground stop until 15:00 UTC / 10:00 Eastern. Edit: Of course as I was typing that, Advisory 028 pops up with a nationwide ground stop until 9:30 Eastern: "GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS" https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_date=01112023&facId=DCC&title=NATIONWIDE+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...
- perihelions 4y ago
  - "All US domestic flights have been grounded for several hours because of a glitch with the flight control system." https://www.bbc.co.uk/news/world-us-canada-64236047 https://www.bbc.co.uk/news/world-us-canada-64236047
  - chucksmash 4y ago
    Yep, the situation has changed since the post was made. Parent comment was correct at the time, however. At 5:40am Eastern, the agent at my departure gate told us that there was a ground stop affecting all flights, which was reflected in my original title. This was not reflected in the NOTAM I posted, as was pointed out by several early commenters, and in fact my flight was able to board and depart around 7:00am Eastern before the ground stop took effect.
deleted 4y ago
[deleted]
nexus7556 4y ago
Any source for the claim of a ground stop? I see plenty of flights still taking off.
- kzrdude 4y ago
  The most recent one seems to say something like that. ATCSCC ADVZY 023 DCC 01/11/2023 ALL UAL MAIN AND SUBS GROUND STOP Not that I know how to read it. The event time is three minutes ago. https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_date=01112023&facId=DCC&title=ALL+UAL+MAIN+AND+SUBS+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_dat...
  - deleted 4y ago
    [deleted]
  - UhUhUhUh 4y ago
    Reads United Airlines apparently.
  - nexus7556 4y ago
    My understanding is that applies to United Airlines and it’s regional partners. It seems each airline is making their own decision. This Reddit post seems to have more info: https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_outage https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...
  - eftychis 4y ago
    Air Traffic Control System Command Center Advisory 23 by the Traffic Management Command Center (DCC) at date [..] - all united airlines ... ground stop. Here is about ground stops: https://www.faa.gov/air_traffic/publications/atpubs/foa_html/chap18_section_13.html https://www.faa.gov/air_traffic/publications/atpubs/foa_html...
- alamortsubite 4y ago
  I can see that East Coast UA flights are currently delayed because of this.
sneak 4y ago
Nothing in the linked page confirms the ground stop mentioned in the title. Reddit comments say that there is no FAA ground stop, but airlines are making individual decisions about it. https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_outage/j3vo6cv/ https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...
- Klonoar 4y ago
  There was a brief ground stop (I'm currently flying and it momentarily held us up) but Delta at the least isn't stopping entirely.
- emn13 4y ago
  https://www.reuters.com/business/aerospace-defense/us-faa-says-flight-personnel-alert-system-not-processing-updates-after-outage-2023-01-11/ https://www.reuters.com/business/aerospace-defense/us-faa-sa...
harrego 4y ago
I really like the advisory page -- all info, no sugar.
- deleted 4y ago
  [deleted]
- gonzo41 4y ago
  https://lite.cnn.com/en https://lite.cnn.com/en It's the best way to get the news. No filler, no adds and it works on a 2g mobile.
  - deleted 4y ago
    [deleted]
  - harrego 4y ago
    http://68k.news http://68k.news is a fun one too.
  - deleted 4y ago
    [deleted]
  - ceejayoz 4y ago
    See also: https://text.npr.org/ https://text.npr.org/
  - kfrzcode 4y ago
    See also: https://en.m.wikinews.org/wiki/Main_Page https://en.m.wikinews.org/wiki/Main_Page
  - kfrzcode 4y ago
    Not sure CNN is the "best" way to get the news. Try this: https://en.wikinews.org/wiki/Main_Page https://en.wikinews.org/wiki/Main_Page No ads, no adds, no filler, no filters. Make the news objective again. Repeal Smith-Mundt. https://www.congress.gov/bill/112th-congress/house-bill/5736 https://www.congress.gov/bill/112th-congress/house-bill/5736
chucksmash 4y ago
Updated title from "FAA Ground Stop for All North America Flights Due to NOTAM Outage" to reflect comments that initial title was not accurate.
- xattt 4y ago
  Confusing title, as Canadian departures with Canadian destinations do not appear to be affected.
testplzignore 4y ago
If I'm understanding the situation correctly: the system stopped processing updates yesterday afternoon and it wasn't a big deal, then they tried restarting it this morning to fix it and it didn't come back up, so now older NOTAMs from before the issue started are not available. Is that right? Do airlines/planes keep a cached copy of these? How large is the NOTAM database?
- stevehawk 4y ago
  Not really. And for most commercial pilots this may come down to whether or not ForeFlight caches theirs or not. I'd actually be really curious to see unofficial commentary from ForeFlight developers about this because it's a pretty unique scenario in the age of electronic flight bags.
- p_l 4y ago
  The problem isn't keeping a copy, it's sending updates that are very time critical (for example restricted areas around POTUS, any impactful event, etc.)
WinstonSmith84 4y ago
[flagged]
- ceejayoz 4y ago
  A small amount, last month? Yes, probably unrelated.
AYBABTME 4y ago
Are 2020s the years where the cold war is back as a lukewarm war, and also all of NA aviation needs to reconvene with its decrepit software? Next plot-twist, it was all ready to blow, but ultimately triggered by Russian/Chinese/North Korean hackers. Ok, taking off my tinfoil hat. Still, maybe I'm suffering of nostalgia bias but the last few years really make for a lot of prime source material for docudramas.
- deleted 4y ago
  [deleted]
- RagnarD 4y ago
  No need for a tinfoil hat. Such systems are prime targets for states hostile to the U.S. If this one wasn't before, it will be now.
- heisenbit 4y ago
  Forensic analysis will reveal that there were multiple hidden failures over the past years that were covered up by foreign hackers to ensure their backdoors were not revealed in an in-depth investigation.
  - peteradio 4y ago
    Free maintenance!
- yehosef 4y ago
  don't forget about this beauty https://techcrunch.com/2021/02/23/solarwinds-hackers-targeted-nasa-federal-aviation-administration-networks/ https://techcrunch.com/2021/02/23/solarwinds-hackers-targete...
ceejayoz 4y ago
https://twitter.com/AUStinAirport/status/1613134228321087488 https://twitter.com/AUStinAirport/status/1613134228321087488 > An FAA system outage is causing ground stops at AUS and other airports across the country. Arriving & departing passengers can expect delays this morning & through the day. If you’re flying today, it’s gonna be a rough one. edit: BBC now confirming a full ground stop. https://twitter.com/BBCBreaking/status/1613151796955856896 https://twitter.com/BBCBreaking/status/1613151796955856896 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_date=01112023&facId=DCC&title=NATIONWIDE+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...
deleted 4y ago
[deleted]
yehosef 4y ago
in unrelated news - https://techcrunch.com/2021/02/23/solarwinds-hackers-targeted-nasa-federal-aviation-administration-networks/ https://techcrunch.com/2021/02/23/solarwinds-hackers-targete... ;) /humor
- oldgradstudent 4y ago
  Next time my server is down I'll blame the Russians and Chinese too.
  - yehosef 4y ago
    If your org was hacked by the Russians/Chinese and was a national safety system, then you might be right to start there.
kzrdude 4y ago
ATC can tell pilots what they need to know, right?
- dx034 4y ago
  The question is, how will ATC know? I'm not that familiar with their systems, but maybe they rely on them to get critical information?
  - ceejayoz 4y ago
    ATC will be fine to get everyone on the ground safely. There are procedures for everything including ATC itself being down.
    - dx034 4y ago
      I never doubted that. The question is if they could've worked without a ground stop by having ATC give them all info.
      - ceejayoz 4y ago
        No. ATC gets busy enough as is, each plane needs NOTAMs covering their entire route, and if the system is down ATC isn’t getting them either anyways.
      - sokoloff 4y ago
        If I print the NOTAMs for a 600nm flight, they're going to typically go 4+ pages of NOTAMs that are fairly specific to my flight (just for departure, along route, and destination airports) and 15+ pages if I print all available NOTAMs in a fairly small font.
    - p_l 4y ago
      ATC can get them back down, but no new flights should start.
- McDyver 4y ago
  NOTAM notices are not only for planes but for other people and equipment using airspace, like drones, skydivers, ... Those are not always connected with ATC, but need the information from NOTAM to plan their activities
dx034 4y ago
There's a ground stop for all flights now. GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS DESTINATION AIRPORT; ALL FACILITIES INCLUDED: ALL GROUND STOP PERIOD: UNTIL 1430Z REASON: EQUIPMENT OUTAGE REMARKS: US NOTAMS SYSTEM DOWN
- Tepix 4y ago
  Why don't they use lower case letters?
  - yrro 4y ago
    https://fixingnotams.org/wp-content/uploads/2019/11/Field-Guide-to-Notams-OpsGroup.pdf https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...
- srmarm 4y ago
  This might be a silly question, but is that a NOTAM saying that NOTAM system is down? How does it get out?
  - zrail 4y ago
    As I understand it there's a hotline and periodic teleconferences and webinars today that for sure everyone at the airlines is participating it.
  - ceejayoz 4y ago
    The website and the system that gets it out to all the planes are related but not the same.
    - jonhohle 4y ago
      With all the hate in HN regarding SOA, here’s a great example of adding resilience by separating concerns at a systems level.
  - capekwasright 4y ago
josephd79 4y ago
Gonna be a bad day for those sys admins
- causality0 4y ago
  May none of our fuckups be bad enough to get an article written about them. Amen.
  - chasd00 4y ago
    heh, my boss was on a public sector project that went so far south she was stalked by news crews leaving the office. The experience left a pretty deep scar on her. This was before my time thank babyjeebus.
  - josephd79 4y ago
    Amen.
- acdha 4y ago
  Kinda wondering the odds of there being one crying into their coffee over a stack of memos saying they should do something before it fails, or some greybeard angrily pointing to their quality control concerns on a new system. Both are pretty common with federal IT.
nkcmr 4y ago
Why do all aviation IT systems feel like they are so fragile? It feels like not a single month can pass these days without some "Airline X has major outage and ruins tons of people's days!" story on the news. What is the tech behind these frail systems?
- lovecg 4y ago
  When was the last time NOTAMs went down? It seems remarkably resilient actually.
- dx034 4y ago
  More modern systems are also down once in a while. With aviation it's just more visible because one system can affect a lot of people. I'm not sure when ATC systems (which I would count NOTAM to be one of) have last failed, I always had the feeling they're very stable. It's usually airlines that struggle with their applications.
  - joshuahedlund 4y ago
    I never know how well it applies but headline outages always make me think of that “The Coming Software Apocalypse” Atlantic article from a few years back[0] [0] https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/ https://www.theatlantic.com/technology/archive/2017/09/savin...
- namdnay 4y ago
  I think it's just a question of media visibility. Aviation outage = queues in airports = easy article to write
- VLM 4y ago
  All hyper optimized systems become fragile until they collapse. The NOTAM system is ridiculous, no surprise it collapsed ... you can lose your ticket for flying thru a MOA that went active AFTER you took off. What could possibly go wrong with a "push" notification system for every tiny little detail of aerospace scheduling? No force is as destructive as the good idea fairy.
samwillis 4y ago
Edit 2 - * Ground stop Lifted * Update from FAA: >Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted. >We continue to look into the cause of the initial problem https://mobile.twitter.com/FAANews/status/1613171400801394688 https://mobile.twitter.com/FAANews/status/161317140080139468... --- Edit - Update from FAA: > Update 4: The FAA is making progress in restoring its Notice to Air Missions system following an overnight outage. Departures are resuming at @EWRairport and @ATLairport due to air traffic congestion in those areas. We expect departures to resume at other airports at 9 a.m. ET. https://mobile.twitter.com/FAANews/status/1613162735721746432 https://mobile.twitter.com/FAANews/status/161316273572174643... --- Flightradar24 are reporting: > Flights to US airports have been groundstopped by the FAA until at least 09:30 ET (1430 UTC). This means any flight not in the air already will not be allowed to depart. Image: flights over US now compared to same time last week. About 700 fewer flights airborne now. > Flights are being held as the FAA works to bring the NOTAM (NOtice To Air Missions) system back online. https://mobile.twitter.com/flightradar24/status/1613153479219986437 https://mobile.twitter.com/flightradar24/status/161315347921... Latest FAA update: > Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage. > The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information. https://mobile.twitter.com/FAANews/status/1613148579270459393 https://mobile.twitter.com/FAANews/status/161314857927045939... Statement from the white house: > The President has been briefed by the Secretary of Transportation this morning on the FAA system outage. There is no evidence of a cyberattack at this point, but the President directed DOT to conduct a full investigation into the causes. The FAA will provide regular updates. https://mobile.twitter.com/PressSec/status/1613153561289932800 https://mobile.twitter.com/PressSec/status/16131535612899328... I believe this means that nothing can take off, but aircraft already in the air can continue to their planned destinations.
- dx034 4y ago
  Looks like they published some kind of a "master NOTAM" now with all important information to restart as soon as possible. https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_date=01112023&facId=DCC&title=OPERATIONS+PLAN&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_dat...
rwmj 4y ago
Fun stuff about NOTAMs: https://fixingnotams.org/wp-content/uploads/2019/11/Field-Guide-to-Notams-OpsGroup.pdf https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...
- eternalban 4y ago
  "There are three parts to the problem: the system, the format, and the content. The system is actually quite amazing. The AFTN network connects every country in the world, and Notam information, once added, is immediately available to every user. Coupled with the internet, delivery is immediate. The format is, at best, forgivable. It’s pretty awful. It’s a trip back in time to when Notams were introduced. You might think that was the 1960’s, or the 50’s. In fact, it’s 1924, when 5-bit ITA2 was introduced. The world shifted to ASCII in 1963, bring ing the Upper and Lower case format that every QWERTY key board uses today, but we didn’t follow – nope, we’ll stick with our 1924 format, thank you." https://web.archive.org/web/20230104130153/https://www.faa.gov/air_traffic/flight_info/aeronav/notams/ https://web.archive.org/web/20230104130153/https://www.faa.g... "The FAA is changing the format for Notice to Air Missions (NOTAM) to align with international standards. The transition to the new format will ensure U.S. NOTAMs are compliant with standards set by the International Civil Aviation Organization (ICAO) and promote further global harmonization among neighboring Air Navigation Service Providers (ANSPs)." ICAO NOTAM format https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf Example! (decoder ring not included) "Q) EGTT/QMRXX/IV/NBO/A/000/999/5129N00028W005"
  - mzs 4y ago
    I want to point out that lower-case is possible with ITA2. Start at 4.6 of ITU-T Recommendation S.2 / 11/1988 in the Blue Book, basically 11111 & 11011 11111 switch case in locking/unlocking mode.
  - wstuartcl 4y ago
    Its even more amazing when you really sit down and think about all of the systems (many highly resistant to change like aircraft navigation systems) that all connect and utilize this service. What appear like simple changes and proposals in the aerospace realm are extremely hard to execute and many times take decades (multiple) to come into play. Global services/systems are even harder.
reaperducer 4y ago
Very entertaining to watch the local TV news this morning. The reporter keeps pulling people aside as they arrive at the airport and asking them what they're going to do with all the planes grounded. None of them know it's happening. So far, they have all admitted that they either don't look at the news, or don't look at the local news, except for one woman who defiantly said, "It wasn't on Facebook," as if that meant that since the grounded flights weren't in her social media feed that it wasn't real.
unionemployee 4y ago
The NOTAM system is embarrassing. This just tops it off. Actually, our whole ATC system and the way the FAA operates is embarrassing. It all works kind of like waste management in NYC. If they were to add new tech and efficiency, fewer people would have jobs, so we keep doing it the old way to the detriment of everyone else.
- dev_daftly 4y ago
  So basically it's the same as every other government entity?
- kfrzcode 4y ago
  Shirley you must be joking.
  - thesuperbigfrog 4y ago
    Rumack : I won't deceive you, Mr. Striker. We're running out of time. Ted Striker : Surely there must be something you can do. Rumack : I'm doing everything I can... and stop calling me Shirley!
    - jmbwell 4y ago
      > There’s a problem with the NOTAMs… >> The NOTAMs? What is it? > It’s a system for alerting pilots to hazards. >> I see, thank you (ETA: haha aw, well I thought it was funny)
- acdha 4y ago
  It’s easy to say things are about jobs but do we have any evidence of that? There’s a modernization project underway at the FAA so I’d bet that if you looked there’s someone who’s been trying for years to get attention on it before something like this happened.
- quantified 4y ago
  What embarrasses you about it, esp compared with more modern systems? This seems like the kind of critical infrastructure whose uptime requirements can make upgrading challenging once in place.
watersb 4y ago
REALLY IMPORTANT NOTAMS tend to alert pilots to RESTRICTED AIRSPACE or AREAS OF UNUSUAL AIR OPERATIONS. You can't fly anywhere near the President of the United States, the movements of which can change from day to day. We get NOTAMS about military aircraft operations; often military airspace which is usually permitted may be shut down for a week or so. Wildfire tanker operations or other natural emergencies can close an airspace. It's quite dynamic, all on top of the scheduled flight plans and routes.
- ru552 4y ago
  They are also for notifying pilots of potential caution areas. I used to operate a 550+ meter tower and when I had a tower light out, I had to file a NOTAM.
  - watersb 4y ago
    Cool, you filed NOTAMs; I just read them. If you like lots of daily mail, become responsible for the operation of a Cessna 172 (Skyhawk), the type of aircraft with the most units built of any, ever. Endless variations. Unending stream of paper NOTAMS via US first class post office... at least, when I had a 172, 20 years ago. These notices are exactly like a recall notice on an automobile: get someone to validate the steering wheel etc. I would suppose that many HN readers have gone through the car hardware remediation process. I would suppose that this class of NOTAM, although in my particular case the most common, are not immediately relevant to a total Ground Stop Order for the USA. But -- consider that airline and other commercial operators may have their own internal requirements that could ground a fleet in the absence of an ability to state that they have complied with all relevant FAA NOTAMS. Oh, the liability!!
  - watersb 4y ago
    (Holy cow, just did remedial maths: 550m is a Large Number! Would like to know more details.)
    - ru552 4y ago
      The tower was used by a TV station to broadcast their channels. You can see the curvature of the earth at the top when it's a clear day.
jxf 4y ago
Q: Is there anything like a systems architecture diagram for how NOTAM works? I'm curious about which parts might have failed and what the moving pieces are.
- tyingq 4y ago
  Some here, mixed in with other info: https://www.faa.gov/air_traffic/technology/swim/users_forum/media/SWIM%20Users%20Forum%2026%20Briefing.pdf https://www.faa.gov/air_traffic/technology/swim/users_forum/... https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%20Summit%20-%20Session%201%20-%20Connecting%20to%20NOTAM%20Data.pdf https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%...
sumanthvepa 4y ago
Was it database failure or did they get hacked?
snake_doc 4y ago
For anyone looking for previous FAA outages: In 2014, an U-2 plane’s flight plan crashed LAX’s ERAM system because the system ran out of memory. https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20to%20Address%20ERAM%20Outages%20Final%20Report%5E11-07-18.pdf https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20...
- GalenErso 4y ago
  Why do secretive spy planes need to file flight plans?
  - ceejayoz 4y ago
    Because they're not always doing secretive stuff, and even their secretive missions may start and end in airspace they have to share with friendly airliners. Sometimes it's a deliberate "we see you", too. https://twitter.com/Gerjon_/status/1475444704250630148 https://twitter.com/Gerjon_/status/1475444704250630148
  - pmontra 4y ago
    That was a training flight, it's in the PDF.
    - mikece 4y ago
      I imagine operational flights might "officially" start and end as routine training flights, at least on paper.
  - bobinator606 4y ago
    [dead]
- controlller 4y ago
  LAX, an air traffic control tower, does not employ ERAM. Los Angeles Center (ZLA), the overlying air route traffic control center, does.
belter 4y ago
https://twitter.com/FAANews/status/1613171400801394688 https://twitter.com/FAANews/status/1613171400801394688 " Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted...We continue to look into the cause of the initial problem"

422 points by chucksmash 4y ago

steponlego 4y ago

They're resuming operations despite "still searching for a cause." Comforting.

ceejayoz 4y ago
You've never rebooted a computer to get it working again, without immediately knowing why it crashed?
- virusuy 4y ago
  I made that my career
- steponlego 4y ago
  Not since I stopped using Microsoft Windows.

jerpint 4y ago

Looks like it’s been fixed: https://twitter.com/BBCBreaking/status/1613174866047340545 https://twitter.com/BBCBreaking/status/1613174866047340545

sandworm101 4y ago

I've got a trip to take in a couple weeks. I have a choice: 4 hours of driving followed by two regional flights. Or just forget flying and drive 16 hours point to point. Between today's issues and all the headaches at YVR over the holidays, driving 16 hours seems more comfortable and less stressful than rolling the dice with regional airline travel. I would rather be stuck driving my car for a day than spend a day sleeping on the floor of an airport lobby.

outericky 4y ago
Depending on the length of the stay and whether you need a car, I'd do the road trip in a heartbeat.
- mgkimsal 4y ago
  Whenever I opt for 'roadtrip', almost inevitably, about 4-5 hours in, I wish I were flying. When I fly, I don't usually say "oh, I wish I'd driven" (unless there's a delay or problem). As I've driven more, I need to allot myself more stops. If a trip is over 10 hours, I'll probably split it in 2 days, and stay overnight some place. 20 years ago, I would have 'pressed on', but... it's usually not worth it any more. But some of the 'cost savings' of driving go away when you factor in hotels.
toast0 4y ago
How many people in the car and how many drivers? What do you expect for the weather? Anywhere you want to stop somewhere in the middle? Probably less effort and time to fly, but a lot of waiting without agency.
lost_tourist 4y ago
99% chance this won't happen to your flight.
- sandworm101 4y ago
  It happened last time on a nearly identical itinerary. First flight was late. Missed connection. Landed at second airport too late for the bus (military to base where my car was parked). Had to spend night in hotel waiting for bus the following day. Time before that, I arrived on time but my bag did not. Had to wait at airport for six hours for bag. Missed bus. Hotel. Bus the following day.
buffington 4y ago
I used to need to be in a city every other week that was only a 1.5 hour flight. I decided to see if driving might get me there in comparable time if I compared when I closed my front door to when I opened the door of my destination. I was surprised to find that the time it took as almost the same. That's because flying requires: * start drive to airport (45 minutes) * park and get on a shuttle (15 minutes) * get to gate (30 minutes) * wait (60 minutes because I'm early by an hour) * fly (1:45) * taxi to gate (5-45 minutes) * walk to light rail stop (10 minutes) * ride light rail too stop (45 minutes) * walk to destination (15 minutes) Door to door flying: about 6 hours Driving: * start driving (6 hours) * fuel stop (15 minutes) * park (15 minutes) * arrive If I didn't obey speed limits, I could cut an hour or more off that 6 hours. That said, I still preferred flying. Hard to take a nap or space out while driving for 6 hours.

ineedasername 4y ago

I'm surprised there's not more activity on this here on HN. I guess info is limited, so I found what I could from ATCSCC notices for today [1]. These are all the notices but portions only w/ redundant bits removed. See link in citation if you want full messages. Times here appear to be GMT: 004 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 00:47 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT OUTAGE ISSUES FOR THE UNITED STATES NOTAM SYSTEM. 013 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 04:18 THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. SINCE THEN NO NEW NOTAMS OR AMENDMENTS HAVE BEEN PROCESSED. TECHNICIANS ARE CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR RESTORATION OF SERVICE AT THIS TIME. 014 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 04:19 HE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT OUTAGE ISSUES... 015 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 07:42 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT... 016 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 07:44 THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. TECHNICIANS ARE CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR RESTORATION OF SERVICE AT THIS TIME. THERE IS CURRENTLY A HOTLINE IN EFFECT WHICH HAS NAIMES/FAA FACILITIES/STAKEHOLDERS IN ATTENDANCE. 020 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 10:58 THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED... 030 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 12:37 THE NOTAM OUTAGE HOTLINE HAS BEEN TERMINATED. In short, the terminology being used so far is "equipment failure". [1] 1/11/2023 link to the day's messages w/ index numbering for each one per my above quotes" https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATCSCC&AdvisoryCategory=All&dates=Wednesday%2C+01-11-2023&AirFlow=AirFlow&Ctop=Ctop&Gstop=Gstop&Gdelay=Gdelay&Route=Route&Other=Other https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATC...

zitterbewegung 4y ago
Probably because it’s not about a package manager , a social network, Cloud Service and a chat application.
- eternalban 4y ago
  [heads-up: 'r/He\ is/I\ am' in your personal landing page]

eigenform 4y ago

Space weather?

adolph 4y ago

Where are the blockchain advocates? This seems like a perfect use case for a distributed ledger.

TecoAndJix 4y ago

In case you were like me and did not know what a NOTAM is: https://www.faa.gov/about/initiatives/notam/what_is_a_notam https://www.faa.gov/about/initiatives/notam/what_is_a_notam

dreamcompiler 4y ago

TIL that NOTAM now stands for "Notice to Air Missions." Back when I was flying it was "Notice to Airmen." Wondered when they would get around to fixing that. Guess they finally did.

wstuartcl 4y ago
Both are still in use. For instance during your pilots tests either is considered an equivalent answer.
- NelsonMinar 4y ago
  that's the test that makes you a Temporary Airman.
pookha 4y ago
They JUST changed it. Made a point of signifying to the masses about it last month: https://generalaviationnews.com/2021/12/07/notams-are-now-notices-to-air-mission/ https://generalaviationnews.com/2021/12/07/notams-are-now-no... What this tells me is a group of government busy-workers are playing around with complex systems that can dramatically impact millions of people.

anigbrowl 4y ago

Wondering if this is just standard aging infrastructure embarrassment or could involve a cyberattack? Cause if I were a state actor with a beef this is exactly the sort of soft-but-expensive injury I'd want to inflict: extremely disruptive, but also hard to assess or explain to the public.

TechBro8615 4y ago

[flagged]

tjohns 4y ago
Most glitches are localized, or there's backup systems, or you can fall back to manual human processes. If pilots/dispatchers can't get current NOTAMs, the aircraft can't safely depart - especially if they're flying IFR. There's no way around it. (One could argue about whether it's legal for general aviation traffic to depart VFR - the regs say you need "all available information", and the information isn't available. But that's arguing semantics.) There was a NOTAM outage about a year ago where Leidos' (the Flight Service contractor in the lower 48) computer system failed and prevented entering new notices, but as a backup they were able to call Alaska FSS on the phone. Since Alaska's FSS is still run by the federal government and used a different computer system, their system was still online. But in this case the outage is on the FAA's (upstream) side.
themitigating 4y ago
Evidence?
- TechBro8615 4y ago
  Do you have any evidence of a technical glitch? Or evidence that there has never been such a technical glitch since 2001?
  - smlacy 4y ago
    Do you? Your arguments are classic whataboutism, and conspiracy/fearmongering. Read up on the Burden of Proof. If you make outlandish claims, they have to come with substantial evidence.
    - TechBro8615 4y ago
      I think it's an outlandish claim that a technical glitch happened today that caused such a degree of widespread outage that the FAA had to take a step it hadn't taken since September 11, 2001.

sidcool 4y ago

What is the root cause?

allen-munsch 4y ago

I was wondering if this might be related to elastic's "several customer facing deployments deleted across multiple regions" issue from earlier today? The timing seemed to line up a bit. https://status.elastic.co/ https://status.elastic.co/

throwaway290 4y ago

Could it have been a nuclear option used by a federal agency to hold someone's possible escape? Perhaps they were not sure which plane, so they had to hold all. Or it was just the most immediate route

zrail 4y ago

As of 06:24 US Eastern the appears to no longer be true. There is a NOTAM outage but no full ground stop is mentioned, just a few specific due to weather. Edit: obviously a developing situation, see responses to this comment for current advisories.

emn13 4y ago
https://www.reuters.com/business/aerospace-defense/us-faa-says-flight-personnel-alert-system-not-processing-updates-after-outage-2023-01-11/ https://www.reuters.com/business/aerospace-defense/us-faa-sa... Clearly still developing; we'll see.
callahad 4y ago
I don't know how these things work, but the most recent advisory, 027, seems to imply that United Airlines has a full ground stop until 15:00 UTC / 10:00 Eastern. Edit: Of course as I was typing that, Advisory 028 pops up with a nationwide ground stop until 9:30 Eastern: "GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS" https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_date=01112023&facId=DCC&title=NATIONWIDE+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...
perihelions 4y ago
- "All US domestic flights have been grounded for several hours because of a glitch with the flight control system." https://www.bbc.co.uk/news/world-us-canada-64236047 https://www.bbc.co.uk/news/world-us-canada-64236047
- chucksmash 4y ago
  Yep, the situation has changed since the post was made. Parent comment was correct at the time, however. At 5:40am Eastern, the agent at my departure gate told us that there was a ground stop affecting all flights, which was reflected in my original title. This was not reflected in the NOTAM I posted, as was pointed out by several early commenters, and in fact my flight was able to board and depart around 7:00am Eastern before the ground stop took effect.

deleted 4y ago

[deleted]

nexus7556 4y ago

Any source for the claim of a ground stop? I see plenty of flights still taking off.

kzrdude 4y ago
The most recent one seems to say something like that. ATCSCC ADVZY 023 DCC 01/11/2023 ALL UAL MAIN AND SUBS GROUND STOP Not that I know how to read it. The event time is three minutes ago. https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_date=01112023&facId=DCC&title=ALL+UAL+MAIN+AND+SUBS+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=23&adv_dat...
- deleted 4y ago
  [deleted]
- UhUhUhUh 4y ago
  Reads United Airlines apparently.
- nexus7556 4y ago
  My understanding is that applies to United Airlines and it’s regional partners. It seems each airline is making their own decision. This Reddit post seems to have more info: https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_outage https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...
- eftychis 4y ago
  Air Traffic Control System Command Center Advisory 23 by the Traffic Management Command Center (DCC) at date [..] - all united airlines ... ground stop. Here is about ground stops: https://www.faa.gov/air_traffic/publications/atpubs/foa_html/chap18_section_13.html https://www.faa.gov/air_traffic/publications/atpubs/foa_html...
alamortsubite 4y ago
I can see that East Coast UA flights are currently delayed because of this.

sneak 4y ago

Nothing in the linked page confirms the ground stop mentioned in the title. Reddit comments say that there is no FAA ground stop, but airlines are making individual decisions about it. https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_outage/j3vo6cv/ https://old.reddit.com/r/flying/comments/108yjgz/faa_notams_...

Klonoar 4y ago
There was a brief ground stop (I'm currently flying and it momentarily held us up) but Delta at the least isn't stopping entirely.
emn13 4y ago
https://www.reuters.com/business/aerospace-defense/us-faa-says-flight-personnel-alert-system-not-processing-updates-after-outage-2023-01-11/ https://www.reuters.com/business/aerospace-defense/us-faa-sa...

harrego 4y ago

I really like the advisory page -- all info, no sugar.

deleted 4y ago
[deleted]
gonzo41 4y ago
https://lite.cnn.com/en https://lite.cnn.com/en It's the best way to get the news. No filler, no adds and it works on a 2g mobile.
- deleted 4y ago
  [deleted]
- harrego 4y ago
  http://68k.news http://68k.news is a fun one too.
- deleted 4y ago
  [deleted]
- ceejayoz 4y ago
  See also: https://text.npr.org/ https://text.npr.org/
- kfrzcode 4y ago
  See also: https://en.m.wikinews.org/wiki/Main_Page https://en.m.wikinews.org/wiki/Main_Page
- kfrzcode 4y ago
  Not sure CNN is the "best" way to get the news. Try this: https://en.wikinews.org/wiki/Main_Page https://en.wikinews.org/wiki/Main_Page No ads, no adds, no filler, no filters. Make the news objective again. Repeal Smith-Mundt. https://www.congress.gov/bill/112th-congress/house-bill/5736 https://www.congress.gov/bill/112th-congress/house-bill/5736

chucksmash 4y ago

Updated title from "FAA Ground Stop for All North America Flights Due to NOTAM Outage" to reflect comments that initial title was not accurate.

xattt 4y ago
Confusing title, as Canadian departures with Canadian destinations do not appear to be affected.

testplzignore 4y ago

If I'm understanding the situation correctly: the system stopped processing updates yesterday afternoon and it wasn't a big deal, then they tried restarting it this morning to fix it and it didn't come back up, so now older NOTAMs from before the issue started are not available. Is that right? Do airlines/planes keep a cached copy of these? How large is the NOTAM database?

stevehawk 4y ago
Not really. And for most commercial pilots this may come down to whether or not ForeFlight caches theirs or not. I'd actually be really curious to see unofficial commentary from ForeFlight developers about this because it's a pretty unique scenario in the age of electronic flight bags.
p_l 4y ago
The problem isn't keeping a copy, it's sending updates that are very time critical (for example restricted areas around POTUS, any impactful event, etc.)

WinstonSmith84 4y ago

[flagged]

ceejayoz 4y ago
A small amount, last month? Yes, probably unrelated.

AYBABTME 4y ago

Are 2020s the years where the cold war is back as a lukewarm war, and also all of NA aviation needs to reconvene with its decrepit software? Next plot-twist, it was all ready to blow, but ultimately triggered by Russian/Chinese/North Korean hackers. Ok, taking off my tinfoil hat. Still, maybe I'm suffering of nostalgia bias but the last few years really make for a lot of prime source material for docudramas.

deleted 4y ago
[deleted]
RagnarD 4y ago
No need for a tinfoil hat. Such systems are prime targets for states hostile to the U.S. If this one wasn't before, it will be now.
heisenbit 4y ago
Forensic analysis will reveal that there were multiple hidden failures over the past years that were covered up by foreign hackers to ensure their backdoors were not revealed in an in-depth investigation.
- peteradio 4y ago
  Free maintenance!
yehosef 4y ago
don't forget about this beauty https://techcrunch.com/2021/02/23/solarwinds-hackers-targeted-nasa-federal-aviation-administration-networks/ https://techcrunch.com/2021/02/23/solarwinds-hackers-targete...

ceejayoz 4y ago

https://twitter.com/AUStinAirport/status/1613134228321087488 https://twitter.com/AUStinAirport/status/1613134228321087488 > An FAA system outage is causing ground stops at AUS and other airports across the country. Arriving & departing passengers can expect delays this morning & through the day. If you’re flying today, it’s gonna be a rough one. edit: BBC now confirming a full ground stop. https://twitter.com/BBCBreaking/status/1613151796955856896 https://twitter.com/BBCBreaking/status/1613151796955856896 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_date=01112023&facId=DCC&title=NATIONWIDE+GROUND+STOP&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...

deleted 4y ago

[deleted]

yehosef 4y ago

in unrelated news - https://techcrunch.com/2021/02/23/solarwinds-hackers-targeted-nasa-federal-aviation-administration-networks/ https://techcrunch.com/2021/02/23/solarwinds-hackers-targete... ;) /humor

oldgradstudent 4y ago
Next time my server is down I'll blame the Russians and Chinese too.
- yehosef 4y ago
  If your org was hacked by the Russians/Chinese and was a national safety system, then you might be right to start there.

kzrdude 4y ago

ATC can tell pilots what they need to know, right?

dx034 4y ago
The question is, how will ATC know? I'm not that familiar with their systems, but maybe they rely on them to get critical information?
- ceejayoz 4y ago
  ATC will be fine to get everyone on the ground safely. There are procedures for everything including ATC itself being down.
  - dx034 4y ago
    I never doubted that. The question is if they could've worked without a ground stop by having ATC give them all info.
    - ceejayoz 4y ago
      No. ATC gets busy enough as is, each plane needs NOTAMs covering their entire route, and if the system is down ATC isn’t getting them either anyways.
    - sokoloff 4y ago
      If I print the NOTAMs for a 600nm flight, they're going to typically go 4+ pages of NOTAMs that are fairly specific to my flight (just for departure, along route, and destination airports) and 15+ pages if I print all available NOTAMs in a fairly small font.
  - p_l 4y ago
    ATC can get them back down, but no new flights should start.
McDyver 4y ago
NOTAM notices are not only for planes but for other people and equipment using airspace, like drones, skydivers, ... Those are not always connected with ATC, but need the information from NOTAM to plan their activities

dx034 4y ago

There's a ground stop for all flights now. GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS DESTINATION AIRPORT; ALL FACILITIES INCLUDED: ALL GROUND STOP PERIOD: UNTIL 1430Z REASON: EQUIPMENT OUTAGE REMARKS: US NOTAMS SYSTEM DOWN

Tepix 4y ago
Why don't they use lower case letters?
- yrro 4y ago
  https://fixingnotams.org/wp-content/uploads/2019/11/Field-Guide-to-Notams-OpsGroup.pdf https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...
srmarm 4y ago
This might be a silly question, but is that a NOTAM saying that NOTAM system is down? How does it get out?
- zrail 4y ago
  As I understand it there's a hotline and periodic teleconferences and webinars today that for sure everyone at the airlines is participating it.
- ceejayoz 4y ago
  The website and the system that gets it out to all the planes are related but not the same.
  - jonhohle 4y ago
    With all the hate in HN regarding SOA, here’s a great example of adding resilience by separating concerns at a systems level.
- capekwasright 4y ago

josephd79 4y ago

Gonna be a bad day for those sys admins

causality0 4y ago
May none of our fuckups be bad enough to get an article written about them. Amen.
- chasd00 4y ago
  heh, my boss was on a public sector project that went so far south she was stalked by news crews leaving the office. The experience left a pretty deep scar on her. This was before my time thank babyjeebus.
- josephd79 4y ago
  Amen.
acdha 4y ago
Kinda wondering the odds of there being one crying into their coffee over a stack of memos saying they should do something before it fails, or some greybeard angrily pointing to their quality control concerns on a new system. Both are pretty common with federal IT.

nkcmr 4y ago

Why do all aviation IT systems feel like they are so fragile? It feels like not a single month can pass these days without some "Airline X has major outage and ruins tons of people's days!" story on the news. What is the tech behind these frail systems?

lovecg 4y ago
When was the last time NOTAMs went down? It seems remarkably resilient actually.
dx034 4y ago
More modern systems are also down once in a while. With aviation it's just more visible because one system can affect a lot of people. I'm not sure when ATC systems (which I would count NOTAM to be one of) have last failed, I always had the feeling they're very stable. It's usually airlines that struggle with their applications.
- joshuahedlund 4y ago
  I never know how well it applies but headline outages always make me think of that “The Coming Software Apocalypse” Atlantic article from a few years back[0] [0] https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/ https://www.theatlantic.com/technology/archive/2017/09/savin...
namdnay 4y ago
I think it's just a question of media visibility. Aviation outage = queues in airports = easy article to write
VLM 4y ago
All hyper optimized systems become fragile until they collapse. The NOTAM system is ridiculous, no surprise it collapsed ... you can lose your ticket for flying thru a MOA that went active AFTER you took off. What could possibly go wrong with a "push" notification system for every tiny little detail of aerospace scheduling? No force is as destructive as the good idea fairy.

samwillis 4y ago

Edit 2 - * Ground stop Lifted * Update from FAA: >Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted. >We continue to look into the cause of the initial problem https://mobile.twitter.com/FAANews/status/1613171400801394688 https://mobile.twitter.com/FAANews/status/161317140080139468... --- Edit - Update from FAA: > Update 4: The FAA is making progress in restoring its Notice to Air Missions system following an overnight outage. Departures are resuming at @EWRairport and @ATLairport due to air traffic congestion in those areas. We expect departures to resume at other airports at 9 a.m. ET. https://mobile.twitter.com/FAANews/status/1613162735721746432 https://mobile.twitter.com/FAANews/status/161316273572174643... --- Flightradar24 are reporting: > Flights to US airports have been groundstopped by the FAA until at least 09:30 ET (1430 UTC). This means any flight not in the air already will not be allowed to depart. Image: flights over US now compared to same time last week. About 700 fewer flights airborne now. > Flights are being held as the FAA works to bring the NOTAM (NOtice To Air Missions) system back online. https://mobile.twitter.com/flightradar24/status/1613153479219986437 https://mobile.twitter.com/flightradar24/status/161315347921... Latest FAA update: > Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage. > The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information. https://mobile.twitter.com/FAANews/status/1613148579270459393 https://mobile.twitter.com/FAANews/status/161314857927045939... Statement from the white house: > The President has been briefed by the Secretary of Transportation this morning on the FAA system outage. There is no evidence of a cyberattack at this point, but the President directed DOT to conduct a full investigation into the causes. The FAA will provide regular updates. https://mobile.twitter.com/PressSec/status/1613153561289932800 https://mobile.twitter.com/PressSec/status/16131535612899328... I believe this means that nothing can take off, but aircraft already in the air can continue to their planned destinations.

dx034 4y ago
Looks like they published some kind of a "master NOTAM" now with all important information to restart as soon as possible. https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_date=01112023&facId=DCC&title=OPERATIONS+PLAN&titleDate=01/11/23 https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_dat...

rwmj 4y ago

Fun stuff about NOTAMs: https://fixingnotams.org/wp-content/uploads/2019/11/Field-Guide-to-Notams-OpsGroup.pdf https://fixingnotams.org/wp-content/uploads/2019/11/Field-Gu...

eternalban 4y ago
"There are three parts to the problem: the system, the format, and the content. The system is actually quite amazing. The AFTN network connects every country in the world, and Notam information, once added, is immediately available to every user. Coupled with the internet, delivery is immediate. The format is, at best, forgivable. It’s pretty awful. It’s a trip back in time to when Notams were introduced. You might think that was the 1960’s, or the 50’s. In fact, it’s 1924, when 5-bit ITA2 was introduced. The world shifted to ASCII in 1963, bring ing the Upper and Lower case format that every QWERTY key board uses today, but we didn’t follow – nope, we’ll stick with our 1924 format, thank you." https://web.archive.org/web/20230104130153/https://www.faa.gov/air_traffic/flight_info/aeronav/notams/ https://web.archive.org/web/20230104130153/https://www.faa.g... "The FAA is changing the format for Notice to Air Missions (NOTAM) to align with international standards. The transition to the new format will ensure U.S. NOTAMs are compliant with standards set by the International Civil Aviation Organization (ICAO) and promote further global harmonization among neighboring Air Navigation Service Providers (ANSPs)." ICAO NOTAM format https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf Example! (decoder ring not included) "Q) EGTT/QMRXX/IV/NBO/A/000/999/5129N00028W005"
- mzs 4y ago
  I want to point out that lower-case is possible with ITA2. Start at 4.6 of ITU-T Recommendation S.2 / 11/1988 in the Blue Book, basically 11111 & 11011 11111 switch case in locking/unlocking mode.
- wstuartcl 4y ago
  Its even more amazing when you really sit down and think about all of the systems (many highly resistant to change like aircraft navigation systems) that all connect and utilize this service. What appear like simple changes and proposals in the aerospace realm are extremely hard to execute and many times take decades (multiple) to come into play. Global services/systems are even harder.

reaperducer 4y ago

Very entertaining to watch the local TV news this morning. The reporter keeps pulling people aside as they arrive at the airport and asking them what they're going to do with all the planes grounded. None of them know it's happening. So far, they have all admitted that they either don't look at the news, or don't look at the local news, except for one woman who defiantly said, "It wasn't on Facebook," as if that meant that since the grounded flights weren't in her social media feed that it wasn't real.

unionemployee 4y ago

The NOTAM system is embarrassing. This just tops it off. Actually, our whole ATC system and the way the FAA operates is embarrassing. It all works kind of like waste management in NYC. If they were to add new tech and efficiency, fewer people would have jobs, so we keep doing it the old way to the detriment of everyone else.

dev_daftly 4y ago
So basically it's the same as every other government entity?
kfrzcode 4y ago
Shirley you must be joking.
- thesuperbigfrog 4y ago
  Rumack : I won't deceive you, Mr. Striker. We're running out of time. Ted Striker : Surely there must be something you can do. Rumack : I'm doing everything I can... and stop calling me Shirley!
  - jmbwell 4y ago
    > There’s a problem with the NOTAMs… >> The NOTAMs? What is it? > It’s a system for alerting pilots to hazards. >> I see, thank you (ETA: haha aw, well I thought it was funny)
acdha 4y ago
It’s easy to say things are about jobs but do we have any evidence of that? There’s a modernization project underway at the FAA so I’d bet that if you looked there’s someone who’s been trying for years to get attention on it before something like this happened.
quantified 4y ago
What embarrasses you about it, esp compared with more modern systems? This seems like the kind of critical infrastructure whose uptime requirements can make upgrading challenging once in place.

watersb 4y ago

REALLY IMPORTANT NOTAMS tend to alert pilots to RESTRICTED AIRSPACE or AREAS OF UNUSUAL AIR OPERATIONS. You can't fly anywhere near the President of the United States, the movements of which can change from day to day. We get NOTAMS about military aircraft operations; often military airspace which is usually permitted may be shut down for a week or so. Wildfire tanker operations or other natural emergencies can close an airspace. It's quite dynamic, all on top of the scheduled flight plans and routes.

ru552 4y ago
They are also for notifying pilots of potential caution areas. I used to operate a 550+ meter tower and when I had a tower light out, I had to file a NOTAM.
- watersb 4y ago
  Cool, you filed NOTAMs; I just read them. If you like lots of daily mail, become responsible for the operation of a Cessna 172 (Skyhawk), the type of aircraft with the most units built of any, ever. Endless variations. Unending stream of paper NOTAMS via US first class post office... at least, when I had a 172, 20 years ago. These notices are exactly like a recall notice on an automobile: get someone to validate the steering wheel etc. I would suppose that many HN readers have gone through the car hardware remediation process. I would suppose that this class of NOTAM, although in my particular case the most common, are not immediately relevant to a total Ground Stop Order for the USA. But -- consider that airline and other commercial operators may have their own internal requirements that could ground a fleet in the absence of an ability to state that they have complied with all relevant FAA NOTAMS. Oh, the liability!!
- watersb 4y ago
  (Holy cow, just did remedial maths: 550m is a Large Number! Would like to know more details.)
  - ru552 4y ago
    The tower was used by a TV station to broadcast their channels. You can see the curvature of the earth at the top when it's a clear day.

jxf 4y ago

Q: Is there anything like a systems architecture diagram for how NOTAM works? I'm curious about which parts might have failed and what the moving pieces are.

tyingq 4y ago
Some here, mixed in with other info: https://www.faa.gov/air_traffic/technology/swim/users_forum/media/SWIM%20Users%20Forum%2026%20Briefing.pdf https://www.faa.gov/air_traffic/technology/swim/users_forum/... https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%20Summit%20-%20Session%201%20-%20Connecting%20to%20NOTAM%20Data.pdf https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%...

sumanthvepa 4y ago

Was it database failure or did they get hacked?

snake_doc 4y ago

For anyone looking for previous FAA outages: In 2014, an U-2 plane’s flight plan crashed LAX’s ERAM system because the system ran out of memory. https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20to%20Address%20ERAM%20Outages%20Final%20Report%5E11-07-18.pdf https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20...

GalenErso 4y ago
Why do secretive spy planes need to file flight plans?
- ceejayoz 4y ago
  Because they're not always doing secretive stuff, and even their secretive missions may start and end in airspace they have to share with friendly airliners. Sometimes it's a deliberate "we see you", too. https://twitter.com/Gerjon_/status/1475444704250630148 https://twitter.com/Gerjon_/status/1475444704250630148
- pmontra 4y ago
  That was a training flight, it's in the PDF.
  - mikece 4y ago
    I imagine operational flights might "officially" start and end as routine training flights, at least on paper.
- bobinator606 4y ago
  [dead]
controlller 4y ago
LAX, an air traffic control tower, does not employ ERAM. Los Angeles Center (ZLA), the overlying air route traffic control center, does.

belter 4y ago

https://twitter.com/FAANews/status/1613171400801394688 https://twitter.com/FAANews/status/1613171400801394688 " Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted...We continue to look into the cause of the initial problem"