5 ms·

Has anything happened regarding IT staffing at the FAA recently? There was the NOTAM outage last week [1], and now this! [1] https://news.ycombinator.com/item?

• wiz21c 4y ago
  Maybe there's an ongoing cyber attack ? :-)
  • bitxbitxbitcoin 4y ago
    That isn’t being admitted to?
    • ProAm 4y ago
      Happens more frequently than you think.
    • eh9 4y ago
      Admitting attacks can lead to more attacks if you’re not prepared
  • idiotsecant 4y ago
    Nope, just the regular old spending money on new projects that benefit the careers of powerful politicians but refusing to spend on maintenance attack.
• acdha 4y ago
  The NOTAM system was apparently running on old SPARC hardware so I’d easily believe that they haven’t been given enough budget to hire engineers and are basically keeping the lights on. One peril of the government funding model is that it’s effective at shoveling money to contracting companies but not having skilled oversight, so you tend to see lots of big projects which founder under their own weight.
  • hackmiester 4y ago
    I would love to see more information about the underlying architecture of the NOTAM system, if you can recall where you saw that.
    • acdha 4y ago
      Not sure how reliable this guy is but it’s certainly easy to believe: https://twitter.com/PatrickMoorhead/status/1613937512002772992 https://twitter.com/PatrickMoorhead/status/16139375120027729...
  • pookha 4y ago
    NOTAM could be run from a pre-historic Toshiba mainframe. it's static. Doesn't need billions in funding.
    • nhtsamera 4y ago
      It's static, until a contractor ignores the post-it note on their monitor that says, "IMPORTANT: ALWAYS add an extra newline BEFORE saving the file". Once that happens, you have to hope that your backups work - why bother having disaster recovery drills for a static system? Once you realize that your backups are hosed, you may experience a creeping suspicion that software is never truly static when human operators are involved.
    • acdha 4y ago
      It doesn’t need billions but it needs enough for dedicated staffing. Many federal agencies have the problem that they’re not given general funding for staff but are given money for large contract projects. It’s not uncommon for that to be structured in a way that, say, a big project has rooms full of developers, PMs, testers, etc. but something small has like one dude keeping it going when they can find time and a stream of unanswered requests to management asking go either fund or decommission it, hire a replacement for people who retired, etc. or that the big project which was supposed to replace it is years overdue. I would not be surprised to learn that this was the case here, too.
  • 4y ago
• envyac 4y ago
  You mean besides the last three years of reduced on-site staffing, and employee travel to support maintenance and modernization? No. $5 says this is the XKCD for this. Corrupted DB file was the culprit, as announced by the FAA. It's a system that relies on user inputs, often manual, and it's all being put into a database. My bet is a user decided to put in a bunch of "fun characters" to make their input easier to read. You can't account for and sanitize all levels of stupidity on user inputs. https://xkcd.com/327/ https://xkcd.com/327/
  • krisoft 4y ago
    > You can't account for and sanitize all levels of stupidity on user inputs. That is false and a defeatist attitude. Sanitization is not the right solution anyway. If you are working with any form of a database and don't know from the top of your head how to avoid query injection attacks then you should look up in the manual of your database. The solution is most often called "parameterized query" or something similar.
  • NavinF 4y ago
    Of course you can account for that. They're just byte strings. That xkcd is poking fun at incompetent programmers, not stupid users.
• xeromal 4y ago
  Probably just more visibility on the matter. Once a major issue happens, things tend to get followed for a while. It can get bandwagoney
• neilv 4y ago
  I don't know anything about these two incidents. But, FWIW, some of the past professional work I'm most proud of was on FAA contracts (on other systems). I'm sure they're handling the immediate situation, and -- consistent with their culture -- they'll analyze what went wrong and can be learned, and follow through on that.
• tjohns 4y ago
  For what it's worth, the METAR network (NADIN) is maintained by the NWS, not the FAA.
  • tjohns 4y ago
    (Following up on this, I was actually mistaken. I believe NADIN is actually an FAA network.)