11 ms·
You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and
by huslage 4y ago
You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be minimized in any way. This is not some government corruption thing. There are hundreds of thousands of lives at stake every day.
I, for one, appreciate the work that the FAA does to keep us safe in the air and appreciate that this was handled appropriately. Everything breaks. It's just a matter of how and when and what we do with it when it does. The FAA handled this outage appropriately and in a timely manner. I feel for the engineers who had to work on this incident.
- greggarious 4y agoIf they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.
- Cthulhu_ 4y agoMaybe the better question to ask is how bad would it be if they *weren't* so risk-averse.
- jjk166 4y agoImagine how bad it would be if Kim Jong Un *weren't* so pro-american. It's a ridiculous hypothetical. Everything can always be worse, that is not a testament to an entity's quality.
- r3trohack3r 4y ago> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.
- justjash 4y agoWorked for a govt contractor years ago and I would imagine the payout from the govt contract was at least 2-3x. There were usually 2-3 layers of people getting paid and I don't think any of them were hurting. Of course no employee was getting a massive payout compared to private. Govt > contractor > sub-contractor > employee was pretty common. I never knew of anyone being a direct contractor to the agency since the contracts were so large and involved a lot of employees.
- bink 4y agoIt's been several years since I've done contracting work (and for the FAA no less), but that can't really happen to the extent you're implying. The open roles for contract positions specify a pay range based on experience (degrees and/or years of experience). You can have a prime and then a sub contractor, but the amount that each can add for management overhead is limited and depends on the contract vehicle. I want to say it was around 5-10%. The actual employer can pay the employed contractor whatever they want, but the rates are published and if they underpay too much the employee will be poached by a competitor. Because the rates are public info the employee can look up how much profit their employer is making any time they want.
- panzagl 4y agoThe contracting company receives usually 2-3x what the actual developer makes- so if the gov't pays $300K for a developer, they get a $100K developer. Some of that overhead is justified, but a lot of it is the company's profit. The old answer used to be for the government to hire directly, but that's been hamstrung for like 40 years by now.
- HillRat 4y agoAt my old firm, our federal project profit margin was a bit lower than median, though once you factor in the sales, contracting, and leg overhead the bottom line was somewhat worse than the project actuals made them look. Since our side of the house did relatively short-burn contracts (6-12 months), federal work was generally not that valuable for us; I used it for filler work when our usual sales pipeline was weak. The real value was for the side of the house that did long-term software and support work or heavy citizen support outsourcing, when contract durations can be measured in decades. Same for a friend who inked a $5bn DOD deal; the margin isn’t great, but it’s a 10-year deal that gives her a stable cash flow basis to grow on. (Also, OP is probably underestimating the full-sheet cost of a federal FTE, as well as the complexities of fund-based budgeting and forecasting.)
- specialp 4y agoI am not sure if you ever dealt with a very entrenched system that has been in place longer than you have been alive, but it is not easy. It is also not easy to hire to deal with these things. Especially at government rates which are lower than the private sector. If you were the FAA and started up an internal startup to find only the best to replace systems that have been running forever you would face a lot of problems. 1. Nobody would give you the funding until something like what happened yesterday did 2. Your developers getting paid more than the FAA directors will get lots of political attention 3. Even a great internal engineering team would most likely take ages to do something like this. This isn't move fast and break stuff with a greenfield, it is painful deconstruction and analysis of a very complicated system. 4. Many people won't want to work on this no matter how much you are paying. It is also out of the wheel house of an agency tasked with flight safety. So expensive contractors arise. Do they do a poor job often? Yes. But these are jobs that are really hard to scope and execute. If it is really a case of overpaid contractors coming in and not doing the work, people here should start a startup and hire a SEAL team six of 1970's computer system rip and replacers and make a lot of money.
- gnarbarian 4y agodemonstrably the policies employed by the FAA do not effectively avert risk (in software). It's more akin to risk aversion theater.
- malfist 4y agoWhich is why we've not had a commercial plane crash since.......2009? Seems FAA is pretty effective at reducing risk.
- nonethewiser 4y agoHow may would there be with an FAA alternative?
- gnarbarian 4y agoa few things 1) mostly thank the airline operators for that. 2) the fact the 737 Max disasters happened outside the US was a matter of chance not exemplary policies by the FAA. FAA policies did not prevent the deployment and roll out of this dangerous aircraft in the United States and did not lead to grounding of the aircraft until multiple events occurred. 3) I'm mostly talking about software here, and I believe a big part of this issue is shoe-horning software into the policies and procedures specifically designed for aviation. there are so many little pointless (in most contexts) requirements that cause the engineers working on these systems to lose the forest through the trees. the FAA creates its own complexity which prevents thinking holistically about our systems in a meaningful or effective manner. in many ways this is a force of entropy. the more lines of code you have to support thousands of requirements and the more revisions you make to those lines of code without a top to bottom refactor the more likely you are to have insidious bugs that pop out like this incident.
- JCM9 4y agoThis demonstrates a gross misunderstanding of how the FAA actually operates in its efforts to address “risk.” It’s far more theatre via bureaucracy and paperwork than actual proven engineering efforts that demonstrably reduce risk.
- gnarbarian 4y agoyou nailed it. Plus the FAA is unable to even perceive the fact that this culture is counterproductive to the goal nor would they know how to address the problem in any way other than adding more layers of paperwork and bureaucracy.
- btown 4y agoIs this conjecture or based on documented shortcomings of their approach though? Much of this paperwork, especially if created in response to NTSB crash post-mortems, could very well be examples of https://fs.blog/chestertons-fence/ https://fs.blog/chestertons-fence/ - is there reason to believe otherwise?
- haswell 4y agoThis is a sweeping claim with really nothing to back it up. Are you saying this from an inside knowledge of the FAA, or is this just an opinion? On the surface, the relative safety of air travel and the lack of major stoppages over a span of 22 years seems like a major counter example. You’re making this statement emphatically and authoritatively, though, so I’m curious to understand where that certainty comes from and how it accounts for the other publicly visible properties of the FAA and air travel.
- njovin 4y agoTwo examples: 1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths (https://www.newyorker.com/news/our-columnists/how-boeing-and-the-faa-created-the-737-max-catastrophe https://www.newyorker.com/news/our-columnists/how-boeing-and...) 2. The FAA's pilot medical vetting process, while thorough, is behind the times. There are people who took ADHD medicine in high school that are unable to obtain a medical certificate due to the FAA's overly-strict policies on prescription drugs. There are current pilots with serious mental issues who are afraid to see a doctor about them due to fear of losing their medical license (https://www.flyingmag.com/why-pilots-dont-want-to-talk-about-mental-health-and-why-they-should/ https://www.flyingmag.com/why-pilots-dont-want-to-talk-about...).
- rattlesnakedave 4y ago> You all seem to think this is similar in value or operation to a web app This line of reasoning is pure, 100%, cope. There is a reason this was broken and grounded flights nationwide. Providing cover for systems that allowed this to happen is not productive.
- haswell 4y ago> This line of reasoning is pure, 100%, cope. What does this even mean? There’s a meaningful difference between providing cover and reminding people that major systems like the one in question are in a different category than the average web app. The fact that such a nationwide stoppage hasn’t occurred since 2001 speaks to the stability of these systems, and I’m not sure what you’re suggesting here. > There is a reason this was broken and grounded flights nationwide. Providing cover for systems that allowed this to happen is not productive. I’m not trying to sound snarky here, but things do tend to break for reasons. Are you suggesting that there is a cure?
- nonethewiser 4y agoPaying exorbitant contracts to companies with weak talent and processes is not "stringent."
- cdiamand 4y agoThe FAA has risk reduction baked into it's very heart and soul and this is evident the more you learn about how the system operates (go look at a flying textbook). It is not perfect organization, but I think it deserves more credit than it is receiving in this thread.
- berniedurfee 4y agoReally. The outage was impactful, but it was remediated in a few hours. That ain’t bad considering how infrequent incidents are in light of the immense complexity of the operation. Much less complex and much more modern systems go down regularly and sometimes don’t come back up for much longer.
- alexb_ 4y agoYou've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.
- bastardoperator 4y agoIf the corrupted file is in the backup, it DID pass through like everything was fine. What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time.
- haswell 4y ago> What's clear to me is that the FAA has no post deployment validation, hasn't tested its DR strategy, and that errors can go unseen for long periods of time. It is possible to have all of those mitigations in place and still experience a failure like this. Post deployment validation is only as good as the validations executed. 99% coverage still leaves the door open to failure. A DR strategy is just that - a strategy. A failure of this sort is not an automatic implication that those things do not exist, just that they failed in this particular case. I would find it incredibly surprising that an organization of that complexity could have survived as long as they did without a major incident if none of those things were in place. They’d be either incredibly lucky, or incredibly competent, and if they are the latter, they would not operate without such mitigations in place. It seems far more believable that an organization of the FAA’s age and complexity missed something along the way.
- bastardoperator 4y agoWe can make excuses all day long. A simple query of the database/table would have produced an error. Sure, the FAA does some complex stuff, but the tech I see in airplanes looks ancient. I'm willing to bet most of the FAA complexity comes from budget (lack thereof) and old computer systems.
- tyingq 4y ago>The FAA handled this outage appropriately and in a timely manner It's still not working as designed, right now. The workarounds in place are fine, but were made up on the fly. Certainly there's room for improvement.
- marmetio 4y agoI guess this is the comment chain where we address the room? I'm seeing a lot of misuse of the word "risk". The FAA prioritizes safety over mission. The mishap that resulted in downtime affected the mission. The common-cause failure of the backup system affected the mission. That is not evidence that they're bad at managing safety risk. Given that there was an article published at the time about removing the dissimilar backup, it's probable that they explicitly accepted this mission risk.
- arcbyte 4y agoThe only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.
- thfuran 4y ago>All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will) Eventually, but not as frequently.
- justinsaccount 4y agoYep, this is the exact point I tried to make above and got heavily downvoted. If you can't move fast when things are working well, you can't move fast when things are broken. Acting like moving slow is going to prevent things from ever breaking is just wishful thinking.
- nucleardog 4y ago> Instead, if you can move fast consistently, you can minimize the total downtime. In safety critical software where _a_ failure can result in loss of life, is “total cumulative duration of downtime” really the metric we’re optimizing for?
- simonh 4y agoDowntime isn’t the metric their procedures are optimised to minimise. It’s optimised to minimise air traffic accidents. Moving fast might minimise total down time (though I seriously doubt that), but what effect would it have on accuracy and reliability? Mistakes mean dead people. In this incident zero people died. You really sure you know you can improve on that?
- DonHopkins 4y agoPlease don't change jobs from Facebook to the air traffic control industry.
- s1dev 4y agoYes. Breaking things in aviation frequently results in a body count
- tristor 4y agoFAA regulations, more so than nearly anything else regulated in the US, are written in blood. They are not written out of government corruption. Many many many hundreds of people have died in airplane accidents since airplanes were invented, and the reason why your average person can board a commercial flight and act like they just got on the bus (with a /better/ risk profile than a bus on public roads) is exactly due to these regulations. The FAA has and continues to lead the entire world in how to do reasonable and meaningful air traffic regulation.
- berniedurfee 4y agoMentor Pilot on YouTube does a great job of discussing aircraft incidents and how the RCA almost always informs new regulations.