6 ms·
Author here. Did not expect to see this on HN at all. Just an engineering war story I shared.
by euler_angles 3y ago
Author here. Did not expect to see this on HN at all. Just an engineering war story I shared.
- tra3 3y agoThis is really cool, thanks for sharing. What's wild to me is that the program started in the late 90s and only now is the F35 fleet up to originally specified? operational capacity. Since then I graduated high school, got a degree, got married etc etc. The time span is mind boggling. Would be interesting to see how continuity is maintained for so long. In software it feels like if a project is more than 6 months old, we throw it out and rewrite it.
- tekla 3y agoYou write shit down and you have career engineers that enforce continuity It's trendy in software to complain about doing annoying work like writing reports and documenting things. But most hard tasks require writing reports and documenting things.
- eitally 3y agoAnd this isn't limited to aerospace. My wife has spent a career in pharma (drug save & pharmacovigilance specifically) and it's the same way there. People complain about rigidity and sluggishness in these industries but there absolutely is an ingrained attitude of documentation and process compliance that pervades. At one point -- and this was just last year -- my wife took over running a monthly safety report that involves manipulating a bunch of data in Excel. Even that has a 9 page instruction guide, and since she now owns the output she also owns maintaining the manual. Too often in the land of software we underestimate the potential negative impact the traditional "move fast and break things" approach to product development can have when it comes to real world use in mission critical systems.
- trhway 3y agoOn the other side this unwillingness and mental non-acceptance of those reports/manuals/etc. as a wasteful activity frequently comes from the understanding that there are more efficient ways of doing things, and that drives the "software eating the world" effect. While I naturally don't know the details of the case you mention and pharma is far from the domains I've been in, yet in many business/enterprise situations the software approach is to code the many-page guide into business logic, including ETL-ing the data instead of manual import, etc. Move fast and break things brings you to the Moon in a decade using primitive tech, where is total process compliance can't do that even in 50 years using much more advanced tech.
- red-iron-pine 3y agoa lot of people died in germany, the ussr, and the us making those rockets work. and in exchange for that we planted a flag there and have a handful of rocks in a glass viewing box. move fast and break things worked real well for the folks who got literally creamed while they were viewing the titanic.
- falcolas 3y agoSo, an amusing anecdote related to your second paragraph - one reason it's taking so long the second time around is everything has to be repeated. They lost the knowledge of how to make rocket stages and engines of that size, and had to re-learn those lessons. It's also quite important to remember how many lives were lost (or nearly lost) because of "breaking things" in the Apollo program. Something that's not nearly as acceptable today than it was at the height of the cold war. Something that directly implies moving more slowly and being more sure that everything works the first time, every time.
- inamberclad 3y agoSeconded. People burned alive until we learned. Surely there is a middle ground that will let us speed up while staying fairly safe, but it's important to remember that outside of software, many rules are written in blood.
- hanche 3y agoHeck, even maintaining my computers at home requires documenting things! I have lost count of the number of hours I’ve lost trying to rediscover how or why I set things up the way I did.
- roughly 3y ago> In software it feels like if a project is more than 6 months old, we throw it out and rewrite it. I think that would be a bad way to operate, but what's worse is what we _actually_ do, which is write the project like it's gonna be replaced in 6 months and instead keep that poorly-documented untested duct-tape contraption around for a decade as the central load-bearing component of critical infrastructure.
- Jenk 3y agoA decade is infancy in that scenario. The world's economies are running on stuff way, way older, for example.
- xattt 3y ago> In software it feels like if a project is more than 6 months old, we throw it out and rewrite it. “The Phoenix pay system is a payroll processing system for Canadian federal government employees, provided by IBM in June 2011 using PeopleSoft software, and run by Public Services and Procurement Canada… By July 2018, Phoenix has caused pay problems to close to 80 percent of the federal government's 290,000 public servants through underpayments, over-payments, and non-payments.“ https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
- nradov 3y agoIronically, many of the agile development practices which are widely used today were pioneered in the Chrysler Comprehensive Compensation (C3) payroll application. It was never able to produce an accurate payroll for Chrysler and couldn't replace the legacy system, although the project was considered at least a partial success in other ways. https://wiki.c2.com/?ChryslerComprehensiveCompensation https://wiki.c2.com/?ChryslerComprehensiveCompensation
- k12sosse 3y agoMy employer still had the balls to pay IBM after the Phoenix snafu to come in and tell us how we could become more efficient and guess what? It was basically word for word what the frontline employees had been screaming for for YEARS that management ignored. And guess what? They ignore the consulting report and did nothing anyways. #MunicipalLife
- tempestn 3y agoThat situation was (is?) absolutely mind-boggling. I personally know government employees that were being underpaid with no recourse for months on end because the software wasn't working and the government apparently had no alternate way to pay. Some people weren't getting paid at all. And as you quoted, it affected 80% of the workforce, hundreds of thousands of people.
- dj_gitmo 3y agoSomeone should get fired for choosing IBM in this case
- jonp888 3y agoI work on software that has a lifetime once installed of about 30 years, and if a safety critical error is found during that time, ideally it needs to fixed with a minimal patch, so we have to maintain the capability to do so. I guess the ethos is quite different to top tech company. We don't get the pay or perks that you would get in Silicon Valley, but we are unionised, and it's a viable option to spend your entire career just on one project so it's very stable. Partly it depends on documentation, but also on thinking long term. There are certain people who are the technical authority for a particular area, and they know that about 5 years before they retire or move on they need to find someone who can take on their role for at least the next decade, to keep their knowledge rolling forward.
- tra3 3y agoThat's fascinating. Is it possible for you to share more details? Industry? Tech stack? If your project started 30 years ago, that means DOS, or Network or maybe one of the IBM behemoths? Then the maintenance includes pacing OS updates and dependency changes?
- HeyLaughingBoy 3y agoNot OP, but I work on medical devices. One product at my last job had an expected service life of 20 years. FDA requires that the manufacturer maintain the ability to support and service a medical device for, IIRC, 5 years after market exit. In the 10 years after release that I was on that project, we went through multiple OS upgrades from Windows NT to XP Embedded, to Windows Embedded Industry (replacement for XP Embedded) and a number of replacement x86 CPU boards had to be qualified as one manufacturer after another exited the market. Since the device is validated as a complete system, we often had to buy a year or two stockpile of existing product to give us time to start the Validation process for replacement hardware. You usually have plenty of warning from a supplier that a product (Windows or a CPU board) is going EOL at a certain point, so you need to start validating whatever the next replacement will be well ahead of time.
- euler_angles 3y agoThe F-35 contract was awarded on October 26, 2001. I was in my freshman year of undergrad, 18 years old. I started on the program in August, 2010. I was 26 years old. The program has just completed its Initial Operational Test & Evaluation, including its runs for score in the Joint Simulation Environment. I am 40 years old.
- wazokazi 3y agoWas there ever any consideration given to building a "testing harness" to physically simulate the F35 landing? Something like the "dead load" testing that the EMALS undergoes. Just in reverse. Anyway, that was great read.
- euler_angles 3y agoThere was a lot of static load testing done, and things like a drop test [0] of a full scale article. But to my knowledge, the only way to test the dynamics of a carrier arrestment is to actually do an arrestment. We do them on land; NAS Patuxent River and NAS Lakehurst (among others) have a full set of Mark 7 arresting gear like you would find on a Nimitz class. Lakehurst also has the advanced arresting gear present on the Ford class. [0] https://www.youtube.com/watch?v=lGPseVNfZO0 https://www.youtube.com/watch?v=lGPseVNfZO0
- fatbird 3y agoHow much of a difference is there between dry land arresting and carrier arresting? I would guess some since the carrier represents a somewhat dynamic surface, and flight conditions might likewise vary. Is there enough that a second round of carrier based testing is required that might trigger significant changes?
- euler_angles 3y agoAll of this was done as a work up to a carrier deployment. In software terms, trying the arrestments on land is deploying to test, doing them on a carrier is production. There were three separate developmental test deployments to carriers for the F-35C. Each deployment sought to expand the understood envelope and and handling procedures. The hook redesign happened before the first deployment. The hard landing story in the post happened during the work up to the third and final deployment.
- dotancohen 3y agoHad the hard landing occurred on a carrier, how would the gentle, flared landing be accomplished? Try to reach an airport on land, ditch the plane in the water, maybe catch the plane with the net on the deck? Our just do a normal carrier landing and hope for the best?
- superjan 3y agoIsn’t there a normally a mechanism that lifts the wire after the landing gear has crossed it?
- euler_angles 3y agoYes, there are pendants that are supposed to keep the wire above the deck, but the short space between the F-35C main landing gear and the tail hook point means that there's not enough time for the pendants to raise the wire above the deck in the manner that the original (erroneous) wire dynamics model would have suggested.
- LorenDB 3y agoHey, would you mind adding an RSS feed to your blog?
- euler_angles 3y agoI am but a grunt who mostly programs radar models, I didn't know Quarto blogs could do that until just now. Yeah, sure, I'll add it.
- euler_angles 3y agoDone!
- LorenDB 3y agoThank you!
- euler_angles 3y agoQuarto made that really easy. Very cool.
- idontwantthis 3y agoSorry, it’s been so long that I’m afraid to ask. What will you do now that it has an RSS feed?
- iab 3y agoAre you actually euler_angles, or are you really tait_bryan_angles
- euler_angles 3y agoDeep down I am indeed tait_bryan_angles
- iab 3y agoI appreciate your candor (and your article)
- tiahura 3y agoInstead of a lot of modeling and testing, wasn't Northrop just allowed to inspect an F18 and measure?
- euler_angles 3y agoThe F-18 tailhook geometry is far different than the F-35 tailhook geometry. F-18 hooks are much farther back from the main landing gear, and are also much longer.
- jtriangle 3y agoWhy exactly did they redesign the tail hook? Surely they could have just used one off any number of other aircraft with some modification? Or are all of those tail hooks bespoke designs because reasons?
- deleted 3y ago[deleted]
- wbeckler 3y agoIt could be related to the fact that they didn't have much space for a normal size tailhook, as stated in the article.
- jtriangle 3y agoI mean more the design of the hook itself, though, I don't know if that design is even atypical to be honest.
- mech987987 3y agoEven if two different aircraft have the same space constraints for the hook (which is a pretty big if), they have different mass and deceleration characteristics (i.e. minimum and maximum approach velocity) during landing- changing the force exerted on the hook. Designing a lighter hook for the lower loaded aircraft is VERY desirable for high tech fighter jets- every ounce saved is better range, better agility, etc. As far as the little lip at the very tip of the hook- it looks to me like the initial design was trying to minimize any risk of digging into the flight deck and causing damage- this is just a guess though.
- cwillu 3y ago“After the LSO finished what he had to say and left the ready room my B/N allowed that he might fly with me again. Me, I was still shaking inside. The next morning I went up on the flight deck before flight ops started and walked to the aft edge of the deck. I was looking for something and found it. About one foot from the end, there was a single, shiny, brand new, solitary hook imprint in the deck.” https://thelexicans.wordpress.com/2013/09/10/one-foot/ https://thelexicans.wordpress.com/2013/09/10/one-foot/
- mlekoszek 3y agoGlad you did.
- aidenn0 3y ago> “Boss,” he says to me, “This fucker ain’t gonna work. Look at this thing. It’s short, it’s too close to the wheels, and look at this dumbass hook shoe they got on it. If the wire don’t hit it exactly right, it’s just gonna go under the hook and you’ll bolter.” Did nobody with practical experience with arrested landings look at the arresting hook design prior to this? Obviously computer models can and do predict extremely novel solutions to existing problems, but it's worth double-checking the model when someone with practical experience says "it will never work" In this case, it seems like a simple slow-motion video of an arresting wire going under the wheels of an F-18 would have been enough to debunk the model.
- icegreentea2 3y ago> Did nobody with practical experience with arrested landings look at the arresting hook design prior to this? I mean... it's very likely that the answer is no. The last new carrier aircraft made was the Super Hornet - and that design was basically done by 1995 (the F-35 tests in question were in 2011/2012). That expertise would also be at McDonald Douglas/Boeing. Northrop Grumman has a long history of carrier aircraft development, but it would have been long dormant by that point. I'm sure there's all sorts of reasons the model's inaccuracy wasn't caught before hand, but sometimes... if you're given a model that's someone says that's been V&V'd, and it produces a result that's only a little weird, you just go with it. There are only so many things you can add extra testing onto in a project. Sometimes you choose wrong. Anyhow, consider that the model results were probably exactly what they were expecting. Remember that the designers would be honing in on the shorter tailhook. You can imagine their mental model going - "ok on legacy aircraft, we have flatter tailhooks because there's enough time for the cable to settle". And then going "ok, with a shorter tailhook, there won't be enough time to settle". And then their model comes out and say "ya, with the shorter tailhook, it won't have enough time to settle - it'll be UP IN THE AIR". Whereas reality is "ya, with the shorter tailhook, it won't have enough time to settle - it'll still be displaced DOWN".
- the_af 3y agoRandom thought: this is a case where someone's intuition matched what actually happened, making us think "why don't they listen to people with common sense?". But what about the many other cases where someone with "common sense" said "this fucker ain't gonna work" but the thing worked as predicted by simulations? Surely they must have happened too.
- throwanem 3y agoThe name "ham-peas" might be familiar. If so, how've you been keeping lately? Been a minute!
- euler_angles 3y agoI have not forgotten your user name here, ha. Yep, been a minute. I should hit you up via email.
- throwanem 3y agoI wouldn't mind that at all. If you don't still happen to have the address, the one in my profile here's good too.
- euler_angles 3y agoI actually still have your personal email address. Expect something from me soon. It took a few years but I work as a developer now (C++, electronic warfare simulator)
- throwanem 3y agoThat's great to hear! I'll keep an eye out, and hoping all's well in the meantime.
- kevin_thibedeau 3y agoHope you have proof that the diagrams aren't export controlled. Otherwise you're going to receive some unexpected visitors soon.
- euler_angles 3y agoThey all came out of a book that anyone can buy called "F-35: From Concept to Cockpit". That book is a compilation of papers presented at an AIAA conference in 2018.
- egorfine 3y agoFirst of all, thank you for the super interesting read! Now, as a Ukrainian I do have a philosophical question of sorts. What we have seen here in a real full-scale combat is that some of the modern machines are way too delicate for actual operations on the ground. For example, I have heard some feedback about the Abrams tank: way too finicky for real use, not durable, not reliable. The same goes about many other western items. (Some hardware demonstrated exceptional reliability, like Bradleys and HIMARS) My question is about modern fighter jets like F-35. Does that level of engineering and the amount of delicate electronics somewhat limit the durability and reliability of the airplane compared to much simpler designs?
- nradov 3y agoThe F-35 was never really intended to be durable or reliable for ongoing use in long wars of attrition. It was designed to be survivable on penetrating strike missions against near-peer adversaries. Essentially to "kick the door in" and destroy high value targets such as air-defense systems during the first few days of a conflict. War games and simulations have shown that simpler designs can't accomplish that mission anymore so concerns over F-35 durability and reliability are somewhat misplaced (although there is certainly room to improve mission readiness rates). No one is even remotely contemplating sending F-35s to Ukraine. Besides and costs and security risks, the Ukrainians unfortunately have nowhere near the infrastructure and logistics to sustain such a complex platform.
- egorfine 3y agoAh, so in that case it looks like reliability and durability are not exactly important or desired features for that kind of missions. Did I get it right?