23 ms·

The F-35’s Software Is So Buggy It Might Ground the Whole Fleet

141 points by Wonnk13 11y ago

sickbeard 11y ago
No it's not http://www.defensenews.com/story/defense/air-space/2016/04/26/f-35-chief-software-bugs-no-longer-threat-ioc/83553372/ http://www.defensenews.com/story/defense/air-space/2016/04/2...
- oxryly1 11y ago
  Different software.
kogus 11y ago
How does this reconcile with this article? https://www.f35.com/news/detail/f-35-chief-software-bugs-no-longer-a-threat-to-ioc https://www.f35.com/news/detail/f-35-chief-software-bugs-no-...
- engi_nerd 11y ago
  Those two articles are talking about two different things. ALIS is the logistics system for the program that has an off-jet and on-jet component. ALIS takes the role of a traditional computerized maintenance management system and expands it to include an on-aircraft prognostics and health monitoring component. The idea is to have the jet tell you what's wrong with itself. The article you linked to is about the F-35's mission systems software, which is really a software suite that encompasses all the computers on the aircraft itself. The MS software load provides all the stuff necessary to make the jet's hardware do what it needs to do to fight.
  - mrweasel 11y ago
    So, the plane will fly, it will just be hard to maintain and operate?
    - throw7 11y ago
      "... Bogdan is saying ALIS is not really critical after all, insisting the F-35 can fly without it for 30 days." So I guess we can wait and see what happens on the 31 day? SNAFU is all that comes to mind.
      - dTal 11y ago
        "Your trial period has expired. Please enter a valid product code."
    - engi_nerd 11y ago
      I don't understand your question. The plane will fly if [_______________]...what, exactly? EDIT: I am merely seeking clarification, not trying to evade answering.
      - mrweasel 11y ago
        Sorry, I meant the F-35 software that ensure that the plane will actually stays in the air is fine, with no real bugs left. The ALIS software, that helps ground crew, maintenance and mission planing, I assume, is still buggy, so you wouldn't want to use it in a war time scenario? I'm not sure I'm making a ton of sense. It's just really interesting, I'm in Denmark and our government are about to pick which fighter plane will replace our old F-16s, and the F-35 is the favorite right now. This is despite experts saying it's a bad plane. What I understood from your comment is that it will fly just fine, but bugs in the ALIS software would make it harder to operate than anticipated.
        engi_nerd 11y ago
        The software that keeps the plane in the air is a totally separate thing from ALIS.
        Retric 11y ago
        My understanding is ALIS was part of the 'pork' side of the F-35. It has relatively simple requirements and was simply used to extract as much money as possible from the government. That said, they have tried to make it essential as possible.
- detritus 11y ago
  Given the low bars of success that seem to have been mentioned in this article, I'd wager there's still a lot wrong with the code. I'm not wholly sure '15 hours uptime' on a computer system is all that confidence-inducing, regardless of whether or not the plane can be reset after however few hours of flight-time it's able to muster.
  - engi_nerd 11y ago
    15 hours of uptime is outstanding when it comes to the avionics that aren't actually keeping the aircraft in the sky. Most legacy fighter avionics required restarts every few hours.
    - alphapapa 11y ago
      That's very interesting. Do you have any examples?
      - engi_nerd 11y ago
        Not from my own experience. Just going from the professional knowledge of friends of mine with years of mission systems engineering support.
        alphapapa 11y ago
        I'm just curious because I find it hard to believe that avionics in combat aircraft are or were so unreliable that they would have to be restarted every few hours. That would not work for e.g. bombers on long missions, fighters on CAP, etc. I can just imagine a pilot thinking, "Okay, we're about to cross the FLOT and I've got 20 minutes until the next avionics reboot. I'd better go ahead and reboot it now before we run into trouble..."
aburan28 11y ago
We should just sell them to Saudi Arabia
- josefresco 11y ago
  Israel might have a problem with that as they have an unofficial, official exclusive.
  - frandroid 11y ago
    The Israelis don't usually care for second-class material though...
- aburan28 11y ago
  I am genuinely interested in who/why I was downvoted for suggesting we sell F35's to Saudi Arabia
  - aeturnum 11y ago
    I didn't downvote you, but some things jump out at me. The suggestion we sell them to Saudi Arabia is orthogonal to the subject of the article (technical problems). Your post and the article do not, from a naive perspective, have a connection. My guess is that you think the F-35 project is not a worthwhile investment for the US defense system, but that it cost a lot of money to make, and so we should get some money out of it. Further, since Saudi Arabia has a lot of oil money and wants weapons, we should sell them this plane (which it seems you think is a failure) - even though Saudi Arabia is nominally an ally and it isn't obvious its a good idea to sell an ally weapons we consider bad investments. My previous paragraph is a just a stab at the thinking that might inform your post - I'm probably totally wrong. Whatever the motivating factors, there's a lot you don't say - either because you don't think it's necessary or you don't want to talk about those background assertions. Obviously, we all believe the people here understand (at least some of) the context, but my guess is that people found your post to not be forthcoming enough.
dsp1234 11y ago
Appears to be a copy of http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-critical-logistics-software.html http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-cr... from April 20th
willtim 11y ago
The software has apparently been written in C++, it all sounds like a bit of a farce. EDIT: to clarify the above comment, the context is the use of C++ and mainstream tools for 8 million lines of safety critical code.
- duaneb 11y ago
  You can write safe C++ (although it's hard). You can write extremely buggy haskell or ada. Are you really that much of a language bigot you can't see past the damn language to provide any interesting conversation?
  - hluska 11y ago
    Your comment would be stronger and you'd make the exact same point if you removed the entire last sentence. That level of aggression has no place here.
  - PeCaN 11y ago
    Speaking from experience, it's a hell of a lot easier to write buggy C++ than buggy Haskell or Ada (both of which can in some cases be formally verified to follow the spec). C++ is unsafe by default and you have to write lots extra checks to make sure your code is bulletproof. Ada is safe by default and you have to be verbose to do powerful-but-dangerous things with pointers. While yes, you can write secure and fault-tolerant code in any language and insecure and buggy code in any language, the sheer amount of C and C++ software that has been demonstrated to be insecure and very buggy in recent years raises some questions about whether that's the tool we want to use. Languages are tools. C is a $5 wrench with a blade for a handle. Ada is a pocketknife where you can pull out the wrench or blade if you need it. Both of them can do the same things, but with one you have to be very, very careful to not cut yourself.
    - sqeaky 11y ago
      > C++ is unsafe by default and you have to write lots extra checks to make sure your code is bulletproof No you don't, just use RAII and you cover almost all the safety issues. Put all memory allocations, file openings, mutex locking and other resource acquisition in constructors and the matching cleanup in deconstructors and it becomes very safe. Now, not only is memory cleaned up and exception safe, but several classes of file handling and multi-threading issues are categorically fixed. There are a few other rules to enhance this to the point of preposterous safety, but with just this technique I see fewer memory leaks than I do with some garbage collected languages. The big safety issues occur when people take the unplanned style of treating C++ as if it were C with Classes. Something that exacerbates this is a plethora of bad training material (mostly old from before C++03) and even university classes that use manual memory allocations outside a constructor (naked new statments), exceptions and manual memory releasing in exception unsafe ways. I am sure Ada can be used incorrectly, now imagine if in an old standard not all Ada compilers supported the same language features. Imagine the subset of common features was dangerous as hell. Imagine creating a memory structure like a linked list because only some compilers provided one prewritten and some are implementations were missing core language features. However, all the compilers ship with dangerous tools to build it wrong. This was C++ in 1995, and many people who learned it then never unlearned their mistakes. C++03 laid down a bunch of technical fixes and a minimum benchmark for decent compilers. It could have been used safely with what I described above, but much of the developer community was change averse. Now with C++11 we are changing and really cool things are happening because the developers are treating it like a different language.
alphapapa 11y ago
Some context is necessary here, because as engi_nerd explained, it's not as if the primary flight control system is crashing and having to be rebooted in mid-air. It's more like a separate system that's used by maintenance personnel on the ground. Of course, if the aircraft's maintenance procedures are designed around it, then deploying a whole wing of them isn't going to work very well, because high-performance aircraft are high-maintenance aircraft. But I can't help but think of Battlestar Galactica. At what point does this software become too complicated? At what point does it become a vulnerability to be exploited by the enemy, or simply a liability reducing readiness below what's needed in wartime? I don't recall hearing stories about the F-16's or F/A-18's computers crashing and needing to be rebooted (or even the F-22's).
- noir_lord 11y ago
  F-22 did have some howlers (the computers crapping out when they flew across an international dateline rings a bell) but as someone who casually follows high tech stuff including military equipment I don't remember anything on the scale of the issues they are having with the F-35. I think a lot of it comes back to scope creep, the F-22 was designed for one branch by one branch to do one thing well, the F-35 was designed for multiple branches by multiple branches to do a bunch of things somewhat well. It's an absolute bear of a programme.
protomyth 11y ago
"In one instance, maintainers even had to manually burn data onto CDs and drive off base to send the massive files across a civilian WiFi network." That is a level of scary, I'm not comfortable with in our military's data. Logistics data flying across a civilian WiFi is really not good.
- maxxxxx 11y ago
  Similar stuff happens in medical sometimes too. Everything is so restricted that at some point if you want to get something done you have to circumvent all security systems in a really bad hack way.
  - striking 11y ago
    So instead of gently decreasing security when it's not possible to be totally secure... it's better to throw the baby out with the bathwater. Wonderful.
    - wutbrodo 11y ago
      Exactly. One of the first things we learned in my UNDERGRAD security class is that "good" security which is unduly onerous will just make users route around it and revert to plain old bad security. The canonical example was ridiculous password requirements which just make people put their password on a post it on their monitor. As you said, security requirements that are that onerous and too rigid to relax when necessary is just shittily designed security to begin with.
      - knodi123 11y ago
        > "good" security which is unduly onerous will just make users route around it and revert to plain old bad security I believe this picture is also a suitably canonical example. http://www.jumpoverthere.com/wp-content/uploads/2012/10/security.jpg http://www.jumpoverthere.com/wp-content/uploads/2012/10/secu...
    - knowaveragejoe 11y ago
chishaku 11y ago
Highlights from the GAO report: * Pilots, maintainers, and administrators at three of the five sites we visited are concerned about ALIS’s ability to deploy and function in forward locations. For example, users are concerned about the large server size and connectivity requirements, and whether the system’s infrastructure can maintain power and withstand a high-temperature environment. * ALIS users at three of the five sites we visited are concerned that a failure in the system’s current infrastructure could degrade the system and ground the fleet. Currently, ALIS information, including data from all U.S. F-35 sites, flows from the Standard Operating Units (SOU) to a single national Central Point of Entry, and then to the lone Autonomic Logistics Operating Unit (ALOU).26 This data flow process has no back up system for continuity of operations if either of these servers were to fail. * Maintainers and pilots at three of the five sites we visited were concerned that ALIS does not have much interoperability with legacy aircraft systems... The ability to share information between ALIS and these legacy systems is vital due to the way the services operate. * Maintainers at four of the five sites we visited told us that the current Action Request (AR) process does not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues. * ALIS users at all five sites we visited are concerned with data accuracy issues within the system, including missing or inaccurate data and inaccessibility of raw data within ALIS. http://www.gao.gov/assets/680/676576.pdf http://www.gao.gov/assets/680/676576.pdf Description of ALIS risks starts on page 15.
- alphapapa 11y ago
  That is amazing: > ALIS does not have redundant infrastructure: ALIS’s current design results in all F-35 data produced across the U.S. fleet to be routed to a Central Point of Entry and then to ALIS’s main operating unit with no backup system or redundancy. If either of these fail, it could take the entire F -35 fleet offline. That sounds like a movie plot [Independence Day] (or Battlestar Galactica)! So, the enemy jams comms or shoots down satellites, and a F-35 squadron deployed to a base on the other side of the world can't do maintenance? An aircraft carrier is attacked and its antennas are destroyed, and they can't do maintenance? The enemy breaks into or takes offline the Central Point of Entry over the Internet (oh, but it's airgapped, of course...), and the entire F-35 fleet around the world can't do maintenance? (Or can't do it at the level required to maintain combat readiness.) Am I misunderstanding something here?
PeCaN 11y ago
Does anyone know what it's written in? Boeing writes (nearly) everything in Ada and as far as I know has never had any software problems of this scale. Most of it is in SPARK as well, a restricted subset of Ada that can be partially formally verified[1]. Frankly, not formally verifying mission-critical software where a failure can potentially threaten someone's life seems rather irresponsible. 1. Contrary to what seems popular opinion, regular Ada is not particularly strict (less so than Rust). SPARK is very, very strict about what you can compile, and if it compiles it follows the spec (which is usually easier to audit).
- topkekz 11y ago
  f-35 part starts at the bottom of page 37 https://www.csiac.org/wp-content/uploads/2016/02/2010_03_30_SoftwareQualityReliabilityandErrorPrediction.pdf https://www.csiac.org/wp-content/uploads/2016/02/2010_03_30_...
- nathancahill 11y ago
  Faulty car software threaten someone's life. Faulty fighter jet software threatens a lot more than that.
  - mbrameld 11y ago
    You're not wrong, you just missed the point.
- fsck--off 11y ago
  I think C++ but I'm not certain about that.
  - deleted 11y ago
    [deleted]
- Jtsummers 11y ago
  C and C++ for the vast majority. Some Ada. Some Fortran, but probably only in physical modules (LRUs) that are being reused (COTS or GOTS) from other projects. http://www.militaryaerospace.com/articles/2013/10/software-code-f-35.html http://www.militaryaerospace.com/articles/2013/10/software-c... -- but a Google search reveals much the same from any article I can find, and just reiterates what I've read in past articles on F-35 software.
tibbon 11y ago
Ok dumb question; if it's so damn broken why are we paying Lockheed Martin for it? Did someone not think to put a term in their contracts saying that if they can't meet basic milestones (working software) that the US Government (and the people) aren't going to pay for it? It's just software...
- bardworx 11y ago
  From my understanding, parts to this jet are made all over the US so its beneficial for US to keep paying which retains jobs in so many states. This plane is highly political and its not really easy to "pull the plug" or not pay since many states will loose workforce. Robert Gates wrote a chapter in his book about it that was fascinating. http://www.amazon.com/Duty-Memoirs-Secretary-at-War/dp/030794963X http://www.amazon.com/Duty-Memoirs-Secretary-at-War/dp/03079...
- jsprogrammer 11y ago
  Why would LM sign such a contract?
  - tormeh 11y ago
    Hopefully they wouldn't, when you look at the end result of their contribution to the project.
- acomjean 11y ago
  You'd have to rebid it, and who would take it? A lot of these are "cost plus" contracts (the government pays the cost + a fixed % profit, and bonuses for meeting delivery dates). This has been railed against as wasteful compared to the private industry [1] This is some of the thought process (from article:) "I've talked several times this year about the importance of having mature technology before you go into production," Thornberry said. "A lot of what happens with cost-plus situations is the technology's not mature so you don't know what it's going to cost because you haven't invented it yet. That's exactly what we're trying to get away from." So the more excuses the companies make the more money make (the spec changed..), as long as you can get someone to sign off on it. The cynic in me notes that a lot of these companies have former military heads as presidents. [1]http://www.defensenews.com/story/breaking-news/2016/02/25/mccain-lrsb-air-force-bomber-block-northrop-boeing-lockheed/80921324/ http://www.defensenews.com/story/breaking-news/2016/02/25/mc...
frandroid 11y ago
Space-X: check. Tesla Motors: check. Hyperloop: in progress. Tesla fighter jet: The missing project in Elon Musk's portfolio...
- wavefunction 11y ago
  I think he prefers being called "Iron Man" rather than "Tony Stark."
- deleted 11y ago
  [deleted]
azinman2 11y ago
How is this still getting funded?!
jlarocco 11y ago
This headline is really misleading. TFA is about the "Autonomic Logistics Information System" (ALIS) system used to automatically order replacements parts and schedule service, NOT the plane itself. It's grounding the plane because they can't schedule maintenance and order parts correctly now that they're moving to this new system, not because the plane itself is buggy. The planes have been flying (mostly bug free) for over 10 years now. A lot of people here are using this to scapegoat C++, but although the plane is written in C++, I don't think the ALIS is...
- drzaiusapelord 11y ago
  Don't bother. Threads like this just become mindless ragefests from people with political axes to grind and sites like vice know how to exploit these feelings for ad impressions. Liberal doves also want anything to attack new weapon systems spending. Its an ugly scene altogether and accurate information and commentary about military programs are lacking. I've lived through a few major weapons systems changes and its all the same. Heck, anyone who has ever worked on a large project knows this stuff is fairly common. A lot of people don't want this thing in the air, especially Russia, China, Iran, and North Korea. Its a very capable weapons system and fills a much needed role. Its crazy a lazily sourced article like this could make it to the front page of HN. It reads like my grandmother wrote it, "THE AIRPLANE IS RUN BY A COMPUTER! TECHNOLOGY IS SCARY!" To compete on this level, automation and complexity are required. Its silly to think we are going to back to how planes were designed 50 years ago.
  - outworlder 11y ago
    > Its a very capable weapons system and fills a much needed role. Which role?
  - hackuser 11y ago
    > Liberal doves also want anything to attack new weapon systems spending I generally agree with your points, but not this one. I've seen no partisan bias in the criticism, and John McCain is the biggest critic of this program, which is being pushed by the Obama administration. In fact, IME most of the criticism comes from the temper-tantrum-of-the-week branch of the GOP. Also, Democrats have become hawkish; almost everyone says they are for a strong military including Clinton and Obama. There aren't many of those knee-jerk doves left - can you name a prominent one?
_of 11y ago
Why was the software not such a big problem with other jets like F-16 and F-22? Is the F-35 software really so much more complicated?
- PeCaN 11y ago
  1) The F-35 has VASTLY more software and several software systems that the F-16 and F-22 didn't have. 2) The F-22 software was largely written in Ada, a sane language that's great for real-time systems that cannot fail. Apparently the F-35 software is mostly written in... C++. Writing fault-tolerant real-time C++ is like walking on a tightrope while gripping a gun with the safety off. You can do it, but it's not particularly pleasant. 3) Apparently the team was considerably expanded/replaced between the F-22 and F-35.
  - PantaloonFlames 11y ago
    ps: the F-22 was also over budget and under-delivered on expectations. So much that the Pentagon canceled the remaining production run after 195 aircraft. https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor What the heck are we doing with these airplane projects? We build stuff we don't need, that doesn't work, and is waaay more expensive than we thought, and the starting cost point was extremely expensive to begin with.
    - hackuser 11y ago
      > the F-22 was also over budget and under-delivered on expectations. So much that the Pentagon canceled the remaining production run after 195 aircraft It was canceled because it was expensive and no longer needed, because the mission it was designed for - fighting a peer enemy - no longer existed. That was 2009, when the U.S.'s primary military concern was fighting two simultaneous wars against enemies who didn't own one airplane or air defense system. Any plane could do the job; we didn't need F-22s. Also, our prior generation planes were still the best in the world. Seven years later, Russia and China have engaged in massive military military buildups, including significantly improving their air combat capabilties, and aggressive behavior. The peer enemy mission is back at the top of the list (and some are now talking about re-starting F-22 production).
esaym 11y ago
I don't really understand how the computer systems of the F-15 and F-14 which were written in 1970's and probably in assembly seemed flawless yet with todays tech we can't even do this?
- willtim 11y ago
  The issue is one of scale. And if by today's tech you mean C++, it also has safety issues that become apparent at scale.
- lostgame 11y ago
  >> probably in assembly >> wondering why it doesn't work now Assembly is instruction-by-instruction, line-by-line. Not that C++ is by nature an unstable language, but I'm sure the margin of error is higher by nature simply by not using ASM. Please, correct me if I'm wrong. I've often thought it programmers could be less afraid to code in ASM we would see the most incredible things on something like a PS4. Of course, that's impractical as all hell due to millions of lines of code turning into a billion or whatever. I used to use ASM all the time for Sega Genesis/GBA home-brew and what-not. I kind of actually miss it. :3
- hackuser 11y ago
  > I don't really understand how the computer systems of the F-15 and F-14 ... seemed flawless Who said they were flawless? Most major new platforms, including many prior fighter planes, had similar teething problems. Novel, huge, highly-complex, mission- (and life- and national-security-) critical systems are going to have a lot of them. Also, modern systems have orders of magnitude more software, due to orders of magnitude more computer capability.
- alkonaut 11y ago
  > I don't really understand how the computer systems of the F-15 and F-14 which were written in 1970's and probably in assembly seemed flawless yet with todays tech we can't even do this? Programming languages haven't evolved much since the 70's, yet the number of lines of code we have to write to fly a 5th generation fighter jet has probably followed the same curve as the number of transistors inside it. And scaling a CPU turned out to be easier than scaling a program. Imagine the difference in complexity between the earliest computers controlling car engines, and what the computers inside a new Tesla are doing. The same with jets. Modern jets can barely stay airborne without a lot of computer power.
cpeterso 11y ago
Here is an interesting talk about the F-35 (Joint Strike Fighter) C++ code given at CppCon 2014 by an engineer working on the project: https://www.youtube.com/watch?v=sRe77Mdna0Y https://www.youtube.com/watch?v=sRe77Mdna0Y The Joint Strike Fighter (JSF) is the first major DOD aircraft program to use C++. Much of this software is either safety critical or mission critical and so must be written in such a way as to be clear, readable, unambiguous, testable, and maintainable. We discuss the driving requirements behind the standard and its evolution. We give a quick overview of our standard and discuss how it differs from later standards such as MISRA C++. We discuss lessons learned over our nine year history of applying the standard to a large embedded software program. We also address ambiguities in rules and difficulties with automated checking of conformance with the standard.
- engi_nerd 11y ago
  As I said in another comment, these remarks apply to the aircraft only, and not to ALIS. To be honest I have no idea what ALIS is written in.
windexh8er 11y ago
As a whole the F-35 is one of those big disappointments that was the direct result of each service branch having a say in requirements of the aircraft. There are more than a few documentaries and articles outlining this in fantastic detail. Ars ran a similar article to this one in January: http://arstechnica.com/information-technology/2016/01/f-35-software-overrun-with-bugs-dod-testing-chief-warns/ http://arstechnica.com/information-technology/2016/01/f-35-s... I was an employee at Lockheed during the time of when the "JSF" vendors were competing for the contract. While I wasn't directly involved in any F-35 programs (I was on the MS2 side supporting NATO programs, particularly ASOC / SCCAN - that's another story in itself...) I was friends with a few programmers responsible for the main flight systems of the early aircraft (I'm unsure if that system was ultimately used in the production version or not) and the attitude seemed to be that the aircraft would never make it to operational use anyway. I was a, relatively, new and naive employee (first employment after college) and it struck me as odd that the plane was a joke internally to the company, at least those "building" pieces of it and it makes me wonder today if the lackadaisical attitude was ripe across all facets of the program. If so it makes sense today of all the failures the program has endured. The aircraft has always underperformed and been overbudget. Ultimately the tax payers foot the bill for a next-gen aircraft that continually gets beat by task-specific variants that are decades older unfortunately. I used to joke around with people about how the $1000 toilet wasn't far off once I learned about how every program's goal was to be cost plus and then milk scope creep to the Nth degree. The programs I was involved with were selling commodity Sun + Cisco + some overpriced serial to Ethernet bridges (http://spt.sunhillo.com/products.php http://spt.sunhillo.com/products.php) tied to some archaic war/peacetime ATC software. It was absolutely astounding what those packages cost and what was actually delivered. I saw some were asking what language for some of these systems and if it were along the lines of what I was exposed to it was a mix of anything and everything. Most critical systems were written in ADA, as suggested, but I saw Java (think awesome Java in the early 2000's), TCL, shell scripts and the like as major components to these systems. In fact I remember a dev having problems getting working out reading serial data from a GPS receiver to set proper system time - so he just bundled ntpd in the build as his own work. Working there was an eye opening experience for both good, and bad reasons. The stereotypical jokes were, unfortunately, true to an extent and I couldn't continue to justify pretending I was working on cool, unspeakable things that were shrouded in James Bond'esque mystique. Ultimately I was proud walking in day one and couldn't get out of there fast enough. The people were great, but the management and true colors of the company were centered on, IMO, calculated enrichment of select groups of people. It was never about the best product - it seemed it was about who was next in line to get fed.

141 points by Wonnk13 11y ago

No it's not http://www.defensenews.com/story/defense/air-space/2016/04/26/f-35-chief-software-bugs-no-longer-threat-ioc/83553372/ http://www.defensenews.com/story/defense/air-space/2016/04/2...

oxryly1 11y ago
Different software.

kogus 11y ago

How does this reconcile with this article? https://www.f35.com/news/detail/f-35-chief-software-bugs-no-longer-a-threat-to-ioc https://www.f35.com/news/detail/f-35-chief-software-bugs-no-...

engi_nerd 11y ago
Those two articles are talking about two different things. ALIS is the logistics system for the program that has an off-jet and on-jet component. ALIS takes the role of a traditional computerized maintenance management system and expands it to include an on-aircraft prognostics and health monitoring component. The idea is to have the jet tell you what's wrong with itself. The article you linked to is about the F-35's mission systems software, which is really a software suite that encompasses all the computers on the aircraft itself. The MS software load provides all the stuff necessary to make the jet's hardware do what it needs to do to fight.
- mrweasel 11y ago
  So, the plane will fly, it will just be hard to maintain and operate?
  - throw7 11y ago
    "... Bogdan is saying ALIS is not really critical after all, insisting the F-35 can fly without it for 30 days." So I guess we can wait and see what happens on the 31 day? SNAFU is all that comes to mind.
    - dTal 11y ago
      "Your trial period has expired. Please enter a valid product code."
  - engi_nerd 11y ago
    I don't understand your question. The plane will fly if [_______________]...what, exactly? EDIT: I am merely seeking clarification, not trying to evade answering.
    - mrweasel 11y ago
      Sorry, I meant the F-35 software that ensure that the plane will actually stays in the air is fine, with no real bugs left. The ALIS software, that helps ground crew, maintenance and mission planing, I assume, is still buggy, so you wouldn't want to use it in a war time scenario? I'm not sure I'm making a ton of sense. It's just really interesting, I'm in Denmark and our government are about to pick which fighter plane will replace our old F-16s, and the F-35 is the favorite right now. This is despite experts saying it's a bad plane. What I understood from your comment is that it will fly just fine, but bugs in the ALIS software would make it harder to operate than anticipated.
      - engi_nerd 11y ago
        The software that keeps the plane in the air is a totally separate thing from ALIS.
      - Retric 11y ago
        My understanding is ALIS was part of the 'pork' side of the F-35. It has relatively simple requirements and was simply used to extract as much money as possible from the government. That said, they have tried to make it essential as possible.
detritus 11y ago
Given the low bars of success that seem to have been mentioned in this article, I'd wager there's still a lot wrong with the code. I'm not wholly sure '15 hours uptime' on a computer system is all that confidence-inducing, regardless of whether or not the plane can be reset after however few hours of flight-time it's able to muster.
- engi_nerd 11y ago
  15 hours of uptime is outstanding when it comes to the avionics that aren't actually keeping the aircraft in the sky. Most legacy fighter avionics required restarts every few hours.
  - alphapapa 11y ago
    That's very interesting. Do you have any examples?
    - engi_nerd 11y ago
      Not from my own experience. Just going from the professional knowledge of friends of mine with years of mission systems engineering support.
      - alphapapa 11y ago
        I'm just curious because I find it hard to believe that avionics in combat aircraft are or were so unreliable that they would have to be restarted every few hours. That would not work for e.g. bombers on long missions, fighters on CAP, etc. I can just imagine a pilot thinking, "Okay, we're about to cross the FLOT and I've got 20 minutes until the next avionics reboot. I'd better go ahead and reboot it now before we run into trouble..."

aburan28 11y ago

We should just sell them to Saudi Arabia

josefresco 11y ago
Israel might have a problem with that as they have an unofficial, official exclusive.
- frandroid 11y ago
  The Israelis don't usually care for second-class material though...
aburan28 11y ago
I am genuinely interested in who/why I was downvoted for suggesting we sell F35's to Saudi Arabia
- aeturnum 11y ago
  I didn't downvote you, but some things jump out at me. The suggestion we sell them to Saudi Arabia is orthogonal to the subject of the article (technical problems). Your post and the article do not, from a naive perspective, have a connection. My guess is that you think the F-35 project is not a worthwhile investment for the US defense system, but that it cost a lot of money to make, and so we should get some money out of it. Further, since Saudi Arabia has a lot of oil money and wants weapons, we should sell them this plane (which it seems you think is a failure) - even though Saudi Arabia is nominally an ally and it isn't obvious its a good idea to sell an ally weapons we consider bad investments. My previous paragraph is a just a stab at the thinking that might inform your post - I'm probably totally wrong. Whatever the motivating factors, there's a lot you don't say - either because you don't think it's necessary or you don't want to talk about those background assertions. Obviously, we all believe the people here understand (at least some of) the context, but my guess is that people found your post to not be forthcoming enough.

dsp1234 11y ago

Appears to be a copy of http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-critical-logistics-software.html http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-cr... from April 20th

willtim 11y ago

The software has apparently been written in C++, it all sounds like a bit of a farce. EDIT: to clarify the above comment, the context is the use of C++ and mainstream tools for 8 million lines of safety critical code.

duaneb 11y ago
You can write safe C++ (although it's hard). You can write extremely buggy haskell or ada. Are you really that much of a language bigot you can't see past the damn language to provide any interesting conversation?
- hluska 11y ago
  Your comment would be stronger and you'd make the exact same point if you removed the entire last sentence. That level of aggression has no place here.
- PeCaN 11y ago
  Speaking from experience, it's a hell of a lot easier to write buggy C++ than buggy Haskell or Ada (both of which can in some cases be formally verified to follow the spec). C++ is unsafe by default and you have to write lots extra checks to make sure your code is bulletproof. Ada is safe by default and you have to be verbose to do powerful-but-dangerous things with pointers. While yes, you can write secure and fault-tolerant code in any language and insecure and buggy code in any language, the sheer amount of C and C++ software that has been demonstrated to be insecure and very buggy in recent years raises some questions about whether that's the tool we want to use. Languages are tools. C is a $5 wrench with a blade for a handle. Ada is a pocketknife where you can pull out the wrench or blade if you need it. Both of them can do the same things, but with one you have to be very, very careful to not cut yourself.
  - sqeaky 11y ago
    > C++ is unsafe by default and you have to write lots extra checks to make sure your code is bulletproof No you don't, just use RAII and you cover almost all the safety issues. Put all memory allocations, file openings, mutex locking and other resource acquisition in constructors and the matching cleanup in deconstructors and it becomes very safe. Now, not only is memory cleaned up and exception safe, but several classes of file handling and multi-threading issues are categorically fixed. There are a few other rules to enhance this to the point of preposterous safety, but with just this technique I see fewer memory leaks than I do with some garbage collected languages. The big safety issues occur when people take the unplanned style of treating C++ as if it were C with Classes. Something that exacerbates this is a plethora of bad training material (mostly old from before C++03) and even university classes that use manual memory allocations outside a constructor (naked new statments), exceptions and manual memory releasing in exception unsafe ways. I am sure Ada can be used incorrectly, now imagine if in an old standard not all Ada compilers supported the same language features. Imagine the subset of common features was dangerous as hell. Imagine creating a memory structure like a linked list because only some compilers provided one prewritten and some are implementations were missing core language features. However, all the compilers ship with dangerous tools to build it wrong. This was C++ in 1995, and many people who learned it then never unlearned their mistakes. C++03 laid down a bunch of technical fixes and a minimum benchmark for decent compilers. It could have been used safely with what I described above, but much of the developer community was change averse. Now with C++11 we are changing and really cool things are happening because the developers are treating it like a different language.

alphapapa 11y ago

Some context is necessary here, because as engi_nerd explained, it's not as if the primary flight control system is crashing and having to be rebooted in mid-air. It's more like a separate system that's used by maintenance personnel on the ground. Of course, if the aircraft's maintenance procedures are designed around it, then deploying a whole wing of them isn't going to work very well, because high-performance aircraft are high-maintenance aircraft. But I can't help but think of Battlestar Galactica. At what point does this software become too complicated? At what point does it become a vulnerability to be exploited by the enemy, or simply a liability reducing readiness below what's needed in wartime? I don't recall hearing stories about the F-16's or F/A-18's computers crashing and needing to be rebooted (or even the F-22's).

noir_lord 11y ago
F-22 did have some howlers (the computers crapping out when they flew across an international dateline rings a bell) but as someone who casually follows high tech stuff including military equipment I don't remember anything on the scale of the issues they are having with the F-35. I think a lot of it comes back to scope creep, the F-22 was designed for one branch by one branch to do one thing well, the F-35 was designed for multiple branches by multiple branches to do a bunch of things somewhat well. It's an absolute bear of a programme.

protomyth 11y ago

"In one instance, maintainers even had to manually burn data onto CDs and drive off base to send the massive files across a civilian WiFi network." That is a level of scary, I'm not comfortable with in our military's data. Logistics data flying across a civilian WiFi is really not good.

maxxxxx 11y ago
Similar stuff happens in medical sometimes too. Everything is so restricted that at some point if you want to get something done you have to circumvent all security systems in a really bad hack way.
- striking 11y ago
  So instead of gently decreasing security when it's not possible to be totally secure... it's better to throw the baby out with the bathwater. Wonderful.
  - wutbrodo 11y ago
    Exactly. One of the first things we learned in my UNDERGRAD security class is that "good" security which is unduly onerous will just make users route around it and revert to plain old bad security. The canonical example was ridiculous password requirements which just make people put their password on a post it on their monitor. As you said, security requirements that are that onerous and too rigid to relax when necessary is just shittily designed security to begin with.
    - knodi123 11y ago
      > "good" security which is unduly onerous will just make users route around it and revert to plain old bad security I believe this picture is also a suitably canonical example. http://www.jumpoverthere.com/wp-content/uploads/2012/10/security.jpg http://www.jumpoverthere.com/wp-content/uploads/2012/10/secu...
  - knowaveragejoe 11y ago

chishaku 11y ago

Highlights from the GAO report: * Pilots, maintainers, and administrators at three of the five sites we visited are concerned about ALIS’s ability to deploy and function in forward locations. For example, users are concerned about the large server size and connectivity requirements, and whether the system’s infrastructure can maintain power and withstand a high-temperature environment. * ALIS users at three of the five sites we visited are concerned that a failure in the system’s current infrastructure could degrade the system and ground the fleet. Currently, ALIS information, including data from all U.S. F-35 sites, flows from the Standard Operating Units (SOU) to a single national Central Point of Entry, and then to the lone Autonomic Logistics Operating Unit (ALOU).26 This data flow process has no back up system for continuity of operations if either of these servers were to fail. * Maintainers and pilots at three of the five sites we visited were concerned that ALIS does not have much interoperability with legacy aircraft systems... The ability to share information between ALIS and these legacy systems is vital due to the way the services operate. * Maintainers at four of the five sites we visited told us that the current Action Request (AR) process does not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues. * ALIS users at all five sites we visited are concerned with data accuracy issues within the system, including missing or inaccurate data and inaccessibility of raw data within ALIS. http://www.gao.gov/assets/680/676576.pdf http://www.gao.gov/assets/680/676576.pdf Description of ALIS risks starts on page 15.

alphapapa 11y ago
That is amazing: > ALIS does not have redundant infrastructure: ALIS’s current design results in all F-35 data produced across the U.S. fleet to be routed to a Central Point of Entry and then to ALIS’s main operating unit with no backup system or redundancy. If either of these fail, it could take the entire F -35 fleet offline. That sounds like a movie plot [Independence Day] (or Battlestar Galactica)! So, the enemy jams comms or shoots down satellites, and a F-35 squadron deployed to a base on the other side of the world can't do maintenance? An aircraft carrier is attacked and its antennas are destroyed, and they can't do maintenance? The enemy breaks into or takes offline the Central Point of Entry over the Internet (oh, but it's airgapped, of course...), and the entire F-35 fleet around the world can't do maintenance? (Or can't do it at the level required to maintain combat readiness.) Am I misunderstanding something here?

PeCaN 11y ago

Does anyone know what it's written in? Boeing writes (nearly) everything in Ada and as far as I know has never had any software problems of this scale. Most of it is in SPARK as well, a restricted subset of Ada that can be partially formally verified[1]. Frankly, not formally verifying mission-critical software where a failure can potentially threaten someone's life seems rather irresponsible. 1. Contrary to what seems popular opinion, regular Ada is not particularly strict (less so than Rust). SPARK is very, very strict about what you can compile, and if it compiles it follows the spec (which is usually easier to audit).

topkekz 11y ago
f-35 part starts at the bottom of page 37 https://www.csiac.org/wp-content/uploads/2016/02/2010_03_30_SoftwareQualityReliabilityandErrorPrediction.pdf https://www.csiac.org/wp-content/uploads/2016/02/2010_03_30_...
nathancahill 11y ago
Faulty car software threaten someone's life. Faulty fighter jet software threatens a lot more than that.
- mbrameld 11y ago
  You're not wrong, you just missed the point.
fsck--off 11y ago
I think C++ but I'm not certain about that.
- deleted 11y ago
  [deleted]
Jtsummers 11y ago
C and C++ for the vast majority. Some Ada. Some Fortran, but probably only in physical modules (LRUs) that are being reused (COTS or GOTS) from other projects. http://www.militaryaerospace.com/articles/2013/10/software-code-f-35.html http://www.militaryaerospace.com/articles/2013/10/software-c... -- but a Google search reveals much the same from any article I can find, and just reiterates what I've read in past articles on F-35 software.

tibbon 11y ago

Ok dumb question; if it's so damn broken why are we paying Lockheed Martin for it? Did someone not think to put a term in their contracts saying that if they can't meet basic milestones (working software) that the US Government (and the people) aren't going to pay for it? It's just software...

bardworx 11y ago
From my understanding, parts to this jet are made all over the US so its beneficial for US to keep paying which retains jobs in so many states. This plane is highly political and its not really easy to "pull the plug" or not pay since many states will loose workforce. Robert Gates wrote a chapter in his book about it that was fascinating. http://www.amazon.com/Duty-Memoirs-Secretary-at-War/dp/030794963X http://www.amazon.com/Duty-Memoirs-Secretary-at-War/dp/03079...
jsprogrammer 11y ago
Why would LM sign such a contract?
- tormeh 11y ago
  Hopefully they wouldn't, when you look at the end result of their contribution to the project.
acomjean 11y ago
You'd have to rebid it, and who would take it? A lot of these are "cost plus" contracts (the government pays the cost + a fixed % profit, and bonuses for meeting delivery dates). This has been railed against as wasteful compared to the private industry [1] This is some of the thought process (from article:) "I've talked several times this year about the importance of having mature technology before you go into production," Thornberry said. "A lot of what happens with cost-plus situations is the technology's not mature so you don't know what it's going to cost because you haven't invented it yet. That's exactly what we're trying to get away from." So the more excuses the companies make the more money make (the spec changed..), as long as you can get someone to sign off on it. The cynic in me notes that a lot of these companies have former military heads as presidents. [1]http://www.defensenews.com/story/breaking-news/2016/02/25/mccain-lrsb-air-force-bomber-block-northrop-boeing-lockheed/80921324/ http://www.defensenews.com/story/breaking-news/2016/02/25/mc...

frandroid 11y ago

Space-X: check. Tesla Motors: check. Hyperloop: in progress. Tesla fighter jet: The missing project in Elon Musk's portfolio...

wavefunction 11y ago
I think he prefers being called "Iron Man" rather than "Tony Stark."
deleted 11y ago
[deleted]

azinman2 11y ago

How is this still getting funded?!

jlarocco 11y ago

This headline is really misleading. TFA is about the "Autonomic Logistics Information System" (ALIS) system used to automatically order replacements parts and schedule service, NOT the plane itself. It's grounding the plane because they can't schedule maintenance and order parts correctly now that they're moving to this new system, not because the plane itself is buggy. The planes have been flying (mostly bug free) for over 10 years now. A lot of people here are using this to scapegoat C++, but although the plane is written in C++, I don't think the ALIS is...

drzaiusapelord 11y ago
Don't bother. Threads like this just become mindless ragefests from people with political axes to grind and sites like vice know how to exploit these feelings for ad impressions. Liberal doves also want anything to attack new weapon systems spending. Its an ugly scene altogether and accurate information and commentary about military programs are lacking. I've lived through a few major weapons systems changes and its all the same. Heck, anyone who has ever worked on a large project knows this stuff is fairly common. A lot of people don't want this thing in the air, especially Russia, China, Iran, and North Korea. Its a very capable weapons system and fills a much needed role. Its crazy a lazily sourced article like this could make it to the front page of HN. It reads like my grandmother wrote it, "THE AIRPLANE IS RUN BY A COMPUTER! TECHNOLOGY IS SCARY!" To compete on this level, automation and complexity are required. Its silly to think we are going to back to how planes were designed 50 years ago.
- outworlder 11y ago
  > Its a very capable weapons system and fills a much needed role. Which role?
- hackuser 11y ago
  > Liberal doves also want anything to attack new weapon systems spending I generally agree with your points, but not this one. I've seen no partisan bias in the criticism, and John McCain is the biggest critic of this program, which is being pushed by the Obama administration. In fact, IME most of the criticism comes from the temper-tantrum-of-the-week branch of the GOP. Also, Democrats have become hawkish; almost everyone says they are for a strong military including Clinton and Obama. There aren't many of those knee-jerk doves left - can you name a prominent one?

_of 11y ago

Why was the software not such a big problem with other jets like F-16 and F-22? Is the F-35 software really so much more complicated?

PeCaN 11y ago
1) The F-35 has VASTLY more software and several software systems that the F-16 and F-22 didn't have. 2) The F-22 software was largely written in Ada, a sane language that's great for real-time systems that cannot fail. Apparently the F-35 software is mostly written in... C++. Writing fault-tolerant real-time C++ is like walking on a tightrope while gripping a gun with the safety off. You can do it, but it's not particularly pleasant. 3) Apparently the team was considerably expanded/replaced between the F-22 and F-35.
- PantaloonFlames 11y ago
  ps: the F-22 was also over budget and under-delivered on expectations. So much that the Pentagon canceled the remaining production run after 195 aircraft. https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor What the heck are we doing with these airplane projects? We build stuff we don't need, that doesn't work, and is waaay more expensive than we thought, and the starting cost point was extremely expensive to begin with.
  - hackuser 11y ago
    > the F-22 was also over budget and under-delivered on expectations. So much that the Pentagon canceled the remaining production run after 195 aircraft It was canceled because it was expensive and no longer needed, because the mission it was designed for - fighting a peer enemy - no longer existed. That was 2009, when the U.S.'s primary military concern was fighting two simultaneous wars against enemies who didn't own one airplane or air defense system. Any plane could do the job; we didn't need F-22s. Also, our prior generation planes were still the best in the world. Seven years later, Russia and China have engaged in massive military military buildups, including significantly improving their air combat capabilties, and aggressive behavior. The peer enemy mission is back at the top of the list (and some are now talking about re-starting F-22 production).

esaym 11y ago

I don't really understand how the computer systems of the F-15 and F-14 which were written in 1970's and probably in assembly seemed flawless yet with todays tech we can't even do this?

willtim 11y ago
The issue is one of scale. And if by today's tech you mean C++, it also has safety issues that become apparent at scale.
lostgame 11y ago
>> probably in assembly >> wondering why it doesn't work now Assembly is instruction-by-instruction, line-by-line. Not that C++ is by nature an unstable language, but I'm sure the margin of error is higher by nature simply by not using ASM. Please, correct me if I'm wrong. I've often thought it programmers could be less afraid to code in ASM we would see the most incredible things on something like a PS4. Of course, that's impractical as all hell due to millions of lines of code turning into a billion or whatever. I used to use ASM all the time for Sega Genesis/GBA home-brew and what-not. I kind of actually miss it. :3
hackuser 11y ago
> I don't really understand how the computer systems of the F-15 and F-14 ... seemed flawless Who said they were flawless? Most major new platforms, including many prior fighter planes, had similar teething problems. Novel, huge, highly-complex, mission- (and life- and national-security-) critical systems are going to have a lot of them. Also, modern systems have orders of magnitude more software, due to orders of magnitude more computer capability.
alkonaut 11y ago
> I don't really understand how the computer systems of the F-15 and F-14 which were written in 1970's and probably in assembly seemed flawless yet with todays tech we can't even do this? Programming languages haven't evolved much since the 70's, yet the number of lines of code we have to write to fly a 5th generation fighter jet has probably followed the same curve as the number of transistors inside it. And scaling a CPU turned out to be easier than scaling a program. Imagine the difference in complexity between the earliest computers controlling car engines, and what the computers inside a new Tesla are doing. The same with jets. Modern jets can barely stay airborne without a lot of computer power.

cpeterso 11y ago

Here is an interesting talk about the F-35 (Joint Strike Fighter) C++ code given at CppCon 2014 by an engineer working on the project: https://www.youtube.com/watch?v=sRe77Mdna0Y https://www.youtube.com/watch?v=sRe77Mdna0Y The Joint Strike Fighter (JSF) is the first major DOD aircraft program to use C++. Much of this software is either safety critical or mission critical and so must be written in such a way as to be clear, readable, unambiguous, testable, and maintainable. We discuss the driving requirements behind the standard and its evolution. We give a quick overview of our standard and discuss how it differs from later standards such as MISRA C++. We discuss lessons learned over our nine year history of applying the standard to a large embedded software program. We also address ambiguities in rules and difficulties with automated checking of conformance with the standard.

engi_nerd 11y ago
As I said in another comment, these remarks apply to the aircraft only, and not to ALIS. To be honest I have no idea what ALIS is written in.

windexh8er 11y ago

As a whole the F-35 is one of those big disappointments that was the direct result of each service branch having a say in requirements of the aircraft. There are more than a few documentaries and articles outlining this in fantastic detail. Ars ran a similar article to this one in January: http://arstechnica.com/information-technology/2016/01/f-35-software-overrun-with-bugs-dod-testing-chief-warns/ http://arstechnica.com/information-technology/2016/01/f-35-s... I was an employee at Lockheed during the time of when the "JSF" vendors were competing for the contract. While I wasn't directly involved in any F-35 programs (I was on the MS2 side supporting NATO programs, particularly ASOC / SCCAN - that's another story in itself...) I was friends with a few programmers responsible for the main flight systems of the early aircraft (I'm unsure if that system was ultimately used in the production version or not) and the attitude seemed to be that the aircraft would never make it to operational use anyway. I was a, relatively, new and naive employee (first employment after college) and it struck me as odd that the plane was a joke internally to the company, at least those "building" pieces of it and it makes me wonder today if the lackadaisical attitude was ripe across all facets of the program. If so it makes sense today of all the failures the program has endured. The aircraft has always underperformed and been overbudget. Ultimately the tax payers foot the bill for a next-gen aircraft that continually gets beat by task-specific variants that are decades older unfortunately. I used to joke around with people about how the $1000 toilet wasn't far off once I learned about how every program's goal was to be cost plus and then milk scope creep to the Nth degree. The programs I was involved with were selling commodity Sun + Cisco + some overpriced serial to Ethernet bridges (http://spt.sunhillo.com/products.php http://spt.sunhillo.com/products.php) tied to some archaic war/peacetime ATC software. It was absolutely astounding what those packages cost and what was actually delivered. I saw some were asking what language for some of these systems and if it were along the lines of what I was exposed to it was a mix of anything and everything. Most critical systems were written in ADA, as suggested, but I saw Java (think awesome Java in the early 2000's), TCL, shell scripts and the like as major components to these systems. In fact I remember a dev having problems getting working out reading serial data from a GPS receiver to set proper system time - so he just bundled ntpd in the build as his own work. Working there was an eye opening experience for both good, and bad reasons. The stereotypical jokes were, unfortunately, true to an extent and I couldn't continue to justify pretending I was working on cool, unspeakable things that were shrouded in James Bond'esque mystique. Ultimately I was proud walking in day one and couldn't get out of there fast enough. The people were great, but the management and true colors of the company were centered on, IMO, calculated enrichment of select groups of people. It was never about the best product - it seemed it was about who was next in line to get fed.