9 ms·
Current challenges with using Linux in aerospace applications
- mikece 3y ago"Linux does not have a safety culture... Linux does not have a quality culture." While this is critical for an airplane (as well as an automobile) I would think the seeds of this would be desirable in corporate application servers as well. While someone might not lose their life if an app server goes down the "move fast and break things" culture only gets you so far before a culture with adult supervision and an eye toward stability is required.
- mschuster91 3y ago> the "move fast and break things" culture only gets you so far before a culture with adult supervision and an eye toward stability is required. You can run multi billion dollar companies that way (Reddit, Twitter, Meta, Tesla), hell you can get richest-man-in-the-world by shitting on everything.
- crabbone 3y agoBut you cannot build airplanes this way, which is the point of the article.
- svleest 3y agoOur point was a but more nuanced. We pointed out the problem (and also value) of the collaborative innovation. Then, we also noted that with some care, we think a carefully customized distribution could be shown to be sufficiently safe.
- ccakes 3y agoIt’s wild to me that software engineers don’t need to be certified in a similar vein to civil engineers. As more and more of our comms move into closed-gardens over the internet, the opportunity for a bug to indirectly result in injury or death grows. All it takes is a bug in a billing system that prevents a delinquent customer from contacting emergency services..
- PrimeMcFly 3y ago> It’s wild to me that software engineers don’t need to be certified in a similar vein to civil engineers. At the moment any coder can claim to be a software engineer. Certification is needed so we can draw a distinction between those who can code, and those can can engineer software solutions, as well as holding them to a higher standard.
- lloeki 3y agoIn France it is: "engineer"-the-title specifically means having a degree from a state-certified engineering school, and when you get the degree the state inscribes you as an engineer in a national registry. Whether it's software engineer or another domain is immaterial, the title "engineer" is heavily protected and making use of it without the degree is liable for pursuit the same way claiming to be a doctor and delivering medical advice when one is not: in the same way as with the Hippocrates oath, engineers are held to a high moral standard of ethics and integrity. The words "élite de la nation" often comes up - and often in derogatory ways from outsiders - but it is borne out from these requirements, that engineers are to be placed in position of great power and corresponding responsibility, meaning they have a duty to call the shots in high stakes situations in ways that go beyond technical knowledge and mastery but also factor in intellectual and moral values. One can get a masters degree in computer science and/or software development though, which is the same "knowledge level" of studies (an engineering degree doubles as a masters degree), but that does not make one an engineer, which is a corps (in the same sense that e.g US Marines are a corps). This dates back to Napoleon and the creation of "classes préparatoires" and public (state-owned) engineer schools, heavily inspired by the military. Case in point Polytechnique is a military school (although the goal is not to produce armymen), and entering any of the three french military corps (the navy, the air force, and the "ground" army) as an officer goes through the same classe préparatoire process. This design has been relaxed over time and for quite a time there are private engineering schools that don't require going through these first two prépa years, but they still require the state certification to legally produce engineers.
- PrimeMcFly 3y ago
- WJW 3y agoWell nobody is ever against "more safety", it's just when you ask how much it is allowed to cost that the sputtering starts. Most companies don't even want to pay for basic maintenance or to pay license costs for a programming language, let alone that they would want to pay for libraries that could also be had for free in a FOSS version. Every FOSS library rightly comes with a license that states in all caps that "THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND". That is pretty directly opposed to the kind of "We guarantee this software will perform within spec 100% of the time" assurances that airplane manufacturers want.
- secondcoming 3y agoDo any CPUs ship with correctness guarantees?
- WJW 3y agoYou can definitely get CPUs which run in pairs and verify amongst themselves that they both come to the same answer when running the same code on the same inputs: https://en.wikipedia.org/wiki/Lockstep_(computing) https://en.wikipedia.org/wiki/Lockstep_(computing). Being physical devices, even if the design adheres to the spec any individual CPU might still be defective of course. You can only test "is it still good" by periodically running one through a test suite and re-certifying that it still returns the correct answers, but there is never any guarantee that it won't break a second later due to freak circumstances. You can get down to acceptable margins though.
- svleest 3y agoUsing redundant systems helps detect (dual redunant) and even correct (triple modular redundant) physical faults in the system, where one system will provide a different answer than the other(s). System redundancy does not detect/correct for design flaws, which are a common mode failure. Catching design flaws is currently done by testing (compared to known correct answers) and peer review (by domain experts). Someday, mathematical proofs might be used (known as "formal methods"), but currently these are only possible to use on very small software projects, such as the seL4 project that formally proved correctness for around 10K lines of code.
- cesarb 3y ago> the "move fast and break things" culture only gets you so far before a culture with adult supervision and an eye toward stability is required. In my opinion, we are still in a "pioneer" phase of software engineering, where expansion (aka "software is eating the world") is prioritized over consolidation. But I think this is already starting to change; one example I've observed (in corporate application servers) being an increased focus on software supply chain management.
- svleest 3y agoYes, that's it. We want to take advantage of the innovative technology in the "expansion" but need to carefully manage what we actually put on the airplane to demonstrate (with evidence) that it is well suited for the intended purpose. Which I think is what you mean by "consolidation".
- dijit 3y agoLinux itself does not have a culture of move fast and break things. Solaris had a more conservative approach than Linux though and people obviously didn't go in for that, thus IllumOS is essentially dead.
- SoftTalker 3y agoSolaris was more conservative in a business sense perhaps, but it was in no way built to avionics standards.
- nottorp 3y ago> "Linux does not have a safety culture... Linux does not have a quality culture." Of course, Linux is mostly competing with Windows and Mac OS, not airplane RTOSes. Thing is, are they expecting someone to do a linux based RTOS for free? Like just pull it off kernel.org and set it up in the airplane?
- svleest 3y agoLinux competes not only on the desktop with Windows and Mac OS, but also on the server and cloud environments, and also in embedded environments. It is highly scalable to many use cases. No, we don't expect to use Linux right off the shelf. As the full presentation noted we would create a carefully curated profile to go on the airplane. And that work to curate and manage it, and producing certification evidence of correctness -- is definitely not free.
- mikece 3y agoWhat operating systems are used for aircraft (or automobile) control/management systems?
- firesteelrain 3y agoVxWorks 653, VxWorks 7 In addition to Synergy BSP
- thecosmicfrog 3y agoThe Boeing 787 is apparently using INTEGRITY-178, a realtime OS from Green Hills Software: https://www.ghs.com/customers/boeing_787.html https://www.ghs.com/customers/boeing_787.html
- highduc 3y agoWhat is Tesla using on their cars?
- foobarbaz33 3y agoI heard Teslas run on a bunch of Javascript in a web browser.
- jpm_sd 3y agoAviation: INTEGRITY, LynxOS, or other certified closed source RTOS. Automotive: https://en.m.wikipedia.org/wiki/AUTOSAR https://en.m.wikipedia.org/wiki/AUTOSAR
- amelius 3y agoWhy don't they simply run the timing-critical stuff on a bunch of MCUs?
- cmvincen 3y agoLinux, usually very old and very well tested Linux. Things that are almost Linux, like LynxOS Lighter RTOSs like VxWorks. If it's simple box it may be bare metal. Oh, and sometimes Windows (but generally not for anything with a safety effect).
- reidacdc 3y agoWe have a related issue in US civilian Federal agencies, where the IT security posture has been moving for some time to a formal compliance scheme. The idea is that to manage things at scale, it's desirable to have certified solutions, and mandate a very broad set of controls. This makes general-purpose Linux systems a hard sell -- Ubuntu and RHEL have e.g. FIPS-validated encryption stacks, but they're generally older releases (currently Ubuntu 20.04 is certified and 22.04 certification is pending), and of course limiting your choice of distro is unwelcome for computational researchers. For data at rest, there are certified self-encrypting hard drives, but they are very hard to source, in part because the FIPS 140-2 suite is also very old, and the newer FIPS 140-3 suite is not yet certified. There are probably ways around this, the diversity and flexibility of Linux cuts both ways, so you can maybe do a FOSS VM infrastructure on top of a certified hypervisor, and get the best of both worlds that way, but it's a lot of work. And unlike in the aviation-safety world, it's not clear that the certified solution is technically better. It has pluses and minuses, but the biggest plus is administrative, not technical -- it's easy to check.
- dlor 3y agoWe're trying to fix this problem at Chainguard. We have our own Linux distro that packages modern versions of software (like minutes or hours after it's released), as well as older versions. We're also working on FIPS 140-2 and 3, and support pretty much every compliance framework we can find.
- c0l0 3y agoAll I have to say from personal experience (some of it gained working for a big bank) is that if you want and seek compliance, you will get neither security, nor saftey - but you will get compliance. :) Whoever tells you otherwise has got a bridge to sell, as well as some compliance- and "security"-facilitating "solutions" on top.
- horsawlarway 3y agoThis has been my exact experience (some of it gained working adjacent to a big bank). At a certain level, folks dropped any real pretense that the compliance regulations in industry were for anything other than shifting liability around and ensuring you can check the right checkbox when doing sales or getting audited. Actual security varied widely, and had zero relation to the compliance checklists.
- PrimeMcFly 3y agoHonestly one of the craziest things is Linus saying security vulnerabilities are just normal bugs, don't deserve special treatment or to be fixed with priority, nor should they even be announced. That's a terrible security approach. The article makes good points.
- cesarb 3y ago> Honestly one of the craziest things is Linus saying security vulnerabilities are just normal bugs, don't deserve special treatment or to be fixed with priority, nor should they even be announced. His position is more nuanced than that. For a full set of relevant quotes, see https://yarchive.net/comp/linux/security_bugs.html https://yarchive.net/comp/linux/security_bugs.html For instance, this quote seems particularly relevant to this article: " [...] In fact, all the boring normal bugs are _way_ more important, just because there's a lot more of them. I don't think some spectacular security hole should be glorified or cared about as being any more "special" than a random spectacular crash due to bad locking. [...] To me, security is important. But it's no less important than everything *else* that is also important! Linus "
- plasticeagle 3y ago"To me, security is important. But it's no less important than everything else that is also important!" I think this should go down in history as one of the most meaningless statements ever made.
- dijit 3y agosoftware that doesn't function at all is 100% secure and has other consequences too. Thus, is it more or less important that the software functions or that it is free of all security issues? If there was a condition that your UI became a different shade of grey in your web browser: you wouldnt buy a domain, create a catchy name, generate a logo and scream from the rooftops. He is just saying: there are important security bugs, there are important functional bugs- they are equally important. Insignificant security bugs are not more important than deadlocks; which if you were depending on the kernel for a life critical system could kill you.
- rcarmo 3y agoA lot of it is open (and friendly) criticism, but “With apologies to LOTR - one does not simply walk into aerospace using Linux” got me to laugh out loud.
- cosmiccatnap 3y agoLet them go implement fuchsia. It will check all their boxes right before it slams into a mountain from a km to mi conversion. I thought we handled this years ago and coming from aviation experts is rather strange that they don't know the industry has migrated away from having a singular operating system that can't die to having a series of redundant fail-safes to fall back to when it does. It's strange to see the places where the microkernel debate still rages on....and how little investment is being made by those complaining multibillion $$ international corporations into projects like fuschia, RTOS, ZephyerOS, GNU Hurd, MIT Mach (or even Darwin), or even Minix! I think these arguments are disingenuous and while they are valid the various organizations making them seem to aggressively not want to find solutions. I smell a strong desire to hold the vanguard of what they have built until they retire and can be unconcerned with compliance...understandable to a degree but harmful in the long run to be going so fast in the wrong direction. Maybe Linux isn't a good fit, that's fine but they clearly don't care about that, they just don't want to implement anything and Linux is a convenient scape goat to not have to contribute back into an open source project even one on a BSD license
- jacooper 3y agowell thats a nice he GNU Hurd can do excell in.
- jacooper 3y agoWow A nice nieche* can excell in* Thanks phone keyboards.
- third_rate_econ 3y agoI would say Boeing doesn't have a safety or quality culture either. Look at the 737 Max and the Starliner.
- svleest 3y agoAs one of the authors of the referenced article, I would be happy to answer further questions. The discussion so far has been vigorous!
- svleest 3y agoHere is the link to the full presentation: https://www.youtube.com/watch?v=skaj70Qo3FA&list=PLbzoR-pLrL6qmJK02QP35go9hj1jawqxy&index=168 https://www.youtube.com/watch?v=skaj70Qo3FA&list=PLbzoR-pLrL...
- svleest 3y agoIf this discussion has caught your interest and you have experience in Linux internals, you might be interested in joining the team at Boeing that is working towards broader use of Linux in aerospace. Here are two links to open positions: https://jobs.boeing.com/job/hazelwood/embedded-linux-senior-software-engineer-virtual/185/51449210064 https://jobs.boeing.com/job/st-louis/embedded-linux-lead-software-architect-chief-virtual/185/51449210736