8 ms·
Might be time to remove working on the 787 from my resume. I feel like the poor thing has been one disaster after another in the news. I can't speak to the qua
by anderspitman 12y ago
Might be time to remove working on the 787 from my resume. I feel like the poor thing has been one disaster after another in the news.
I can't speak to the quality of the A and B level (most critical) code, but the development process for the C level software I was working on definitely could have used a lot of improvement. Messy code, tests and documentation were an afterthought/checkbox item, etc. The incentives were just wrong.
I think there's a ton of room for process innovation in avionics software development. One thing I wanted to build for a long time was a tool for tracing. In theory, every industry requirement (DO-178B, etc) was supposed to trace to a hight level Software Design Document (SDD) requirement, which was supposed to trace to a Software Requirements Document (SRD) requirement, which would trace to a code function. We maintained all of that BY HAND. It was a huge mess. Perfect example of something that could have been an extremely valuable development tool, but ended up just being a hassle to try and maintain.
Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics. Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...). It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011!
- icegreentea 12y agoI work in the medical device field, and we have a similar process requirement (traceability from Design Input Requirements -> Software Specification -> Software Verification Procedure (and implicitly, the actual code function) -> Software Verification Report). We're currently wrangling with a giant-ass spreadsheet to keep track, and it totally sucks.
- anderspitman 12y agoAh yes I completely forgot SRD -> TESTS -> Then Code. Maybe that's because we always did it the other way around... I'm telling you, there's money to be made building tools for this stuff. I think a big part of the reason things aren't being improved is that the people in a position to recognize bad process and tooling maybe aren't the type of people to see an opportunity to make money solving the problem rather than putting up with it. I wouldn't associate most of the engineers I knew at Honeywell as the type to stay up until 2AM every night for 3 months working on a side project to pitch to their boss. I think it's really exciting what's happening in healthcare right now though. The innovative culture is exploding. Ultimately I care much more about what happens in medicine than avionics, as long as planes aren't falling out of the sky every 248 days...
- HeyLaughingBoy 12y agoThere are already tools for this stuff. Problem is that they are all various forms of crappy and the market is so small that there is little incentive to improve them. I work in Medical Devices and one of the tools I'm supposed to use (we find every excuse to avoid it) has a UI like a 2001 Swing app and while it works, it is insanely painful to use due to its absolutely counterintuitive interface. We're actually integrating more and more of our work in to Visual Studio since its tools are excellent. The problem is that the organization needs to validate any tool before we can use it as a part of Quality Management and that process can take forever.
- anderspitman 12y agoVisual Studio is awesome. I'm really excited to see how Code turns out on Linux, especially for things like building GUI apps and 3D stuff.
- lenocinor 12y agoI worked at Honeywell on these kinds of projects. I agree that these people definitely weren't that type. :) Bizarrely now I am also helping (in a small way) with an innovative healthcare tech project. Maybe many people go through similar trajectories in programming without realizing it.
- thmzlt 12y agoI'm curious about this process. Do you have contact info or pointers to other resources about it?
- icegreentea 12y agoUnfortunately for you, I don't. I stay far away from the regulatory stuff - I really only enough to get my job done, mostly in the form of directives from my manager and our quality section. That said, our software development process is really just a mild adaption of all our other development processes (like ones we use for hardware and reagent development). As I've described it above, it's a pretty standard engineering approach to design control, and there are all sorts of variations on them. In fact, the FDA makes note of how broad the field of medical devices are, and does not actually enforce any specific design control process, but rather provides a framework for you to develop a process that meets your developmental and regulatory needs. As part of getting FDA approval, your design control process will be audited, as well as how well you followed it. In addition, medical devices are classified into 3 classes, corresponding to patient risk. Our product is currently classified in the lowest risk level, and so we can avoid a lot of additional requirements. As I recall from skimming through the standard (sorry, we only had a hard copy at work.. costs money apparently), higher risk devices with software components must have certain types of tests performed. In our risk class, we have total freedom to define our testing requirements - that said, the rigour of our testing is still under FDA scrutiny, it just means that there are no specific checkboxes to hit (like integration testing for example). I guess if you want a place to start off, this might be a good place to look: http://en.wikipedia.org/wiki/Design_controls http://en.wikipedia.org/wiki/Design_controls
- dsuth 12y agoIt's essentially the v model [1] for software development. It's commonly used in systems engineering and the industries that implement it, so military, avionics, medical, safety engineering etc. I work on industrial safety systems, and would echo the other posts here - it's a requirement of the industry, but it's rarely done well due to the cost to implement, and the difficulty in implementing tools like Doors. NASA and the U.S. Military have good guides and white papers on it, which you should be able to find online with a bit of digging. [1] http://en.m.wikipedia.org/wiki/V-Model_(software_development) http://en.m.wikipedia.org/wiki/V-Model_(software_development...
- halayli 12y agoDid you consider using dtrace for tracing? Edit: Oops, misread the comment.
- icegreentea 12y agoDifferent type of tracing. This is basically tracing line items from a document to another set of line items on lower level documents, potentially all the way down to specific lines of source code.
- anderspitman 12y agoJinx!
- anderspitman 12y agoDifferent type of tracing. I'm referring to textual tags which are used to associate a design requirement with a piece of code, the idea being that you can show a requirement is fulfilled by showing that it traces to tests and code, and that the existence of any piece of code needs to be justified by tracing back to the requirements. http://en.wikipedia.org/wiki/Requirements_traceability http://en.wikipedia.org/wiki/Requirements_traceability
- walterbell 12y agoBorland's Caliber was once popular for requirements traceability, http://www.borland.com/Products/Requirements-Management/Caliber http://www.borland.com/Products/Requirements-Management/Cali...
- anderspitman 12y agoCaliber looks great. I can only hope they're using something like that nowadays. Of course it has the word "Agile" in it so it's probably about 10 years too young.
- deleted 12y ago[deleted]
- mpweiher 12y ago> Then of course there's language choice. C is king... Is that true for all of Boeing? I recall reading about the Lustre synchronous dataflow language being used by Airbus. It apparently allows all sorts of interesting safety guarantees.
- anderspitman 12y agoNot sure about all of Boeing. I worked for a subcontractor who got a small piece of the project from Honeywell who contracted with Boeing. But we worked on aircraft from Boeing, Gulfstream, Airbus, etc and other than the 787 Ada stuff (which we ported to C), I don't recall seeing anything that wasn't written in C, C w/ classes (heavily stripped C++), or Javascript. Of course it's possible that my boss just asked for those types of projects since we had the most expertise there.
- _jwilde 12y agoCurious: what was Javascript used for? (In terms of general usage contexts.)
- anderspitman 12y agoThat was a joke ;)
- stcredzero 12y agoIn theory, every industry requirement (DO-178B, etc) was supposed to trace to a high level Software Design Document (SDD) requirement, which was supposed to trace to a Software Requirements Document (SRD) requirement, which would trace to a code function. We maintained all of that BY HAND. It was a huge mess. Perfect example of something that could have been an extremely valuable development tool, but ended up just being a hassle to try and maintain. This is a symptom that group culture there is interfering with rational decision making around tool use. Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics. This is another symptom. Perhaps it's excusable if there were preexisting libraries/codebase. It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011! Another indication that group culture is blocking rational decision making about tool/language choice.
- javert 12y ago> This is another symptom. Perhaps it's excusable if there were preexisting libraries/codebase. Uh, what would you use? As far as I know, the choices are: Assembler, Ada, and C. We're talking about avionics code, here. C++ is not going to help you in this case. Java will be too slow/unpredictable. Real-Time Java is kind of sketch in my opinion (I could be wrong there). Most everything else is totally out. Actually I'd probably choose Ada... but there are reasons the industry switched away from Ada.
- jonnybgood 12y agoWhen you say Java I'm assuming you're refering to a JVM? Now what JVM are you refering to?
- stcredzero 12y agoAs far as I know, the choices are: Assembler, Ada, and C. We're talking about avionics code, here. In that case, "group" may refer to the avionics software industry as a whole.
- 12y ago
- rancur 12y ago> the poor thing has been one disaster after another in the news everything this complicated has always had these problems, difference is news jumps on it like a hawk now
- rancur 12y ago> Rust so you want to fly in planes using software written in Rust
- anderspitman 12y agoA subset of it, yes. Any dynamic memory allocation is out, which would pare it down heavily, but you can still reap the safety advantages with the static stuff.
- kibwen 12y agoRust happens to have already defined a subset of itself that forbids dynamic memory allocation, which makes use of a pared-down standard library called libcore.
- anderspitman 11y agoAwesome! I figured such a thing would exist eventually but didn't realize it was already in the wild.
- rancur 12y agoI definitely don't see a problem telling people the plane was built with rust
- CHY872 12y ago> Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics. Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...). It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011! C has the big advantage that having been used for many years, it is very well understood, the compilers that exist are mature, and the formal analysis tools you want to use in a high assurance setting all exist also. Higher level languages usually have none of these things.
- deleted 12y ago[deleted]
- randomsearch 12y ago> C has the big advantage that having been used for many years, it is very well understood, C is probably the least understood language I know of. Vast parts of it are undefined, everyone makes their own decisions and compiler writers define the language. C is a practical choice because it is performant, legacy code exists, and because a lot of people know the language. Personally, I would never take a job writing safety-critical code in C. To give a single example of a far superior language for such applications, I would consider SPARK Ada, amongst others. I don't know how people that work on safety critical systems using C can sleep at night. C is for systems programming where performance is king, not for building software that can kill people.
- CHY872 12y agoAda is a good decision in this field. Nevertheless, the parent was comparing C to Rust, where the language is immature, has no static analysis tools (to pick up the most important bugs), no well defined semantics and is entirely unproven in the field. This is the barrier for entry.
- kibwen 12y ago
- twistedpair 12y ago> Because that's how far behind avionics is. We were using Visual Studio 6 in 2011! Wait, what about autoPilot.js ?
- mkramlich 12y ago> Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...). that's a terrifying statement. to those of us who know what a person's first programming projects are like, quality-wise just terrifying. mindlessly stupid on your management's part
- anderspitman 12y agoKeep in mind we're talking about C-level communications software, not an autopilot system. And the code was eventually tested heavily, just in a very inefficient and expensive manner. But I don't completely disagree with your assessment.
- DigitalJack 12y agoWe use Doors for that. I think we tried TeamCenter for one project but it was a bag of hurt. The UI for doors is a little clunky (looks like a win95 app), but it fulfills it's purpose. Doors equivalent functionality could be built with a web interface and a sql db backend. Maybe any kind of database would work, relational just makes the most sense to me. That said, assuming DO178B is similar to DO254, it's more about making sure you have a design process and that you are following it, and less about how good that process is.
- anderspitman 12y agoAh yes DOORS! I remember using that on one of our projects. Pretty sure it was 787. I had the feeling that if used properly it could be a powerful tool. But I also remember the whole IBM Rational suite being incredibly slow...
- smsm42 12y agoI'm not sure how C is specifically to blame for something like int overflow - while in many languages this specific problem won't happen, in many popular ones it would happen the same way as in C. E.g. doesn't Rust also have limited integer types?
- steveklabnik 12y agoIn Rust, integer overflow is defined behavior, though: https://github.com/rust-lang/rfcs/blob/master/text/0560-integer-overflow.md https://github.com/rust-lang/rfcs/blob/master/text/0560-inte...
- smsm42 12y agoTrue, but as far as I see from this RFC, you have to explicitly check for overflow or run code in debug mode (which of course doesn't help unless your CI includes the one-year-without-reboot test, which would probably do wonders to the production schedules ;). If you thought about it, when writing that code, then you could add the same check in C too! The whole issue is that most people do not make such checks consistently.
- randomsearch 12y ago> I'm not sure how C is specifically to blame for something like int overflow C is not a good choice when safety is paramount. "The low-level nature of C and C++ means that bit- and byte-level manipulation of objects is commonplace; the line between mathematical and bit-level operations can often be quite blurry. Wraparound behavior using unsigned integers is legal and well-defined, and there are code idioms that deliberately use it. On the other hand, C and C++ have undefined semantics for signed overflow and shift past bitwidth: operations that are perfectly well-defined in other languages such as Java." http://www.cs.utah.edu/~regehr/papers/overflow12.pdf http://www.cs.utah.edu/~regehr/papers/overflow12.pdf
- lenocinor 12y agoWeird, I had to do the same thing (porting Ada to C) for a similar project (the JSF). It's an awful idea, because so much is missing in the translation.