7 ms·

How Airbus is debugging the A350

122 points by hencq 13y ago

brownbat 13y ago
The megastructures documentary provides a pretty captivating look at construction: http://www.youtube.com/user/megadocumentary1 http://www.youtube.com/user/megadocumentary1
- chockablock 13y ago
  Link is to doc on A380 not A350.
ChuckMcM 13y ago
I am impressed they have a distributed CAD/CAM system which lets them share the schematics of the planes construction with all the partners. "Source Code Control" in the 3D CAD space was abysmal, got better in the 3D digital feature space as studios created systems for asset management, and it seems to be solidly implemented by Airbus here : (video link: http://videos.airbus.com/video/dc6bd25e7f3s.html http://videos.airbus.com/video/dc6bd25e7f3s.html)
- pwnna 13y ago
  I'm currently working with some 3D CAD software like Inventor and Creo Parametrics. They still save versonioned files (gear.prt.1, gear.prt.2). Not sure what to think about that.
  - danpat 13y ago
    Reminds me of this: http://thedailywtf.com/Comments/The_Best-est_Version_Control.aspx http://thedailywtf.com/Comments/The_Best-est_Version_Control...
- nether 13y ago
  > "Source Code Control" in the 3D CAD space was abysmal, got better in the 3D digital feature space as studios created systems for asset management, and it seems to be solidly implemented by Airbus here I doubt that when technical drawing "versioning" predates CAD by decades, in the form of drawing release/review and configuration management (CM) groups that were highly organized by WW2. This was then implemented by packages like ENOVIA, SolidWorks PLM, which facilitate the review/signoff process to be paperless, but it is basically the same. This is completely different from the entertainment industry, which doesn't care about part compatibility, analyst reviews (stress, aerodynamics, weights, etc.), not to mention the nature of CAD data (rife with engineering metadata on assembly hierarchies, dimensions with tolerances, materials) being very different from the "looks good" graphics of entertainment/art.
  - ChuckMcM 13y ago
    My take on it has always been that 'drawings' (in the draftsperson meaning) were essentially the archive format of designs. So while you could look through a list of change orders in a drawing, seeing what that change was, or more commonly not seeing it, has been the challenge. Boeing made a big deal about this when, as a Sun customer in the '80s, Sun helped them put basic drawings online as living models/schematics. I would agree its less impressive if everyone is forced to used the same CAD package. The video did not state whether or not that was the case.
- achamayou 13y ago
  It's not implemented by Airbus, the revision control is built into CATIA, the commercial software they use (made by Dassault Systemes).
- frik 13y ago
  EADS/Airbus are using CATIA by Dassault Systemes/IBM. Though they had some issues during development of A380 (different software versions, change management, etc): Initial production of the A380 was troubled by delays attributed to the 530 km (330 mi) of wiring in each aircraft. Airbus cited as underlying causes the complexity of the cabin wiring (98,000 wires and 40,000 connectors), its concurrent design and production, the high degree of customisation for each airline, and failures of configuration management and change control. The German and Spanish Airbus facilities continued to use CATIA version 4, while British and French sites migrated to version 5. This caused overall configuration management problems, at least in part because wiring harnesses manufactured using aluminium rather than copper conductors necessitated special design rules including non-standard dimensions and bend radii; these were not easily transferred between versions of the software http://en.wikipedia.org/wiki/A380#Production_and_delivery_delays http://en.wikipedia.org/wiki/A380#Production_and_delivery_de... Nevertheless, CATIA is top-notch is used by many car, ship, aircraft and spacecraft manufactures.
  - ChuckMcM 13y ago
    Thanks for that, so the cost of doing business with EADS/Airbus is you need to buy CATIA license? In the past this was a challenge with people picking up small machine shops and what not since they had their own design flow. So I'm curious if IBM/Dassult made allowances for that or if there is a $50K "membership" fee which you have to pay to get into the EADS club :-)
jbapple 13y ago
Airbus has been one of the success stories commonly told by the static analysis community: http://www.astree.ens.fr/ http://www.astree.ens.fr/ (Here I mean https://en.wikipedia.org/wiki/Static_program_analysis https://en.wikipedia.org/wiki/Static_program_analysis , not https://en.wikipedia.org/wiki/Static_analysis https://en.wikipedia.org/wiki/Static_analysis )
- blah32497 13y ago
  I'm fairly ignorant of the details of static analysis, but why is it being done on programs written in C? Shouldn't they use languages specially suited for this kind of analysis? I remember learning that stateless programing (ie. functional programming) makes this kind of analysis several orders of magnitude easier since it eliminates coupling and control flow dependence. Yet I've never heard of critical software being written in Haskell or whatever.
  - muaddirac 13y ago
    If I had to guess, it's because C lets them model their software closely to how the hardware is designed.
  - achamayou 13y ago
    C99 with some restrictions isn't actually that big a language, it's quite possible to put together a formal semantics for it, especially if you disallow heap allocation. There's at least one fairly mature implementation of a certified compiler out there (CompCert) with only minor restrictions to the language.
  - jbapple 13y ago
    I suspect the arrow of causality goes the other way: the control software was written in C first. Later, Airbus wanted to gain confidence in its correctness. In other words, the static analysis works on C programs because there are more extant (and mission-critical) C programs than Haskell ones, and the authors of the static analysis software wanted their tool to be as useful as possible, so they chose to analyze C.
  - 13y ago
ninjazee124 13y ago
Anyone know how they built their 3D graphic page? http://images.businessweek.com/graphics/airbus-a350-3d-graphic/ http://images.businessweek.com/graphics/airbus-a350-3d-graph... How did they go from the Trimble/Sketchup A350 model to showing the model in the browser in "3D"?
- kalleboo 13y ago
  No idea, but it put this in the Javascript Console... Recommended listening: http://youtu.be/AjzcdvF3gDc?t=3m48s http://youtu.be/mGF_0AcHaGs http://youtu.be/kn6-c223DUU http://youtu.be/eF-4Cr9Iy_8 edit: further investigation looks like they're using http://threejs.org http://threejs.org loading a COLLADA-format file (that can even be QuickLooked on my Mac somehow) http://images.businessweek.com/graphics/airbus-a350-3d-graphic/models/airbus-a350-900-repos.dae http://images.businessweek.com/graphics/airbus-a350-3d-graph...
  - jonalmeida 13y ago
    I love easter eggs like this. I'm going to start checking the JS console for every website from now on..
  - skyebook 13y ago
    OS X has had OpenCOLLADA built into QuickLook (as well as Preview) for a few years now. That said, it can still be fussy with many scenes.
- ninjazee124 13y ago
  To answer my own question, looks like they are using three.js as described here to load the Collada/DAE file. http://tech.vg.no/2013/07/08/webgl-dae-model-viewer/ http://tech.vg.no/2013/07/08/webgl-dae-model-viewer/
- gulbrandr 13y ago
  Here is the script of the 3D scene (not minified) http://images.businessweek.com/graphics/airbus-a350-3d-graphic/js/main1392310992.js http://images.businessweek.com/graphics/airbus-a350-3d-graph...
pmr_ 13y ago
Given for how long we have been developing airplanes and even planes in almost the same size as the A350 the lack of a somewhat standardized development process astounds me. Did newly developed planes used to be less safe and were more problems worked out during actual use? Or did they just not have as many problems to begin with due to less automation and sturdier but heavier materials?
- Serow225 13y ago
  Having worked in the flight test industry in a 'prior work life', nothing that this article describes sounds especially interesting or novel. I think what happened is that when the planes started getting technologically intense and at the same time the development team became highly distributed (geographically and contracted), there was a period of time where things 'got out of control' in that a) the design/simulation tools didn't have good capabilities for dealing with this level of distributed/revisioned work b) it was more important to 'get the job done' than making sure that everyone used the same exact toolchain and was working on the same version of the model etc. Eventually this caught up with them and they experienced some significant issues (like the wiring snafu the article mentioned, and I also recall another issue where fuselage parts wouldn't mate up), which finally made the industry pull back and get serious about fixing these design tool/practice issues.
- branchan 13y ago
  The release cycles for aerospace companies are a lot longer than in software. The previous A330 that they are talking about was originally released back in the early 90's. I assume Airbus didn't think it needed a full development system overall for incremental upgrades but believed it did when it came to designing a whole brand new airplane.
jreichhold 13y ago
This is nothing new and isn't different from Boeing in anything they mentioned in the article. Yes the 787 had issues, but the same types of testing occurred. The 787 was fundamentally different from previous Boeing aircraft with lots of primary components made by subcontractors. Lack of rigor and believing things would just work (too optimistic) from what I have heard on the outside. Iron birds, flight tests, etc are the requirements from the certification authorities. I.e. this is a fluff piece acting as journalism where the title and conclusions don't match the data.
- Uchikoma 13y ago
  To me, not being an expert, the article mentioning a lot of rigor and thoroughness to not run again into the A380 problems, this "This is nothing new and isn't different from Boeing in anything they mentioned in the article." contradicts this "Lack of rigor and believing things would just work (too optimistic) from what I have heard on the outside."
  - deleted 13y ago
    [deleted]
  - jreichhold 13y ago
    This is an article with no attempt to understand the state of the art outside of what Airbus wanted written and is PR spin. Most of the techniques here were used in the 737NG program in the late 90s
    - Uchikoma 13y ago
      So your answer in this discussion is "PR spin. PR spin."
- yread 13y ago
  just look at how the production ramp up looks like (hint airbus has a public plan, there was no such thing for 787). The guys over at airliners.net are tracking the state of building frames. They are ~1100 hours into a 2400 hour test flight program and they have only 4 planes flying (or almost there) and 2 in various states of building. Compare that to 787 which had to fix so many uncomplete frames after finding issues in test flights
tankenmate 13y ago
I posted this story over the weekend and it didn't get traction. So my question is, what is the lag threshold to when it becomes a new submission?
- kristianp 13y ago
  This story has an extra "#p1" at the end of the link, which defeats HNs duplicate detection. The submitter probably did it accidentally, by going to a different page of the story and then back to page 1.
  - tankenmate 13y ago
    Well I'm glad it did slip by the detection; it meant more people got to see the article.
- cpeterso 13y ago
  "The Best Time to Post on Hacker News": http://nathanael.hevenet.com/the-best-time-to-post-on-hacker-news-a-comprehensive-answer/ http://nathanael.hevenet.com/the-best-time-to-post-on-hacker... The short answer is 9:00–10:00 AM EST on a weekday.
rqebmm 13y ago
So if I'm following this correctly, Airbus's breakthrough design philosophy is to use distributed version control to facilitate iterative construction with a heavy emphasis on integration testing?
ajcarpy2005 13y ago
A good fiction book for those interested in aircraft engineering, testing, maintenance, root cause analysis after problems, etc. is "Airframe" by Michael Crichton.
deleted 13y ago
[deleted]
todd8 13y ago
Imagine the difficulty in debugging modern CPUs. Remember the floating point problems Intel had? There are far too many possible edge cases to be confident that testing alone will reveal them. Consequently, both Intel and AMD use formal proof methodologies to verify the correctness of their processors. I know that AMD uses (or used to use) the work of Boyer and Moore for validation of their designs. Intel uses it own prover. [1] [1] Fifteen Years of Formal Property Verification in Intel by L Fix, 2008 [http://www.cs.ucc.ie/~herbert/CS6320/EXS/LimorFix%20Intel%20FM.pdf http://www.cs.ucc.ie/~herbert/CS6320/EXS/LimorFix%20Intel%20...]
aragot 13y ago
Re: "Derisking" Can someone explain me whether/how Agile methodologies would be applied to an Airbus project? I'm asking this because I can't always explain how to do Agile when people pretend there are a lot of reqs, so an industrial project would be a good example to try it on.
- Spearchucker 13y ago
  Not realistic. Agile has many very good practices, but the ones that are missing are exactly the ones needed when you have a zero bug tolerance. You mention derisking - creating a proof of concept (a short sprint that proves you can do something you're unsure about) is one way of eliminating risk. Agile however, includes no practices for identifying, analysing and managing risk. Similarly, absolute reliability requires upfront design, and documentation. See https://www.wittenburg.co.uk/Entry.aspx?id=99bb5987-e08d-4e81-974d-daa5e07c0d5f https://www.wittenburg.co.uk/Entry.aspx?id=99bb5987-e08d-4e8... .

122 points by hencq 13y ago

The megastructures documentary provides a pretty captivating look at construction: http://www.youtube.com/user/megadocumentary1 http://www.youtube.com/user/megadocumentary1

chockablock 13y ago
Link is to doc on A380 not A350.

ChuckMcM 13y ago

I am impressed they have a distributed CAD/CAM system which lets them share the schematics of the planes construction with all the partners. "Source Code Control" in the 3D CAD space was abysmal, got better in the 3D digital feature space as studios created systems for asset management, and it seems to be solidly implemented by Airbus here : (video link: http://videos.airbus.com/video/dc6bd25e7f3s.html http://videos.airbus.com/video/dc6bd25e7f3s.html)

pwnna 13y ago
I'm currently working with some 3D CAD software like Inventor and Creo Parametrics. They still save versonioned files (gear.prt.1, gear.prt.2). Not sure what to think about that.
- danpat 13y ago
  Reminds me of this: http://thedailywtf.com/Comments/The_Best-est_Version_Control.aspx http://thedailywtf.com/Comments/The_Best-est_Version_Control...
nether 13y ago
> "Source Code Control" in the 3D CAD space was abysmal, got better in the 3D digital feature space as studios created systems for asset management, and it seems to be solidly implemented by Airbus here I doubt that when technical drawing "versioning" predates CAD by decades, in the form of drawing release/review and configuration management (CM) groups that were highly organized by WW2. This was then implemented by packages like ENOVIA, SolidWorks PLM, which facilitate the review/signoff process to be paperless, but it is basically the same. This is completely different from the entertainment industry, which doesn't care about part compatibility, analyst reviews (stress, aerodynamics, weights, etc.), not to mention the nature of CAD data (rife with engineering metadata on assembly hierarchies, dimensions with tolerances, materials) being very different from the "looks good" graphics of entertainment/art.
- ChuckMcM 13y ago
  My take on it has always been that 'drawings' (in the draftsperson meaning) were essentially the archive format of designs. So while you could look through a list of change orders in a drawing, seeing what that change was, or more commonly not seeing it, has been the challenge. Boeing made a big deal about this when, as a Sun customer in the '80s, Sun helped them put basic drawings online as living models/schematics. I would agree its less impressive if everyone is forced to used the same CAD package. The video did not state whether or not that was the case.
achamayou 13y ago
It's not implemented by Airbus, the revision control is built into CATIA, the commercial software they use (made by Dassault Systemes).
frik 13y ago
EADS/Airbus are using CATIA by Dassault Systemes/IBM. Though they had some issues during development of A380 (different software versions, change management, etc): Initial production of the A380 was troubled by delays attributed to the 530 km (330 mi) of wiring in each aircraft. Airbus cited as underlying causes the complexity of the cabin wiring (98,000 wires and 40,000 connectors), its concurrent design and production, the high degree of customisation for each airline, and failures of configuration management and change control. The German and Spanish Airbus facilities continued to use CATIA version 4, while British and French sites migrated to version 5. This caused overall configuration management problems, at least in part because wiring harnesses manufactured using aluminium rather than copper conductors necessitated special design rules including non-standard dimensions and bend radii; these were not easily transferred between versions of the software http://en.wikipedia.org/wiki/A380#Production_and_delivery_delays http://en.wikipedia.org/wiki/A380#Production_and_delivery_de... Nevertheless, CATIA is top-notch is used by many car, ship, aircraft and spacecraft manufactures.
- ChuckMcM 13y ago
  Thanks for that, so the cost of doing business with EADS/Airbus is you need to buy CATIA license? In the past this was a challenge with people picking up small machine shops and what not since they had their own design flow. So I'm curious if IBM/Dassult made allowances for that or if there is a $50K "membership" fee which you have to pay to get into the EADS club :-)

jbapple 13y ago

Airbus has been one of the success stories commonly told by the static analysis community: http://www.astree.ens.fr/ http://www.astree.ens.fr/ (Here I mean https://en.wikipedia.org/wiki/Static_program_analysis https://en.wikipedia.org/wiki/Static_program_analysis , not https://en.wikipedia.org/wiki/Static_analysis https://en.wikipedia.org/wiki/Static_analysis )

blah32497 13y ago
I'm fairly ignorant of the details of static analysis, but why is it being done on programs written in C? Shouldn't they use languages specially suited for this kind of analysis? I remember learning that stateless programing (ie. functional programming) makes this kind of analysis several orders of magnitude easier since it eliminates coupling and control flow dependence. Yet I've never heard of critical software being written in Haskell or whatever.
- muaddirac 13y ago
  If I had to guess, it's because C lets them model their software closely to how the hardware is designed.
- achamayou 13y ago
  C99 with some restrictions isn't actually that big a language, it's quite possible to put together a formal semantics for it, especially if you disallow heap allocation. There's at least one fairly mature implementation of a certified compiler out there (CompCert) with only minor restrictions to the language.
- jbapple 13y ago
  I suspect the arrow of causality goes the other way: the control software was written in C first. Later, Airbus wanted to gain confidence in its correctness. In other words, the static analysis works on C programs because there are more extant (and mission-critical) C programs than Haskell ones, and the authors of the static analysis software wanted their tool to be as useful as possible, so they chose to analyze C.
- 13y ago

ninjazee124 13y ago

Anyone know how they built their 3D graphic page? http://images.businessweek.com/graphics/airbus-a350-3d-graphic/ http://images.businessweek.com/graphics/airbus-a350-3d-graph... How did they go from the Trimble/Sketchup A350 model to showing the model in the browser in "3D"?

kalleboo 13y ago
No idea, but it put this in the Javascript Console... Recommended listening: http://youtu.be/AjzcdvF3gDc?t=3m48s http://youtu.be/mGF_0AcHaGs http://youtu.be/kn6-c223DUU http://youtu.be/eF-4Cr9Iy_8 edit: further investigation looks like they're using http://threejs.org http://threejs.org loading a COLLADA-format file (that can even be QuickLooked on my Mac somehow) http://images.businessweek.com/graphics/airbus-a350-3d-graphic/models/airbus-a350-900-repos.dae http://images.businessweek.com/graphics/airbus-a350-3d-graph...
- jonalmeida 13y ago
  I love easter eggs like this. I'm going to start checking the JS console for every website from now on..
- skyebook 13y ago
  OS X has had OpenCOLLADA built into QuickLook (as well as Preview) for a few years now. That said, it can still be fussy with many scenes.
ninjazee124 13y ago
To answer my own question, looks like they are using three.js as described here to load the Collada/DAE file. http://tech.vg.no/2013/07/08/webgl-dae-model-viewer/ http://tech.vg.no/2013/07/08/webgl-dae-model-viewer/
gulbrandr 13y ago
Here is the script of the 3D scene (not minified) http://images.businessweek.com/graphics/airbus-a350-3d-graphic/js/main1392310992.js http://images.businessweek.com/graphics/airbus-a350-3d-graph...

pmr_ 13y ago

Given for how long we have been developing airplanes and even planes in almost the same size as the A350 the lack of a somewhat standardized development process astounds me. Did newly developed planes used to be less safe and were more problems worked out during actual use? Or did they just not have as many problems to begin with due to less automation and sturdier but heavier materials?

Serow225 13y ago
Having worked in the flight test industry in a 'prior work life', nothing that this article describes sounds especially interesting or novel. I think what happened is that when the planes started getting technologically intense and at the same time the development team became highly distributed (geographically and contracted), there was a period of time where things 'got out of control' in that a) the design/simulation tools didn't have good capabilities for dealing with this level of distributed/revisioned work b) it was more important to 'get the job done' than making sure that everyone used the same exact toolchain and was working on the same version of the model etc. Eventually this caught up with them and they experienced some significant issues (like the wiring snafu the article mentioned, and I also recall another issue where fuselage parts wouldn't mate up), which finally made the industry pull back and get serious about fixing these design tool/practice issues.
branchan 13y ago
The release cycles for aerospace companies are a lot longer than in software. The previous A330 that they are talking about was originally released back in the early 90's. I assume Airbus didn't think it needed a full development system overall for incremental upgrades but believed it did when it came to designing a whole brand new airplane.

jreichhold 13y ago

This is nothing new and isn't different from Boeing in anything they mentioned in the article. Yes the 787 had issues, but the same types of testing occurred. The 787 was fundamentally different from previous Boeing aircraft with lots of primary components made by subcontractors. Lack of rigor and believing things would just work (too optimistic) from what I have heard on the outside. Iron birds, flight tests, etc are the requirements from the certification authorities. I.e. this is a fluff piece acting as journalism where the title and conclusions don't match the data.

Uchikoma 13y ago
To me, not being an expert, the article mentioning a lot of rigor and thoroughness to not run again into the A380 problems, this "This is nothing new and isn't different from Boeing in anything they mentioned in the article." contradicts this "Lack of rigor and believing things would just work (too optimistic) from what I have heard on the outside."
- deleted 13y ago
  [deleted]
- jreichhold 13y ago
  This is an article with no attempt to understand the state of the art outside of what Airbus wanted written and is PR spin. Most of the techniques here were used in the 737NG program in the late 90s
  - Uchikoma 13y ago
    So your answer in this discussion is "PR spin. PR spin."
yread 13y ago
just look at how the production ramp up looks like (hint airbus has a public plan, there was no such thing for 787). The guys over at airliners.net are tracking the state of building frames. They are ~1100 hours into a 2400 hour test flight program and they have only 4 planes flying (or almost there) and 2 in various states of building. Compare that to 787 which had to fix so many uncomplete frames after finding issues in test flights

tankenmate 13y ago

I posted this story over the weekend and it didn't get traction. So my question is, what is the lag threshold to when it becomes a new submission?

kristianp 13y ago
This story has an extra "#p1" at the end of the link, which defeats HNs duplicate detection. The submitter probably did it accidentally, by going to a different page of the story and then back to page 1.
- tankenmate 13y ago
  Well I'm glad it did slip by the detection; it meant more people got to see the article.
cpeterso 13y ago
"The Best Time to Post on Hacker News": http://nathanael.hevenet.com/the-best-time-to-post-on-hacker-news-a-comprehensive-answer/ http://nathanael.hevenet.com/the-best-time-to-post-on-hacker... The short answer is 9:00–10:00 AM EST on a weekday.

rqebmm 13y ago

So if I'm following this correctly, Airbus's breakthrough design philosophy is to use distributed version control to facilitate iterative construction with a heavy emphasis on integration testing?

ajcarpy2005 13y ago

A good fiction book for those interested in aircraft engineering, testing, maintenance, root cause analysis after problems, etc. is "Airframe" by Michael Crichton.

deleted 13y ago

[deleted]

todd8 13y ago

Imagine the difficulty in debugging modern CPUs. Remember the floating point problems Intel had? There are far too many possible edge cases to be confident that testing alone will reveal them. Consequently, both Intel and AMD use formal proof methodologies to verify the correctness of their processors. I know that AMD uses (or used to use) the work of Boyer and Moore for validation of their designs. Intel uses it own prover. [1] [1] Fifteen Years of Formal Property Verification in Intel by L Fix, 2008 [http://www.cs.ucc.ie/~herbert/CS6320/EXS/LimorFix%20Intel%20FM.pdf http://www.cs.ucc.ie/~herbert/CS6320/EXS/LimorFix%20Intel%20...]

aragot 13y ago

Re: "Derisking" Can someone explain me whether/how Agile methodologies would be applied to an Airbus project? I'm asking this because I can't always explain how to do Agile when people pretend there are a lot of reqs, so an industrial project would be a good example to try it on.

Spearchucker 13y ago
Not realistic. Agile has many very good practices, but the ones that are missing are exactly the ones needed when you have a zero bug tolerance. You mention derisking - creating a proof of concept (a short sprint that proves you can do something you're unsure about) is one way of eliminating risk. Agile however, includes no practices for identifying, analysing and managing risk. Similarly, absolute reliability requires upfront design, and documentation. See https://www.wittenburg.co.uk/Entry.aspx?id=99bb5987-e08d-4e81-974d-daa5e07c0d5f https://www.wittenburg.co.uk/Entry.aspx?id=99bb5987-e08d-4e8... .