5 ms·

"Internal reviews identified that the heart of the problem was the fact that the different design groups working on the project had used different Computer Aide

"Internal reviews identified that the heart of the problem was the fact that the different design groups working on the project had used different Computer Aided Design (CAD) software to create the engineering drawings. The development of the aircraft was a collaboration between 16 sites spread across 4 different countries. German and Spanish designers had used one version of the software (CATIA version 4), while British and French teams had upgraded to version 5." Really interesting! While I've hardly worked on anything as complex as an A380, I take it as a high priority for all software packages, from dev to test to prod, all be on the exact same version. Python 3.4 on one box and 3.6 on another? Let's not.

sixdimensional 9y ago
I work for an aerospace-related org. You would be shocked at how even within a single org nobody enforces standards across different groups. Even within this one org they use a handful of different CAD systems whether by choice, or as a result of time passing and people and technology changing. Nobody spends the money to bring old drawings forward into newer tech when it changes either, and the CAD files tend to often be proprietary binary formats that are difficult to work with. Really quite annoying and wasteful, but it is the way it is because people don't think about these problems and plan/ spend in advance to avoid them.
- rjsw 9y ago
  The aerospace companies have spent a lot of money on open CAD formats [1], though it wouldn't have helped in this particular case. [1] https://en.wikipedia.org/wiki/ISO_10303 https://en.wikipedia.org/wiki/ISO_10303
obstinate 9y ago
Yep. At my employer, all programs that get deployed to production must be built with either the current version of the toolchain, or the previous version. All builds are hermetic, and it's a requirement that all programs get rebuilt and redeployed at least once every N months, so that the programs out in the wild aren't built with toolchains and libraries that are too wildly different from head. But there's a whole team dedicated to the task of maintaining that state here. I can imagine that at a company where this is not the core competency, there would be some slippage.
davidf18 9y ago
In a way it is ironic because in aeronautical engineering everything is well tested with backup systems when it comes to making the airframes, engines, and the entire airplane. Yet, they did not engineer their software system according to the same standards of residency which is a part of engineering often forgotten about by many. One of the great value-added of firms using open source software is indeed the integration testing of various software packages. Lightbend is one example, they do a full Scala stack https://www.lightbend.com/platform https://www.lightbend.com/platform Databricks for Spark, Cloudera and MapR for Spark/Hadoop, etc.
- davidf18 9y ago
  EDIT: Autocorrect correction: Yet, they did not engineer their software system according to the same standards of ++resiliency++ which is a part of engineering often forgotten about by many.
hliyan 9y ago
What's your opinion on docker in production where every component runs on its own container with its own version of dependencies?
mattmanser 9y ago
Several years ago I worked with a large multinational company where every site had their own IT teams. A lot of mergers, etc. Probably much like BAE. A firm who's major product is info. Some of those teams were extremely incompetent. My client had made a video of the CEO speaking to the whole company which turned out wouldn't fit on their internal sharepoint site because there was a 50 meg limit for videos. The CEO was not at all impressed with the quality of a 20 min video at 50 meg so in desperation, they ended up asking us to host it for them. But then vimeo didn't work for some sites, then wistia for others, in the end we ended up having to set it up on Amazon's cloudfront. So each site had their own firewall rules, own white-lists, bad IT connections, etc. Throughout this debacle, we got claims of our incompetence by IT personnel at their various sites. Really opened my eyes at how disorganised a large organisation can become, as well as how people can hold positions in large orgs that they have no ability to perform by politicking. And that's how you end up with loads of different versions of software at different sites in the same company. EDIT: Actually, I recall incorrectly, we originally used cloudfront, which didn't work, then vimeo was blocked on some sites, but no-one had blocked wistia yet as they were still a fairly new entrant. Random endorsement: Wistia are great, love their API, love their support, great video hosting company.
- maxxxxx 9y ago
  I see the same at my company. I think the reason is that the top guys in non-IT industries don't see IT as core competence but as a cost center that ideally gets outsourced to the lowest bidder. Unless there is a very strong CIO different divisions won't talk to each other and no standardization happens. Even if one division does a good job they usually can't sustain that for long because their budgets will get cut. It's a very unhappy situation for everybody and the best people leave after a while.
- tajen 9y ago
  Not sure politicking is the root reason for Airbus: Catia is just hugely expensive, so upgrading is really a considerable cost. But probably not worth causing $6.1bn delay – Technical debt can bite you in weird ways.
- dakial1 9y ago
gumby 9y ago
> Really interesting! ... I take it as a high priority for all software packages, from dev to test to prod, all be on the exact same version. Python 3.4 on one box and 3.6 on another? Let's not. The canonical way to get around this is unit test and CI. This article shows that airbus did that...and still got screwed. Which shows that your approach is a good one!

sixdimensional 9y ago

I work for an aerospace-related org. You would be shocked at how even within a single org nobody enforces standards across different groups. Even within this one org they use a handful of different CAD systems whether by choice, or as a result of time passing and people and technology changing. Nobody spends the money to bring old drawings forward into newer tech when it changes either, and the CAD files tend to often be proprietary binary formats that are difficult to work with. Really quite annoying and wasteful, but it is the way it is because people don't think about these problems and plan/ spend in advance to avoid them.

rjsw 9y ago
The aerospace companies have spent a lot of money on open CAD formats [1], though it wouldn't have helped in this particular case. [1] https://en.wikipedia.org/wiki/ISO_10303 https://en.wikipedia.org/wiki/ISO_10303

obstinate 9y ago

Yep. At my employer, all programs that get deployed to production must be built with either the current version of the toolchain, or the previous version. All builds are hermetic, and it's a requirement that all programs get rebuilt and redeployed at least once every N months, so that the programs out in the wild aren't built with toolchains and libraries that are too wildly different from head. But there's a whole team dedicated to the task of maintaining that state here. I can imagine that at a company where this is not the core competency, there would be some slippage.

davidf18 9y ago

In a way it is ironic because in aeronautical engineering everything is well tested with backup systems when it comes to making the airframes, engines, and the entire airplane. Yet, they did not engineer their software system according to the same standards of residency which is a part of engineering often forgotten about by many. One of the great value-added of firms using open source software is indeed the integration testing of various software packages. Lightbend is one example, they do a full Scala stack https://www.lightbend.com/platform https://www.lightbend.com/platform Databricks for Spark, Cloudera and MapR for Spark/Hadoop, etc.

davidf18 9y ago
EDIT: Autocorrect correction: Yet, they did not engineer their software system according to the same standards of ++resiliency++ which is a part of engineering often forgotten about by many.

hliyan 9y ago

What's your opinion on docker in production where every component runs on its own container with its own version of dependencies?

mattmanser 9y ago

Several years ago I worked with a large multinational company where every site had their own IT teams. A lot of mergers, etc. Probably much like BAE. A firm who's major product is info. Some of those teams were extremely incompetent. My client had made a video of the CEO speaking to the whole company which turned out wouldn't fit on their internal sharepoint site because there was a 50 meg limit for videos. The CEO was not at all impressed with the quality of a 20 min video at 50 meg so in desperation, they ended up asking us to host it for them. But then vimeo didn't work for some sites, then wistia for others, in the end we ended up having to set it up on Amazon's cloudfront. So each site had their own firewall rules, own white-lists, bad IT connections, etc. Throughout this debacle, we got claims of our incompetence by IT personnel at their various sites. Really opened my eyes at how disorganised a large organisation can become, as well as how people can hold positions in large orgs that they have no ability to perform by politicking. And that's how you end up with loads of different versions of software at different sites in the same company. EDIT: Actually, I recall incorrectly, we originally used cloudfront, which didn't work, then vimeo was blocked on some sites, but no-one had blocked wistia yet as they were still a fairly new entrant. Random endorsement: Wistia are great, love their API, love their support, great video hosting company.

maxxxxx 9y ago
I see the same at my company. I think the reason is that the top guys in non-IT industries don't see IT as core competence but as a cost center that ideally gets outsourced to the lowest bidder. Unless there is a very strong CIO different divisions won't talk to each other and no standardization happens. Even if one division does a good job they usually can't sustain that for long because their budgets will get cut. It's a very unhappy situation for everybody and the best people leave after a while.
tajen 9y ago
Not sure politicking is the root reason for Airbus: Catia is just hugely expensive, so upgrading is really a considerable cost. But probably not worth causing $6.1bn delay – Technical debt can bite you in weird ways.
dakial1 9y ago

gumby 9y ago

> Really interesting! ... I take it as a high priority for all software packages, from dev to test to prod, all be on the exact same version. Python 3.4 on one box and 3.6 on another? Let's not. The canonical way to get around this is unit test and CI. This article shows that airbus did that...and still got screwed. Which shows that your approach is a good one!