5 ms·

The actual report: http://www.fss.aero/accident-reports/dvdfiles/PT/2001-08-24-PT.pdf http://www.fss.aero/accident-reports/dvdfiles/PT/2001-08-24-... One of my

by kqr 4y ago

The actual report: http://www.fss.aero/accident-reports/dvdfiles/PT/2001-08-24-PT.pdf http://www.fss.aero/accident-reports/dvdfiles/PT/2001-08-24-... One of my pet peeves: the first 7 "causes and contributing factors" effectively blames the engine technicians for taking shortcuts. Nowhere does it say anything about the scheduling pressure from management that made the technicians consider shortcuts in the first place. If you have hiring and firing power and you tell someone they need to get the plane back in the air a few days later, they will do things to get the plane back in the air a few days later. This is considered simply a "risk" by the report, and not a direct cause. As a consequence, the actual actions taken basically all belong to the category of "fix the engineers", "fix the flight crew", or "fix the plane". Not a single "fix the stressful maintenance environment".

Shank 4y ago
It really underscores the importance of using a framework like HFACS [0] in a situation like this, because you end up exhausting all possible sources of problems and ensuring that you reach the correct root cause. It seems abundantly clear that the forcing-function was getting the aircraft back into service as fast as possible, not as safely as possible or as carefully as possible. A well-designed organization would have said that any unexpected contingencies during the maintenance process would have led to a full re-evaluation of the situation to understand it. The moment the engine didn't match their understanding, they should have scheduled a replacement plane or cancelled the upcoming flight, and reset expectations about how fast the plane could be back in-service. [0]: https://en.wikipedia.org/wiki/Human_Factors_Analysis_and_Classification_System https://en.wikipedia.org/wiki/Human_Factors_Analysis_and_Cla...
silvestrov 4y ago
Also lack of access to documentation: the technician attempted to access the Rolls-Royce EIPC and the SB’s through the computer network, but was also denied access. This should be a major point for a maintenance shop. Documentation should always be available, both online and in offline versions.
- hef19898 4y ago
  Which it is, tech pubs are huge part of the after market activities of every aerospace manufacturer. That those are not up to date, or not available when or where needed, is a major finding during audits or investigations.
- gambiting 4y ago
  The article says the service bulletin was actually available on site through a different publication, but the mechanic wasn't aware of this(and yet did the right thing of contacting dedicated support, who didn't advise him correctly): " The Rolls Royce SBs were also listed in the Trent Illustrated Parts Catalogue, accessible from any computer at the facility, but he was apparently unaware of this, so he instead switched to plan B and called the Air Transat Maintenance Control Center for help."
- MilStdJunkie 4y ago
  So many problems involving tech pubs process here. I could almost classify this whole incident as a tech pubs problem, since the aircrew faults also involved the operational checklists and their planning/guidance. TechPubs teams often have no control over the compliance (Optional, Mandatory, etc) of the SBs - that, at least in my experience, always came from way up high in the chain, sometimes from a VP who wasn't even tangentially involved in MRO. The required procedures - MX or ACRW - also are often coming from a costs concern, rather than criticality. The whole process of figuring out what's critical and what's not - that's a problem still being smacked into, every day. It's not hard to make a Sanity Check for pubs vs ERP/PDM/CMMIS, but often people involved REALLY DON'T WANT TO SEE THAT. Because it's horrifying. The commenter who mentioned HFACS is right on the money. Unless you implement something equivalent, you're gonna be chasing your own tail. What a great article.
dsfyu404ed 4y ago
What shortcuts? Spec said no touchy between tubes. Tubes no touchy when aircraft left the hanger? Seems good to me. These are the kinds of failures you get when you try to proceduralize everything with foolproof procedures, checklists, run-books, processes, etc. The maintenance people didn't even know they were making a judgement call. Had the technicians sleeved the offending tubes with appropriate chafing gear and secured them to reduce movement as would be common place in more earthly situations instead of trying to get it "technically in spec" without thinking about the implications of what they were actually doing this problem may very well have been averted. But maintenance people in aviation aren't really ever exercising those creative problem solving skills since it's such a book driven workflow they're always dealing with so of course they missed the forest for the trees (not that they would have been allowed to implement a one-off fix like that).
- lbriner 4y ago
  I don't think this is true at all > These are the kinds of failures you get when you try to proceduralize everything with foolproof procedures > Had the technicians The report makes it clear that the problem was their lack of access to the relevant details, not that the process was defective. If they had seen the need to replace the hydraulic hose, they would have presumably changed it or flagged the problem to someone higher up. > The maintenance people didn't even know they were making a judgement call. It seems like they did. If you replace a part and something is touching, you are not trusted in aviation to try and make some space with a bit of fiddling for exactly this reason. They did make a judgment call. If they had relied on the process, again, they could have queried why the new pump was touching when the old one wasn't.
- bambax 4y ago
  > But maintenance people in aviation aren't really ever exercising those creative problem solving skills This is the opposite of what happened. The engineers were confronted with a novel and unexpected problem, and instead of following procedure (which they lacked access to) they applied "creative problem solving skills". From TFA: > Although it was possible to install the post-SB fuel lines and hydraulic pump with a pre-SB hydraulic tube, the tube would rest against one of the fuel lines at a point where it rounded a 90-degree bend close to the pump. Aware that the plane could not be dispatched unless there was clearance between the tubes, the technicians torqued a nut on the end of the hydraulic tube until it rose approximately 0.635 mm off the face of the fuel line. Torquing a nut so that the hydraulic tube isn't technically rubbing against the fuel line is pretty creative and seems to solve the problem. But it didn't take into account pressure and vibration while in flight, and almost caused the death of 306 people.
peteradio 4y ago
Replace the use of the word "risk" with "gamble" and then it would paint the scene correctly.
- q1w2 4y ago
  Semantics don't solve problems.
btrettel 4y ago
That's a pet peeve of mine too. Here's a relevant HN comment I wrote about my experience as a patent examiner that applies more broadly: https://news.ycombinator.com/item?id=31315234 https://news.ycombinator.com/item?id=31315234 > There are way too many jobs where management doesn't give the front-line worker an appropriate amount of time and/or resources, and when things inevitably go wrong, the front-line worker is blamed, and the problem is studied as if the only factor was the front-line worker.
mannykannot 4y ago
In this case, there is no evidence here of any ontoward pressure on the maintenance crew, and there is not even any evidence that anyone in the chain of command outside of those involved in the task had any inkling that this might develop into a schedule disruption. On the contrary, the evidence strongly suggests that the maintenance crew did not feel they were under any special pressure: in their minds, they had found the correct solution, one which would allow them to complete the job on time, so any risk of flights being delayed or canceled had been solved. You do not even have to work in the airline industry to see that unscheduled maintenance risks creating delays, and that is less than optimal for everyone involved - no-one has to be told it. If that knowledge creates too much stress and conflict of interest to be safe, commercial aviation would simply be untenable.
- dkarl 4y ago
  This reminds me of times in the first couple of years of my career, where me and some other junior people were terrified that we weren't going to get a project done on time. We would be wracking our brains for out-of-the-box ways to get it done, whispering to each other, having trouble sleeping, and then we would be amazed that the lead developer would simply go to our manager (while we were still desperately brainstorming) and tell him we wouldn't finish on time. And nothing awful would happen. The lead developer was fine. We were fine. The project finished late. Nobody got fired. Now as a senior developer it's hard for me see why I was so terrified. Some managers yell and bluster. (Or, they used to, back in the day.) Some managers shame and insinuate. Some managers obliquely threaten. But they all, no matter how they express their authority, have to face up to the unpredictability of project planning. The question is not whether your manager is nice or mean. The question is, do they really think they're better off firing you and replacing you? Whether because they think they can get an upgrade or simply to pin the blame on you for something. After you've been around the block once or twice, you understand that a manager who is chill to your face is ultimately subject to the same motivations as a manager who uses stress as a weapon. I wonder what it's like for airline mechanics? They are represented by unions. I would think that safety infractions would be the real potential career killers, but that's just me guessing.
  - 4y ago
sidewndr46 4y ago
Management cannot be at fault because they are not responsible for safely flying the aircraft or safely maintaining the aircraft. That is the job of the employees, not management. Now if management had for example knowingly procured inferior replacement parts or hired non-certified personnel to work on the aircraft, they would be at fault.
- deleted 4y ago
  [deleted]

by kqr 4y ago

Shank 4y ago

It really underscores the importance of using a framework like HFACS [0] in a situation like this, because you end up exhausting all possible sources of problems and ensuring that you reach the correct root cause. It seems abundantly clear that the forcing-function was getting the aircraft back into service as fast as possible, not as safely as possible or as carefully as possible. A well-designed organization would have said that any unexpected contingencies during the maintenance process would have led to a full re-evaluation of the situation to understand it. The moment the engine didn't match their understanding, they should have scheduled a replacement plane or cancelled the upcoming flight, and reset expectations about how fast the plane could be back in-service. [0]: https://en.wikipedia.org/wiki/Human_Factors_Analysis_and_Classification_System https://en.wikipedia.org/wiki/Human_Factors_Analysis_and_Cla...

silvestrov 4y ago

Also lack of access to documentation: the technician attempted to access the Rolls-Royce EIPC and the SB’s through the computer network, but was also denied access. This should be a major point for a maintenance shop. Documentation should always be available, both online and in offline versions.

hef19898 4y ago
Which it is, tech pubs are huge part of the after market activities of every aerospace manufacturer. That those are not up to date, or not available when or where needed, is a major finding during audits or investigations.
gambiting 4y ago
The article says the service bulletin was actually available on site through a different publication, but the mechanic wasn't aware of this(and yet did the right thing of contacting dedicated support, who didn't advise him correctly): " The Rolls Royce SBs were also listed in the Trent Illustrated Parts Catalogue, accessible from any computer at the facility, but he was apparently unaware of this, so he instead switched to plan B and called the Air Transat Maintenance Control Center for help."
MilStdJunkie 4y ago
So many problems involving tech pubs process here. I could almost classify this whole incident as a tech pubs problem, since the aircrew faults also involved the operational checklists and their planning/guidance. TechPubs teams often have no control over the compliance (Optional, Mandatory, etc) of the SBs - that, at least in my experience, always came from way up high in the chain, sometimes from a VP who wasn't even tangentially involved in MRO. The required procedures - MX or ACRW - also are often coming from a costs concern, rather than criticality. The whole process of figuring out what's critical and what's not - that's a problem still being smacked into, every day. It's not hard to make a Sanity Check for pubs vs ERP/PDM/CMMIS, but often people involved REALLY DON'T WANT TO SEE THAT. Because it's horrifying. The commenter who mentioned HFACS is right on the money. Unless you implement something equivalent, you're gonna be chasing your own tail. What a great article.

dsfyu404ed 4y ago

What shortcuts? Spec said no touchy between tubes. Tubes no touchy when aircraft left the hanger? Seems good to me. These are the kinds of failures you get when you try to proceduralize everything with foolproof procedures, checklists, run-books, processes, etc. The maintenance people didn't even know they were making a judgement call. Had the technicians sleeved the offending tubes with appropriate chafing gear and secured them to reduce movement as would be common place in more earthly situations instead of trying to get it "technically in spec" without thinking about the implications of what they were actually doing this problem may very well have been averted. But maintenance people in aviation aren't really ever exercising those creative problem solving skills since it's such a book driven workflow they're always dealing with so of course they missed the forest for the trees (not that they would have been allowed to implement a one-off fix like that).

lbriner 4y ago
I don't think this is true at all > These are the kinds of failures you get when you try to proceduralize everything with foolproof procedures > Had the technicians The report makes it clear that the problem was their lack of access to the relevant details, not that the process was defective. If they had seen the need to replace the hydraulic hose, they would have presumably changed it or flagged the problem to someone higher up. > The maintenance people didn't even know they were making a judgement call. It seems like they did. If you replace a part and something is touching, you are not trusted in aviation to try and make some space with a bit of fiddling for exactly this reason. They did make a judgment call. If they had relied on the process, again, they could have queried why the new pump was touching when the old one wasn't.
bambax 4y ago
> But maintenance people in aviation aren't really ever exercising those creative problem solving skills This is the opposite of what happened. The engineers were confronted with a novel and unexpected problem, and instead of following procedure (which they lacked access to) they applied "creative problem solving skills". From TFA: > Although it was possible to install the post-SB fuel lines and hydraulic pump with a pre-SB hydraulic tube, the tube would rest against one of the fuel lines at a point where it rounded a 90-degree bend close to the pump. Aware that the plane could not be dispatched unless there was clearance between the tubes, the technicians torqued a nut on the end of the hydraulic tube until it rose approximately 0.635 mm off the face of the fuel line. Torquing a nut so that the hydraulic tube isn't technically rubbing against the fuel line is pretty creative and seems to solve the problem. But it didn't take into account pressure and vibration while in flight, and almost caused the death of 306 people.

peteradio 4y ago

Replace the use of the word "risk" with "gamble" and then it would paint the scene correctly.

q1w2 4y ago
Semantics don't solve problems.

btrettel 4y ago

That's a pet peeve of mine too. Here's a relevant HN comment I wrote about my experience as a patent examiner that applies more broadly: https://news.ycombinator.com/item?id=31315234 https://news.ycombinator.com/item?id=31315234 > There are way too many jobs where management doesn't give the front-line worker an appropriate amount of time and/or resources, and when things inevitably go wrong, the front-line worker is blamed, and the problem is studied as if the only factor was the front-line worker.

mannykannot 4y ago

In this case, there is no evidence here of any ontoward pressure on the maintenance crew, and there is not even any evidence that anyone in the chain of command outside of those involved in the task had any inkling that this might develop into a schedule disruption. On the contrary, the evidence strongly suggests that the maintenance crew did not feel they were under any special pressure: in their minds, they had found the correct solution, one which would allow them to complete the job on time, so any risk of flights being delayed or canceled had been solved. You do not even have to work in the airline industry to see that unscheduled maintenance risks creating delays, and that is less than optimal for everyone involved - no-one has to be told it. If that knowledge creates too much stress and conflict of interest to be safe, commercial aviation would simply be untenable.

dkarl 4y ago
This reminds me of times in the first couple of years of my career, where me and some other junior people were terrified that we weren't going to get a project done on time. We would be wracking our brains for out-of-the-box ways to get it done, whispering to each other, having trouble sleeping, and then we would be amazed that the lead developer would simply go to our manager (while we were still desperately brainstorming) and tell him we wouldn't finish on time. And nothing awful would happen. The lead developer was fine. We were fine. The project finished late. Nobody got fired. Now as a senior developer it's hard for me see why I was so terrified. Some managers yell and bluster. (Or, they used to, back in the day.) Some managers shame and insinuate. Some managers obliquely threaten. But they all, no matter how they express their authority, have to face up to the unpredictability of project planning. The question is not whether your manager is nice or mean. The question is, do they really think they're better off firing you and replacing you? Whether because they think they can get an upgrade or simply to pin the blame on you for something. After you've been around the block once or twice, you understand that a manager who is chill to your face is ultimately subject to the same motivations as a manager who uses stress as a weapon. I wonder what it's like for airline mechanics? They are represented by unions. I would think that safety infractions would be the real potential career killers, but that's just me guessing.
- 4y ago

sidewndr46 4y ago

Management cannot be at fault because they are not responsible for safely flying the aircraft or safely maintaining the aircraft. That is the job of the employees, not management. Now if management had for example knowingly procured inferior replacement parts or hired non-certified personnel to work on the aircraft, they would be at fault.

deleted 4y ago
[deleted]