5 ms·

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same r

I seem to remember another problem at NATS which had the same effect. Primary fell over so they switched over to a secondary that fell over for the exact same reason. It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do. The bit that makes me feel a bit sick though is that they didn't have a method called "ValidateFlightPlan" that throws an error if for any reason it couldn't be parsed and that error could be handled in a really simple way. What programmer would look at a processor of external input and not think, "what do we do with bad input that makes it fall over?". I did something today for a simple message prompt since I can't guarantee that in all scenarios the data I need will be present/correct. Try/catch and a simple message to the user "Data could not be processed".

d1sxeyes 3y ago
Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.
- zaphar 3y ago
  I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.
  - mixdup 3y ago
    It's very hard to ensure you capture every single possible failure mode. Yes, the engineering control is important but it's not the most critical. What to do if it does fail (for any reason) is the truly critical control, because it solves for the possibility of not knowing every possible way something might fail and therefore missing some way to prevent a failure
  - jeffrallen 3y ago
    One or more of three results can come from the engineering exercise of trying to keep something from breaking in the first place: 1. You could know the solution, but it would be too heavy. 2. You could know the solution, but it would include more parts, each of which would need the same process on it, and the process might fail the same way 3. You miss something and it fails anyway, so your "what if this fails" path better be well rehearsed and executed. Real engineering is facing the tradeoffs head on, not hand waving them away.
  - d1sxeyes 3y ago
    The engineering controls don't independently make systems safe, they make things more reliable and cost-effective, and hopefully reduce the number of times the process controls kick in. The process controls do however independently make things safe. The reason for this is that there are 'unknown unknowns'—we accept that our knowledge and skills are imperfect, and there may be failures that occur which could have been eliminated with the proper engineering controls, but we, as imperfect beings and organisations, did not implement the engineering controls because we did not identify this possible failure mode. There are also known errors, where the cost of implementing engineering controls may simply outweigh the benefits when adequate process controls are in place.
  - sheepshear 3y ago
    Everyone uses slightly different terminology and groups things differently but this will give you the gist. https://en.m.wikipedia.org/wiki/Hierarchy_of_hazard_controls https://en.m.wikipedia.org/wiki/Hierarchy_of_hazard_controls
- samus 3y ago
  It was in a bad state, but in a very inane way: a flight plan in its processing queue was faulty. The system itself was mostly fine. It was just not well-written enough to distinguish an input error from an internal error, and thus didn't just skip the faulty flight plan.
  - Twirrim 3y ago
    at the risk of nitpicking: "a flight plan in its processing queue was faulty" isn't true, the flight plan was fine. It couldn't process it. I mention this only because the Daily Mail headline pissed me off with it's usual bullshit foreigner fear mongering crap.
    - samus 3y ago
      Indeed, that intention is quite transparent in this case. Anyways, I suspect that invalid input exists that would have made the system react in a similar way
- deleted 3y ago
  [deleted]
1970-01-01 3y ago
Yep. In electrical terms, you replaced the fuse to watch it blow again. There are no more fuses in your shop. Progress?
- adamauckland 3y ago
  Stick a nail in
sheepshear 3y ago
Failing over is correct because there's no way to discern that the hardware is not at fault. They should have designed a better response to the second failure to avoid the knock-on effects.
- anentropic 3y ago
  I don't think anything in this incident pointed to a hardware fault The software raised an exception because a "// TODO: this should never happen" case happened A hardware fault would look like machines not talking to each other or corrupted data file unreadable
  - sheepshear 3y ago
    Retroactive inspection revealed that it wasn't a hardware failure, but the computer didn't know that at the time, and hardware failure can look like anything, so it was correct to exercise its only option.
j_mo 3y ago
No validation, anddd this point from the article stood out to me: --- The programming style is very imperative. Furthermore, the description sounds like the procedure is working directly on the textual representation of the flight plan, rather than a data structure parsed from the text file. This would be quite worrying, but it might also just be how it is explained. --- Given that description, I'd be surprised if it wasn't just running a regex / substring matches against the text and there's no classes / objects / data structure involved. Bearing in mind this is likely decades old C code that can't be rewritten or replaced because the entirety of the UK's aviation runs on it.
- jameshh 3y ago
  > Bearing in mind this is likely decades old C code that can't be rewritten or replaced because the entirety of the UK's aviation runs on it. It's new code, from 2018 :) Quote from the report: > An FPRSA sub-system has existed in NATS for many years and in 2018 the previous FPRSA sub- system was replaced with new hardware and software manufactured by Frequentis AG, one of the leading global ATC System providers.
philjohn 3y ago
The Ariane 5 launch failure[1] was a similar issue, albeit with a more spectacular outcome. Primary suffers integer overflow, fails. Secondary is identical, which also overflows. Angle of attack increases, boosters separate. Rocket goes boom. [1] https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88

d1sxeyes 3y ago

Well, if the primary is known not to be in a good state, you might as well fail over and hope that the issue was a fried disk or a cosmic bit flip or something. The real safety feature is the 4 hour lead time before manual processing becomes necessary. One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.

zaphar 3y ago
I'm no aviation safety controls expert but it seems to me that there are two types of controls that should be in place: 1. Process controls: What do we do when this breaks for any reason. 2. Engineering controls: What can we do to keep this from breaking in the first place? Both of them seem to be somewhat essential for a truly safe system.
- mixdup 3y ago
  It's very hard to ensure you capture every single possible failure mode. Yes, the engineering control is important but it's not the most critical. What to do if it does fail (for any reason) is the truly critical control, because it solves for the possibility of not knowing every possible way something might fail and therefore missing some way to prevent a failure
- jeffrallen 3y ago
  One or more of three results can come from the engineering exercise of trying to keep something from breaking in the first place: 1. You could know the solution, but it would be too heavy. 2. You could know the solution, but it would include more parts, each of which would need the same process on it, and the process might fail the same way 3. You miss something and it fails anyway, so your "what if this fails" path better be well rehearsed and executed. Real engineering is facing the tradeoffs head on, not hand waving them away.
- d1sxeyes 3y ago
  The engineering controls don't independently make systems safe, they make things more reliable and cost-effective, and hopefully reduce the number of times the process controls kick in. The process controls do however independently make things safe. The reason for this is that there are 'unknown unknowns'—we accept that our knowledge and skills are imperfect, and there may be failures that occur which could have been eliminated with the proper engineering controls, but we, as imperfect beings and organisations, did not implement the engineering controls because we did not identify this possible failure mode. There are also known errors, where the cost of implementing engineering controls may simply outweigh the benefits when adequate process controls are in place.
- sheepshear 3y ago
  Everyone uses slightly different terminology and groups things differently but this will give you the gist. https://en.m.wikipedia.org/wiki/Hierarchy_of_hazard_controls https://en.m.wikipedia.org/wiki/Hierarchy_of_hazard_controls
samus 3y ago
It was in a bad state, but in a very inane way: a flight plan in its processing queue was faulty. The system itself was mostly fine. It was just not well-written enough to distinguish an input error from an internal error, and thus didn't just skip the faulty flight plan.
- Twirrim 3y ago
  at the risk of nitpicking: "a flight plan in its processing queue was faulty" isn't true, the flight plan was fine. It couldn't process it. I mention this only because the Daily Mail headline pissed me off with it's usual bullshit foreigner fear mongering crap.
  - samus 3y ago
    Indeed, that intention is quite transparent in this case. Anyways, I suspect that invalid input exists that would have made the system react in a similar way
deleted 3y ago
[deleted]

1970-01-01 3y ago

Yep. In electrical terms, you replaced the fuse to watch it blow again. There are no more fuses in your shop. Progress?

adamauckland 3y ago
Stick a nail in

sheepshear 3y ago

Failing over is correct because there's no way to discern that the hardware is not at fault. They should have designed a better response to the second failure to avoid the knock-on effects.

anentropic 3y ago
I don't think anything in this incident pointed to a hardware fault The software raised an exception because a "// TODO: this should never happen" case happened A hardware fault would look like machines not talking to each other or corrupted data file unreadable
- sheepshear 3y ago
  Retroactive inspection revealed that it wasn't a hardware failure, but the computer didn't know that at the time, and hardware failure can look like anything, so it was correct to exercise its only option.

j_mo 3y ago

No validation, anddd this point from the article stood out to me: --- The programming style is very imperative. Furthermore, the description sounds like the procedure is working directly on the textual representation of the flight plan, rather than a data structure parsed from the text file. This would be quite worrying, but it might also just be how it is explained. --- Given that description, I'd be surprised if it wasn't just running a regex / substring matches against the text and there's no classes / objects / data structure involved. Bearing in mind this is likely decades old C code that can't be rewritten or replaced because the entirety of the UK's aviation runs on it.

jameshh 3y ago
> Bearing in mind this is likely decades old C code that can't be rewritten or replaced because the entirety of the UK's aviation runs on it. It's new code, from 2018 :) Quote from the report: > An FPRSA sub-system has existed in NATS for many years and in 2018 the previous FPRSA sub- system was replaced with new hardware and software manufactured by Frequentis AG, one of the leading global ATC System providers.

philjohn 3y ago

The Ariane 5 launch failure[1] was a similar issue, albeit with a more spectacular outcome. Primary suffers integer overflow, fails. Secondary is identical, which also overflows. Angle of attack increases, boosters separate. Rocket goes boom. [1] https://en.wikipedia.org/wiki/Ariane_flight_V88 https://en.wikipedia.org/wiki/Ariane_flight_V88