6 ms·

All the articles I read on this state that the issue was that a flight plan had ambiguous data. The real issue is that a single erroneous flight plan should no

by Closi 3y ago

All the articles I read on this state that the issue was that a flight plan had ambiguous data. The real issue is that a single erroneous flight plan should not stop the full system.

beardyw 3y ago
Yes,but we've all been there. > two separate waypoints which – although geographically distant – had identical designators.
- adhvaryu 3y ago
  Imagine the dread if they find out who wrote the code. Also to be honest, shouldn't this have been better unit tested? Considering aviation is a ~ £60 billion industry for the UK.
  - monooso 3y ago
    In their defence, they did manage to process 15 million flight paths before encountering this problem. Some bugs are just really difficult to predict.
  - krisoft 3y ago
    > Also to be honest, shouldn't this have been better unit tested? We don't know how well it was tested. (unit or otherwise) What we know that it has processed 15 millions of flight plans previously without revealing this flaw. So they must have got some things right.
  - throwaway154 3y ago
    Not a unit test problem. Relying on it's unit tested strong systems does not make. There's a lot more to testing.
  - petesergeant 3y ago
    > shouldn't this have been better unit tested Entirely possible there was a unit test that confirmed the system would error out in this particular condition. This sounds more like a requirements issue.
throwbadubadu 3y ago
Depends.. yes, why was the flight plan not rejected individually or before (and where is at least the proper logging which allows to find the root cause and restart quicker)? But the system is likely more complex and distributed, and I can also see that defensively programmed safety critical systems can and should do that at a certain point where certain input or output is just unexpected and wrong. To recover in some way or silently go on with eventually unexpected (even if not undefined) behaviour then is no option.
- blitzar 3y ago
  A defensively programmed safety critical system should reject (loudly) the offending flight plan (and its permission to fly in the airspace) rather than shut itself off and leave all the planes flying (currently and in the foreseable future) to figure things out for themselves.
  - throwbadubadu 3y ago
    Said that and yes is always an easy and apparent statement to make as an armchair outsider.. read the full report? ;) (Hint again: Something certainly could have been done much better there in this system, but the limited component itself that all caused this maybe even just had no other option at that point. Been there once in those many component systems talking with each other across with too many interfaces, people, etc involved, part of them safety critical and also subject to endless requirements and certifications? Can recommend..)
  - atcalan 3y ago
    Our flight plan validator still gives me nightmares.
    - blitzar 3y ago
      All the more reason that a single flight plan that fails path validation doesnt (by design) shut down the entire air space of a nation for 24-48 hours.
zimpenfish 3y ago
> The real issue is that a single erroneous flight plan should not stop the full system. But the flight plan wasn't erroneous - it was perfectly valid. It just happened to have a weird coincidence of two identically tagged airports spanning the UK. If someone had put an extra waypoint in between the two, nothing would have broken (AIUI).
- krisoft 3y ago
  > But the flight plan wasn't erroneous - it was perfectly valid. Well. Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. It should flag that flight plan for problem resolution and keep on working with the rest of the flight plans. Would result in a slightly inconvenienced flight dispatcher, or worst case an inconvenienced flight, instead of a nationwide shutdown.
  - ChrisRR 3y ago
    We know. That's clear to everyone here. But clearly in their design of the system they hadn't considered it. I'm sure no doubt they will now fix the issue, but you can only aim to handle faults that you predict during development, plan for unexpected faults, and hope that something super unexpected doesn't cause cascading faults
    - zimpenfish 3y ago
      > But clearly in their design of the system they hadn't considered it. They might well have considered it the possibility of duplicate tags in the same flight plan but since identical tags are supposed to geographically distinct, they probably didn't further consider a flight plan where the entry and exit points from the UK (which is a relatively small airspace, I think) are the same tag (since it sounds from the report they did that this was an extremely weird flight plan.)
    - krisoft 3y ago
      > That's clear to everyone here. Are you sure? Have you also consulted with zimpenfish? From their comment it seems they disagree. > hope that something super unexpected doesn't cause cascading faults But that is the thing. Processing failing isn’t and shouldn’t be “unexpected”. Here the individual processing items are flight plans, but the concept is more general. They could be “requests” to a web server, they could be “tasks” in an async worker node, they could be “transactions” in a database, if you do any compute with a work item it can fail. You need to put a big “try-catch” around it and add a field in your database or queue or remote procedure call system to mark them as failed. One work item failing shouldn’t be “super unexpected”. Even if you can’t name the exact conditions under witch it might fail.
gnfargbl 3y ago
I don't know anything about ATC systems. Why should it be possible to continue when a single flight plan is wrong? Naïvely it sounds like having even a single plane in an unknown position would make safe automated control impossible. Is that wrong?
- jleahy 3y ago
  Well they can reject the bad flight plan and the plane will not be allowed to follow it.
  - nirse 3y ago
    Do you know what the consequences of that would be?
    - jleahy 3y ago
      Yes, it happens to people every day for all kinds of boring reasons. Normally because they made a mistake in their flight plan. Remember anyone flying IFR needs a flight plan, lots of private pilots etc. Plenty of people make mistakes.
- krisoft 3y ago
  > Why should it be possible to continue when a single flight plan is wrong? To make the system resilient. Note: When the error happened the flight was not in the air yet. The system in question received the flight plan 4 hours before departure time. If the system would have flagged the flight plan as bad they could have called them and told them that they can't fly. If not that they could have refused them when they were entering the airspace. Can happen any time for any reason. > Naïvely it sounds like having even a single plane in an unknown position would make safe automated control impossible. Flight plans are not for knowing the position of the plane. > Is that wrong? Slightly.
  - trollied 3y ago
    > The system in question received the flight plan 4 hours before departure time. This is not true. The plan is transferred 4 hours before it is due to enter UK airspace. Big difference. Flights can already be in the air when the plan is transferred.
ChrisRR 3y ago
From reading these comments I can see a huge number of people have never had to conduct risk analysis in developing a product. Sure, the unexpected identical naming should not cause an issue, but when you're developing software like this you conduct multiple risk analyses and FMEAs to attempt to design a system that can handle faults that you can predict You can have failovers, safe states, recovery procedures up to the eyeballs, but if a situation occurs that you had not predicted then you're in unknown territory. Your system may handle the fault and fail safely, or it may cause an unexpected chain of events. TLDR: You can plan for faults all you want, but you can never guarantee it's 100% bombproof

by Closi 3y ago

All the articles I read on this state that the issue was that a flight plan had ambiguous data. The real issue is that a single erroneous flight plan should not stop the full system.

beardyw 3y ago

Yes,but we've all been there. > two separate waypoints which – although geographically distant – had identical designators.

adhvaryu 3y ago
Imagine the dread if they find out who wrote the code. Also to be honest, shouldn't this have been better unit tested? Considering aviation is a ~ £60 billion industry for the UK.
- monooso 3y ago
  In their defence, they did manage to process 15 million flight paths before encountering this problem. Some bugs are just really difficult to predict.
- krisoft 3y ago
  > Also to be honest, shouldn't this have been better unit tested? We don't know how well it was tested. (unit or otherwise) What we know that it has processed 15 millions of flight plans previously without revealing this flaw. So they must have got some things right.
- throwaway154 3y ago
  Not a unit test problem. Relying on it's unit tested strong systems does not make. There's a lot more to testing.
- petesergeant 3y ago
  > shouldn't this have been better unit tested Entirely possible there was a unit test that confirmed the system would error out in this particular condition. This sounds more like a requirements issue.

throwbadubadu 3y ago

Depends.. yes, why was the flight plan not rejected individually or before (and where is at least the proper logging which allows to find the root cause and restart quicker)? But the system is likely more complex and distributed, and I can also see that defensively programmed safety critical systems can and should do that at a certain point where certain input or output is just unexpected and wrong. To recover in some way or silently go on with eventually unexpected (even if not undefined) behaviour then is no option.

blitzar 3y ago
A defensively programmed safety critical system should reject (loudly) the offending flight plan (and its permission to fly in the airspace) rather than shut itself off and leave all the planes flying (currently and in the foreseable future) to figure things out for themselves.
- throwbadubadu 3y ago
  Said that and yes is always an easy and apparent statement to make as an armchair outsider.. read the full report? ;) (Hint again: Something certainly could have been done much better there in this system, but the limited component itself that all caused this maybe even just had no other option at that point. Been there once in those many component systems talking with each other across with too many interfaces, people, etc involved, part of them safety critical and also subject to endless requirements and certifications? Can recommend..)
- atcalan 3y ago
  Our flight plan validator still gives me nightmares.
  - blitzar 3y ago
    All the more reason that a single flight plan that fails path validation doesnt (by design) shut down the entire air space of a nation for 24-48 hours.

zimpenfish 3y ago

> The real issue is that a single erroneous flight plan should not stop the full system. But the flight plan wasn't erroneous - it was perfectly valid. It just happened to have a weird coincidence of two identically tagged airports spanning the UK. If someone had put an extra waypoint in between the two, nothing would have broken (AIUI).

krisoft 3y ago
> But the flight plan wasn't erroneous - it was perfectly valid. Well. Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. It should flag that flight plan for problem resolution and keep on working with the rest of the flight plans. Would result in a slightly inconvenienced flight dispatcher, or worst case an inconvenienced flight, instead of a nationwide shutdown.
- ChrisRR 3y ago
  We know. That's clear to everyone here. But clearly in their design of the system they hadn't considered it. I'm sure no doubt they will now fix the issue, but you can only aim to handle faults that you predict during development, plan for unexpected faults, and hope that something super unexpected doesn't cause cascading faults
  - zimpenfish 3y ago
    > But clearly in their design of the system they hadn't considered it. They might well have considered it the possibility of duplicate tags in the same flight plan but since identical tags are supposed to geographically distinct, they probably didn't further consider a flight plan where the entry and exit points from the UK (which is a relatively small airspace, I think) are the same tag (since it sounds from the report they did that this was an extremely weird flight plan.)
  - krisoft 3y ago
    > That's clear to everyone here. Are you sure? Have you also consulted with zimpenfish? From their comment it seems they disagree. > hope that something super unexpected doesn't cause cascading faults But that is the thing. Processing failing isn’t and shouldn’t be “unexpected”. Here the individual processing items are flight plans, but the concept is more general. They could be “requests” to a web server, they could be “tasks” in an async worker node, they could be “transactions” in a database, if you do any compute with a work item it can fail. You need to put a big “try-catch” around it and add a field in your database or queue or remote procedure call system to mark them as failed. One work item failing shouldn’t be “super unexpected”. Even if you can’t name the exact conditions under witch it might fail.

gnfargbl 3y ago

I don't know anything about ATC systems. Why should it be possible to continue when a single flight plan is wrong? Naïvely it sounds like having even a single plane in an unknown position would make safe automated control impossible. Is that wrong?

jleahy 3y ago
Well they can reject the bad flight plan and the plane will not be allowed to follow it.
- nirse 3y ago
  Do you know what the consequences of that would be?
  - jleahy 3y ago
    Yes, it happens to people every day for all kinds of boring reasons. Normally because they made a mistake in their flight plan. Remember anyone flying IFR needs a flight plan, lots of private pilots etc. Plenty of people make mistakes.
krisoft 3y ago
> Why should it be possible to continue when a single flight plan is wrong? To make the system resilient. Note: When the error happened the flight was not in the air yet. The system in question received the flight plan 4 hours before departure time. If the system would have flagged the flight plan as bad they could have called them and told them that they can't fly. If not that they could have refused them when they were entering the airspace. Can happen any time for any reason. > Naïvely it sounds like having even a single plane in an unknown position would make safe automated control impossible. Flight plans are not for knowing the position of the plane. > Is that wrong? Slightly.
- trollied 3y ago
  > The system in question received the flight plan 4 hours before departure time. This is not true. The plan is transferred 4 hours before it is due to enter UK airspace. Big difference. Flights can already be in the air when the plan is transferred.

ChrisRR 3y ago

From reading these comments I can see a huge number of people have never had to conduct risk analysis in developing a product. Sure, the unexpected identical naming should not cause an issue, but when you're developing software like this you conduct multiple risk analyses and FMEAs to attempt to design a system that can handle faults that you can predict You can have failovers, safe states, recovery procedures up to the eyeballs, but if a situation occurs that you had not predicted then you're in unknown territory. Your system may handle the fault and fail safely, or it may cause an unexpected chain of events. TLDR: You can plan for faults all you want, but you can never guarantee it's 100% bombproof