4 ms·

> The real issue is that a single erroneous flight plan should not stop the full system. But the flight plan wasn't erroneous - it was perfectly valid. It jus

> The real issue is that a single erroneous flight plan should not stop the full system. But the flight plan wasn't erroneous - it was perfectly valid. It just happened to have a weird coincidence of two identically tagged airports spanning the UK. If someone had put an extra waypoint in between the two, nothing would have broken (AIUI).

krisoft 3y ago
> But the flight plan wasn't erroneous - it was perfectly valid. Well. Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. It should flag that flight plan for problem resolution and keep on working with the rest of the flight plans. Would result in a slightly inconvenienced flight dispatcher, or worst case an inconvenienced flight, instead of a nationwide shutdown.
- ChrisRR 3y ago
  We know. That's clear to everyone here. But clearly in their design of the system they hadn't considered it. I'm sure no doubt they will now fix the issue, but you can only aim to handle faults that you predict during development, plan for unexpected faults, and hope that something super unexpected doesn't cause cascading faults
  - zimpenfish 3y ago
    > But clearly in their design of the system they hadn't considered it. They might well have considered it the possibility of duplicate tags in the same flight plan but since identical tags are supposed to geographically distinct, they probably didn't further consider a flight plan where the entry and exit points from the UK (which is a relatively small airspace, I think) are the same tag (since it sounds from the report they did that this was an extremely weird flight plan.)
  - krisoft 3y ago
    > That's clear to everyone here. Are you sure? Have you also consulted with zimpenfish? From their comment it seems they disagree. > hope that something super unexpected doesn't cause cascading faults But that is the thing. Processing failing isn’t and shouldn’t be “unexpected”. Here the individual processing items are flight plans, but the concept is more general. They could be “requests” to a web server, they could be “tasks” in an async worker node, they could be “transactions” in a database, if you do any compute with a work item it can fail. You need to put a big “try-catch” around it and add a field in your database or queue or remote procedure call system to mark them as failed. One work item failing shouldn’t be “super unexpected”. Even if you can’t name the exact conditions under witch it might fail.
- zimpenfish 3y ago
  > Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. Sure but "flight data is safety critical information that is passed to ATCOs the system must be sure it is correct and could not do so in this case" is a reasonably valid stance in a super-critical system that must absolutely defer to safe operations in the case of error.
  - krisoft 3y ago
    So what are you saying? Are you saying that it is reasonable to bring down the whole system in a hard to debug way for one item being out of order? That is not reasonable. And it won't become reasonable by alluding to how "super-critical" or "safety-critical" the system is.
    - zimpenfish 3y ago
      It's not "one item being out of order" though - it's "the system cannot verify this flight plan as being correct". Stopping in the face of being unable to verify that safety critical information is correct is a perfectly good choice, especially when the good consequences are fairly light like "some people are a bit late with their flights" and the bad consequences may be 10s or 100s of people dying.
      - krisoft 3y ago
        > consequences are fairly light like "some people are a bit late with their flights" and the bad consequences may be 10s or 100s of people dying I think this might be the root of the misunderstanding between us. The consequences of an airplane not having a flight plan is not “10s or 100s of people dying”. The consequences of rejecting a flight plan are: - if caught on the ground:the one flight is delayed. They don’t take off until they manage to file a fligtplan the system accepts. - if caught at the airspace boundary when they arrive without a flight plan in the system there are two options: if ATC want to be hard-ass they will be refused entry and have to land at an alternate airport. If ATC is acomodating they will be asked at every new controller “<flightnumber> state your intentions”. Either way the system is fine and dandy. This is not what keeps airplanes from coliding with each other or the ground. Obviously the first option, catching the problem on the ground, is better because it is not increasing the workload of the ATC personel. Here is the thing, you are talking about “the system cannot verify this flight plan as being correct". And you are right! It cannot. So it shouldn’t try. It should give up. On that flight plan. I totally agree with that. But there were lots of other flight plans it could and should have verified. Have you done web development? What would you think of a server where a single weird edge case request would stop the whole service for hours? Would you say it is well engineered?
SideburnsOfDoom 3y ago
> If someone had put an extra waypoint in between the two, nothing would have broken (AIUI). I don't see it said anywhere that the two same-named waypoints were consecutive, with no other points between them. Just that they were "approximately 4000 nautical miles apart". Or that being consecutive duplicates rather than just duplicates was the cause. Since they don't name the waypoints, it is very vague.
- zimpenfish 3y ago
  As best I understand page 9 of [0], it sounds like the search for UK entry and exit points led to the A and B being found with no specified entry/exit/waypoints between them, and therefore "the software could not extract a valid UK portion of flight plan between these two points". [0] https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
  - SideburnsOfDoom 3y ago
    > the A and B being found with no specified entry/exit/waypoints between them No waypoints that enter UK airspace between them, since "In this specific event, both of the waypoints were located outside of the UK". (1) So there could be any number from zero up, of international waters waypoints between them? 1) https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
    - zimpenfish 3y ago
      > So there could be any number from zero up, of international waters waypoints between them? For me, it read like "there was no route between Aa and Ab because there were no waypoints available between them". But yeah, without a more detailed breakdown, information, and/or a better understanding of the precise format of the files and procedures, it is going to be a bit of assumption and guesswork.
atcalan 3y ago
To a human. To a computer, it was degenerate.

krisoft 3y ago

> But the flight plan wasn't erroneous - it was perfectly valid. Well. Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. It should flag that flight plan for problem resolution and keep on working with the rest of the flight plans. Would result in a slightly inconvenienced flight dispatcher, or worst case an inconvenienced flight, instead of a nationwide shutdown.

ChrisRR 3y ago
We know. That's clear to everyone here. But clearly in their design of the system they hadn't considered it. I'm sure no doubt they will now fix the issue, but you can only aim to handle faults that you predict during development, plan for unexpected faults, and hope that something super unexpected doesn't cause cascading faults
- zimpenfish 3y ago
  > But clearly in their design of the system they hadn't considered it. They might well have considered it the possibility of duplicate tags in the same flight plan but since identical tags are supposed to geographically distinct, they probably didn't further consider a flight plan where the entry and exit points from the UK (which is a relatively small airspace, I think) are the same tag (since it sounds from the report they did that this was an extremely weird flight plan.)
- krisoft 3y ago
  > That's clear to everyone here. Are you sure? Have you also consulted with zimpenfish? From their comment it seems they disagree. > hope that something super unexpected doesn't cause cascading faults But that is the thing. Processing failing isn’t and shouldn’t be “unexpected”. Here the individual processing items are flight plans, but the concept is more general. They could be “requests” to a web server, they could be “tasks” in an async worker node, they could be “transactions” in a database, if you do any compute with a work item it can fail. You need to put a big “try-catch” around it and add a field in your database or queue or remote procedure call system to mark them as failed. One work item failing shouldn’t be “super unexpected”. Even if you can’t name the exact conditions under witch it might fail.
zimpenfish 3y ago
> Then the real issue is that a single flight plan, erroneous or not, should not stop the full system. Sure but "flight data is safety critical information that is passed to ATCOs the system must be sure it is correct and could not do so in this case" is a reasonably valid stance in a super-critical system that must absolutely defer to safe operations in the case of error.
- krisoft 3y ago
  So what are you saying? Are you saying that it is reasonable to bring down the whole system in a hard to debug way for one item being out of order? That is not reasonable. And it won't become reasonable by alluding to how "super-critical" or "safety-critical" the system is.
  - zimpenfish 3y ago
    It's not "one item being out of order" though - it's "the system cannot verify this flight plan as being correct". Stopping in the face of being unable to verify that safety critical information is correct is a perfectly good choice, especially when the good consequences are fairly light like "some people are a bit late with their flights" and the bad consequences may be 10s or 100s of people dying.
    - krisoft 3y ago
      > consequences are fairly light like "some people are a bit late with their flights" and the bad consequences may be 10s or 100s of people dying I think this might be the root of the misunderstanding between us. The consequences of an airplane not having a flight plan is not “10s or 100s of people dying”. The consequences of rejecting a flight plan are: - if caught on the ground:the one flight is delayed. They don’t take off until they manage to file a fligtplan the system accepts. - if caught at the airspace boundary when they arrive without a flight plan in the system there are two options: if ATC want to be hard-ass they will be refused entry and have to land at an alternate airport. If ATC is acomodating they will be asked at every new controller “<flightnumber> state your intentions”. Either way the system is fine and dandy. This is not what keeps airplanes from coliding with each other or the ground. Obviously the first option, catching the problem on the ground, is better because it is not increasing the workload of the ATC personel. Here is the thing, you are talking about “the system cannot verify this flight plan as being correct". And you are right! It cannot. So it shouldn’t try. It should give up. On that flight plan. I totally agree with that. But there were lots of other flight plans it could and should have verified. Have you done web development? What would you think of a server where a single weird edge case request would stop the whole service for hours? Would you say it is well engineered?

SideburnsOfDoom 3y ago

> If someone had put an extra waypoint in between the two, nothing would have broken (AIUI). I don't see it said anywhere that the two same-named waypoints were consecutive, with no other points between them. Just that they were "approximately 4000 nautical miles apart". Or that being consecutive duplicates rather than just duplicates was the cause. Since they don't name the waypoints, it is very vague.

zimpenfish 3y ago
As best I understand page 9 of [0], it sounds like the search for UK entry and exit points led to the A and B being found with no specified entry/exit/waypoints between them, and therefore "the software could not extract a valid UK portion of flight plan between these two points". [0] https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
- SideburnsOfDoom 3y ago
  > the A and B being found with no specified entry/exit/waypoints between them No waypoints that enter UK airspace between them, since "In this specific event, both of the waypoints were located outside of the UK". (1) So there could be any number from zero up, of international waters waypoints between them? 1) https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
  - zimpenfish 3y ago
    > So there could be any number from zero up, of international waters waypoints between them? For me, it read like "there was no route between Aa and Ab because there were no waypoints available between them". But yeah, without a more detailed breakdown, information, and/or a better understanding of the precise format of the files and procedures, it is going to be a bit of assumption and guesswork.

atcalan 3y ago

To a human. To a computer, it was degenerate.