10 ms·

Coincidentally-identical waypoint names foxed UK air traffic control system

34 points by simonjgreen 3y ago

edwinjm 3y ago
"which were 4,000nm apart" This is written by Americans, right?
jiofj 3y ago
[flagged]
- blitzar 3y ago
  Brexiteers already beat you too it - it was the French trying to thwart our glorious brexit. https://www.dailymail.co.uk/news/article-12454779/Did-blunder-French-airline-spark-air-traffic-meltdown-Officials-probe-single-badly-filed-travel-plan-caused-UKs-entire-flight-control-collapse-worst-outage-decade-1-000-flights-cancelled-chaos-set-DAYS.html https://www.dailymail.co.uk/news/article-12454779/Did-blunde...
  - zimpenfish 3y ago
    https://archive.ph/MSbCL https://archive.ph/MSbCL
  - deleted 3y ago
    [deleted]
- throwaway154 3y ago
  No. It was immigrants in boats that did it. Brexit done properly would have ensured this would have never happened.
Closi 3y ago
All the articles I read on this state that the issue was that a flight plan had ambiguous data. The real issue is that a single erroneous flight plan should not stop the full system.
- beardyw 3y ago
  Yes,but we've all been there. > two separate waypoints which – although geographically distant – had identical designators.
  - adhvaryu 3y ago
    Imagine the dread if they find out who wrote the code. Also to be honest, shouldn't this have been better unit tested? Considering aviation is a ~ £60 billion industry for the UK.
    - monooso 3y ago
      In their defence, they did manage to process 15 million flight paths before encountering this problem. Some bugs are just really difficult to predict.
    - krisoft 3y ago
      > Also to be honest, shouldn't this have been better unit tested? We don't know how well it was tested. (unit or otherwise) What we know that it has processed 15 millions of flight plans previously without revealing this flaw. So they must have got some things right.
    - throwaway154 3y ago
      Not a unit test problem. Relying on it's unit tested strong systems does not make. There's a lot more to testing.
    - petesergeant 3y ago
      > shouldn't this have been better unit tested Entirely possible there was a unit test that confirmed the system would error out in this particular condition. This sounds more like a requirements issue.
- throwbadubadu 3y ago
  Depends.. yes, why was the flight plan not rejected individually or before (and where is at least the proper logging which allows to find the root cause and restart quicker)? But the system is likely more complex and distributed, and I can also see that defensively programmed safety critical systems can and should do that at a certain point where certain input or output is just unexpected and wrong. To recover in some way or silently go on with eventually unexpected (even if not undefined) behaviour then is no option.
amacalac 3y ago
This reads like a 0-day exploit bringing down an entire system. I’m usually not the paranoid type :D
- blueflow 3y ago
  Its a "bug": https://en.wikipedia.org/wiki/Bug_(engineering) https://en.wikipedia.org/wiki/Bug_(engineering)
- throwitaway156 3y ago
  Rule #1 about 0-days: If you learned about it and nobody is dead or arrested, its not a 0-day.
matthewsinclair 3y ago
It turns out, naming things is hard [0]. [0]: https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html
- atcalan 3y ago
  [dead]
wkat4242 3y ago
So what was the waypoint I wonder? And who filed it? Not that they did anything wrong, I'm just wondering why it's so scarce on details.
- SideburnsOfDoom 3y ago
  I heard waypoint ALVIN is a candidate https://mastodon.social/@Edent/111017144236825037 https://mastodon.social/@Edent/111017144236825037 https://opennav.com/waypoint/US/ALVIN https://opennav.com/waypoint/US/ALVIN https://opennav.com/waypoint/UK/ALVIN https://opennav.com/waypoint/UK/ALVIN But this suggests not (as one of those ALVINs is inside the UK): > Although there has been work by ICAO and other bodies to eradicate non-unique waypoint names there are duplicates around the world. In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced. In this specific event, both of the waypoints were located outside of the UK, one towards the beginning of the route and one towards the end; approximately 4000 nautical miles apart https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
  - bloak 3y ago
    > such identical designators should be geographically widely spaced Yes, though I wouldn't normally regard "4,000nm" (mentioned in the article) as widely spaced in geographical terms. Unless they meant nautical miles (which should be abbreviated as M, NM, Nm or nmi according to en.wiktionary.org) rather than nanometres. OK, perhaps that comment wasn't really worth making ... On the other hand, perhaps next year we'll read about some other system (aviation or silicon design) collapsing because the software mixed up nanometres and nautical miles.
    - SideburnsOfDoom 3y ago
      I think in "approximately 4000 nautical miles apart", as I quoted above, the "nautical miles" means ... nautical miles? IDK what else to say to make it clearer. Bottom of page 8 of this article that i was referring to here https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
roydivision 3y ago
I'm guessing a software test plan has been updated recently with a new case.
robin_reala 3y ago
You’d hope that the people building these systems have now got a staging server running with AFL or another fuzzer pointed at it.
zarzavat 3y ago
> Given that the system could not reconcile the error, the fail-safe software logic intervened to prevent the incorrect data being passed to air traffic controllers, and the FPRSA-R primary system – as designed – suspended its functioning and handed its tasks to a back-up system. But the back-up system applied the same logic to the flightplan, with the same result, and similarly suspended itself. This seems to be the root cause. My reading of this is that the system initially validated the flight plan, but while processing the plan it hit something it didn’t understand. It should have treated this as a late validation error, rejected the plan and continued its work, but instead it treated it as an internal error and crashed. I guess it probably hit some kind of assertion.
- panki27 3y ago
  It's explained in the article. Two waypoints had identical designators, which is an edge case not implemented in the software - not sure how this is a "late validation error" if the data is in fact valid. The root cause to me seems to be that whatever they are using as designators (possibly strings shudder) is not guaranteed to be unique.
  - mst 3y ago
    I believe when OP said 'treated this as a late validation error', they were acknowledging the inevitability of bugs and root cause-ing "why did a single failure to validate cause the entire system to fall over rather than reject the single flight plan in question." The words 'treated this as' are load bearing here.
Taniwha 3y ago
At least it didn't lock up following the flightplan in a loop ...... (or is that why it really shut down)
deleted 3y ago
[deleted]
atcalan 3y ago
I've literally chased this kind of bug down in our ATC simulation tool. People don't think Adaptation be like it is, but it do. Just wait till they hear of context dependent fixes with the same name. I.E. airport, airway, fix, all named 'BUD'. This stuff was written when airplanes were bicycles with handmade motors and canvas covered wings.
KuiN 3y ago
Here's the full preliminary incident report if anyone wants to read it: https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident... Obviously concerning that a single (perfectly valid) flight plan can take down both the primary and backup. Reject the flight plan that the system can't understand, you've got 4 hours for someone on front-line support to be able to work out the correct path and enter it manually? Meanwhile it'd be good if the system continued to operate. Futher concerns about first and second line support being unable to find in the logs the cause or even the flight plan being processed when the systems failed. Had to bring in the 3rd party developers to look at "lower-level" logs to find out what happened. If your monitoring/logging isn't good enough that the first responder can't work out at least what the system was doing when it failed, that's a significant problem.
- trebligdivad 3y ago
  Most of the system did continue to operate - but it couldn't accept new flightplans automatically; the flight plans were given 4 hours in advance, so they only put in the restrictions after a couple of hours. Still I agree it would have been better if it had continued - but yes the most important bit is how long it took to find the bad plan.
- gnfargbl 3y ago
  Yep. From the report: At 0832 both systems failed and the controllers started to empty the four-hour buffer. At 1100, systems still weren't back and so to avoid the hard cutover that looked to be coming at at 1230, they began the switch to manual mode. It took them until 1336 to restore the automatic systems, and until 1803 to fully switch out of manual mode. Given those operational constraints, it sounds like the support teams basically have two hours to resolve a critical system failure.
doctor_eval 3y ago
“Myths engineers believe about waypoint names…”

34 points by simonjgreen 3y ago

edwinjm 3y ago

"which were 4,000nm apart" This is written by Americans, right?

jiofj 3y ago

[flagged]

blitzar 3y ago
Brexiteers already beat you too it - it was the French trying to thwart our glorious brexit. https://www.dailymail.co.uk/news/article-12454779/Did-blunder-French-airline-spark-air-traffic-meltdown-Officials-probe-single-badly-filed-travel-plan-caused-UKs-entire-flight-control-collapse-worst-outage-decade-1-000-flights-cancelled-chaos-set-DAYS.html https://www.dailymail.co.uk/news/article-12454779/Did-blunde...
- zimpenfish 3y ago
  https://archive.ph/MSbCL https://archive.ph/MSbCL
- deleted 3y ago
  [deleted]
throwaway154 3y ago
No. It was immigrants in boats that did it. Brexit done properly would have ensured this would have never happened.

Closi 3y ago

All the articles I read on this state that the issue was that a flight plan had ambiguous data. The real issue is that a single erroneous flight plan should not stop the full system.

beardyw 3y ago
Yes,but we've all been there. > two separate waypoints which – although geographically distant – had identical designators.
- adhvaryu 3y ago
  Imagine the dread if they find out who wrote the code. Also to be honest, shouldn't this have been better unit tested? Considering aviation is a ~ £60 billion industry for the UK.
  - monooso 3y ago
    In their defence, they did manage to process 15 million flight paths before encountering this problem. Some bugs are just really difficult to predict.
  - krisoft 3y ago
    > Also to be honest, shouldn't this have been better unit tested? We don't know how well it was tested. (unit or otherwise) What we know that it has processed 15 millions of flight plans previously without revealing this flaw. So they must have got some things right.
  - throwaway154 3y ago
    Not a unit test problem. Relying on it's unit tested strong systems does not make. There's a lot more to testing.
  - petesergeant 3y ago
    > shouldn't this have been better unit tested Entirely possible there was a unit test that confirmed the system would error out in this particular condition. This sounds more like a requirements issue.
throwbadubadu 3y ago
Depends.. yes, why was the flight plan not rejected individually or before (and where is at least the proper logging which allows to find the root cause and restart quicker)? But the system is likely more complex and distributed, and I can also see that defensively programmed safety critical systems can and should do that at a certain point where certain input or output is just unexpected and wrong. To recover in some way or silently go on with eventually unexpected (even if not undefined) behaviour then is no option.

amacalac 3y ago

This reads like a 0-day exploit bringing down an entire system. I’m usually not the paranoid type :D

blueflow 3y ago
Its a "bug": https://en.wikipedia.org/wiki/Bug_(engineering) https://en.wikipedia.org/wiki/Bug_(engineering)
throwitaway156 3y ago
Rule #1 about 0-days: If you learned about it and nobody is dead or arrested, its not a 0-day.

matthewsinclair 3y ago

It turns out, naming things is hard [0]. [0]: https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html

atcalan 3y ago
[dead]

wkat4242 3y ago

So what was the waypoint I wonder? And who filed it? Not that they did anything wrong, I'm just wondering why it's so scarce on details.

SideburnsOfDoom 3y ago
I heard waypoint ALVIN is a candidate https://mastodon.social/@Edent/111017144236825037 https://mastodon.social/@Edent/111017144236825037 https://opennav.com/waypoint/US/ALVIN https://opennav.com/waypoint/US/ALVIN https://opennav.com/waypoint/UK/ALVIN https://opennav.com/waypoint/UK/ALVIN But this suggests not (as one of those ALVINs is inside the UK): > Although there has been work by ICAO and other bodies to eradicate non-unique waypoint names there are duplicates around the world. In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced. In this specific event, both of the waypoints were located outside of the UK, one towards the beginning of the route and one towards the end; approximately 4000 nautical miles apart https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...
- bloak 3y ago
  > such identical designators should be geographically widely spaced Yes, though I wouldn't normally regard "4,000nm" (mentioned in the article) as widely spaced in geographical terms. Unless they meant nautical miles (which should be abbreviated as M, NM, Nm or nmi according to en.wiktionary.org) rather than nanometres. OK, perhaps that comment wasn't really worth making ... On the other hand, perhaps next year we'll read about some other system (aviation or silicon design) collapsing because the software mixed up nanometres and nautical miles.
  - SideburnsOfDoom 3y ago
    I think in "approximately 4000 nautical miles apart", as I quoted above, the "nautical miles" means ... nautical miles? IDK what else to say to make it clearer. Bottom of page 8 of this article that i was referring to here https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident...

roydivision 3y ago

I'm guessing a software test plan has been updated recently with a new case.

robin_reala 3y ago

You’d hope that the people building these systems have now got a staging server running with AFL or another fuzzer pointed at it.

zarzavat 3y ago

> Given that the system could not reconcile the error, the fail-safe software logic intervened to prevent the incorrect data being passed to air traffic controllers, and the FPRSA-R primary system – as designed – suspended its functioning and handed its tasks to a back-up system. But the back-up system applied the same logic to the flightplan, with the same result, and similarly suspended itself. This seems to be the root cause. My reading of this is that the system initially validated the flight plan, but while processing the plan it hit something it didn’t understand. It should have treated this as a late validation error, rejected the plan and continued its work, but instead it treated it as an internal error and crashed. I guess it probably hit some kind of assertion.

panki27 3y ago
It's explained in the article. Two waypoints had identical designators, which is an edge case not implemented in the software - not sure how this is a "late validation error" if the data is in fact valid. The root cause to me seems to be that whatever they are using as designators (possibly strings shudder) is not guaranteed to be unique.
- mst 3y ago
  I believe when OP said 'treated this as a late validation error', they were acknowledging the inevitability of bugs and root cause-ing "why did a single failure to validate cause the entire system to fall over rather than reject the single flight plan in question." The words 'treated this as' are load bearing here.

Taniwha 3y ago

At least it didn't lock up following the flightplan in a loop ...... (or is that why it really shut down)

deleted 3y ago

[deleted]

atcalan 3y ago

I've literally chased this kind of bug down in our ATC simulation tool. People don't think Adaptation be like it is, but it do. Just wait till they hear of context dependent fixes with the same name. I.E. airport, airway, fix, all named 'BUD'. This stuff was written when airplanes were bicycles with handmade motors and canvas covered wings.

KuiN 3y ago

Here's the full preliminary incident report if anyone wants to read it: https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident%20Investigation%20Preliminary%20Report.pdf https://publicapps.caa.co.uk/docs/33/NERL%20Major%20Incident... Obviously concerning that a single (perfectly valid) flight plan can take down both the primary and backup. Reject the flight plan that the system can't understand, you've got 4 hours for someone on front-line support to be able to work out the correct path and enter it manually? Meanwhile it'd be good if the system continued to operate. Futher concerns about first and second line support being unable to find in the logs the cause or even the flight plan being processed when the systems failed. Had to bring in the 3rd party developers to look at "lower-level" logs to find out what happened. If your monitoring/logging isn't good enough that the first responder can't work out at least what the system was doing when it failed, that's a significant problem.

trebligdivad 3y ago
Most of the system did continue to operate - but it couldn't accept new flightplans automatically; the flight plans were given 4 hours in advance, so they only put in the restrictions after a couple of hours. Still I agree it would have been better if it had continued - but yes the most important bit is how long it took to find the bad plan.
gnfargbl 3y ago
Yep. From the report: At 0832 both systems failed and the controllers started to empty the four-hour buffer. At 1100, systems still weren't back and so to avoid the hard cutover that looked to be coming at at 1230, they began the switch to manual mode. It took them until 1336 to restore the automatic systems, and until 1803 to fully switch out of manual mode. Given those operational constraints, it sounds like the support teams basically have two hours to resolve a critical system failure.

doctor_eval 3y ago

“Myths engineers believe about waypoint names…”