3 ms·

NATS publishes preliminary report on technical incident of 8 September

1 points by asplake 9d ago

zrail 9d ago
Note: NATS the UK airspace manager, not NATS the pubsub system.
- verst 8d ago
  And here I was surprised they would use NATS in their implementation to assign squawk codes.
fotta 9d ago
link to the report (pdf): https://www.nats.aero/wp-content/uploads/2026/09/NATS-Preliminary-Investigation-Report-into-NAS-Incident-on-08-Sept-2026-Issued-16-Sept-2026.pdf https://www.nats.aero/wp-content/uploads/2026/09/NATS-Prelim...
- fotta 8d ago
  > The incident was triggered by a valid manual request for a squawk code. This manual request was made correctly and there was nothing abnormal or invalid about the associated flight plan. > While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted. > The reason this scenario has not occurred before is because: > 1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond. > 2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value. > 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally. > Post-incident investigation has identified that when processing of the squawk allocation request resumed, the data associated with it had been corrupted and affected some subsequent flight data updates.
  - dtf 8d ago
    > 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally. Well, that's comforting to know. BBC: "Flight chaos caused by software defect in space of a millisecond, report says" Sky: "'Millisecond' software error caused air traffic outage that grounded thousands of flights" The Guardian: "Flight chaos for hundreds of thousands was caused in ‘millisecond’ by software error" Sounds like pure bad luck.
    - philipwhiuk 8d ago
      Or insufficient testing?
      - nradov 8d ago
        Testing isn't an effective way to catch most race conditions. Code reviews, static analysis, and rigorous enforcement of concurrent coding standards is usually a better approach.
        jiehong 8d ago
        Perhaps something like what TigerBeetle does: deterministic simulation testing.
      - dtf 8d ago
        Or maybe even just reviewing logic that is subject to pre-emption. Maybe I'm being too harsh.. on the plus side the system has at least failed hard every time there's been a fault. Nobody has died. But it's been 3 times now in the past couple of years, and two of those times resulted in over 2000 flights cancelled and days of backlog, and misery for hundreds of thousands. It's really not acceptable.
        chrisjj 8d ago
        > the system has at least failed hard every time there's been a fault You don't know that.
macguillicuddy 8d ago
Interesting! Looks like a race condition (with a 1ms window) during squawk allocation (the process that gives each aircraft a 4 digit identifier for ATC purposes) caused data corruption. They present that window as being small but 1ms is a decent chunk of time in our work lol
- crote 8d ago
  More specifically: it sounds like something which was supposed to be an atomic operation was split into two by the preempting task, with corruption happening because that task also interacted with the same piece of (now-temporarily-invalid) data.
- JdeBP 8d ago
  It's a decent chunk in most people's. But we're talking about a system that is also older than most people. It was first put into operation in 1967 in the U.S.A., and brought over to the U.K. in 1974. It's formally called 'NAS En Route Stage A', and is written in a 1950s language named JOVIAL. It originally ran on an IBM 9020. It runs on IBM 9020 compatible systems today. For some reason, a couple of years ago someone added a lengthy unverifiable description of it to Wikipedia's IBM 9020 article, even though that very description said that NAS hadn't run on a 9020 for 34 years at the time of writing, but had been running on a 4381.
lukaslueg 8d ago
Would be interesting if the report actually got into the details. The curiously specific "approx. 1ms window" sounds like an off-the-shelf memory race condition in combination with a preempting thread, for which 1ms is an absolute eternity and Guaranteed To Happen By Tuesday This Week™. But the "database corruption" sounds like database IDs being generated from timestamps with 1ms granularity.
- LtdJorge 8d ago
  They might have to switch to ULID or similar.
deleted 8d ago
[deleted]
ExoticPearTree 8d ago
A few years back, NATS went down because a flight plan waypoint confused the system and it crashed. They had to manually find the problematic flight plan, remove it and start the whole system back up. Considering the last issue they encountered, it looks like in more than one place, there is no error catching and graceful resolution for those errors. I would assume a system of such importance to handle issues without hiccups and alert the operators of what did not work. Like “this input caused this problem”, not just crash.
- pixelesque 8d ago
  Back in 2023 when the previous issue happened, it didn't actually "crash", it detected what it perceived as an inconsistency (which was due to invalid waypoint logic for waypoint codes in multiple countries) and put itself in "maintenance mode".
  - ExoticPearTree 8d ago
    My point was that it should not have happenned in the first place. It should have just rejected the flight plan, sent a notice to whatever coordination center would have been in charge of the flight and kept running. The fact that it entered in maintenance mode still ruined a lot of people’s days. I don’t think any flight crew or passenger cared about semantics back then.
hiveops80lab 8d ago
[flagged]
mrlonglong 7d ago
I rather suspect the current IBM system the software runs on is much quicker than the original system it used to run on. That can increase the probability of a data race occuring.

1 points by asplake 9d ago

zrail 9d ago

Note: NATS the UK airspace manager, not NATS the pubsub system.

verst 8d ago
And here I was surprised they would use NATS in their implementation to assign squawk codes.

fotta 9d ago

link to the report (pdf): https://www.nats.aero/wp-content/uploads/2026/09/NATS-Preliminary-Investigation-Report-into-NAS-Incident-on-08-Sept-2026-Issued-16-Sept-2026.pdf https://www.nats.aero/wp-content/uploads/2026/09/NATS-Prelim...

fotta 8d ago
> The incident was triggered by a valid manual request for a squawk code. This manual request was made correctly and there was nothing abnormal or invalid about the associated flight plan. > While this request was being processed, the NAS received a message for a higher priority activity to be undertaken which resulted in the squawk code allocation being paused while the system processed the higher priority message. Switching between different activities in response to prioritised requests is a normal function of the system; however, when the processing of the squawk allocation request resumed, the software defect meant it did not resume correctly and the resulting output was corrupted. > The reason this scenario has not occurred before is because: > 1. The defect existed in a specific subsection of code within a software module, with an exposure window estimated as approximately one millisecond. > 2. For the fault to occur, a higher-priority request had to arrive during that exact millisecond while the original request was part-way through updating a value. > 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally. > Post-incident investigation has identified that when processing of the squawk allocation request resumed, the data associated with it had been corrupted and affected some subsequent flight data updates.
- dtf 8d ago
  > 3. Had the higher-priority request arrived even one millisecond earlier or later, the update would have completed normally. Well, that's comforting to know. BBC: "Flight chaos caused by software defect in space of a millisecond, report says" Sky: "'Millisecond' software error caused air traffic outage that grounded thousands of flights" The Guardian: "Flight chaos for hundreds of thousands was caused in ‘millisecond’ by software error" Sounds like pure bad luck.
  - philipwhiuk 8d ago
    Or insufficient testing?
    - nradov 8d ago
      Testing isn't an effective way to catch most race conditions. Code reviews, static analysis, and rigorous enforcement of concurrent coding standards is usually a better approach.
      - jiehong 8d ago
        Perhaps something like what TigerBeetle does: deterministic simulation testing.
    - dtf 8d ago
      Or maybe even just reviewing logic that is subject to pre-emption. Maybe I'm being too harsh.. on the plus side the system has at least failed hard every time there's been a fault. Nobody has died. But it's been 3 times now in the past couple of years, and two of those times resulted in over 2000 flights cancelled and days of backlog, and misery for hundreds of thousands. It's really not acceptable.
      - chrisjj 8d ago
        > the system has at least failed hard every time there's been a fault You don't know that.

macguillicuddy 8d ago

Interesting! Looks like a race condition (with a 1ms window) during squawk allocation (the process that gives each aircraft a 4 digit identifier for ATC purposes) caused data corruption. They present that window as being small but 1ms is a decent chunk of time in our work lol

crote 8d ago
More specifically: it sounds like something which was supposed to be an atomic operation was split into two by the preempting task, with corruption happening because that task also interacted with the same piece of (now-temporarily-invalid) data.
JdeBP 8d ago
It's a decent chunk in most people's. But we're talking about a system that is also older than most people. It was first put into operation in 1967 in the U.S.A., and brought over to the U.K. in 1974. It's formally called 'NAS En Route Stage A', and is written in a 1950s language named JOVIAL. It originally ran on an IBM 9020. It runs on IBM 9020 compatible systems today. For some reason, a couple of years ago someone added a lengthy unverifiable description of it to Wikipedia's IBM 9020 article, even though that very description said that NAS hadn't run on a 9020 for 34 years at the time of writing, but had been running on a 4381.

lukaslueg 8d ago

Would be interesting if the report actually got into the details. The curiously specific "approx. 1ms window" sounds like an off-the-shelf memory race condition in combination with a preempting thread, for which 1ms is an absolute eternity and Guaranteed To Happen By Tuesday This Week™. But the "database corruption" sounds like database IDs being generated from timestamps with 1ms granularity.

LtdJorge 8d ago
They might have to switch to ULID or similar.

deleted 8d ago

[deleted]

ExoticPearTree 8d ago

A few years back, NATS went down because a flight plan waypoint confused the system and it crashed. They had to manually find the problematic flight plan, remove it and start the whole system back up. Considering the last issue they encountered, it looks like in more than one place, there is no error catching and graceful resolution for those errors. I would assume a system of such importance to handle issues without hiccups and alert the operators of what did not work. Like “this input caused this problem”, not just crash.

pixelesque 8d ago
Back in 2023 when the previous issue happened, it didn't actually "crash", it detected what it perceived as an inconsistency (which was due to invalid waypoint logic for waypoint codes in multiple countries) and put itself in "maintenance mode".
- ExoticPearTree 8d ago
  My point was that it should not have happenned in the first place. It should have just rejected the flight plan, sent a notice to whatever coordination center would have been in charge of the flight and kept running. The fact that it entered in maintenance mode still ruined a lot of people’s days. I don’t think any flight crew or passenger cared about semantics back then.

hiveops80lab 8d ago

[flagged]

mrlonglong 7d ago

I rather suspect the current IBM system the software runs on is much quicker than the original system it used to run on. That can increase the probability of a data race occuring.