4 ms·

US halts flights at busy East Coast airports, says fiber line cut

81 points by allanbreyes 5d ago

sc68cal 5d ago
There is also a new ATC system being rolled out, as early as today https://www.airwaysmag.com/new-post/faa-smart-first-deployment-washington https://www.airwaysmag.com/new-post/faa-smart-first-deployme...
- whatever1 5d ago
  I suspect it is vibe coded. Contract won in June. Deploying in September. Good luck to all of us.
  - macintux 5d ago
    Deploying first to DC? Yeah, that’ll end well.
  - gkedzierski 5d ago
    I can guarantee you that it was not. Check out DO-278A, and its standards and guidelines that must be followed.
    - tencentshill 5d ago
      ...Or what? The US government is in move-fast-and-break-things mode, and is running on a skeleton crew besides.
quickthrowman 5d ago
It has been conjectured that the easiest way to locate a buried fiber optic cable is to start digging. (The actual easiest method is to install the conduit or direct burial cable with tracer wire)
- dboreham 5d ago
  Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers. And of course there are marker posts that say "buried fiber optic cable do not dig".
  - pixl97 5d ago
    Once you get out farther from civilization these signs become less common and there are more situations where they can get removed. Most cheap hired labor doesn't realize you shouldn't dig up the market posts and throw them in the dumpster on construction sites. Add a little bit of rain and suddenly the contractor showing up with an excavator that was told everything was properly marked, and can see markings in other places but not where they are digging, and disaster occurs. Also soils aren't static, I've been on sites where the midpoint of a buried telecommunications cable had drifted over 8 feet from where the markers were a few hundred yards apart. Just looking at the markers and assuming a straight line isn't a safe bet. Looking at the fence rows to the left and right of the cable and you could see a matching bow in the fence rows.
- bluGill 5d ago
  Fiber was all installed long after we knew locating it was important. I suspect it all had a tracer wire installed with it. Of course sometimes the tracer wire can break. Before fiber a lot of things were put into the ground with no thought how you would find it again.
atonse 5d ago
I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc? Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?
- bri3d 5d ago
  * Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work. * There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately. The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.
  - wrs 5d ago
    I’ve had enough double-redundant links fail - and I mean pretty thoroughly double-redundant, different provider, different media, different physical path - that I’m quite surprised something like the air traffic control system only has two links.
    - pixl97 5d ago
      One thing that needs to be reviewed is if they are dual links or redundant links. In any system where both sides of redundancy always carry traffic then loss of one link can cause congestion failure if any link fails. This is a very common means of failure in electrical networks that requires load shedding. Well, you can't load shed air traffic. A more complex system that I like, but comes with it's own set of constraints and implementation issues is a system where both lines carry all the traffic at all times. This way the default state of the system is always working and your first failure isn't invisibly critical. But this is very hard as we see in TCP when things get out of order and high latency creeps in. You have to manage a lot more state at the data level.
cube00 5d ago
> When it went to flip into the backup, we discovered that the backup fiber had a break Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it. I wonder how long it was down? Days, weeks, months?
- readthenotes1 5d ago
  A lot of people don't check their backups until they need to restore. A lot of people are incompetent.
  - thewebguyd 5d ago
    A backup without a restore test isn't a backup at all
    - bonoboTP 5d ago
      You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
      - deleted 5d ago
        [deleted]
  - andrewjf 5d ago
    Nobody wants backups as a feature. The feature is restore.
- toast0 5d ago
  Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
tomasphan 5d ago
ATC seems like a great place to have redundancy with a starlink dish on the roof.
- peri-cl 5d ago
  I think this the same system that was the topic of a major contractor dispute a few months ago, between Verizon (incumbent fiber contractor) and SpaceX (lobbying to replace it). https://news.ycombinator.com/item?id=43205435 https://news.ycombinator.com/item?id=43205435 ("Starlink to take over $2.4B contract to overhaul air traffic control comms (theverge.com)")
dboreham 5d ago
[flagged]
dctoedt 5d ago
Not to be conspiratorial, but about an hour ago I saw this in The Economist: "Russia’s grey-zone attacks on Europe are growing more brazen" [QUOTE:] Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless. [/QUOTE] And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters .... https://www.economist.com/europe/2026/09/21/russias-grey-zone-attacks-on-europe-are-growing-more-brazen?giftId=YWViOTU2ZWYtMDJmZC00MTVmLWExOTItYTZiOGY1ODFjZTlm&utm_campaign=gifted_article https://www.economist.com/europe/2026/09/21/russias-grey-zon...
cyanydeez 5d ago
So of course if the FAA is going to put AI into the watchtowers, they're obviously going to use local AI and not rely on some random Amazon cloud infra in some middle eastern country. right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.
kqgnkqgn 5d ago
Building multiple diverse paths and monitoring for fiber cuts is not hard. Having 2 fiber paths is insufficient for even only moderately important workloads at a tech company. Overlapping fiber cuts happen. For something with significant economic and safety impact this is just crazy. Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.
- woodruffw 5d ago
  They had two diverse paths, but apparently did not have a (reliable?) process for ensuring the backup path was functional during fail-over. I imagine there will be some soul-searching over that. Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).
baggachipz 5d ago
Maybe Sean Duffy was just binge-watching his old reality tv shows and gobbled up the bandwidth?
chresko 5d ago
Are we sure this wasn't Claude? Has Dario made an announcement yet?

81 points by allanbreyes 5d ago

There is also a new ATC system being rolled out, as early as today https://www.airwaysmag.com/new-post/faa-smart-first-deployment-washington https://www.airwaysmag.com/new-post/faa-smart-first-deployme...

whatever1 5d ago
I suspect it is vibe coded. Contract won in June. Deploying in September. Good luck to all of us.
- macintux 5d ago
  Deploying first to DC? Yeah, that’ll end well.
- gkedzierski 5d ago
  I can guarantee you that it was not. Check out DO-278A, and its standards and guidelines that must be followed.
  - tencentshill 5d ago
    ...Or what? The US government is in move-fast-and-break-things mode, and is running on a skeleton crew besides.

quickthrowman 5d ago

It has been conjectured that the easiest way to locate a buried fiber optic cable is to start digging. (The actual easiest method is to install the conduit or direct burial cable with tracer wire)

dboreham 5d ago
Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers. And of course there are marker posts that say "buried fiber optic cable do not dig".
- pixl97 5d ago
  Once you get out farther from civilization these signs become less common and there are more situations where they can get removed. Most cheap hired labor doesn't realize you shouldn't dig up the market posts and throw them in the dumpster on construction sites. Add a little bit of rain and suddenly the contractor showing up with an excavator that was told everything was properly marked, and can see markings in other places but not where they are digging, and disaster occurs. Also soils aren't static, I've been on sites where the midpoint of a buried telecommunications cable had drifted over 8 feet from where the markers were a few hundred yards apart. Just looking at the markers and assuming a straight line isn't a safe bet. Looking at the fence rows to the left and right of the cable and you could see a matching bow in the fence rows.
bluGill 5d ago
Fiber was all installed long after we knew locating it was important. I suspect it all had a tracer wire installed with it. Of course sometimes the tracer wire can break. Before fiber a lot of things were put into the ground with no thought how you would find it again.

atonse 5d ago

I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc? Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?

bri3d 5d ago
* Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work. * There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately. The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.
- wrs 5d ago
  I’ve had enough double-redundant links fail - and I mean pretty thoroughly double-redundant, different provider, different media, different physical path - that I’m quite surprised something like the air traffic control system only has two links.
  - pixl97 5d ago
    One thing that needs to be reviewed is if they are dual links or redundant links. In any system where both sides of redundancy always carry traffic then loss of one link can cause congestion failure if any link fails. This is a very common means of failure in electrical networks that requires load shedding. Well, you can't load shed air traffic. A more complex system that I like, but comes with it's own set of constraints and implementation issues is a system where both lines carry all the traffic at all times. This way the default state of the system is always working and your first failure isn't invisibly critical. But this is very hard as we see in TCP when things get out of order and high latency creeps in. You have to manage a lot more state at the data level.

cube00 5d ago

> When it went to flip into the backup, we discovered that the backup fiber had a break Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it. I wonder how long it was down? Days, weeks, months?

readthenotes1 5d ago
A lot of people don't check their backups until they need to restore. A lot of people are incompetent.
- thewebguyd 5d ago
  A backup without a restore test isn't a backup at all
  - bonoboTP 5d ago
    You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
    - deleted 5d ago
      [deleted]
- andrewjf 5d ago
  Nobody wants backups as a feature. The feature is restore.
toast0 5d ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.

tomasphan 5d ago

ATC seems like a great place to have redundancy with a starlink dish on the roof.

peri-cl 5d ago
I think this the same system that was the topic of a major contractor dispute a few months ago, between Verizon (incumbent fiber contractor) and SpaceX (lobbying to replace it). https://news.ycombinator.com/item?id=43205435 https://news.ycombinator.com/item?id=43205435 ("Starlink to take over $2.4B contract to overhaul air traffic control comms (theverge.com)")

dboreham 5d ago

[flagged]

dctoedt 5d ago

Not to be conspiratorial, but about an hour ago I saw this in The Economist: "Russia’s grey-zone attacks on Europe are growing more brazen" [QUOTE:] Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless. [/QUOTE] And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters .... https://www.economist.com/europe/2026/09/21/russias-grey-zone-attacks-on-europe-are-growing-more-brazen?giftId=YWViOTU2ZWYtMDJmZC00MTVmLWExOTItYTZiOGY1ODFjZTlm&utm_campaign=gifted_article https://www.economist.com/europe/2026/09/21/russias-grey-zon...

cyanydeez 5d ago

So of course if the FAA is going to put AI into the watchtowers, they're obviously going to use local AI and not rely on some random Amazon cloud infra in some middle eastern country. right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.

kqgnkqgn 5d ago

Building multiple diverse paths and monitoring for fiber cuts is not hard. Having 2 fiber paths is insufficient for even only moderately important workloads at a tech company. Overlapping fiber cuts happen. For something with significant economic and safety impact this is just crazy. Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.

woodruffw 5d ago
They had two diverse paths, but apparently did not have a (reliable?) process for ensuring the backup path was functional during fail-over. I imagine there will be some soul-searching over that. Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).

baggachipz 5d ago

Maybe Sean Duffy was just binge-watching his old reality tv shows and gobbled up the bandwidth?

chresko 5d ago

Are we sure this wasn't Claude? Has Dario made an announcement yet?