6 ms·

I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to

I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.

kube-system 6y ago
I wouldn't doubt it. Single misconfigured routers have caused issues of similar scale.
- im3w1l 6y ago
  I believe the internet is designed to handle broken links, but is not very robust against misconfigured (or malicious) routers.
  - ArchOversight 6y ago
    If a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly. Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
  - mikem170 6y ago
    Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!
neogodless 6y ago
While I'm also not qualified to diagnose the problem, I would imagine it's something like this... The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic. The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
- overkill28 6y ago
  The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).
  - betterunix2 6y ago
    What makes you think it is only connected to Manhattan? Long Island has submarine cables connected to New England, various points along the East Coast south of NYC, and several transatlantic cables.
mlyle 6y ago
So, for clarity, it's never one fiber line being cut. It's a big massive bundle of fiber carrying different circuits for different carriers different places that were run in the same trench and all got cut at once. Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one. Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
- korethr 6y ago
  And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effect the repair, and you're probably going to have to rotate through techs to avoid fatigue-induced errors slowing things down even more.
  - fcsp 6y ago
    Would it maybe be an idea to, going forward, color the cables individually before putting them into the ground?
    - jzwinck 6y ago
      Of course they are colored. Like this: http://www.evertopcomms.com/144-fiber-count-gyts-fiber-optic-cable-internet-cable-wire-thin-pe-inner-sheath-41.html http://www.evertopcomms.com/144-fiber-count-gyts-fiber-optic... It is still a lot of work, and so far not automated.
      - bobthepanda 6y ago
        It is also, as a matter of practicality, going to be very difficult to find 144 colors that can‘t be mistaken for one another, particularly in conditions that i would imagine wouldn‘t have great lighting.
jorblumesea 6y ago
My naive explanation: fiber networks handle incredible loads, and a single fiber line might correspond to 10+ Tbs (terabytes per second). So all that data needs to go somewhere else, and if it gets shunted, packets get backed up, resulting in a cascading failure. Note: I am not a fiber technician nor do I work for a large ISP.

I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.

kube-system 6y ago

I wouldn't doubt it. Single misconfigured routers have caused issues of similar scale.

im3w1l 6y ago
I believe the internet is designed to handle broken links, but is not very robust against misconfigured (or malicious) routers.
- ArchOversight 6y ago
  If a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly. Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
- mikem170 6y ago
  Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!

neogodless 6y ago

While I'm also not qualified to diagnose the problem, I would imagine it's something like this... The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic. The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)

overkill28 6y ago
The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).
- betterunix2 6y ago
  What makes you think it is only connected to Manhattan? Long Island has submarine cables connected to New England, various points along the East Coast south of NYC, and several transatlantic cables.

mlyle 6y ago

So, for clarity, it's never one fiber line being cut. It's a big massive bundle of fiber carrying different circuits for different carriers different places that were run in the same trench and all got cut at once. Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one. Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.

korethr 6y ago
And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effect the repair, and you're probably going to have to rotate through techs to avoid fatigue-induced errors slowing things down even more.
- fcsp 6y ago
  Would it maybe be an idea to, going forward, color the cables individually before putting them into the ground?
  - jzwinck 6y ago
    Of course they are colored. Like this: http://www.evertopcomms.com/144-fiber-count-gyts-fiber-optic-cable-internet-cable-wire-thin-pe-inner-sheath-41.html http://www.evertopcomms.com/144-fiber-count-gyts-fiber-optic... It is still a lot of work, and so far not automated.
    - bobthepanda 6y ago
      It is also, as a matter of practicality, going to be very difficult to find 144 colors that can‘t be mistaken for one another, particularly in conditions that i would imagine wouldn‘t have great lighting.

jorblumesea 6y ago

My naive explanation: fiber networks handle incredible loads, and a single fiber line might correspond to 10+ Tbs (terabytes per second). So all that data needs to go somewhere else, and if it gets shunted, packets get backed up, resulting in a cascading failure. Note: I am not a fiber technician nor do I work for a large ISP.