5 ms·

>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Me

>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Meanwhile google, zoom, aws, and microsoft went down? There isn't any shared datacenters here. There's at least 3 distinct datacenters. So what's the truth? what actually went down?

jasonjayr 6y ago
Traffic is passing, although slowly. I think this revealed that there is insufficient failover capacity.
- mikem170 6y ago
  Could be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc. I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm... I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system. As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two. (Typed over my phone, cuz my broadband is down!)
ArchOversight 6y ago
If 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow. Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
- mikem170 6y ago
  Yep, and it's possible that running low on ipv4 address space exacerbated the situation, so many more small routes than there used to be, maybe not all getting summarized during an outage they way they used to be, causing problems on peer bgp routers like you said.
qeternity 6y ago
Uh presuming that the connection is multihomed which is not the case for residential connections in NY.
ASalazarMX 6y ago
It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented. Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.
- kridsdale1 6y ago
  Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?
  - FlyMoreRockets 6y ago
    Occam's razor: when presented with competing hypotheses about the same prediction, one should select the solution with the fewest assumptions. https://en.wikipedia.org/wiki/Occam%27s_razor https://en.wikipedia.org/wiki/Occam%27s_razor Why is it that conspiracy theories seem to always require a more complex answer than the truth?
    - ASalazarMX 6y ago
      Occam's razor is a guide to start with the simpler explanation. If the simpler explanation is not sufficient, you add more elements until it is. My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet. The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
    - midasuni 6y ago
      Conspiracy theories are more fun to read. One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years

jasonjayr 6y ago

Traffic is passing, although slowly. I think this revealed that there is insufficient failover capacity.

mikem170 6y ago
Could be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc. I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm... I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system. As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two. (Typed over my phone, cuz my broadband is down!)

ArchOversight 6y ago

If 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow. Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.

mikem170 6y ago
Yep, and it's possible that running low on ipv4 address space exacerbated the situation, so many more small routes than there used to be, maybe not all getting summarized during an outage they way they used to be, causing problems on peer bgp routers like you said.

qeternity 6y ago

Uh presuming that the connection is multihomed which is not the case for residential connections in NY.

ASalazarMX 6y ago

It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented. Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.

kridsdale1 6y ago
Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?
- FlyMoreRockets 6y ago
  Occam's razor: when presented with competing hypotheses about the same prediction, one should select the solution with the fewest assumptions. https://en.wikipedia.org/wiki/Occam%27s_razor https://en.wikipedia.org/wiki/Occam%27s_razor Why is it that conspiracy theories seem to always require a more complex answer than the truth?
  - ASalazarMX 6y ago
    Occam's razor is a guide to start with the simpler explanation. If the simpler explanation is not sufficient, you add more elements until it is. My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet. The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
  - midasuni 6y ago
    Conspiracy theories are more fun to read. One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years