6 ms·

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, whi

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card. > Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes. And then > By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service. And finally > The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes. Looks like the root cause analysis has a way to go. Addendum says: > The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner. The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment. The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.

dec0dedab0de 8y ago
Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.
- lykr0n 8y ago
  Yep. It sounds like the packets were coming from this nic and the backbone routers had no logic to filter out those kinds of invalid packets. No SRC/DST/TTL should be a big red flag, but it seems like they didn't have a rule for that. Talk about a needle in a hay stack.
  - jandrese 8y ago
    Traditional IP filters are typically useless against packet storms because all of the packets are technically valid, there are just way too many of them.
- linker3000 8y ago
  I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,
  - sitkack 8y ago
    Could be a totally valid solution ... for an equipment rental place. Pretty sure people died that didn't have to with this 911 outage.
  - cyberpunk 8y ago
    Oh god, I had to deal with some procurve switches in a past life. They were a nightmare to manage and unreliable. We actually ended up replacing them on 2 racks with 24port unmanaged gbit 3com ones we bought from some high street electronics shop one night during an outage... Not a fun memory.. (Of course, those were later replaced with some proper equipment)
deleted 8y ago
[deleted]
mysuperlogin 8y ago
It's affect, not effect. https://www.grammarly.com/blog/affect-vs-effect/ https://www.grammarly.com/blog/affect-vs-effect/
tyingq 8y ago
One example might be "ip helper" setups in a Cisco router. People use it to forward DHCP requests over a WAN. One broken SIP device, no flood control, and...bam, your WAN is hosed.
- packet_nerd 8y ago
  Ip helper usually only shows up in the context of DHCP so its easy to forget it forwards a lot more than just DHCP broadcasts.
CamTin 8y ago
It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as it is to gawk, I don't think we'll ever get a good picture of what happened and what CenturyLink did wrong, if anything, to allow such an outage to take place.
- extrapickles 8y ago
  My reading of the tea leaves is that the invalid packets where for some low level control protocol. I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad packets, so after a delay it would enable the trunk again and get the invalid packets again.
  - jandrese 8y ago
    I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes... I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on big networks.
    - drcross 8y ago
      >I have a very low opinion of STP. That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.
      - 8y ago
bch 8y ago
Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...
- hinkley 8y ago
  The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s. There are lots of proximate reasons that a failure can avoid all of your sanity checks, but the simplest is when your organization is a little insane. I try to steer the conversation toward avoiding surprises, other times toward ergonomics (don’t rely so much on humans in the moment to do the right thing). Often people don’t fight me too much on that, but sometimes it’s a near thing. Lots of senior devs are senior because they keep everybody else down, so blaming human error for an outage seems perfectly reasonable to them.
  - bch 8y ago
    > The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s. Agreed. I think it’s a case for running a Joy Driven Development operation first, and looking to organizational responsibility first (versus looking for somewhere to cast blame), but practically, that probably too utopian to bear out in practice. No reason not to keep it in mind, though.
Jach 8y ago
From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); } if (conn != null) { conn.close(); } } } close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.
- cpeterso 8y ago
  "Release It" is an awesome book. I contributed a couple FindBug Java checkers to warn about some of the problems the book describes. The case you mention above might have been prevented by using checked Java exceptions. Our programming languages and tools could be doing a lot more to catch these problems at compile time or make them impossible by language or API design.
  - Deinumite 8y ago
    I think that was taken into account when the "try with resources" statements were added to Java. As you said it would be nice if the languages could make these situations impossible. I think its a bit of a chicken and egg problem because some language issues don't come to light until people are using it and it is too late.
- throw0u1t 8y ago
  I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!
  - 8y ago

dec0dedab0de 8y ago

Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.

lykr0n 8y ago
Yep. It sounds like the packets were coming from this nic and the backbone routers had no logic to filter out those kinds of invalid packets. No SRC/DST/TTL should be a big red flag, but it seems like they didn't have a rule for that. Talk about a needle in a hay stack.
- jandrese 8y ago
  Traditional IP filters are typically useless against packet storms because all of the packets are technically valid, there are just way too many of them.
linker3000 8y ago
I used to look after an office location that had three racks with older generation HP Procurve switches. About once every 4-6 months, the whole lot would go haywire and the only way to settle down the network was to switch them all off, wait, and then power up again. We never did track down the root cause and could never make it happen on demand,
- sitkack 8y ago
  Could be a totally valid solution ... for an equipment rental place. Pretty sure people died that didn't have to with this 911 outage.
- cyberpunk 8y ago
  Oh god, I had to deal with some procurve switches in a past life. They were a nightmare to manage and unreliable. We actually ended up replacing them on 2 racks with 24port unmanaged gbit 3com ones we bought from some high street electronics shop one night during an outage... Not a fun memory.. (Of course, those were later replaced with some proper equipment)

deleted 8y ago

[deleted]

mysuperlogin 8y ago

It's affect, not effect. https://www.grammarly.com/blog/affect-vs-effect/ https://www.grammarly.com/blog/affect-vs-effect/

tyingq 8y ago

One example might be "ip helper" setups in a Cisco router. People use it to forward DHCP requests over a WAN. One broken SIP device, no flood control, and...bam, your WAN is hosed.

packet_nerd 8y ago
Ip helper usually only shows up in the context of DHCP so its easy to forget it forwards a lot more than just DHCP broadcasts.

CamTin 8y ago

It will be almost impossible to get a good idea of the actual RCA from public reports, since even if they are accurate, we would need a complete understanding of the topology of the network in question to understand them. That said, the public descriptions will be filtered through so many layers of PR and lawyers as to be incomprehensible. "invalid frame packet"? Were they invalid frames or invalid packets? As fun as it is to gawk, I don't think we'll ever get a good picture of what happened and what CenturyLink did wrong, if anything, to allow such an outage to take place.

extrapickles 8y ago
My reading of the tea leaves is that the invalid packets where for some low level control protocol. I’ve had invalid spanning tree (protocol typically used to prevent loops in networks) packets cause a trunk link to flap as the invalid packet made the switch think its only trunk link (the link to the rest of the network) was part of a loop and shut it down. When the link went down, it could no longer get the bad packets, so after a delay it would enable the trunk again and get the invalid packets again.
- jandrese 8y ago
  I've seen plenty of networks where there is a packet storm, then STP disables the link and fixes it, then 30 seconds later STP re-enables the link and the packet storm resumes... I have a very low opinion of STP. I'm also not a fan of the fad to have very flat networks where there are very few routers and instead everything is switches in one gigantic subnet. Packet storms are notoriously difficult to track down on big networks.
  - drcross 8y ago
    >I have a very low opinion of STP. That's not a very radical opinion in network engineering circles. No one ever liked it due to it's non forwarding links for loop prevention but it was good enough to work until we discovered it's successor.
    - 8y ago

bch 8y ago

Every time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/ https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

hinkley 8y ago
The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s. There are lots of proximate reasons that a failure can avoid all of your sanity checks, but the simplest is when your organization is a little insane. I try to steer the conversation toward avoiding surprises, other times toward ergonomics (don’t rely so much on humans in the moment to do the right thing). Often people don’t fight me too much on that, but sometimes it’s a near thing. Lots of senior devs are senior because they keep everybody else down, so blaming human error for an outage seems perfectly reasonable to them.
- bch 8y ago
  > The quality of the outcome really depends on the participants. As they say, blaming the humans is easy. But it’s not five why’s. Agreed. I think it’s a case for running a Joy Driven Development operation first, and looking to organizational responsibility first (versus looking for somewhere to cast blame), but practically, that probably too utopian to bear out in practice. No reason not to keep it in mind, though.

Jach 8y ago

From the book "Release It!", the author describes an incident where an airline's entire check-in system went down for three hours, grounding its hundreds of planes and causing a pretty big backlog for hours more. The 'root cause' was code on the flight search server: lookupByCity(...) { .... try { conn = connectionPool.getConnection(); stmt = conn.createStatement(); ... } finally { if (stmt != null) { stmt.close(); } if (conn != null) { conn.close(); } } } close() can throw, and in the circumstance of the outage it did for stmt, leading to the connection not getting closed and eventually the pool being exhausted with every thread blocked waiting for a connection. It's an interesting chain of failures, arguably the presence of such a chain is the real root cause, rather than the unhandled sql exception.

cpeterso 8y ago
"Release It" is an awesome book. I contributed a couple FindBug Java checkers to warn about some of the problems the book describes. The case you mention above might have been prevented by using checked Java exceptions. Our programming languages and tools could be doing a lot more to catch these problems at compile time or make them impossible by language or API design.
- Deinumite 8y ago
  I think that was taken into account when the "try with resources" statements were added to Java. As you said it would be nice if the languages could make these situations impossible. I think its a bit of a chicken and egg problem because some language issues don't come to light until people are using it and it is too late.
throw0u1t 8y ago
I know people like to hate on errors as values as in Go, but I think exceptions are worse when it comes to unexpected side effects and this is a prime example!
- 8y ago