16 ms·

CenturyLink 911 outage was caused by a single network card sending bad packets

328 points by EamonnMR 8y ago

mikeash 8y ago
They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days.
- JustSomeNobody 8y ago
  > ... can’t happen again Gotta love this. > No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days. Yep, they put a bandage on it and called it good.
- SlowRobotAhead 8y ago
  >a single piece of malfunctioning hardware Just wait until writes Bloomberg that this was because China had put a secret microchip on a network card. Months of entertainment to follow.
- jka 8y ago
  Worth giving some credit though - mitigation is important for something as critical as 911 service. Hopefully they're also tracking the design flaws, and yes that's worth following (and asking whether they're planning to do so?), but bear in mind people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future).
  - eeeeeeeeeeeee 8y ago
    I completely disagree. Mitigation with a patch applied in a few hours is acceptable, but mitigation DAYS after a multi-day outage of critical services that have life or death consequences is completely unacceptable.
  - mikeash 8y ago
    I completely agree. I wouldn’t expect them to have this fix already, since it probably requires some major work. But some mention that they’ll be looking into it would certainly be nice.
  - vmchale 8y ago
    > people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future). yes, clearly knocking out 911 service for millions of people isn't a problem. won't someone think of the poor programmers??
- patrickg_zill 8y ago
  I don't have any inside info. But it sounds like either a control plane or a mesh/fabric board that was messing with all the other such boards in their system.
tonetheman 8y ago
Wonder how they found it... I bet that was some unhappy hours looking for it.
- sverige 8y ago
  That's the "I found a 15-year-old bug in our NIC" story that I'd love to read but which will probably never be told. And whoever designed the system is busy obfuscating that on their resume this morning.
  - Aloha 8y ago
    I's more likely the card went rogue and failed.
- dsfyu404ed 8y ago
  Once they knew what they were looking for it was probably pretty quick to find. A network card that's just vomiting all over the network is loud and visible by definition.
TheWoodsy 8y ago
Someone was displeased - https://fuckingcenturylink.com/ https://fuckingcenturylink.com/ I'd love to see an in-depth technical analysis of the outage.
- nubb 8y ago
  This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng
  - azinman2 8y ago
    I always heard L3 was quite good. Who is if they’re not?
    - zamadatix 8y ago
      There is no such thing as a good national carrier right now, L3 just happened to be the least shit of the group to deal with. Your best bet is to try to find a decent local/regional carrier as your primary circuit provider and then use L3/The Devil/ATT (in that order) as your secondary. Yes your regional/local is still going to hand off to those guys but at least they'll be doing all of them and you get real support for local issues.
    - nubb 8y ago
      They used to be. It's been a clusterfuck for a few years now with them.
  - scruffyherder 8y ago
    I have a uh .. 'client' that uses CL/L3 and I swear they have outages WEEKLY. Like they should give notice when we can use the circuit vs when it's going down. again.
  - Scoundreller 8y ago
    “Probably”? Fuckinglevel3.com redirects to fuckingcenturylink.com right now.
Someone1234 8y ago
Makes you wonder how secure their backhaul really is? If the whole thing is a single flat logical network (one that could allow bad packets to propagate as we witnessed) that would suggest it is also quite vulnerable to malicious actions. It is all well and good applying a filter, but that seems like a bandaid fix. Why is equipment even able to talk that has no reason to do so? Seems like they've put convenience over good network governance.
- ecp9 8y ago
  Much networking equipment is not designed to handle malicious or bizarre traffic. TCP/IP is amazingly brittle, and often fails on me in surprising ways that the standards say should never ever happen.
  - drcross 8y ago
    I don't think that's fair to say. Billions of people unlock their phone or log in to their computers every morning and everything works, pretty much all of the time.
    - ecp9 8y ago
      If you're a hipster in a major city near fiber, sure, it all works great. For the rest of us, no, daily failures are the reality.
      - drcross 8y ago
        That´s not a failure of TCP.
crispyambulance 8y ago
The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers. Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching? Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?
- extrapickles 8y ago
  They generally only get dropped if they have an invalid checksum. Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.
  - kijiki 8y ago
    If cut-through is enabled in the switch, it won't even drop on bad checksum, since by the time the switch can tell the checksum is wrong, it has already forwarded the entire packet.
    - loeg 8y ago
      It is really unfortunate that the checksum is on the opposite end of the frame from the routing information on Ethernet frames.
      - topranks 8y ago
        Why? If it was at the start it’d remain useless until the whole frame was received anyway. Makes sense to have it at the end.
        kiallmacinnes 8y ago
        It also allows you to calculate the checksum as your serialising the packet data onto the send buffer, without having to get the whole packet in memory, checksum it, write the checksum, then finally write the packet data.
carlsborg 8y ago
How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card. > Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes. And then > By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service. And finally > The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes. Looks like the root cause analysis has a way to go. Addendum says: > The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner. The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment. The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.
- dec0dedab0de 8y ago
  Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.
  - lykr0n 8y ago
yosefzeev 8y ago
Did the network card in question also send blue flashes up into the sky?
- Crosseye_Jack 8y ago
  They got the gaming branded network card with all the RGB.
cesarb 8y ago
Reminds me of an AT&T outage 30 years ago: https://catless.ncl.ac.uk/risks/9.62.html#subj2 https://catless.ncl.ac.uk/risks/9.62.html#subj2 https://catless.ncl.ac.uk/risks/9.63.html#subj3 https://catless.ncl.ac.uk/risks/9.63.html#subj3
TagAsDelusion 8y ago
"bad packets"
solarengineer 8y ago
I had faced something like this a few years ago: https://dynamicproxy.livejournal.com/46862.html https://dynamicproxy.livejournal.com/46862.html Summary: the specific Symantec disk imaging software was partly loaded via PXE boot, and that machine started to flood the network with bad packets. Switching the computer off for a few seconds didn't help, since the SMPS capacitors still held current - enough to keep the card alive and for the sysadmins to not suspect that computer!
CydeWeys 8y ago
I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink, as well as my SO giving them a Roku for the holidays and setting them up with streaming services, which they got lots of practice using in the two whole days the cable TV didn't work. Guess who's cutting the cord and switching to Internet-only service from Verizon FIOS at a much cheaper rate now? This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: You get huge costly outages revealing your basic incompetence and customers fleeing to superior competitors forever.
- jandrese 8y ago
  Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.
  - waynesonfire 8y ago
    centurylink fiber is pretty great as well
    - deleted 8y ago
      [deleted]
    - loeg 8y ago
      Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps. And the price is extremely competitive with Comcast ($80/mo for "gigabit"), unlike their DSL offerings, which are complete crap. The major downside is that their IPv6 support is nonexistent and they have no plans for rolling it out.
bandrami 8y ago
Packets or frames? The report mentions both (including "packet frames", which, ugh)
- topranks 8y ago
  These terms are interchangeable. Although typically one or other is used when speaking about a specific technology. Also see datagram, cell and probably others I’m forgetting right now.
  - bandrami 8y ago
    But packets and frames as implemented are different things. One encapsulates the other.
  - packet_nerd 8y ago
    > terms are interchangeable No, use frame when your talking about layer 2, packet when your talking about layer 3, and segment for layer 4. Datagram and protocol data unit (PDU) are general terms that can apply to any layer. A switch forwards frames, while a router routes packets. https://stackoverflow.com/questions/31446777/difference-between-packets-and-frames https://stackoverflow.com/questions/31446777/difference-betw...
    - icedchai 8y ago
      You are technically correct. However, the term “Ethernet packet” is commonly used colloquially... it does not seem worth arguing about.
      - packet_nerd 8y ago
        Yeah, but the point is you wouldn't expect a competent network engineer to use the words "packet frames". And incompetence would seem to fit with the facts of the case, i.e. 911 was down for 2 days.
        8y ago
fmeyer 8y ago
I once took down an entire network with a Dev node running a misconfigured DHCP server. Second time with a snmp v2 ddos. If your network isn't properly configured these things can happen easily.
- yjftsjthsd-h 8y ago
  > If your network isn't properly configured these things can happen easily. Absolutely true. However, if you are an ISP, then not correctly configuring your network is... unimpressive.
ddingus 8y ago
One NIC? I have tracked that kind of thing down before. "Line noise adapters", otherwise known as former NICs, can be a pita. But taking down the whole service, or for a pretty big region? I am off to read the details!
bogomipz 8y ago
>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this. A mangled ethernet frame would be dropped if the CRC was incorrect. A "show int" on a switch would have shown drop counters incrementing. If it was a broadcast storm it also should have been obvious which device was sending an outsized amount of traffic to the all 1's address. Management networks are generally low traffic - ssh and some SNMP. It would have should have been obvious looking at interface graphs by TX on the management network. Further any modern switch from a major vendor has a storm control setting which disables a port when it goes beyond a certain threshold for either broadcast, multicast or unicast. Even if storm control wasn't enabled it would have been trivial to do so, find the offending port and work backwards from there. >"A polling filter was applied to adjust the way packets were received in the network equipment" "Polling filter" is not even an idiomatic network engineering term. I'll assume this means an access list. So it took them 50 hours to apply an ACL? And this required engaging the hardware vendor? This is a garbage RFO even if its not meant for a technical audience. It sounds like the real RFO is due to incompetence, bad network design and probably a horrid corporate culture shaped by fear, silos and CYA at this company.
- jauer 8y ago
  > A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of impressively clowny behavior.
  - bogomipz 8y ago
    >"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A decision was made to isolate a device in San Antonio, TX from the network as it seemed to be broadcasting traffic and consuming capacity." Lightwave gear most certainly does have any concept of broadcasts.
drcross 8y ago
Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. A single interface shutdown command would have been all that was needed if they had remote access.
- mschuster91 8y ago
  > Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. Or that said infrastructure may exist, but not redundant... I mean, RS232-over-IP or RS232-over-ISDN boxes are no secret sauce, but when their access line is routed over the same thing the box is supposed to remote-manage, then one has problems.
heyjudy 8y ago
Shocking that a. There was no error monitoring. b. That a SPoF existed. c. That it wasn't found sooner. The FCC, with their Verizon lackey Ajit Pai, should fine them $100 million bucks to get their attention, but they won't because corporate welfare.
cf498 8y ago
This is infrastructure on which lives depend. Who exactly is liable here outside of a FCC fine? Its just insane how software errors are apparently more or less considered equivalent to "higher power" losses. This all not mentioning the lack of a backup plan for a 50h outage.
aviv 8y ago
What is amazing here is not that a single network card caused this mess, but that most folks here actually believe this story.

328 points by EamonnMR 8y ago

They’re now filtering bad packets so this can’t happen again. No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days.

JustSomeNobody 8y ago
> ... can’t happen again Gotta love this. > No mention of fixing the design flaw in the system that allows a single piece of malfunctioning hardware to knock out 911 service for millions of users for two days. Yep, they put a bandage on it and called it good.
SlowRobotAhead 8y ago
>a single piece of malfunctioning hardware Just wait until writes Bloomberg that this was because China had put a secret microchip on a network card. Months of entertainment to follow.
jka 8y ago
Worth giving some credit though - mitigation is important for something as critical as 911 service. Hopefully they're also tracking the design flaws, and yes that's worth following (and asking whether they're planning to do so?), but bear in mind people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future).
- eeeeeeeeeeeee 8y ago
  I completely disagree. Mitigation with a patch applied in a few hours is acceptable, but mitigation DAYS after a multi-day outage of critical services that have life or death consequences is completely unacceptable.
- mikeash 8y ago
  I completely agree. I wouldn’t expect them to have this fix already, since it probably requires some major work. But some mention that they’ll be looking into it would certainly be nice.
- vmchale 8y ago
  > people have limited time and resources, so don't be too hard on them (or they'll be less willing to help and investigate in future). yes, clearly knocking out 911 service for millions of people isn't a problem. won't someone think of the poor programmers??
patrickg_zill 8y ago
I don't have any inside info. But it sounds like either a control plane or a mesh/fabric board that was messing with all the other such boards in their system.

tonetheman 8y ago

Wonder how they found it... I bet that was some unhappy hours looking for it.

sverige 8y ago
That's the "I found a 15-year-old bug in our NIC" story that I'd love to read but which will probably never be told. And whoever designed the system is busy obfuscating that on their resume this morning.
- Aloha 8y ago
  I's more likely the card went rogue and failed.
dsfyu404ed 8y ago
Once they knew what they were looking for it was probably pretty quick to find. A network card that's just vomiting all over the network is loud and visible by definition.

TheWoodsy 8y ago

Someone was displeased - https://fuckingcenturylink.com/ https://fuckingcenturylink.com/ I'd love to see an in-depth technical analysis of the outage.

nubb 8y ago
This is probably the same guy from fuckinglevel3.com. We used level3 for years and sent each other that link almost daily. I guess he had to update after CL bought them :] Don't expect a technical report from CL/L3. We had 60+ mpls/vpls circuits from them and all our reports were very high level. Source: am neteng
- azinman2 8y ago
  I always heard L3 was quite good. Who is if they’re not?
  - zamadatix 8y ago
    There is no such thing as a good national carrier right now, L3 just happened to be the least shit of the group to deal with. Your best bet is to try to find a decent local/regional carrier as your primary circuit provider and then use L3/The Devil/ATT (in that order) as your secondary. Yes your regional/local is still going to hand off to those guys but at least they'll be doing all of them and you get real support for local issues.
  - nubb 8y ago
    They used to be. It's been a clusterfuck for a few years now with them.
- scruffyherder 8y ago
  I have a uh .. 'client' that uses CL/L3 and I swear they have outages WEEKLY. Like they should give notice when we can use the circuit vs when it's going down. again.
- Scoundreller 8y ago
  “Probably”? Fuckinglevel3.com redirects to fuckingcenturylink.com right now.

Someone1234 8y ago

Makes you wonder how secure their backhaul really is? If the whole thing is a single flat logical network (one that could allow bad packets to propagate as we witnessed) that would suggest it is also quite vulnerable to malicious actions. It is all well and good applying a filter, but that seems like a bandaid fix. Why is equipment even able to talk that has no reason to do so? Seems like they've put convenience over good network governance.

ecp9 8y ago
Much networking equipment is not designed to handle malicious or bizarre traffic. TCP/IP is amazingly brittle, and often fails on me in surprising ways that the standards say should never ever happen.
- drcross 8y ago
  I don't think that's fair to say. Billions of people unlock their phone or log in to their computers every morning and everything works, pretty much all of the time.
  - ecp9 8y ago
    If you're a hipster in a major city near fiber, sure, it all works great. For the rest of us, no, daily failures are the reality.
    - drcross 8y ago
      That´s not a failure of TCP.

crispyambulance 8y ago

The descriptions so far about this problem are either at 30000 foot vagueness or they're in technical shorthand that just assumes the audience is 100% pro network engineers. Don't "bad packets" get dropped at the first switch? Isn't that one of the main benefits of packet based switching? Was this even an ethernet packet or something else like an optical transport protocol (eg OTN)?

extrapickles 8y ago
They generally only get dropped if they have an invalid checksum. Since checksums are hardware accelerated, the invalid packet probably had a valid checksum applied to it.
- kijiki 8y ago
  If cut-through is enabled in the switch, it won't even drop on bad checksum, since by the time the switch can tell the checksum is wrong, it has already forwarded the entire packet.
  - loeg 8y ago
    It is really unfortunate that the checksum is on the opposite end of the frame from the routing information on Ethernet frames.
    - topranks 8y ago
      Why? If it was at the start it’d remain useless until the whole frame was received anyway. Makes sense to have it at the end.
      - kiallmacinnes 8y ago
        It also allows you to calculate the checksum as your serialising the packet data onto the send buffer, without having to get the whole packet in memory, checksum it, write the checksum, then finally write the packet data.

carlsborg 8y ago

How does a single network card emitting bad packets effect other sites? > investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card. > Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes. And then > By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service. And finally > The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes. Looks like the root cause analysis has a way to go. Addendum says: > The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner. The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment. The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.

dec0dedab0de 8y ago
Packet storms are real, and they are tricky to troubleshoot. Modern tech has made it less likely to happen, but that just means when it does happen noone suspects it.
- lykr0n 8y ago

yosefzeev 8y ago

Did the network card in question also send blue flashes up into the sky?

Crosseye_Jack 8y ago
They got the gaming branded network card with all the RGB.

cesarb 8y ago

Reminds me of an AT&T outage 30 years ago: https://catless.ncl.ac.uk/risks/9.62.html#subj2 https://catless.ncl.ac.uk/risks/9.62.html#subj2 https://catless.ncl.ac.uk/risks/9.63.html#subj3 https://catless.ncl.ac.uk/risks/9.63.html#subj3

TagAsDelusion 8y ago

"bad packets"

solarengineer 8y ago

I had faced something like this a few years ago: https://dynamicproxy.livejournal.com/46862.html https://dynamicproxy.livejournal.com/46862.html Summary: the specific Symantec disk imaging software was partly loaded via PXE boot, and that machine started to flood the network with bad packets. Switching the computer off for a few seconds didn't help, since the SMPS capacitors still held current - enough to keep the card alive and for the sysadmins to not suspect that computer!

CydeWeys 8y ago

I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.) This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink, as well as my SO giving them a Roku for the holidays and setting them up with streaming services, which they got lots of practice using in the two whole days the cable TV didn't work. Guess who's cutting the cord and switching to Internet-only service from Verizon FIOS at a much cheaper rate now? This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: You get huge costly outages revealing your basic incompetence and customers fleeing to superior competitors forever.

jandrese 8y ago
Honestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.
- waynesonfire 8y ago
  centurylink fiber is pretty great as well
  - deleted 8y ago
    [deleted]
  - loeg 8y ago
    Yeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps. And the price is extremely competitive with Comcast ($80/mo for "gigabit"), unlike their DSL offerings, which are complete crap. The major downside is that their IPv6 support is nonexistent and they have no plans for rolling it out.

bandrami 8y ago

Packets or frames? The report mentions both (including "packet frames", which, ugh)

topranks 8y ago
These terms are interchangeable. Although typically one or other is used when speaking about a specific technology. Also see datagram, cell and probably others I’m forgetting right now.
- bandrami 8y ago
  But packets and frames as implemented are different things. One encapsulates the other.
- packet_nerd 8y ago
  > terms are interchangeable No, use frame when your talking about layer 2, packet when your talking about layer 3, and segment for layer 4. Datagram and protocol data unit (PDU) are general terms that can apply to any layer. A switch forwards frames, while a router routes packets. https://stackoverflow.com/questions/31446777/difference-between-packets-and-frames https://stackoverflow.com/questions/31446777/difference-betw...
  - icedchai 8y ago
    You are technically correct. However, the term “Ethernet packet” is commonly used colloquially... it does not seem worth arguing about.
    - packet_nerd 8y ago
      Yeah, but the point is you wouldn't expect a competent network engineer to use the words "packet frames". And incompetence would seem to fit with the facts of the case, i.e. 911 was down for 2 days.
      - 8y ago

fmeyer 8y ago

I once took down an entire network with a Dev node running a misconfigured DHCP server. Second time with a snmp v2 ddos. If your network isn't properly configured these things can happen easily.

yjftsjthsd-h 8y ago
> If your network isn't properly configured these things can happen easily. Absolutely true. However, if you are an ISP, then not correctly configuring your network is... unimpressive.

ddingus 8y ago

One NIC? I have tracked that kind of thing down before. "Line noise adapters", otherwise known as former NICs, can be a pita. But taking down the whole service, or for a pretty big region? I am off to read the details!

bogomipz 8y ago

>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this. A mangled ethernet frame would be dropped if the CRC was incorrect. A "show int" on a switch would have shown drop counters incrementing. If it was a broadcast storm it also should have been obvious which device was sending an outsized amount of traffic to the all 1's address. Management networks are generally low traffic - ssh and some SNMP. It would have should have been obvious looking at interface graphs by TX on the management network. Further any modern switch from a major vendor has a storm control setting which disables a port when it goes beyond a certain threshold for either broadcast, multicast or unicast. Even if storm control wasn't enabled it would have been trivial to do so, find the offending port and work backwards from there. >"A polling filter was applied to adjust the way packets were received in the network equipment" "Polling filter" is not even an idiomatic network engineering term. I'll assume this means an access list. So it took them 50 hours to apply an ACL? And this required engaging the hardware vendor? This is a garbage RFO even if its not meant for a technical audience. It sounds like the real RFO is due to incompetence, bad network design and probably a horrid corporate culture shaped by fear, silos and CYA at this company.

jauer 8y ago
> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of impressively clowny behavior.
- bogomipz 8y ago
  >"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A decision was made to isolate a device in San Antonio, TX from the network as it seemed to be broadcasting traffic and consuming capacity." Lightwave gear most certainly does have any concept of broadcasts.

drcross 8y ago

Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. A single interface shutdown command would have been all that was needed if they had remote access.

mschuster91 8y ago
> Needing to dispatch local field engineers is telling because it shows they do not have/prioritise remote login capabilities in their access layer switching infrastructure. Or that said infrastructure may exist, but not redundant... I mean, RS232-over-IP or RS232-over-ISDN boxes are no secret sauce, but when their access line is routed over the same thing the box is supposed to remote-manage, then one has problems.

heyjudy 8y ago

Shocking that a. There was no error monitoring. b. That a SPoF existed. c. That it wasn't found sooner. The FCC, with their Verizon lackey Ajit Pai, should fine them $100 million bucks to get their attention, but they won't because corporate welfare.

cf498 8y ago

This is infrastructure on which lives depend. Who exactly is liable here outside of a FCC fine? Its just insane how software errors are apparently more or less considered equivalent to "higher power" losses. This all not mentioning the lack of a backup plan for a 50h outage.

aviv 8y ago

What is amazing here is not that a single network card caused this mess, but that most folks here actually believe this story.