4 ms·

>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet

>"A CenturyLink network management card in Denver, CO was propagating invalid frame packets across devices" This is of course gibberish. A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. A NIC on their management subnet disrupted their entire network? There are so many levels of absurdity to this. A mangled ethernet frame would be dropped if the CRC was incorrect. A "show int" on a switch would have shown drop counters incrementing. If it was a broadcast storm it also should have been obvious which device was sending an outsized amount of traffic to the all 1's address. Management networks are generally low traffic - ssh and some SNMP. It would have should have been obvious looking at interface graphs by TX on the management network. Further any modern switch from a major vendor has a storm control setting which disables a port when it goes beyond a certain threshold for either broadcast, multicast or unicast. Even if storm control wasn't enabled it would have been trivial to do so, find the offending port and work backwards from there. >"A polling filter was applied to adjust the way packets were received in the network equipment" "Polling filter" is not even an idiomatic network engineering term. I'll assume this means an access list. So it took them 50 hours to apply an ACL? And this required engaging the hardware vendor? This is a garbage RFO even if its not meant for a technical audience. It sounds like the real RFO is due to incompetence, bad network design and probably a horrid corporate culture shaped by fear, silos and CYA at this company.

jauer 8y ago
> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of impressively clowny behavior.
- bogomipz 8y ago
  >"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A decision was made to isolate a device in San Antonio, TX from the network as it seemed to be broadcasting traffic and consuming capacity." Lightwave gear most certainly does have any concept of broadcasts.
  - jauer 8y ago
    > Also do you that they would use DWDM for their management network? Yes, in the form of the Optical Supervisory Channel (OSC), which is built into DWDM gear and generally implemented as Ethernet over SONET. The OSC can also carry management traffic for other devices (aka datawire). It's Ethernet, so it has broadcasts...
    - bogomipz 8y ago
      Even if its OCS we're talking about the supervisory channel is truly out of band in that its generally on a proprietary wavelength isolated from your other channels carrying customer traffic. In this sense its no different than how a copper ethernet management VLAN should not be able to take down your entire production network.
      - jauer 8y ago
        "should not be able to" being the operative phrase :) Optical control plane generally hasn't benefited from the hardening that's happened in the IP world. Things like CoPP haven't become common practice yet.
        bogomipz 8y ago
        >"Optical control plane generally hasn't benefited from the hardening that's happened in the IP world" OK, but I imagine we can probably both agree that proper network design is orthogonal to the pace of development in optical transmission gear ;)

jauer 8y ago

> A "frame" is ethernet or L2 concept, packets are transport layer. Using the term "Frame packets" in an official RFO is laughable. OTU also has frames. Optical gear is generally happy to pass along mangled packets :/ > any modern switch from a major vendor has a storm control setting Look at the switches embedded in optical transport gear. They are pretty rudimentary. Optical transport gear (L1 networks) are full of impressively clowny behavior.

bogomipz 8y ago
>"OTU also has frames." Yes but STS "frames" and OTN are layer 1 concerns. There would still never be "frame packets." It's just as egregious. Also do you believe anyone would use DWDM for their management network? Management interfaces seldom require anything more than a few megabits of bandwidth. Burning an entire wavelength for a management network would be pretty crazy. In the RFO Centurylink also mentions - "A decision was made to isolate a device in San Antonio, TX from the network as it seemed to be broadcasting traffic and consuming capacity." Lightwave gear most certainly does have any concept of broadcasts.
- jauer 8y ago
  > Also do you that they would use DWDM for their management network? Yes, in the form of the Optical Supervisory Channel (OSC), which is built into DWDM gear and generally implemented as Ethernet over SONET. The OSC can also carry management traffic for other devices (aka datawire). It's Ethernet, so it has broadcasts...
  - bogomipz 8y ago
    Even if its OCS we're talking about the supervisory channel is truly out of band in that its generally on a proprietary wavelength isolated from your other channels carrying customer traffic. In this sense its no different than how a copper ethernet management VLAN should not be able to take down your entire production network.
    - jauer 8y ago
      "should not be able to" being the operative phrase :) Optical control plane generally hasn't benefited from the hardening that's happened in the IP world. Things like CoPP haven't become common practice yet.
      - bogomipz 8y ago
        >"Optical control plane generally hasn't benefited from the hardening that's happened in the IP world" OK, but I imagine we can probably both agree that proper network design is orthogonal to the pace of development in optical transmission gear ;)