3 ms·

> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt< Cost-cutting didn't seem to have pl

by scurvy 8y ago

> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt< Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this outage on cost cutting. You're simply bashing a large company that had a bad day. They might deserve your vitriol for plenty of other reasons in other cases, but you can't blame this outage on cost cutting.

CydeWeys 8y ago
It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as that seems like a plausible thing an IT/networking department would've gotten around to with more funding, and more and higher-skilled, higher-paid veteran staff who could pre-emptively identify and fix this issue through experience and testing. Any time something really bad happens that could have been prevented, it's reasonable to ask why it wasn't prevented. This wasn't some black swan event, just a routine hardware failure that they didn't have defenses against.
- scurvy 8y ago
  Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it. To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
  - wpietri 8y ago
    We don't, and apparently they didn't either. But it was their job to do so. I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause. We can't know, of course. But I think it would be hard to argue that this is optimal, and that nobody at CenturyLink ever suggested it could be better if they spent a bit more money.
    - scurvy 8y ago
      It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one. Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as many "legacy" technologies as they do, things aren't very cut and dry. NTT probably has the best managed global network with their extensive use of SDN. They run a tight ship and everything goes through their automation frameworks. This requires a huge investment in R&D. Even then, it doesn't cover 100% of their network because "there's that T320 for that customer in Chicago, and the stuff up in Michigan for Dorian's T1." Multiply that times 30 or so to consider size differences between NTT and CenturyLink....and yeah, stuff happens.
      - wpietri 8y ago
        I agree it's hard. Anything interesting is. But your last paragraph specifically describes technical debt as part of the problem. If even the best org (NTT) has technical debt and CenturyLink isn't the best org, then I think it's safe to suspect that a) CenturyLink has significant technical debt, and b) they have underinvested in that technical debt compared with industry best practices.
LeftTurnSignal 8y ago
This issue is almost the same issue I had when I had an outage that took down the internet connection for a local govt I worked at. The issue was a single bad port on a switch about 3 hours away. It took two days of downtime for them to figure it out. Now the place I work at just deals with having fibre cut 2-3 hours away and losing a days worth of work for it. At least now I don't have to rely on insiders to tell me what actually is going on. But Centurylink does offer a 4g service for backup connections.... I'm not surprised at all with this. Business as usual for Centurylink.

by scurvy 8y ago

CydeWeys 8y ago

It seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as that seems like a plausible thing an IT/networking department would've gotten around to with more funding, and more and higher-skilled, higher-paid veteran staff who could pre-emptively identify and fix this issue through experience and testing. Any time something really bad happens that could have been prevented, it's reasonable to ask why it wasn't prevented. This wasn't some black swan event, just a routine hardware failure that they didn't have defenses against.

scurvy 8y ago
Who's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it. To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
- wpietri 8y ago
  We don't, and apparently they didn't either. But it was their job to do so. I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause. We can't know, of course. But I think it would be hard to argue that this is optimal, and that nobody at CenturyLink ever suggested it could be better if they spent a bit more money.
  - scurvy 8y ago
    It's hard to work on network problems when the management network is also having problems. That requires dispatching people locally to sites to see what's going on. You're blind in both eyes, not just one. Look, I'm not defending them or their choices. I'm just saying that this more more nuanced than "They suck they should have known better or spent more money." When you run a network as large as this and with as many "legacy" technologies as they do, things aren't very cut and dry. NTT probably has the best managed global network with their extensive use of SDN. They run a tight ship and everything goes through their automation frameworks. This requires a huge investment in R&D. Even then, it doesn't cover 100% of their network because "there's that T320 for that customer in Chicago, and the stuff up in Michigan for Dorian's T1." Multiply that times 30 or so to consider size differences between NTT and CenturyLink....and yeah, stuff happens.
    - wpietri 8y ago
      I agree it's hard. Anything interesting is. But your last paragraph specifically describes technical debt as part of the problem. If even the best org (NTT) has technical debt and CenturyLink isn't the best org, then I think it's safe to suspect that a) CenturyLink has significant technical debt, and b) they have underinvested in that technical debt compared with industry best practices.

LeftTurnSignal 8y ago

This issue is almost the same issue I had when I had an outage that took down the internet connection for a local govt I worked at. The issue was a single bad port on a switch about 3 hours away. It took two days of downtime for them to figure it out. Now the place I work at just deals with having fibre cut 2-3 hours away and losing a days worth of work for it. At least now I don't have to rely on insiders to tell me what actually is going on. But Centurylink does offer a 4g service for backup connections.... I'm not surprised at all with this. Business as usual for Centurylink.