7 ms·

I'm unable to read the article on my phone (ad?) so I apologize if it is answered there, but why is this a long lasting problem? My past two or three (not sure

I'm unable to read the article on my phone (ad?) so I apologize if it is answered there, but why is this a long lasting problem? My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. (I don't want to ignore the pre- and intra- effort by ops, I'm just saying it paid off) Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place? Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.

jpollock 8y ago
Software is hard, and backbones tend to be mono-cultures. This means that there will be bugs which only become apparent when the full fleet has been deployed. This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything. This is also why emergency services should not be using cell phones for reliable service during emergencies. Here's another example from 1990, where the phone network got stuck in a reset loop. https://catless.ncl.ac.uk/Risks/9/62#subj2 https://catless.ncl.ac.uk/Risks/9/62#subj2 Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.
- flexer2 8y ago
  My dad spent his career as a natural gas pipeline technician. For a long time they maintained their own radio towers and VHF or UHF (I forget which) radio network. Each district had their own comms guy who maintained the network, and it was very resilient and reliable, allowing the entire company to be in contact essentially anywhere. In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T. After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
  - iseeyoubydesign 8y ago
    yea ... but thats really their purpose. And even in a logical sense the company itself should strive for such measures. Learn where you "can" outsource then do it. The thorough process and underlying principal isnt bad. Its the execution here that really caused the problem, choosing a provider not as reliable as their own. Same thing could happen in house. Manager comes in decides to cut cost , sees redundant employees, decides to reduce communications engineers by half. Same problem.
  - julianlam 8y ago
    That makes me sad to hear, because not only was this company left with an inferior solution, the man hours and effort that went into the original project (including purchasing part of the spectrum!) was essentially nullified. It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?
    - scruffyherder 8y ago
      But then we wouldn't have single threaded wonders like node.js It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again. Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?
      - XorNot 8y ago
        Sure except you've managed to bitch about most major programming languages here without proposing any alternatives.
        salawat 8y ago
        The poster's entire point was that all of them came from a desire to find another option when a reasonable alternative already existed.
- BuildTheRobots 8y ago
  Great write-up, thanks for the link. TLDR: Race condition when switches reboot during high (read normal) traffic load. I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal. > AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.
  - iforgotpassword 8y ago
    Race conditions strike again. It's just hard, whether it's multi threading or distributed systems. https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
reaperducer 8y ago
My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. Because your workplaces don't rely on funding from taxpayers.
- jamp897 8y ago
  CenturyLink isn’t exactly an underfunded provider though.
  - ujvighe 8y ago
    Since you misunderstood the polite version, here's the reality: 911 failed because there isn't an option to support a competing service, instead people are compelled using violence to pay for a service that doesn't really care about their well being and is usually used to send state thugs to murder people without recourse. HN has so much bias
  - deleted 8y ago
    [deleted]
  - yomodsyohi 8y ago
    Comments to your post are not allowed. Wow so even with show dead I can. Barely even read the comments that the HN community has deemed improper.
  - autodeadhihi 8y ago
    hN has decided that my comments are auto dead, IE, my comments are always dead. And this is my first comment. Awesome sauce!
- kodablah 8y ago
  > Because your workplaces don't rely on funding from taxpayers. Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.
briffle 8y ago
We used a vendor to put up two MPLS drops to every office. different providers, different routes from the building, etc. Turns out, further up the chain, an ISP was buying transit from another ISP, and they all went on the same shared fiber that a nice backhoe operator found for us, about 800 miles away.
- jrwr 8y ago
  Trace route can help with that somewhat, I've noticed that a few times with redundant connections
  - kortilla 8y ago
    The problem with traceroute is that it won’t show that two different ISPs are leasing strands out of the same bundle. All of the unexpected shared infra issues I’ve observed have been down at this lower level that never shows up at the IP layer.
    - nickpsecurity 8y ago
      Is there a directory or service one can use to get that information?
      - chucky_z 8y ago
        In my experience this exists for ISPs, but it's not easily available. I've used ISP brokers in the past and they've had this information in some proprietary systems when I've asked for quotes and proposals.
        walrus 8y ago
        My experience corroborates this. I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.
technion 8y ago
In Australia at least, this is becoming harder as time goes on. Several good alternatives have been purchased by one conglomerate. I know several businesses that did everything right in buying two or even three sets of connectivity from different suppliers, and now what they have are different trading names for the same company providing a service on the same network.
cynoclast 8y ago
Because they don't test failover.

jpollock 8y ago

Software is hard, and backbones tend to be mono-cultures. This means that there will be bugs which only become apparent when the full fleet has been deployed. This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything. This is also why emergency services should not be using cell phones for reliable service during emergencies. Here's another example from 1990, where the phone network got stuck in a reset loop. https://catless.ncl.ac.uk/Risks/9/62#subj2 https://catless.ncl.ac.uk/Risks/9/62#subj2 Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.

flexer2 8y ago
My dad spent his career as a natural gas pipeline technician. For a long time they maintained their own radio towers and VHF or UHF (I forget which) radio network. Each district had their own comms guy who maintained the network, and it was very resilient and reliable, allowing the entire company to be in contact essentially anywhere. In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T. After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
- iseeyoubydesign 8y ago
  yea ... but thats really their purpose. And even in a logical sense the company itself should strive for such measures. Learn where you "can" outsource then do it. The thorough process and underlying principal isnt bad. Its the execution here that really caused the problem, choosing a provider not as reliable as their own. Same thing could happen in house. Manager comes in decides to cut cost , sees redundant employees, decides to reduce communications engineers by half. Same problem.
- julianlam 8y ago
  That makes me sad to hear, because not only was this company left with an inferior solution, the man hours and effort that went into the original project (including purchasing part of the spectrum!) was essentially nullified. It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?
  - scruffyherder 8y ago
    But then we wouldn't have single threaded wonders like node.js It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again. Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?
    - XorNot 8y ago
      Sure except you've managed to bitch about most major programming languages here without proposing any alternatives.
      - salawat 8y ago
        The poster's entire point was that all of them came from a desire to find another option when a reasonable alternative already existed.
BuildTheRobots 8y ago
Great write-up, thanks for the link. TLDR: Race condition when switches reboot during high (read normal) traffic load. I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal. > AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.
- iforgotpassword 8y ago
  Race conditions strike again. It's just hard, whether it's multi threading or distributed systems. https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25

reaperducer 8y ago

My past two or three (not sure) workplaces have had to failover to alternate providers for office traffic when something broke in the primary, and both had contracts in place and the technical infrastructure to do so in relatively short order. Because your workplaces don't rely on funding from taxpayers.

jamp897 8y ago
CenturyLink isn’t exactly an underfunded provider though.
- ujvighe 8y ago
  Since you misunderstood the polite version, here's the reality: 911 failed because there isn't an option to support a competing service, instead people are compelled using violence to pay for a service that doesn't really care about their well being and is usually used to send state thugs to murder people without recourse. HN has so much bias
- deleted 8y ago
  [deleted]
- yomodsyohi 8y ago
  Comments to your post are not allowed. Wow so even with show dead I can. Barely even read the comments that the HN community has deemed improper.
- autodeadhihi 8y ago
  hN has decided that my comments are auto dead, IE, my comments are always dead. And this is my first comment. Awesome sauce!
kodablah 8y ago
> Because your workplaces don't rely on funding from taxpayers. Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.

briffle 8y ago

We used a vendor to put up two MPLS drops to every office. different providers, different routes from the building, etc. Turns out, further up the chain, an ISP was buying transit from another ISP, and they all went on the same shared fiber that a nice backhoe operator found for us, about 800 miles away.

jrwr 8y ago
Trace route can help with that somewhat, I've noticed that a few times with redundant connections
- kortilla 8y ago
  The problem with traceroute is that it won’t show that two different ISPs are leasing strands out of the same bundle. All of the unexpected shared infra issues I’ve observed have been down at this lower level that never shows up at the IP layer.
  - nickpsecurity 8y ago
    Is there a directory or service one can use to get that information?
    - chucky_z 8y ago
      In my experience this exists for ISPs, but it's not easily available. I've used ISP brokers in the past and they've had this information in some proprietary systems when I've asked for quotes and proposals.
      - walrus 8y ago
        My experience corroborates this. I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.

technion 8y ago

In Australia at least, this is becoming harder as time goes on. Several good alternatives have been purchased by one conglomerate. I know several businesses that did everything right in buying two or even three sets of connectivity from different suppliers, and now what they have are different trading names for the same company providing a service on the same network.

cynoclast 8y ago

Because they don't test failover.