3 ms·

The sad thing about this happening to Optus is that it _could_ have happened to any of the big telcos, this sort of technical cascading fault, but because it ha

The sad thing about this happening to Optus is that it _could_ have happened to any of the big telcos, this sort of technical cascading fault, but because it happened to Optus on the back of the data breach all the focus is on Optus, not on the wider lessons that ought to be learned. Would’ve been great to come out of an event like this looking at whether catastrophic fault conditions like this exist elsewhere in our national infra, but it feels like all that’ll happen is Optus gets the shit kicked out of them while other providers count their blessings

paranoidrobot 3y ago
What annoys me isn't so much that they had this outage. It's that they took so long to restore. Carrier systems are supposed to be multiply fault tolerant. Even if you have to put warm bodies physically in front of them, you should have enough of those people around to get the core networks up and going again within an hour or so by rolling back to a known-good configuration. Even if they've somehow managed to brick the systems, they should have enough hot/cold spares, and the ability to call $VENDOR to hand-deliver new ones if need be.
- simfree 3y ago
  The model for network delivery has changed. Networks like Rogers in Canada, Optus in Australia, and dish in the USA outsource the core knowledge needed to recover their network to vendors like Nokia and Cisco. The employees locally lack the knowledge and the access to restore the network without guidance from external vendors outside the country. From an operating cost perspective, this is the right choice, but for reliability and sovereignty it's terrible. Sadly, with RCS, 5G Standalone and other new technologies that require operating servers with leading edge software, operators repeatedly choose to outsource the entire stack to an external vendor like Nokia rather than replicate that knowledge locally.
  - deleted 3y ago
    [deleted]

paranoidrobot 3y ago

What annoys me isn't so much that they had this outage. It's that they took so long to restore. Carrier systems are supposed to be multiply fault tolerant. Even if you have to put warm bodies physically in front of them, you should have enough of those people around to get the core networks up and going again within an hour or so by rolling back to a known-good configuration. Even if they've somehow managed to brick the systems, they should have enough hot/cold spares, and the ability to call $VENDOR to hand-deliver new ones if need be.

simfree 3y ago
The model for network delivery has changed. Networks like Rogers in Canada, Optus in Australia, and dish in the USA outsource the core knowledge needed to recover their network to vendors like Nokia and Cisco. The employees locally lack the knowledge and the access to restore the network without guidance from external vendors outside the country. From an operating cost perspective, this is the right choice, but for reliability and sovereignty it's terrible. Sadly, with RCS, 5G Standalone and other new technologies that require operating servers with leading edge software, operators repeatedly choose to outsource the entire stack to an external vendor like Nokia rather than replicate that knowledge locally.
- deleted 3y ago
  [deleted]