5 ms·
> Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables. 30 minutes? This is absolutely mind-bog
by pen2l 7y ago
> Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables.
30 minutes?
This is absolutely mind-boggling to me. 30 minutes -- and you can identify exactly where the cut was, deep in the water, access it, fix it in the water? To me that sounds like a 30-month job not a 30-second job.
Anyone here with insight into this fix-process: could you please shed light on how this is done and how it can be so fast?
- __jal 7y agoI'd guess traffic was rerouted.
- kordlessagain 7y agoI'd guess it was done on purpose, so that it be rerouted.
- pjmorris 7y agoMy first guess would be re-routing traffic away from the damaged cable(s)
- oogali 7y agoIt’s about a 1-3 month repair job to acquire replacement cable, load it into a ship, and sail out to the break point. However, service is restored sooner than that by rerouting service over a different fiber cable or undamaged set of fibers.
- oyashirochama 7y agoThey don't replace the cable typically, they fuse it with epoxy and it usually works pretty efficiently, mostly taking the time scheduling the boat and equipment the repair is likely a couple hour job at most and 12-24 for the epoxy to dry.
- doikor 7y agoThe cable is not fully replaced. They lift the cable up to a boat and repair it (replacing the damaged bit and joining it to the original) and lower it back down. Here is an animation describing the process https://m.youtube.com/watch?v=ZJblHOGXKo4 https://m.youtube.com/watch?v=ZJblHOGXKo4
- wwarner 7y agoAmazing really that the optics at the repair site are still good enough to carry the bandwidth.
- lossolo 7y agoNo one fixed the cables, they have rerouted traffic through other links. Most of the traffic was automatically rerouted, that's how BGP works.
- walrus01 7y agoIf it took 30 minutes that's not BGP, that's some people at ISPs manually provisioning new circuits on DWDM systems, using a moderate amount of automation tools, and bringing the circuits up facing routers at both ends. Or doing something like creating a few new VLANs in existing larger 100/200/400Gbps intercity transport links, and trunking those VLANs through to routers on each end. A topology change in BGP is a couple of seconds.
- lossolo 7y agoThat's why I've specifically wrote "most", of course it's not all BGP, BGP routing algorithms do not need 30 minutes to change routes.
- iso947 7y agoDo international peers tend to use BFD?
- oogali 7y agoBGP may or may not still be involved. BGP itself has no real facility for dealing with circuit congestion. So, I am willing to bet that dynamic routing was certainly in place, but the secondary/eBGP multipath destinations were full when the primary path went down. And that necessitates manual action in instances where you aren't doing MPLS-TE, or don't have enough standby capacity, or both.
- walrus01 7y agoKeeping sufficient hot standby submarine capacity live and in contract is very costly in $/month.
- deleted 7y ago[deleted]
- walrus01 7y agoNo, it's a process of ISPs rerouting transport traffic. Onto other routes at OSI layer 1. Physical repairs of broken submarine fiber takes weeks to a month.
- rmetzler 7y agoAn event like this, where multiple fibers are disrupted and magically fixed less than an hour later just smells like the NSA, one of the other five eyes or a friendly telco installed a fiber optic beam splitter. https://en.wikipedia.org/wiki/Beam_splitter https://en.wikipedia.org/wiki/Beam_splitter
- jethro_tell 7y agoI don't have any traceroutes so this is speculation but the probably just route around it if it only takes 30 minutes. I was under the impression that a beam splitter can be installed without disruption. If they fucked up and broke the cable installing a beam splitter, a 30 minute fix is still pretty impressive.
- rmetzler 7y agoI‘m by no means an expert but I did some datacenter work. I wonder how this would be possible to install a beam splitter without disruption? Don’t you need to plug the fibers out, put the splitter in and then plug the fibers back in the splitter? I was searching YouTube but couldn’t find a video.
- Hikikomori 7y agoYou don't need a beam splitter to listen in on traffic.
- xvilka 7y agoThey bend the cables, thus allowing part of it be tracked with some residue light coming out of the fibre.
- rmetzler 7y agoI agree, this might be possible but I wonder how good the quality is then? A splitter will give you 50% light. That limits the length of the cable too, doesn’t it? How many errors will be there when you bend the cable and get maybe 15% light of encrypted traffic, and you can’t signal any retransmission?
- skybrian 7y agoRerouted to a backup, I'd guess?
- jacquesm 7y agoIdentifying the break can be done really quick: https://en.wikipedia.org/wiki/Optical_time-domain_reflectometer https://en.wikipedia.org/wiki/Optical_time-domain_reflectome... But then you still have to go there and fix it. I can't even get a pizza delivered in 30 minutes so I suspect the fix did not actually repair any cables.
- zupreme 7y agoI understand your astonishment, but if they had properly configured dynamic routing, it would not have taken that long. Sadly some international ISP’s don’t like to share (even sanitized) routing tables...