14 ms·
Google Cloud: Major packet loss between multiple regions – Europe and Asia
- nanankcornering 4y agoaffecting other T1 providers too, NTT, Arelion (ex-Telia) also cloud providers OVH who have dark fibers between EU <-> Asia
- Tinkeringz 4y agoMaybe a shared undersea cable or landing point issue since it’s multiple
- njibhu 4y agoSource for OVH: https://network.status-ovhcloud.com/incidents/pphdyqq9cgyl https://network.status-ovhcloud.com/incidents/pphdyqq9cgyl Couldn't find for Arelion
- gundmc 4y agoThis link has been updated to say the root cause is a fiber cut.
- dontbenebby 4y ago>affecting other T1 providers too Then maybe the headline should be more accurate rather than give free marketing to what some might term a monopoly.
- deleted 4y ago[deleted]
- sophacles 4y agoThis was reported before it was clear that cables were cut. As with many things symptoms appeared before the cause was understood and that knowledge filter through the NOCs of the world, let alone out to users. Also, how is the headline not accurate? Did Google Cloud not in fact have major packet loss between multiple regions? How does "this service is down" count as marketing, isn't it giving people bad impressions?
- dontbenebby 4y ago>This was reported before it was clear that cables were cut. That's rude. Who did that?
- mynameisvlad 4y agoWhat is this snarky attitude? Be civil. As the parent comment mentions, the service providers are obviously going to see an issue the second it happens, but it might take minutes/hours for information to disseminate on the root cause. Just like... Practically any incident. If you immediately know the root cause, it shouldn't have been an incident in the first place.
- joatmon-snoo 4y agoI think there's a /s on the comment you're responding to :)
- dontbenebby 4y ago>What is this snarky attitude? Be civil. How am I beint uncivil? If I reply in short sentences I'm uncivil, longer ones have my words twisted and picked apart. "That's rude. Who cut the cable?" is a legitimate question and perfectly civil. If you are perceiving something as "snarky", that is on you. You cannot see into my mind. (If I didn't care, I'd just smirk to myself and not even ask who cut the cable.) Now, do you think the above is uncivil? If so why? And, as I asked previously, do you know who cut the cable?
- deleted 4y ago[deleted]
- mynameisvlad 4y agoYou actually asked "That's rude. Who did that?" after commenting "Then maybe the headline should be more accurate rather than give free marketing to what some might term a monopoly." The other interpretation of that statement, given your antagonistic first comment, is asking "Who issued the incident", as evidenced by the second half of my reply.
- nanankcornering 4y agooutage caused by AAE-1 submarine cable cut in Egypt.. well.. let's see how long it takes https://twitter.com/CloudflareRadar/status/1534201030430363650 https://twitter.com/CloudflareRadar/status/15342010304303636...
- ArtWomb 4y agoactually logged in to make a joke about how the everyday vpc hiccups we live with are becoming indistinguishable from reported downage but of course hn has more current info than the gcp team i was chatting with and if this trend of cutting undersea cables continues i'm getting a shortwave ;)
- Victerius 4y agoThe Internet really is a series of tubes after all.
- mgdlbp 4y agoBut it's now a bit stuck.
- randmeerkat 4y agoThe tubes are full.
- iso1631 4y agoAnd SEA-ME-WE-5 too apparently. I had a Wiocc outage on London-Kenya last night, didn't think much about it, maybe they rerouted from break last night, and that's why I had such big problems today If they both went together than that feels like an anchor, if one went last night and another today that's a nasty coincidence. I'm not sure which is worse.
- RedShift1 4y agoWell they are clearly not dark fibers as the cut is causing issues?
- jthrowsitaway 4y agoIndeed. Not sure why that term is misused so much. There's nothing dark about fiber that's being used. > A dark fibre or unlit fibre is an unused optical fibre, available for use in fibre-optic communication
- toast0 4y agoMore or less, someone with a cable leases you dark fiber, and then you light it. What are you using? lit fiber (obviously), what did you lease? dark fiber. When it's severed, it's not exactly lit anymore either ;)
- dpe82 4y agoBy that logic, all fiber is "dark fiber". :) As I recall, "dark fiber" came into use after the dotcom bust left lots of overbuilt and unused network infrastructure around that companies could buy up for years after for pennies on the dollar. Buying "dark fiber" in that context had meaning - it meant you were buying already built-out and unused fiber, compared to running your own fiber lines at full cost as had previously been more common.
- toast0 4y agoWell, I was joking a bit, but I would consider it dark fiber when you lease the fiber and run the equipment on both ends, and lit fiber when the owner of the fiber is running the equipment on both ends and selling you IP transit.
- sophacles 4y ago> When it's severed, it's not exactly lit anymore either ;) It literally is lit though, the lasers at both ends are still trying to send bits, or at least sending pulses to do fault location.
- NeckBeardPrince 4y agoAnother day, another huge GCP outage that impacts every region.
- davidkuennen 4y agoI don't remember having any outage so far on GCP and I'm hosting there for years now. Even now no disruption for my service (europe-west-1)
- wussboy 4y agoI can't recall a GCP outage in my 2 years using it. I can recall 3 separate AWS outages that affected one of our SaaS platforms.
- davidkuennen 4y agoYou can say about Google what you want, but GCP has been an awesome product so far and the status page game is top-notch compared to other players in the field.
- mhoad 4y agoAre you thinking of AWS maybe?
- johndfsgdgdfg 4y agoYes, exactly my thinking. Google invades our privacy and keeps users hostages for more money. We need to boycott Google at any cost.
- m1keil 4y agoI had issues between Sydney and Eu earlier in AWS.
- dontbenebby 4y ago
- nixcraft 4y agoI am pretty sure Google Ads, search core business, Gmail (workspace), YouTube, and their main web properties are not affected. Does that mean they use a different network/IDC for Google properties? Maybe all of their web properties are designed with multiple HA zones. I don't know.
- javajosh 4y agoKind of interesting to speculate what kind of traffic needs to transit and what doesn't. It would be a major efficiency failure if the services you mentioned needed to transit the atlantic for any reason. "Maybe all of their web properties are designed with multiple HA zones." I don't think it's physically possible that Google could do what it does without a comprehensive edge network.
- Chabsff 4y agoThey sure do have one: https://peering.google.com/#/infrastructure https://peering.google.com/#/infrastructure
- nixcraft 4y agoWhat fantastic info that answers my questions. Thanks for sharing it.
- deleted 4y ago[deleted]
- jeffbee 4y agoI would expect mail delivery flows cross the entire network on average because the mail arrives from places that are in no way related to where the user account is homed.
- the-rc 4y agoI did capacity planning for a large user facing service more than a decade ago. Not only we had global N+2 redundancy (we could still serve peak traffic if we lost the two largest clusters worldwide), but I also made sure we had at least N+1 per "continent". That's how I managed to get resources for us in the infamous, reserved and now long gone Budapest cluster(s), whose only other tenant was Gmail.
- 908B64B197 4y agoWhat are the odds this brings GCP closer to https://killedbygoogle.com/ https://killedbygoogle.com/ ? Nobody pays retail for GCP anyways. Always wondered how much they were losing on it.
- oneepic 4y agoIMO those odds are literally zero. Do you think AWS or Azure would get canned because of a single outage? There are lots of customers on GCP, and Google also appears to have a pretty big buy-in already -- just look at all the regions they have available in GCP.
- nknealk 4y agoThey actually break it out now in their financial reporting. It’s a loss of 931 million for the most recent quarter on revenues of 5.8 billion. Likely most of the loss stems from R&D to develop the product offering.
- asjkaehauisa 4y agoThey won't. GCP is a investment. Building DC/backbone infrastructure is a lot harder than x application. Still they need computing power to own products. Maybe in the paper they're losing money but in reality they're securing position in the market
- toast0 4y agoA packet loss incident due to cut fiber that lasted just over three hours according to the status page is not really that big of a deal. Fiber cuts happen, and are the only reliable method to determine if your redundant fibers are in the same bundle. As a non-customer, their status updates seem ok, one hour to open the incident isn't great, and transparency is near zero, but at least they provided regular updates once it was open.
- davidkuennen 4y agoSeems like it's been resolved.
- gfv 4y agoI think the issue started a bit earlier. If you'd tracerouted from Asia to Europe at roughly 13UTC today, you could see that multiple ISPs started carrying traffic the wrong way around the world, eastward through North America, instead of the usual direct undersea cable. The AMS-IX[0] traffic graph shows a non-insignificant drop too. [0]: https://www.ams-ix.net/ams https://www.ams-ix.net/ams
- hulitu 4y agoBGP hijacking ?
- Linda703 4y ago[dead]
- olliej 4y agoMaybe should be retitled? It seems to be a problem with the AAE1 and SMW5 submarine cables[1] and effects multiple providers, not just GCP [1] https://twitter.com/netblocks/status/1534211064673587204?s=20&t=Mhjr0TpyWDog4NHndCHO-g https://twitter.com/netblocks/status/1534211064673587204?s=2...
- deleted 4y ago[deleted]
- iso1631 4y agoMy links from London to Delhi (Tata) and Nairobi (WIOCC) had issues starting 12:24:21 GMT, losing leased lines and internet over the course of a couple of seconds then. Delhi internet resumed quickly (less than 2 seconds) with an extra 9ms rtt. Private wire resumed about 15 minutes later. Nairobi however remained unreachable for hours -- outgoing internet traffic to some places worked -- if I tracerouted from the office it would die in LINX, but from home it would run from London, New York, MSP, Seatle, Tokyo, Singapore, Djubouti and down to Nairobi, mostly on Hurricane Electric WIOCC tell me a fibre cut between Marseille and Djibouti. They've got my circuits back up at least, although still routing on WIOCC as far as London.