31 ms·
Google Cloud Europe service disruption
- bugsense 3y ago[flagged]
- deleted 3y ago[deleted]
- xrayarx 3y agoFrom what can be seen on the Status page, more than just Europe seems to have a problem.
- jostein 3y agoI think their status page briefly showed more regions affected, but I have not noticed any problems in europe-north1 or europe-west1 where I have systems running.
- xrayarx 3y agoWater intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region https://twitter.com/GCP_Incidents https://twitter.com/GCP_Incidents
- belter 3y agoHow large is the flood? How far away are these zones?
- wut42 3y agoIt happened at GlobalSwitch Clichy, near Paris. From what I gathered from a french forum[1], it started with a flood and then a fire. No rooms have been affected, apparently. [1]: https://lafibre.info/datacenter/incendie-maitrise-globalswitch-clichy/ https://lafibre.info/datacenter/incendie-maitrise-globalswit...
- bilekas 3y agoI'm getting horrible flashbacks of OVH DC's those many years ago.
- Jamie9912 3y agoTHAT was many years ago? Felt like yesterday
- rpadovani 3y agoA bit more than 2 years: https://www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/ https://www.datacenterdynamics.com/en/news/fire-destroys-ovh... I thought it was less than 12 months...
- bilekas 3y agoIt does feel like yesterday but yeah was a couple years back !
- mablopoule 3y agoWe were impacted at a previous company, luckily we had solid backup, so everything was back online a few hours after. Still, it was kinda fun to go to work and learn that the corporate website literally went up in flames.
- BonoboIO 3y agoWhat a disaster. A datacenter made out of wood, what could go wrong ...
- alibert 3y agoCooling pump failure lead to water leaking into the UPS room. Batteries caught fire and firefighters can't access the room. Fire is contained although. (this was at ~10-11 am GMT+2 time) Edit: Fire is extinguished (~3pm GMT+2) https://www.mail-archive.com/frnog@frnog.org/msg72320.html https://www.mail-archive.com/frnog@frnog.org/msg72320.html https://www.mail-archive.com/frnog@frnog.org/msg72323.html https://www.mail-archive.com/frnog@frnog.org/msg72323.html https://www.mail-archive.com/frnog@frnog.org/msg72327.html https://www.mail-archive.com/frnog@frnog.org/msg72327.html
- aeyes 3y agoBut europe-west9-a is only one zone, why does the whole region fall over as a consequence?
- throwoutway 3y agoProbably some dependencies they did not plan for
- londons_explore 3y agoSwitching off all of 1 zone and checking the others aren't impacted is literally step 1 of checking your organisation is truly zonally redundant... Someone as big as Google ought to have been practicing this automatically every week in a staging environment, and probably at least annually in production.
- bushbaba 3y agoGCP has multiple zones in the same physical building. Not all cloud providers have distinct physical buildings for each Availability Zone.
- aeyes 3y agoDo they have an official description what a zone is somewhere? Back in the days when we had our own data centers a zone was defined as a "fire section" meaning that it should not be impacted if any other zone of the data center had a fire. This obviously means that you can't call 3 floors of a building a zone. Edit: The information on this site https://cloud.google.com/docs/geography-and-regions#regions_and_zones https://cloud.google.com/docs/geography-and-regions#regions_... clearly states that a zone is "physically distinct" so they have some explaining to do. Edit 2: Sneaky... They changed the status page to say "europe-west9" instead of "europe-west9-a".
- bushbaba 3y agoPhysically distinct could refer to distinct hardware in the same building and cage space. It’s “physically distinct”. Google makes no promises that the zones are in different buildings or separated by N feet/miles of space.
- eodafbaloo 3y agoIt seems that it went back to normal except for Paris
- sgt 3y agoIs that so? All of them are showing network alert.
- p_l 3y agoThere's network alert on the whole of EU because one of the regions in EU is out. A case of "need to drill down"
- skrebbel 3y agoI love how well google cloud status reflects real life.
- rippercushions 3y agoAt least Paris is red, what's the last time you saw more than a green dot with a little exclamation mark on the AWS status page?
- skrebbel 3y agoI was referring to how things being “back to normal except for Paris” is a common occurrence here in Europe
- londons_explore 3y agoThey should just stick a big "on strike" banner across the status page...
- base 3y agoService Health Status: https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
- deleted 3y ago[deleted]
- rippercushions 3y agoTitle is incorrect, this is not a general outage. There are two separate issues: europe-west-9 (Paris) has been physically flooded with water somehow and is hard down. This is obviously bad if you're using the region in question, but has zero impact elsewhere. https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf... There is a separate issue stopping changes to HTTP load balancers across most of GCP, but it has no impact on serving and they're rolling out a fix already. https://status.cloud.google.com/incidents/uSjFxRvKBheLA4Zr5qHs#oKGDLmoneB3YgW79P7vm https://status.cloud.google.com/incidents/uSjFxRvKBheLA4Zr5q...
- bsdz 3y agoI'm not sure if it's a separate issue but I've had trouble creating new VM instances in Google Cloud Console or listing GPU types using their CLI and I'm in europe-west-2. The ticket I was following originally got merged with the Paris flood ticket (by Google). It was working until midnight (London) last night but went down before 8am before recovering about 1h ago for me. Not sure why an outage at one regional data centre can affect services elsewhere in the zone. Perhaps it's when pooling together meta data from different data centers for listing options?
- aristus 3y agoAlso, consider everyone either automatically or manually trying to make up for the lost capacity in eu.
- martius 3y agoCloud Console is having issues related to the outage in europe-west9 > Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY#73mBtVKKfeJGJ1yaY7hV https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
- zerd 3y ago
- Takennickname 3y agoI thought this title meant cancelled I literally felt the blood leave my face
- cal85 3y agoI thought it meant “now available” and was surprised it wasn’t already.
- iainmerrick 3y agoYes, it would be less confusing to say “down”.
- brookst 3y agoOr just "released, down, and canceled" to tighten up the news cycle.
- LadyCailin 3y agoI did too, and was not at all surprised.
- belter 3y ago[flagged]
- nonethewiser 3y agoI read it that way too. Not sure why. Maybe the difference between out and down.
- tvanantwerp 3y agoThis was my first thought too. Shows how Google has trained us to expect the worst from them...
- davidkuennen 3y agoTBH, GCP isn't operating like Google does. I'm a long time customer and have only good things to tell so far.
- fukawi_tribe 3y ago[flagged]
- orwin 3y agoWell, actually... The firemen flooded the DC. The air conditionner stopped working at around 4 PM from what i got, the firemen were called at around 5, then once arrived asked for a shutdown and put off the fire with water. Lot of it.
- golemiprague 3y ago[dead]
- femtozer 3y agoIt seems that there is a second issue due to a fire in a GlobalSwitch datacenter where Google host Edge cache locations (article in french): https://dcmag.fr/breve-un-depart-dincendie-dans-un-batiment-de-global-switch-a-clichy/ https://dcmag.fr/breve-un-depart-dincendie-dans-un-batiment-...
- nielsole 3y ago> Water intrusion in europe-west9-a > We expect general unavailability of the europe-west9 region. Why would emergency shutdown of a single AZ lead to general unavailability of a region? Isn't that the point of multiple AZs? > There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage yikes
- numbsafari 3y agoFrom other comments here, it sounds like multiple zones in that region are located in the same datacenter? If so, that's ... not good.
- hf_twink 3y agothat’s how GCP does zones, firewalled off with separate networks/power in the same physical location.
- mschuster91 3y agoOuch. Isn't part of separate zones being protected against something, say, like a terrorist attack or a natural disaster that can take down a whole datacenter?
- anonymousDan 3y agoAFAIK if you dig into the details, the different cloud providers have very different concepts of what constitutes an AZ with respect to the types of faults that are isolated.
- lastangryman 3y agoI always felt a bit scammed with AWS Multi-AZ on RDS that basically doubles your cost. If their set up is anything like this, I now feel vindicated in turning it off....
- perfmode 3y agothe flooding meant to provide cover for intelligence services to infiltrate? have i been watching too much espionage media?
- dannyw 3y agoGoogle is an intelligence service, more or less.
- deleted 3y ago[deleted]
- testemailfordg2 3y agoWater intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage. Customers are advised to failover to other regions if they are impacted.
- KomoD 3y agoTerrible title, can it be changed?
- drag0s 3y agoLooks like it is/was not only Europe. Today we had issues in the US too, and some other regions still affected as well (us-east1, asia-northeast1, asia-south1 & asia-southeast2)
- bagacrap 3y agoSeems related: https://dcmag.fr/breve-un-depart-dincendie-dans-un-batiment-de-global-switch-a-clichy/ https://dcmag.fr/breve-un-depart-dincendie-dans-un-batiment-...