12 ms·
Google loses data as lightning strikes
- jneander 11y agoThis could potentially explain a lingering error on my Google Drive. Might there be movement within Google to contact the owners of data that was lost?
- Filligree 11y agoThe article mentions Compute Engine, which is an external offering that isn't really used internally. It's hard to say what else might have been affected, if anything, but going by the article this shouldn't be causing your problem.
- jneander 11y agoFair enough. One can hope.
- swah 11y agoFile syncing is a hard problem..
- JupiterMoon 11y agoOnly because we make it hard.
- AnimalMuppet 11y agoIf you've got a way to make it easy, we'd like to hear it.
- kuschku 11y agoSimilar to git, but if you have multiple branches of the same file, display them visually and allow the user to merge just that file?
- xeromal 11y agoIf people have to do anything it's not good enough.
- kuschku 11y agoThat is then mathematically impossible. What if I edit the same pixel, in the same image, in the same nanosecond on two different devices, and then sync? There are situations where conflicts are necessary. The focus should be on making conflict resolution easily accessible, not on trying to be smart and overriding files at random
- JupiterMoon 11y agoActually quite a lot of serious by non-power-users do want this. You only have to look at a random persons laptop and see and the file names with "final version.docx", "final version 2.docx" "final final version.docx" etc to see this. They certainly don't want a console based solution but a cross between git and word style track changes would make lots of people pretty happy.
- JupiterMoon 11y agoWell image if file systems, file formats and version control systems were built from the ground up together. Currently file syncing is harder because we expect our file server to handle an infinite variety of legacy file formats and the file sever cannot try to know anything about the document structure (whereas a version control system can examine the contents of plain text files). Surely this is why Google docs can manage collaborative working more easily -- because they can control of the whole stack. Better than the OP would be file syncing of arbitrary files between arbitrary systems is hard.
- jankeymeulen 11y agoCan't say for sure but I'd doubt it. (Disclaimer: I work there but have absolutely no knowledge about how Drive stores data)
- nkrisc 11y agoThe odds of it being your data is probably the same as you being struck by lightning four times...
- mathattack 11y agoBeyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.
- mankyd 11y agoThe article states that it was "some recently written data [...] located on storage systems which were more susceptible to power failure". Perhaps it hadn't made it to a backup.
- ismavis 11y agoYou find more details in the Google incident report: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
- bsurmanski 11y agoAs mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server. Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info. For 99% of cases (and pretty well all user cases), this would not cause data lose. The key here is that the data was generated on the servers and did not have a chance to duplicate before the event. [0] http://research.google.com/archive/bigtable-osdi06.pdf http://research.google.com/archive/bigtable-osdi06.pdf
- miander 11y agoSounds like this wasn't caused by power surges reaching the equipment but rather an effect of repeated power loss to drive arrays not fully designed to handle it. The article is pretty unclear. Still sounds like an infrastructure problem though.
- magicalist 11y agoThe incident report ismavis posted below (and linked in the article) has far more information: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
- deleted 11y ago[deleted]
- darkr 11y agoAFAIK google doesn't use hardware-based arrays in their servers. I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as 10 seconds), which gives the diesel power generators enough time to come online and take over from there. What I guess is that might have happened is that grid power was lost, they switched over to UPS fine the first time, grid power came back so they swapped back, repeat a few times and on one of those times the batteries didn't have enough charge to keep things going for the generator swap-over.
- thrownaway2424 11y agoNope. http://www.buildablade.com/images/Google_data_centers-1.jpg http://www.buildablade.com/images/Google_data_centers-1.jpg
- dekhn 11y ago
- circa 11y ago"Lightning crashes, an old server dies...."
- jmartinpetersen 11y agoOh, I see you're Selling the Drama ...
- impostervt 11y agoCurious - how do they know lightning hit four times? Was someone outside counting?
- sz4kerto 11y agoBuilding power management systems can detect this.
- cdr 11y agoAs sz4kerto said, it was probably tracked by building systems. But beyond that, weather services track every single lightning strike in most of the developed world. Eg BELLS: http://radar.meteo.be/en/3337408-Lightning+detection.html#ppt_10819449 http://radar.meteo.be/en/3337408-Lightning+detection.html#pp... This is how, for example, you can know whether a wildfire was started by lightning - once a point of origin is determined, simply check the data for strikes. https://en.wikipedia.org/wiki/Lightning_detection https://en.wikipedia.org/wiki/Lightning_detection http://www.lightningmaps.org/ http://www.lightningmaps.org/
- larrys 11y agoCould also be security cameras in addition to what others mentioned.
- chinathrow 11y agoYes, lots of folks are counting. http://www.blitzortung.org/ http://www.blitzortung.org/
- keithpeter 11y agoThanks very much for posting this link. Could be really handy for physics teaching next year.
- chatmasta 11y ago> Google said that just 0.000001% of disk space was permanently affected. Assuming 1 petabyte of total storage at the datacenter, that equates to about 100mb. I wonder how much storage they have there.
- sbecrab4 11y agoAssuming the wild 1PB guess above... that could be anything from 100mb of a single volume lost, all the way up to 100 million volumes each losing one byte of data. There isn't enough information to determine the number of volumes affected. 0.000001% of disk space isn't very informative. Publishing the percentage and/or number of persistent volumes affected would give us an idea of the scale of the problem.
- ucaetano 11y ago0.000001% of Persistent Disk space, which is probably a small fraction of the total datacenter storage.
- decker 11y agoI think your estimate is too low by 1000x. 1PB is 500 2T drives. There's no way Google would run a data center with only 25-250 hosts.
- bentpins 11y agohttps://what-if.xkcd.com/63/ https://what-if.xkcd.com/63/ XKCD guessed 15EB in 2013
- atlbeer 11y agoThe real clouds are getting angry at companies misappropriating their name
- lgleason 11y agoIn other news, real clouds file lawsuit against cloud providers for trademark and copyright infringement. :)
- idlewords 11y agoRelevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", as if you will be making repeated ten-million-year trials of this effectively brand new technology. The point is that once you get into several nines of reliability, really rare events that are impossible to model start to dominate your risk budget.
- hyperpape 11y ago"just 0.000001% of disk space was permanently affected." So Google just exhausted their 11 9's for centuries to come.
- MichaelGG 11y agoWhich would be embarrassing if this had happened to a product offering such an SLA. Instead, this is just zone-local HDD storage. It will only affect customers that decided to run a system in a single site, making no snapshots or using cross-region replication, etc. Which may be quite a few seeing as there seems to be this misunderstanding that VMs "in the cloud" have physics-defying redundancy attributes. (Not saying you have this idea, just that multiple clients have expressed surprise when their single-VM setup goes down because "isn't this what the cloud is for?".)
- deleted 11y ago[deleted]
- ck2 11y agoI sense a future Mr. Robot plot idea.
- necessity 11y agoAnd magnets! Everyone loves magnets.
- deleted 11y ago[deleted]
- vanderZwan 11y agoWouldn't successive lightning strikes be more likely due to ionisation of the air?
- xvedejas 11y agoWell, there's the opposite argument: "Wouldn't successive lightning strikes be less likely due to reduced local electric potential?" It's not readily apparent to me which of the effects will be larger.
- rtkwe 11y agoThe volume of air ionized by a lightening strike is small and dissipates quickly because it's so hot.
- Florin_Andrei 11y agoAssuming they don't deplete the charge too much, maybe.
- gdulli 11y agoI've lost attachments to old gmail messages before, I never thought it was impossible or unlikely for Google to lose data. I'm sure the data wasn't truly lost. If I'd called them up and they'd made it their priority to find my old files, they could have done so, having so many redundant backups. But of course no one at Google is taking calls like that or acting on individual requests. The data was effectively lost, not technically lost. But I'm sure it's uncommon.
- calyhre 11y agoGlad to be part of the 0.000001%. We had a tough night because of this outage :(
- linkregister 11y agoGoogle's actual statement with RCA: https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
- ishanr 11y agoThats why you should always write it down folks...
- Aloha 11y agoI work on cell sites, grounding system design and repair is a primary design element, even then, the presumption in the industry is that if a site takes a direct hit - or for that matter a nearby strike - the equipment is a total loss. The surge suppression gear we put in (lead ins at power feeds, RF feed, etc) is mostly to prevent a fire and to ensure the extra energy goes largely to ground.. but it won't prevent dead gear.
- logicallee 11y agoAre you saying essentially that "There is no such thing as a surge protector, they don't physically exist. Only surge reducers exist." Because that's what it sounds like to me. EDIT: All right, I'll rephrase. According to Google's infobox from nat'l geographic, lightning generates up to 1 billion volts. -> Are surge protectors at even the highest-end data centers simply not rated to a billion volts of surge protection?
- PhantomGremlin 11y agoThere's a "protector", but the protection isn't guaranteed to be 100% effective. That's an OK use of the language in this context and has many parallels in other fields. E.g. using a condom as protection against pregnancy and disease. Many people have learned the hard way that it's not 100%.
- Aloha 11y agoEffectively yes, when you're on lightning scale, yes. I use an isobar myself to power reduce noise, but I'm under no illusion that it will protect my shit from a direct strike.
- apparitions 11y agoWow. NSA attacks are really leveling up.
- djhworld 11y agoDo they know what customers were affected? I use google drive a lot, I don't track what's in my drive, should I be worried?
- bskap 11y agoI think it only impacted people who were using Compute Engine and storing data on a hard drive instead of using Cloud Storage. I suspect that Drive data has redundant copies stored in multiple data centers in addition to frequent backups.
- toocute2care 11y agoThe wrath of the Gods.
- tedchs 11y agoKey point from Google's status incident page that clarifies some of the partially-incorrect statements in the press and comments about this: "In a very small fraction of cases (less than 0.000001% of PD space in europe-west1-b), there was permanent data loss." https://status.cloud.google.com/incident/compute/15056#5719570367119360 https://status.cloud.google.com/incident/compute/15056#57195...
- sbecrab4 11y agoQuoting the data lost as a percentage of disk space is both accurate and misleading. It makes the impact sound tiny because only recent writes were affected. Obviously writes that were in flight at the time of the incident are going to be a tiny percentage of overall storage. What they don't tell us is what percentage of persistent disks which were in use at the time were affected. That percentage is likely far higher. If only 0.000001% of volumes in use were affected it would never have made the news.
- ai_ja_nai 11y agoSo Google has no redundancy at datacenter level?
- wmf 11y agoNot for block storage (aka persistent disks): "we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data." Achieving RPO=0 generally requires synchronous replication to a different datacenter which adds significant latency.
- pacala 11y agoAmazon builds datacenter redundancy in the same geographic locale. You can then setup synchronous replication between datacenters without atrocious latency, as all the disks are fairly close by, albeit powered [in an emergency] by independent generators. OTOH, Google does datacenter redundancy across different locales, which make synchronous replication perform much worse, like you noted.
- wmf 11y agoGCE appears to have a similar region/zone structure as Amazon, but neither provider replicates block storage across zones.
- pacala 11y agoI stand corrected. Looking at https://cloud.google.com/compute/docs/zones?hl=en https://cloud.google.com/compute/docs/zones?hl=en, it seems that GCE has multiple zones in the same locale (region). I might have confused with Google's internal setup, which during my tenure there, was heavy on using replication across locales, up to systems like Megastore and the 200ms synchronously replicated commit.
- iradik 11y agoAt Amazon, when resolving an issue in our internal ticketing system I recall there being "Act of God" as a reason code. Seems applicable here.
- JupiterMoon 11y agoI think this is a technical term in the insurance world.
- danepowell 11y agoThis raises a relevant concern that's been on my mind: what's the best way to back up cloud services? Given that services like S3 and Google Drive have many more nines of durability than any local storage system I could devise, are backups even worth the trouble? There are a lot of cloud-to-cloud backup services out there, but to me that seems like the blind leading the blind, especially with regards to malicious data destruction. For instance, I've recently been experimenting with Cloudally to automatically back up Google Drive, which seems like a good solution at first- until you think about the fact that Cloudally uses Google accounts for authentication (and doesn't use 2FA for native authentication). In other words, an attacker with access to my primary data (Google Drive) would also have access to my backups. Worse than that, Cloudally actually increases the attack surface, since its lack of 2FA presumably makes it easier to crack than my Google account. Similarly, I'm guessing a lot of cloud backup services share data centers with the services they are backing up.
- njharman 11y agodurability is not same as "protection". It doesn't include items like government seizure, hacking. Your own drives, in your physical possession fill in that gap (somewhat).
- nemo1618 11y agoIf you really care about durability, your best best is erasure-coding + a wide geographic distribution of shards. For example, you could encode 1 TB of data into four shards, each shard containing 500 GB. You distribute these to servers in SF, NYC, Berlin, and Sydney. The key here is that you only need two shards to recover your 1 TB of data, and they can be any two shards. So if lightning strikes Berlin, and the Big One hits SF, your data is still safe. And thanks to erasure-coding, you can achieve this with only 2x redundancy (instead of 4x).
- mirimir 11y agoIt's been a while since I followed this. I see over 100 Reed-Solomon erasure-coding projects on GitHub. Which would you recommend?
- rcthompson 11y agoIt sounds like they mostly lost recently created/stored data that hadn't yet been fully replicated to the required degree of redundancy.
- upbeatlinux 11y agoThe cloud strikes back.
- pliu 11y ago"The Google Computer Engine (GCE) service allows..." Did anyone else cringe?
- marcelocamanho 11y agoZeus killing some porn
- okadaka 11y agoSo Google said: "...although... the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain. In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk." I thought battery is supposed to cover writing the entire write buffer cache to disk in case of power loss. Sounds like they had some badly designed gear which did no account for partial battery charge which should downsize the cache to battery's capacity.
- optimum75 11y agoYep! Sometimes shit happens. That's why I am using Spinbackup to back up my Google data to a secure storage.
- optimum75 11y agoYep! Sometimes shit happens. That's why I am using Spinbackup to back up my Google data to a secure storage.
- sas_gbi 11y agoYes, shit happens. I personally backup my Google data with Spinbackup - a cloud-to-cloud backup and recovery solutions provider.