10 ms·
Google calls Drive data loss "fixed," locks forum threads saying otherwise
- tlogan 3y agoI can confirm that some files were somehow deleted in march. I suspect the problem is related to announced that 5m files is not limit anymore [1]. I was thinking that some of our team members messed up but now I feel bad. [1] https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35329135
- klysm 3y agoThat’s one way to close out those tickets.
- code_duck 3y agoI had a problem using it just yesterday. I uploaded an image using the phone app, then tried to find it and download it on my desktop. It wasn’t in the Recent list. I found it by searching, tried to download it and the website told me “You have selected a file that does not exist”.
- jacquesm 3y agoWhen I ask people if they have out-of-cloud backups of their data they look at me as if I'm mad. The cloud can't lose data. Until it does. And then what?
- donmcronald 3y agoOr what if "the cloud" is pushed as a backup solution while encouraging you to use "files on demand" to delete your local data? Hello OneDrive! Lol.
- toomuchtodo 3y agohttps://en.wikipedia.org/wiki/Five_stages_of_grief https://en.wikipedia.org/wiki/Five_stages_of_grief
- ysofunny 3y agohttps://www.buttersafe.com/2010/05/20/the-five-stages/ https://www.buttersafe.com/2010/05/20/the-five-stages/
- toomuchtodo 3y agoI concede yours is the better link :)
- wmf 3y agoLocal backups can also fail.
- ska 3y agoThat's why you need redundancy.
- aidenn0 3y agoIt's also why you need redundancy.
- bcrosby95 3y agoI prefer even more redundancy. In general, I have 4 copies of any piece of data I care about: the machine, dropbox, backblaze, and a local external hard drive.
- dylan604 3y agoIf you have multiple backups all on the same format, you only have one backup according to THE rules
- 1letterunixname 3y agoI take it you've never written DR bare metal recovery procedures or been in involved in detailed, living business continuity planning. You have zero backups if they're untested. If they're tested and separate by vendor and/or distance, if they're in an accessible medium or format, then they are different.
- dylan604 3y agoI take it you like to read into people's comments things that aren't there.
- kjrose 3y ago
- PeterisP 3y agoI think it is generally fine to not have out-of-cloud backups of data as long as you still have the primary copy locally (as opposed to data being only in the cloud), so you're screwed only if the cloud provider loses data the same day as you happen to lose your local data.
- mschulkind 3y agoAnd what happens if Google Drive accidentally loses a file on the cloud and then syncs that deletion locally?
- rsync 3y ago"... they look at me as if I'm mad. The cloud can't lose data." We used to run an ad on reddit that read something like: Your infra is on Amazon AWS and your backups are on Amazon AWS ... you're doing it wrong ..." ... and we had to stop because it made people angry. They were quite irate and combative at the very notion that there was any non-zero risk whatsoever* at AWS.
- orf 3y agoThis makes me irate, in a way that’s hard to express. They are not doing it wrong. What’s the threat model you’re saying people are not accounting for? eu-west-1 getting nuked? “AWS” also isn’t a monolithic entity: for all intents and purposes AWS Backup is a separate vendor from AWS RDS, just with a unified billing and management pane. I’d rather use that vendor, integrated with my AWS resources and managed with the same access controls, encryption, billing, etc that I use for everything else than ship it off to a random third party and maintain that connection. Because the risk factor of multiple, isolated and separate AWS teams running different products with different infra having simultaneous large data loss incidents boils down to “nukes”. So maybe people get irate in the same way as they might do with people who say stuff like “the cloud is a scam, why use it when you can host things on servers in a closet?”
- jacquesm 3y agoInteresting comment. So, what would you do if you were called by someone whose cloud account got hacked and wiped? Or whose cloud account got closed because of 'unspecified suspicious activity on your account'? Of course that never happens, right?
- crazygringo 3y agoI feel like we haven't yet come up with the right models and vocabulary to express these kinds of risks, or maybe I just haven't heard it. Especially because your credentials to AWS are likely stored somewhere that would also store your credentials to your separate backup vendor -- so why would two cloud vendors provide any more protection than two products within one cloud vendor? It's clear to me how to model hardware failure, or accidental data loss. It's not clear how to model "hackers gain access to one set of credentials but not another" or "provider closes your account and won't give you your data".
- minerva23 3y agoI like the expression, "Your data doesn't exist unless it exists in three places, and at least one of those places should be under your direct control."
- overboard2 3y agohttps://rachelbythebay.com/fun/som/ https://rachelbythebay.com/fun/som/
- SushiHippie 3y agoI like the text at the bottom of the page if you don't have javascript enabled: > Hey NoScript peeps, (or other users without Javascript), I dig the way you roll. That's why this page is mostly static and doesn't generate the list dynamically. The only things you're missing are a progressive descent into darkness and a happy face who gets sicker and sicker as you go on. Oh, and there's a total at the bottom, but anyone who uses NoScript can surely count for themselves. > Rock on!
- toasted-subs 3y agoWorking at Google, I had my cls modified without changes from my end. Managers would get mad at me for not delivering things on time, when I could say "the cat ate my homework" seems like people sabotaged my career got to the rest of Google. The worst thing knowing while my code changes were being sabotaged everybody else's seemed to be fine.
- gretch 3y agoYou're saying that your CLs were invisibly modified? Like no other author is attributed to have made the changes?
- stevemk14ebr 3y agoI don't believe you.
- pimlottc 3y agoCLS?
- Terr_ 3y ago> CL: Stands for "changelist", which means one self-contained change that has been submitted to version control or which is undergoing code review. Other organizations often call this a "change", "patch", or "pull-request". [0] https://github.com/google/eng-practices https://github.com/google/eng-practices
- FearlessNebula 3y agoGoogle uses a proprietary version control system called Piper
- galkk 3y agoHonestly that really sounds weird, especially given that there are periodic snapshots and you can restore older revisions.
- toasted-subs 3y ago
- magicalhippo 3y agoGoogle has taken some inspiration from the bastard operator from hell[1], it would seem... [1]: https://bofh.bjash.com/bofh/bofh1.html https://bofh.bjash.com/bofh/bofh1.html
- crazygringo 3y agoThis is so strange, this is exactly how you don't handle problems like this. Write a blog post, explain what happened, explain who's affected and to what extent, explain if it can be fixed and what you're doing, and explain what you'll do to make sure it doesn't happen again. Putting out a supposed hidden fix in the Drive for Desktop client, to see if it can recover files locally (?!) when the entire issue is files disappearing from the cloud, doesn't seem like it makes any sense. Or if the problem really is solely with files that should have been uploaded but weren't, and nothing from the cloud ever actually got deleted, then explain and justify that as well -- because that's not what people are saying. I don't understand what's going on at Google. If actual data loss occurred, trying to pretend it didn't happen is never the answer. As the saying goes, "the cover-up is worse than the crime". Why Google is not fully and transparently acknowledging this issue baffles me. The corporate playbook for these types of situations is well known, and it involves being transparent and accountable.
- FridgeSeal 3y agoThat’s what any normal company who is used to dealing with customers would do, but google isn’t that. Google is entirely unacquainted with the concept of “customer relations”. I’m half convinced that google-the-business sees customers as convenient peasants that purchase whatever it deigns to sell. The idea of supporting customers is basically antithetical to them: look at all the stories of people trying to get support for GCP as a great case in point.
- EarlKing 3y agoAnd this is why no one should have anything to do with Google. If the last fifteen years haven't taught people that, well, there's just no hope for you. For everyone else: Stay away from their offerings as if they were the plague.
- asdff 3y agoIts amazing how you can say as much for almost every product they have. Especially as they get more ambitious. Google Fiber. Sidewalk Toronto. The list goes on. Why believe anything they pitch to you when they've proven time and time again that they aren't likely to deliver as promised, or even at all?
- layoric 3y agoTurns out "Cloud" is someone else's computer, with the same possible issues. Saying that, I would be very surprised if the same ever happened on S3. Not saying it won't, but if the specs are to be believe, seems pretty rock solid. On a separate note, really hope the industry either makes cloud compute cheaper or starts using better priced competitors/on premisis soon, the big 3 clouds are crazy expensive IMO.
- jacquesm 3y agoInterestingly: S3 loses based on their own figures some data every year. They store 100 trillion objects and they say they are 99.999999999% reliable on an annual basis. So every year they lose some objects, but the chance that these are your objects is vanishingly small.
- layoric 3y agoAhh interesting, I thought this was a calculated reliability rate, not an actualized one.
- jacquesm 3y agoYou'd hope that with such a massive statistical base that the two would approach each other?
- klysm 3y agoWell if they’ve never lost an object then how would you go about doing that.
- tux3 3y agoStatistics can do that. "If we were only X% durable, what are the odds we would have made it so far with no data loss?" Then pick a threshold p value you like and that X% will be a reasonably confident estimate If other words if you think you have 99.999999999 but you really have 99.9, Nature will correct you rather quickly. If reality persistently lets you get away with it, you can counterfactually deduce you must have more nines.
- goles 3y agoI feel this is increasingly becoming the era of the NAS. A lot of the time I go looking for shows or movies it’s no longer offered on the same service if quite literally at all. Many of my liked YouTube videos are now just [deleted]. Not to mention any data you store in the cloud when engineer(s) experience career altering events.
- deleted 3y ago[deleted]
- crazygringo 3y agoWhat is interesting is that we've gone through 3 phases of consumer NAS: 1) Western Digital (MyBook I think was the name?) with built-in HD -- notoriously buggly and unreliable pieces of junk 2) Then just... no NAS as cloud seemed to be the answer 3) Now, Synology NAS enclosures and their custom OS (DSM) exists which is just a dream of ease-of-use If there is a new era of NAS, I'd say it's single-handedly being enabled by Synology, which is really quite remarkable.
- cheeze 3y agoI run my own RaidZ3 server at home. It's rock solid. Uses old surplus hardware and a whole bunch of 8TB drives. Synology is cool but too limited in what it can do for a homserver, IMO.
- crazygringo 3y agoI'm talking about for regular users, not the HN crowd who can build their own NAS out of anything. :)
- judge2020 3y agoMost Americans don't have enough data to warrant a dedicated NAS. And even if they do, the cost is a major dealbreaker, $400+ including drives is way more than most would ever consider spending on computer hardware.
- crtified 3y agoWhether or not Google truly 'lost data', any backup is only as good as the last verification done by the datas owner. Simply putting varying degrees of closed-source Syncing App in place long term, then expecting them to just keep working perfectly in all circumstances, through all possible future PEBKAC (either by the user or the corporation) - to the extent that they're reliable as a persons long term and only data backup - is a false expectation that the masses have bought into as part of the whole mobile/cloud world hype.
- nazgulsenpai 3y agoOr one day being locked out of your account for violating some nebulous Terms of Service with no recourse. The horror stories appear here too often.
- sterlind 3y agoAre we sure this is limited to Drive? The Ars article mentions that some users reportedly lost data without using the desktop app at all, which seems to imply that (one of?) the bugs was inside Google's infra. I wonder if they might have suffered some invisible data corruption issue in Colossus or whatever they use now, and the effects on Drive just happen to be the most visible. Though presumably whatever broke wasn't part of GCP or we would have noticed by now, right?
- gruez 3y ago> Are we sure this is limited to Drive? The Ars article mentions that some users reportedly lost data without using the desktop app at all, which seems to imply that (one of?) the bugs was inside Google's infra. Seems much more plausible that there's something wrong with the backend code for google drive (the product).
- jeffbee 3y agoSeriously. The system in question is a distributed one with multiple participants and the resolution, which is gated to specific versions of the Windows sync client, strongly suggests the bug is in that client.
- mook 3y agoThe Ars article mentions people losing things in Sheets where they didn't have a Windows client at all, though. Of course, we have inadequate intimation to determine if it's the same issue.
- RachelF 3y agoHard to tell. Google does lose data, but they aren't going to own up to it. Anecdotally, I use a different Google Drive client (Syncdocs) and haven't lost anything. However, I don't know what % of users are affected.
- chris_wot 3y agoI am increasingly wondering why anyone trusts Google for anything other than ads and search.
- _cenw 3y agonot very good at search either these days
- chris_wot 3y agoSo it's all ads now?
- ShadowBanThis01 3y agoThe more you look at Google products, the more the illusion of their "eliteness" evaporates. Shitty branding. Shitty UI. Shitty functional design. And most of all, shitty attitude. I had the luxury of telling a Google recruiter thanks, but NOPE. I had friends who are highly competent programmers with FAANG resumes treated like trash by Google interviewers, and another who actually worked there who gave detailed accounts of a toxic culture that pitted peers against each other. Google sucks. They're coasting on their entrenchment and that's that.
- NiallBunting 3y agoI'm currently battling their support. Tried opening a Play account to publish an app to the app store but missed an email to verify my identity. Now the link in the email no longer works because "Google couldn't verify your identity" as I've had "too many tries" and my account is restricted meaning I can't publish the app. Support just repeats the same things back that I've had too many tries and my account is restricted and I can't get a refund. It's woeful how bad the support is to get such a simple thing sorted out. Don't miss that email if setting up a developer play account! (If anyone can fix this my developer play account ID is: 7827257533299144892)
- beretguy 3y agoDon’t make apps. Make a PWA.
- awinter-py 3y agoproducts will get so much better if user groups can figure out how to own the bugzilla
- xyst 3y agoHaven’t used these “cloud” storage systems to store anything critical for a looong time. I was burned a long time ago when I had college assignments stored on a university backup system (I think they outsourced it to MS?). Lost a half semester of work because a stupid sys admin(s) for the university nuked the primary blob storage (instead of months old records) in an attempt to save $$$.
- computer23 3y agoI had an issue the other day where I deleted a file through the web interface and saved a file with the same name using the Windows client. The file disappeared and was never uploaded. Would that be a manifestation of the problem, or is the problem a "files at rest are disappearing" issue?
- smegsicle 3y agotrust the experts you guys
- maxbond 3y agoSo last time I saw this discussed on HN [1], people said there were tape backups of all these files and that they would be fully recovered. Some people said they worked adjacent to this tape backup system. I'm inclined to believe they weren't lying and that the tape backup system does exist. So why weren't they able to recover from tape? Is the tape backup more limited than people reported, and this data wasn't backed up? Was it just too difficult and expensive to scan the tapes and decide which was the canonical version of each file? [1] https://news.ycombinator.com/item?id=38427864 https://news.ycombinator.com/item?id=38427864
- e40 3y agoIf the number of users with lost files google may take a long time or not want to do the recovery.
- rokkitmensch 3y agoImagine. "Your data is backed up idempotently and permanently to tape, but you are too unimportant for us to ever recover your data loss from tape." Charitably, the whole system is on tape and piecemeal restoration is impossible. This is why professionals practice their restoration protocols regularly. Not to imply that it's reasonable to treat customers of commodity SaaS as professionals.
- jpc0 3y agoReading this thread more people need to learn about the 3-2-1 backup strategy. If it's only in the cloud it's not backed up, its stored. If its on the cloud and on your local PC it's not backed up, it's copied. If it's on your local PC and two independent clouds, or on your pc, on your local NAS and on the cloud it is backed up. 3 copies, two different mediums, at least one off site. In most interpretations cloud is considered a different mediu..
- RecycledEle 3y agoGoogle Drive is as broken as Microsoft Windows Search, Microsoft Windows Virtual Memory, and Microsoft Windows' transparency.
- toasted-subs 3y agoGoogle doesn't want to admit fault instead it tries to silence the dissidents. Idk seems way too off to me. Maybe we should have other organizations step in and smack the big G around until the call uncle.