11 ms·
Please delete 2M files to continue using your Google Drive account
- ethicalsmacker 4y ago[flagged]
- crazygringo 4y agoI certainly can, but I can't imagine why anyone would have 5M in cloud storage meant for manipulation by end users. Locally you could process those pretty quick. Or in cloud block storage. But the amount of latency involved in each cloud API operation to create a new file... yikes.
- morkalork 4y agoLots of software has export to and import from gdrive features.
- bingobongodude 4y agoThis is meant and advertised for backup.
- bhhaskin 4y agoHonest question, but where do they state it's for backups?
- bingobongodude 4y agoIn all their docs concerning Android devices for example.
- crazygringo 4y agoBut not for backing up 5M files. Again, the latency per-file will absolutely kill you. Just for performance, if you insist on backing up the content of 5M files to Google Drive, you really need to write a custom script to tarball and compress whole directories on-the-fly or something. I mean, 5M files is like multiple sets of map tiles for the entire world. Sometimes people definitely need to work with those, but using cloud backup through the Google Drive API interface is just not the way to go there...
- bingobongodude 4y agoThis is not for you or Google to decide. And no, I can sync changes without having to tarball. I might only touch one of those 5M files a week actually, depending on my structure.
- crazygringo 4y agoWell of course it is for Google to decide. They're the ones running the service. ;) But again, if you ever want to retrieve all 5M files, I hope you're willing to wait a few weeks-to-months, because that's what it's going to take... Which is why it's a terrible idea not to tarball them first to something more manageable in terms of latency per file operation.
- bingobongodude 4y agoIt was not for them to decide up until now. And yes of course users were willing to wait $duration for it until now. If you are suggesting a work around that is okay. It is not okay for google to tell their users to tarball "now" "because".
- serf 4y agohearing the argument "This is for data, not for backups." strikes me as so strange. I can understand why a company would care about the access policies; if you are storing some kind of database that is accessed constantly and routinely then it's nice to know -- but when did we become okay with categorizing bits beyond that? I don't really care for the future of needing a subscription to backup cat photos, specifically. I'd rather bits be bits except in limited instances where the data is priced based upon access frequency/retention/guarantees. (yeah, it stands to reason that cold storage that has limited access should be a fair bit cheaper than hot storage where the data is ready to go ASAP.) This kind of thing just accelerates the stenography arms-race wherein people try to store their terabytes of data as cat pictures.
- bingobongodude 4y agoWhere is the difference though seriously? If it is for "data" I might be accessing it every other minute. If it is for backups I might be accessing it every other minute as well. I do look at my cat pictures every other second though and yes I do encrypt them before and after I do so. The whole spiel of cold/hot storage ..
- javier2 4y agoYou are right about the problem, but they did also sell it as a backup product.
- coreyp_1 4y agoReally? Then you lack imagination! Google Drive is a backup solution. I can think of a lot of reasons to have a lot of small files, whether they be for logging purposes, analysis, training data, etc. In fact, I can think of more legit uses than nefarious ones! Especially since an illegal use would be easy to trace back to you personally!
- yebyen 4y agoCan you think of a good reason to design a retention policy, as well? (I can!)
- bhhaskin 4y agoWhere do they state it's a backup solution? Edit: why the down votes? It was an honest question.
- dragonwriter 4y ago> Where do they state it’s a backup solution? Backup is a built-in function of the Drive desktop app (previously, “Backup and Sync” was one of two separate Drive desktop apps.) Backup to drive is a built-in core feature of Android.
- poolopolopolo 4y agoThe moment they are okay with wsp backing up things there?
- ethicalsmacker 4y ago5 million files is a lot of files. You could back up windows several times over and that’s saying a lot. What normal user has millions of typical data laying around? Not many, if any.
- TheHappyOddish 4y agoThere are billions of internet users. The "abnormal" group may be hundreds of thousands or millions, and still be < 1%.
- aobdev 4y agoFrequent backups that include node_modules.
- 0cf8612b2e1e 4y agoI was thinking similarly. Any kind of programming language environment can leave all sorts of real + cached files lying around. In the Python word, I just created a new virtual environment on Linux, and that had 1339 files. I then loaded it up with a few standard players (numpy, pandas, scipy, scikit-learn, matplotlib, dask, and django) - this ballooned the file count to 16,945.
- ethicalsmacker 4y agoYou got me, node modules would. But who puts node modules in google drive? I wish google ill but even I wouldn’t wish that on them.
- plasticsoprano 4y agoPeople use these cloud agnostic backup systems that just dump everything into the cloud account. Add version history to it and you can rack it up.
- bingobongodude 4y agoWhy would you care about reasons, if you offer a certain amounts of space? There is a size limit. Why would you implement an item limit? I have about 15M files in my Dropbox. The limit of my box is the size of my files combined AND their own size not their number. "Files uploaded to dropbox.com must be 50 GB or smaller. All files uploaded to your Dropbox must be smaller than your storage space. For example, if your account has a storage quota of 2 GB, you can upload one 2 GB file or many files that add up to 2 GB. If you are over your storage quota, Dropbox will stop syncing."
- bingobongodude 4y agoI mean I know why they want to restrict the amount. It adds computing expenses to their bill. But wow.
- crazygringo 4y agoCan I ask what the bulk of the files are? Only things I can think of are mapping tiles. Because while logs can take up space they don't generally create that many separate files...
- bingobongodude 4y agoOf course you an ask, but the answer is "no". It does not concern you, unless you are my host.
- crazygringo 4y agoI'm not asking for private data, I'm just trying to understand the use case...
- bingobongodude 4y ago[flagged]
- 4y ago
- overthrow 4y agoWhat about a business with 5M customers? Storing one file per customer seems reasonable.
- yebyen 4y agoIf you're managing this in a free Google Drive account, shame on you (seriously)
- upon_drumhead 4y agoThese are paid, large plans. 5 million files on a 20tb plan seems unreasonable.
- bingobongodude 4y agoWhy would it seem unreasonable?
- maccard 4y agoThat's 4MB/file, which is not exactly large. That's also a small use case. 20TB is 4 people on Google Workspace. Our company is 40 people, so our limit is 200TB (which we pay for), but 5 million files.
- thombat 4y agoThe issue is common to all account types, ie also affects paid accounts.
- overthrow 4y agoPaid accounts are limited too. https://issuetracker.google.com/issues/268606830?pli=1 https://issuetracker.google.com/issues/268606830?pli=1 > What it should say is that no matter how much you pay you have to adhere to the 5m file limit.
- bingobongodude 4y agoThis is not about the free tier.
- radicaldreamer 4y agoWhy should anyone have to think about number of items at all? For most of computing history, the only thing people really care about is storage space, not items stored...
- Symbiote 4y agoNoticeable limits on item count were fairly common until around the mid 1990s. FAT16 has a limit of 65460 total files. FAT32 limits 65535 files in a directory. The RISC OS ADFS filesystem had a limit of 77 files in a directory; I used this until about 1994.
- treesciencebot 4y agoThat is the part that baffles me. There might be an argument against when this becomes unsustainable (e.g. the number of bytes inside a file is actually smaller than the number of bytes the metadata associated with that file takes) but that can be solved quite intuitively by making user pay for it (essentially they should be counted towards the same usage limits, if they aren't already?).
- ethicalsmacker 4y agoBecause even a zero byte file consumes space and if you give people the option someone will create several petabytes of zero byte files.
- akvadrako 4y agoThat isn't really true. Not only do most filesystems give you free byte counts, they also give you free inode counts, which are like number of files. There are also limits to number of files in a directory.
- serf 4y agoa brand new minimal install of Ubuntu MATE hovers around 700,000 files. If you can't imagine how that might inflate, I don't know what to tell you.
- crazygringo 4y agoWait, really? Coming from Windows and macOS that boggles my mind. What on earth are the main contributors to that file count? I can't even begin to imagine, but I'm incredibly curious now. Also, that seems like an enormous waste of disk space. Assuming a 4 KB block size means roughly an average of 2 KB of space is wasted per file, which for 700K files would total well over a gigabyte of wasted space. Yikes.
- pxoe 4y agowell, it's not like windows is too far off - on my w10 install, c:/windows/ comes up to about 280k files. (and in total, that small-ish system drive, 40 gb, has about 380k files on it. i'd imagine macos has numbers that go something like that as well, since it is a *nix system.
- marwis 4y agoMany filesystems, ext4 included, support inlining of small files directly in filesystem metadata to avoid such waste.
- dmux 4y agoRelated thread from a couple of days ago: "5M item limit for Google Drive: File unable to generate or upload due to 403" https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35329135
- thevagrant 4y agoDoes Google not have a smart layer that compresses certain rarely accessed files into an archive (hidden from the user)?
- bshipp 4y agoProbably not, but as soon as I get back to my laptop I'll queue a server containing rclone to haul individual files into an archive, squish them, and upload the zip file.
- iameli 4y agoAny of y'all worked at Google? I'm fascinated by the internal processes that would lead to something like this happening with no warning for any of their users. Everywhere I've worked before someone would be like "hey we should probably send an email at least"
- riceart 4y agoWhat’s the incentive? None. There is no internal process. You’ve been working for a company that has had for its entire history a cash cow that essentially has no customer facing contact for its survival - you wouldn’t think to go there. Keep google search and YouTube running, don’t fuck up chrome too much - everything else is basically unimportant.
- sneak 4y agoYou forgot Maps, Android, Gmail, and GCP, each of which is a multibillion dollar company on its own.
- macintux 4y agoIs Android turning any significant profit? I'd be surprised.
- AlotOfReading 4y agoI imagine the licensing revenues are a literal rounding error, but it's hugely profitable if we include things like Play services. The latest numbers I could find accounted for $48B in revenue in 2021, compared to $256B for all of alphabet. It's pretty substantial.
- sowbug 4y agoEven if it doesn't directly bring in a dollar in revenue (let alone profit), I suspect there's significant value in Android running on 3 of every 4 phones worldwide.
- 4y ago
- justinator 4y agoAnybody have luck deleting your entire Gmail inbox? I have a test account that's over it's limit, and I can't delete enough emails in a day to get it safely under. Selecting ever single email message in the inbox and then hitting the trash can just fails. Is there an actual nuclear option?
- mastax 4y agoThere's probably an email client that could do that. If the SMTP endpoint won't allow it there are email clients that talk the Gmail API directly. You could probably ask ChatGPT to write a program using the Gmail API to delete all the emails in your inbox.
- justinator 4y agohn is my OG ChatGPT*. Usually I just offer a wrong answer using my spare account and someone will chime in with a better answer. *(this post is satire)
- a3n 4y agoI think you are ChatGPT﹡. I think you posting the wrong answer is Step N-1. Step N is when you, ChatGPT, harvest the better answer, package it, and send it to your human prompter, or to the other instance of ChatGPT that prompted you. ﹡(this post is also satire)
- dnadler 4y agoI had to deal with this a while back and I recall finding some Google scripting thing that allowed me to delete chunks of emails at a time. I don’t recall exactly what it was though :(
- cjbgkagh 4y agoDelete via IMAP protocol using libraries in probability just about any programming language.
- tyingq 4y ago
- ajb 4y agoGoogle Drive lost all our HR files, everyone's contracts etc. Huge panic. We only got them back by using Vault (the legal discovery back door, which doesn't return them in the same format or folder structure). And yes, no one deleted them - I checked the audit logs. If we weren't firefighting so bad here I would be pushing for us to switch.
- V__ 4y agoI'm not judging, but I just don't understand why someone would save business critical data to Google Drive only. Is this a US thing? Aren't you afraid of privacy or trade secret leaks and potential legal ramifications?
- ajb 4y agoI'm not in the US. One of the consequences of SAAS being easier to use to than old school local IT is that these decisions are often made by the user of the data directly, for whom these are often not the most salient risks. In the old days you couldn't really operate beyond a few people without a sysadmin who would know that they would be fired if something like this happened and so would be much more keen to make sure long tail risks are taken care of. With SAAS it's difficult to persuade people that we need to pay for a third party backup service when the SAAS does their own backups for free. Regarding privacy or trade secret risks, Google provides adequate permission mechanisms for that if operated correctly. For example, HR have their own shared drive which other users can't see. Threats like Google itself getting hacked or deciding to look at our docs are basically in the "act of God" category for our business.
- traveler01 4y agoAt this point I’m astonished people still trust in Google services. The fact that this targets even paid users…
- andrewmcwatters 4y agoI remember looking for some saved emails on GMail once only to realize that at some point, emails I tagged as Saved were just gone, missing, one day beyond a particular point in time. If you don't own your bytes, someone else does, and they can and do go missing. Everyone has to buy some storage sometime. It's super naive to think companies and organizations are "professional" and don't just straight up lose things. It's just life. Did you know GitHub has an error page for when they cannot retrieve a repository from their database? Yeah, your emails can just go missing. Your repos can just go missing. I shudder to think someone's deed to their home somewhere out here in the states has at some point in time just straight up gone missing, but I would not be surprised. There's an accredited university in New York that's closing soon. All of those people's degrees--how will you verify you actually graduated from there? I guess you had better kept copies of your transcripts and your physical diploma. Dust to dust, ashes to ashes.
- mrzimmerman 4y agoFirst: Friendly reminder to everyone to back up your data and then back up your backups to a separate location. Second: I’m the person in my family who wakes up in a cold sweat worrying if our photos are backed up properly, or my music collection. I’ve got a handful of backups at the house that I double check periodically to make sure they haven’t decayed, and some stuff backed up to some cloud services. I’ve been talking to some family members and close friends about building some simple home lab raid devices that we can use to back up our stuff locally and to one another’s locations. I’m not enough of a network engineer or server admin to know the best way to set it up, but I’m hoping we’ll figure out a simple, cost effective way to handle it. I’m hoping it’s a fun project either way and we can all avoid entropy for just a bit longer.
- rendaw 4y agoHow do you back up emails?
- pfarrell 4y agoI use google take out to do a dump of email and back up the download file. In order to clear out a bunch of space on my gmail account, I did a full export and then deleted conversations based on date. I ended up having to use a script which ran over a few days, due to limits on how fast I could access messages. This was all to avoid signing up for a paid account. https://takeout.google.com/ https://takeout.google.com/
- jonathantf2 4y agoJust because it’s in the “cloud” doesn’t mean it’s backed up, you’re still vulnerable to malicious and accidental deletion, corruption etc.
- Reason077 4y agoA 5 million file limit might be quite reasonable if you're paying for the basic, 100 GB storage tier. But Google Drive offers up to 30 TB of storage. 30 million MB! That means if your average file size is less than 6MB, which is very likely if you're storing JPEGs, audio files, text records, or whatever, you'll never be able to fill your Google Drive storage. Surely the file count limit should scale with the storage limit?!
- AwaAwa 4y agoWhen you use cloud services, uninformed and unscheduled changes in pattern are simply par for the course.
- mcculley 4y ago> “We have a business critical operational system in the animal health space which is currently affected by this.” I am astounded that people put anything more important than cat photos on Google Drive.
- deely3 4y agoLooks like victim blaming. > I am astounded that people put anything more important than cat photos on Google, Apple, Microsoft, Linux, Unix, general cloud provider, mobile phone, HDD, PC, etc. etc. Also, sorry, but I have to say: there nothing wrong with caring about cat photos.
- mcculley 4y agoI do feel bad for the victims. I am still astounded. I run two businesses. It never would have occurred to me that Google Drive is something I would use for operational data. [Edit: I get the impression my original comment was interpreted as snarky. I am 51. I did not grow up in the world of ubiquitous Google Drive. I have only heard how hard it is to get customer support out of Google. I am genuinely surprised that people rely on Google Drive and other Google services for business. This has me wondering if there is a market opportunity to sell more serious services to businesses that never thought about operational reliability.]
- perryizgr8 4y agoEvery company that uses GSuite probably uses Google drive as their shared file storage, which obviously includes a lot of files critical to company's operation. What else would you expect?
- mcculley 4y agoIt also had not occurred to me that serious businesses use GSuite. I cannot imagine depending on Google, given their history. I realize I am out of touch. I remain astounded. Market opportunity!
- 4y ago
- maximinus_thrax 4y agoI cannot imagine a business using Google at this point. I'm not saying Google is not customer focused, they actually appear to show actual hostility. It's amazing people still think it's a good idea to keep critical stuff on Google infrastructure.