16 ms·
Google Drive does a surprise rollout of file limits, locking out some users
- chrisbolt 4y agoPrevious: https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35395001 https://news.ycombinator.com/item?id=35395001
- dang 4y agoThanks! Macroexpanded: Please delete 2M files to continue using your Google Drive account - https://news.ycombinator.com/item?id=35395001 https://news.ycombinator.com/item?id=35395001 - March 2023 (109 comments) 5M item limit for Google Drive: File unable to generate or upload due to 403 - https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35329135 - March 2023 (133 comments)
- bigiain 4y agoHa! “ a safeguard to prevent misuse of our system in a way that might impact the stability and safety of the system." Google: We have identified modern web development as a threat to our systems, and have taken measures to ensure npm users cannot store their npm_modules directories on GoogleDrive. Please consider rewriting your nodejs projects in Go.
- kube-system 4y agoWhy do people put their NPM modules in google drive?
- drusepth 4y agoThere's a surprisingly large number of groups of people that use Google Drive, Dropbox, and similar for "source control" and/or collaboration on projects instead of real VCS.
- lazide 4y agoThat seems really illadvised.
- nickphx 4y agoeveryone is differently abled.
- bigiain 4y agoCynical view, a lot about npm seems really ill advised. One additional bad decision after that choice seems easy enough to go along with… ;-)
- tomjen3 4y agoA lot of it is. And yet, it works so well it conquered the world.
- bigiain 4y agoSo did Mc Donald’s and Starbucks. I’m not entirely happy with the idea of living in a world conquered by the lowest common denominator solutions.
- listenallyall 4y agoGet over yourself. McDonald's is far from "lowest common denominator." Go travel the world a bit and see how real people live and what they actually eat every day.
- wincy 4y agoWe had a drive thru grass fed burger place open up across the street from McDonald’s and it was gross. It lasted maybe a year. McDonald’s makes pretty good burgers, not what I want every day for sure but in a pinch they aren’t bad.
- 4y ago
- deleted 4y ago[deleted]
- thunderbong 4y agoThe reason this happens is Google Drive is marketed as a backup solution, so people add their work folders to it which consists of all these files.
- Takennickname 4y agowouldn't package.json be enough?
- codetrotter 4y agoThey probably have a folder that auto syncs with Google Drive https://support.google.com/drive/answer/10838124?hl=en https://support.google.com/drive/answer/10838124?hl=en Same way that Dropbox works So they are not manually choosing what to sync, they just put all of the work and other files they want to automatically sync inside of that folder. This ends up including things like node modules and things like that
- NoZebra120vClip 4y agoI don't sync folders. I mount my accounts as network drives. As for corruption or overwriting files, I have rarely encountered issues, but Drive has file revision histories as well as a Trash can, so if you notice the problem and it's not spread over 5M files, you can roll back to last-known-good versions.
- codetrotter 4y ago> I don't sync folders. I mount my accounts as network drives. Implementation-wise under the hood is it really much different? In my simplistic, probably overly naive idea it seems that a “network drive” built on top of Google Drive would still cache the files in memory, and improper shutdown or loss of network access could still leave your files in an inconsistent state. Probably they have a lot of advanced logic on top, that could roll back partial updates, or keep track of things so that the updates are not applied before sync is complete. But anyways from this point of view sync vs network mounted drive feels like sync is a kind of network mounted drive that flushes to local disk in addition to working with the remote. One of these days I want to build such things from scratch myself so that I can experience first-hand the horrors that lie underneath sync / network disk service setups modelled on my idea of Dropbox / Google Drive etc
- tomjen3 4y agoIf I was teaching anybody programming I would teach them to copy the folder over every time they added a new feature, rather than teach them git at the same time. With some C++ libraries or node_modules that can run up fast.
- q7xvh97o2pDhNrh 4y agoWhat? Why would you teach someone to do things the wrong way?
- renewiltord 4y agoI must apologize. We have purposely trained them wrong. As a joke.
- Xylakant 4y agoYou can’t teach all of the things at the same time. People will just freak out, be confused and learn nothing. There’s a lot of the things that you’ll teach “wrong” at the beginning. You’ll never talk about CI/CD at that stage, deployment, packaging, etc. either, despite a lot of this being a requirement for a lot of software engineering projects.
- jbverschoor 4y agoWhy would you have nodemodules in every project? It shouldn’t be a local directory anyway
- ikiris 4y agoThey’re probably a math teacher and want them to learn about punch cards before introducing complex concepts like a floppy disk.
- Brian_K_White 4y agoIt's not teaching them the wrong way. It's teaching them a simple version of the right way. You can't do everything at once and git is counter-productive on day one, and the most important thing was they find programming remotely interesting at all. The next most important thing is always working on a new copy. How exactly you make those new copies barely matters at all compared to just having them at all. Using something arcane and unforgiving and inscrutable and complex like git to accomplish those copies is way way down on the list. The new student is barely interested in the actual coding to make a game if you're lucky. They would rather mow lawns than learn administriva like git. It could even be argued as irresponsible to hand git to babies and expect everything to be fine. Just because we all have about 5 git commands we use 99.9% of the time, and have no problem 99% of the time because we learned how to "hold it right" and just always carefully do things in the right order, does not mean git can actually be made safe and simple. Those 5 or so commands aren't actually enough.
- kbumsik 4y agoI mean, why not? I use both git and Dropbox, because I have multiple devices to sync: laptops, a desktop, and linux servers. If you use a desktop and a laptop back and forth, git is quite inconvenient. You don't need to commit and push from the desktop to switch to the laptop with a folder level realtime sync solution. Also, git only protects files that are staged, but Dropbox can restore any files to any point of file event (not periodic snapshot, but it keeps all file events). So combining both you are safe from any kinds of mistakes. I use Dropbox because Google Drive sync wasn't stable enough for this kind of workflow but I am not sure if it has been improved lately.
- stinos 4y agoI mean, why not? Because git combined with Dropbox and the likes can be slow, lead to broken repositories, mad sync conflicts and so on? At least that is what I've witnessed with colleagues. This was years ago though, and I assume you dont have these issues or maybe a different workflow, so this is more like a word of warning for others: test this properly first before just diving into it.
- kbumsik 4y agoHmm I haven't have significant issues for almost 10 years. And a few years ago Dropbox introduced a folder-level rewind feature so you can easily restore any broken folder. Anyway, YMMV.
- youngtaff 4y agoYup, I’ve been running this config for personal projects for over 10 years too
- Dylan16807 4y agoMaybe if you shut a laptop while it's syncing and then start editing the same things on a different computer without considering that you should finish the sync, but that seems pretty niche. Are you sure those people were using their own personal dropbox folders? If they were sharing with each other that's a different thing entirely and requires a lot more care to avoid problems.
- _v7gu 4y agoI don’t like to do it, but I like to sync my projects folder and google drive does not offer me an “exclude folders named node_modules” option
- bluedino 4y agoI worked with a guy that put all of our Rails apps in his Dropbox
- RobotToaster 4y agoI'm guessing, but github desktop by default uses the documents folder. By default Google drive backs up the documents folder.
- NoZebra120vClip 4y agoI use Google Workspaces at work, and I have configured my notebook more like a netbook. Every possible file and folder gets stashed in Drive, and the agent runs to integrate with the Windows filesystem. This is especially important for Company Data and Confidential Materials. I have modest storage needs in my role, but imagine a developer, Big Data user, or video editor used it. As long as they could tolerate the latencies, they might stash quite a bit in there.
- throwaway689236 4y agoI even exclude them from my TimeMachine backup, they're pinned.
- inamberclad 4y agoOut of curiosity, I checked how many files were in my node based autocompleter: $ find .vim/plugged/coc.nvim/node_modules/ | wc 12715 12715 952711
- epilys 4y agoShould be wc -l
- switch007 4y agoThe first column of the output is the lines
- ElectricalUnion 4y agoWithout parameters, wc prints `newlines`, `words` and `bytes`.
- rovr138 4y ago~/Development master ± find . | wc 1029821 1237285 118990227 ~/Development contains 3 projects I worked on this past week. Only 3 because this is a new laptop. I'd hate to think what this looks like on my old laptop. But I'll run it later.
- paulddraper 4y agohttps://preview.redd.it/tfugj4n3l6ez.png?auto=webp&v=enabled&s=b919a7283e56d929e37758af3b34e532c1a71453 https://preview.redd.it/tfugj4n3l6ez.png?auto=webp&v=enabled...
- onion2k 4y agoGoogle: We have identified modern web development as a threat to our systems, and have taken measures to ensure npm users cannot store their npm_modules directories on GoogleDrive. Please consider rewriting your nodejs projects in Go. That's a bit unfair on Node and NPM. The modules directory rarely grows to more than 4 million files.
- yjftsjthsd-h 4y ago> The modules directory rarely grows to more than 4 million files. The article says the limit is 5 million files, so... 2 NPM projects? That's not really better...
- hnfong 4y agoI think the GP is sarcasm, but I only have like 70% confidence.
- hedora 4y ago“In practice, the number of impacted users here is vanishingly small." Well, yeah, I imagine they’re moving elsewhere. Seriously though, do people actually trust them not to randomly intentionally break stuff at this point?
- andyp-kw 4y agoGoogle have a history of acting now and communicating to customers later. Just look up the number of issues developers have with the Play Store.
- throwaway_ab 4y agoI pay for 5 TB and planned to use the drive to store a copy of my data. Things I store that have lots of files: - The frames for my Timelapse videos = 400,000 files - The files in my Eagle app photo database = 400,000 files - Other image files, my programming repositories, documents, music, stable diffusion Deforum frames = 400,000 files 80% of these files I've accumulated in the last 12 months and can see myself easily hitting this 5,000,000 file limit well before I run out of TB's So now I know I will never be able to use all the space I'm paying for, I'm going to stop uploading my files and instead search for a proper backup service, something I should of researched in the first place. Anyone here have any recommendations for a backup service?
- shitlord 4y agoI've used Backblaze for years and never had a problem. There's no upload file/data limit, but downloading it back can be slow. There's an option for them to mail you a hard drive but I never tried it.
- axelsvensson 4y agoBackblaze is terrible, but you will likely only find out when you need to restore. When backup excluded some files, they blamed antivirus software and recommended to go without. They actively deleted their only backup because the client couldn't read the original files (which was due to disk error). They admitted that there isn't any guarantee or even effort to make the backup consistent - after asking me to explain the concept. The only reason I still have my data is because of the very expensive cleanroom disk rescue that they of course refused to pay for - because why try to do what you can to compensate for failing at your only job?
- lasftew 4y agoIt sounds like these problems are related to their end-user backup solution - can't comment on that as I've never used it. However, when referring to Backblaze, I think most people here refer to their nice (and cheap) S3-like cloud storage solution, which works perfectly with the likes of restic, rclone and friends. That's probably what you should use if you care about control.
- fencepost 4y agoIf the number of users affected is as 'vanishingly small' as a Google spokesman indicated then you'd think they'd be able to contact them - at least the paying customers?
- eviks 4y agoThat's what they are doing - via an article in a reputable news organization!
- MikeDelta 4y agoMakes me wonder: if the amount of users affected is vanishingly small, then why implement it, as it sounds like a vanishingly small problem to me?
- _a9 4y agoProbably a very small amount of people abusing it somehow (How a lot of files can be abused i dont know), like how a small amount of people abused the old gsuite unlimited plan to store petabytes for $12/m
- alpaca128 4y agoThen why doesn't Google use their standard solution and just ban those vanishingly few users, pointing to an ambiguous "terms of service violation" and then ignoring all their support requests for a lifetime?
- fsh 4y agoVery interesting that google chose to do this instead of fixing the software that caused the limitation. No wonder that their products are seen as a joke in the business world.
- pleb_nz 4y agoBlows my mind people still trust them TBH. How many years of poor behaviors and bad decisions does it take for people in the know to move on. Understand there are a lot of non tech savvy users who wouldn't know, but I would expect hackernews users to be better informed on average and would have moved on
- Gigachad 4y agoI’ve been using drive since it came out and I’m yet to run in to a single problem with it. It’s been the most reliable storage system I’ve ever had.
- pleb_nz 4y agoI'm not talking about drive specifically, although now it seems it too can be added to the lists of shitty giggle practices and decisions.
- karmasimida 4y agoFixing software for like 0.01% of their whole user base, who plays flat fee anyway seems bad investment I have to agree with Google in this case, 5 million files for an account is borderline abuse for object storage system.
- topicseed 4y agoWe need a .googledriveignore file then.
- sqldba 4y agoSo just don’t be one of the vanishingly small paid users who this affects. Easy. I wonder what vanishingly small is. 0.001% of a million is still thousands.
- nathants 4y agohello rclone and s3.
- knorker 4y agoNot fair to compare Drive with s3. Completely different products. The Google equivalent of s3 is GCS.
- stinos 4y agoAnyone knows how this works legally? You buy a service, suddenly without notice the services changes features. Does the small print allow for that? And how is this 'ok' in software but probably not anywhere else (pretty sure a service contract for an elevator doesn;t allow the service company to just say "we're going to limit the ?amount of times your elevator goes up and down to 100 times a day now")
- burnished 4y agoThe limit is 5 million files. The article feels a little overblown in its reaction.
- ungamedplayer 4y agoNumbers add up quickly.
- Dylan16807 4y agoIt's easy to hit that many files, and they made the change without warning. If you're on the $10 2TB plan and your files average 100KB, 5 million files means you can only use a quarter of the space you're paying for. And before you call that unrealistic, my system drive averages 200KB per file and my main personal data drive is close to 300KB per file. Both would hit the limit if I wasn't using something fancy to pack files together. And then there's the $18 5TB plan with the same limit on file count. Even completely ignoring the option to pay for extra terabytes.
- burnished 4y ago..could you not split it up over multiple, smaller drives then? I get that this could be frustrating (being surprised specifically), but it seems a pretty reasonable restriction so it seems odd to criticize it like this.
- numpad0 4y agoGoogle Drive is not a device, it’s a cloud storage service. There’s no metaphorized “drive device” either, it’s all under the same virtual root folder.
- dragonwriter 4y ago> could you not split it up over multiple, smaller drives then? Well, you could create multiple Google Accounts and create all the files in the same Drive with sharing enabled, because the limit isn’t on what is in one user’s Drive, but what one user can create across all Drives.
- jeron 4y agoTimely reminder of this great GitHub repo: https://github.com/awesome-selfhosted/awesome-selfhosted https://github.com/awesome-selfhosted/awesome-selfhosted
- nickcw 4y agoRclone users noticed this new limit back in February. Here is a thread discussing it on the rclone forum: https://forum.rclone.org/t/new-limit-unlocked-on-google-drive/36136 https://forum.rclone.org/t/new-limit-unlocked-on-google-driv... It would be nice to have official confirmation of the limit rather than relying on speculation.
- deleted 4y ago[deleted]
- anothernewdude 4y agoGoogle: "We are too fucking useless or lazy to make file count not an issue since we already limit you by space."
- tyingq 4y agoWell, also too lazy to provide exclude patterns or some other tool to let you work within our limit.
- eviks 4y agoOn a tangent and out of curiosity, which filesystems are great at working with a gazillion of tiny files?
- cvccvroomvroom 4y agoThe Google Search and StackOverflow filesystems.
- Vecr 4y agoBTRFS, at least technically. It can get really fragmented and slow though.
- eviks 4y agoHow is "slow" great?
- Beldin 4y ago"Vanishingly small": a number of users small enough to be downplayed, but large enough so that neither an individual approach to the problem would work, nor that the problems could be ignored. Suspected to be a complex number.
- anjel 4y ago"affecting" is quite the euphemism, here
- thrdbndndn 4y agoWait, is it new? I believe Google Drive for Workspace always have a file count limit, and IIRC it's as low as 500k or something, despite having "unlimited" capacity. To be totally fair to Google, I know this precisely because there are communities of data hoarders that actively abuse various cloud storages. In Google Drive's case, They have ways to create "free" Google Workspace accounts via registration exploitation from various institutions. People use them to store PB-level data. (For the interested, there are also ways to apply free MS developer accounts that are supposed to expire in 3 months but can be re-refresh indefinitely. This comes with 5TB "free" cloud storage x 5 (10?) separate sub-accounts.)
- Kwpolska 4y agoWhat if Microsoft changes their mind and you wake up one day with your cloud backup being nuked due to the account expiring or due to your use being non-development-related?
- thrdbndndn 4y agoAt this point people are just hoarding for the sake of hoarding. The data itself isn't really that important: collecting the entire Netflix catalog is pretty neat, but if it's gone it's fine, you can basically re-find anything in piracy world. And they usually have multiple backups.
- cableshaft 4y agoI have multiple backups, but they're all on drives in my house. If my house burns down, there go all my backups. That's why I keep all the important stuff on a cloud backup service as well.
- mort96 4y agoHmm, there was a HN thread about this a few days ago [1] where everyone seemed to attack people for even considering the idea of storing 5M files in a cloud storage solution, going so far as to argue that even disclosing such a limit would be unreasonable to expect. In this thread, the prevailing thought seems to be that having a 5M file limit is unreasonable and adding it without disclosing it is egregious. Just a curious thing I noticed. [1]: https://news.ycombinator.com/item?id=35329135 https://news.ycombinator.com/item?id=35329135
- layer8 4y agoDifferent time zones maybe, or weekday vs. weekend.
- midland_trucker 4y agoGoes to show that nailing down the 'HN crowd' opinion is more difficult than it seems!
- kmos17 4y agoInteresting observation, I have noticed too that contradictory opinions are very frequent. My completely subjective impression has been that there is a loud contingent that just likes to have a contradictory opinion, a form of mansplaining that is frequent in engineering circles. Or maybe it’s just that people in general are driven to give their opinion on things they disagree with.
- ben_w 4y ago> people in general are driven to give their opinion on things they disagree with I’m definitely more likely to reply when I disagree than when I agree. I do try to make a habit of giving positive replies also (and not just building on what was said before, as I’m doing here, nor just upvote silently), as constant contrarianism is… kinda dull, especially from the outside.
- drstewart 4y agoWhat does gender have to do with here?
- Overtonwindow 4y agoOh good lord, I have so many files spread across a dozen drive accounts. It's free storage, I'm going to use it.
- SergeAx 4y agoI wonder what jury-rigged solution may lead to breaking into 5M limit? I can't believe it was just digital hoarding. In the end, hoarders know better to keep things in zip-archives.
- jmyeet 4y agoSome people will let technical limitations define a product. Others will have the product dictate the technical design. This, to me, is an example of the former. I don't know the serverside implementation of Google Drive but imagine the files on your Drive correspond to files on something like an ext4 filesystem. In this scenario, each file has a cost (eg in the inode table) and there is wastage depending on what your block size is. Whatever the case, Drive seems to treat files as first-class objects. Compare this to something like Git. In Git (and other DVCSs like Mercurial), you need to track changes between files so the base unit is not files, it's the repo itself. It maps to your local filesystem as files but Git is really tracking the repo as a whole. So if you were designing Google Drive, you could seamlessly detect a directory full of small files and track that directory as one "unit" if you really wanted to. That would be the way you make the product dictate the design.
- nashashmi 4y agoI wish there were a "zip" standard for storing a million small files in one package. NPM and other open source programs badly need this.
- Gigachad 4y agoIsn’t that just tar?
- ElectricalUnion 4y ago.jar/.apk (internally a zip archive) comes to mind. AppImage (internally is either a ISO 9660 Rock Ridge or a SquashFS filesystem), .deb (internally ar archive), .rpm (internally a cpio archive) are I think relevant examples too.
- Xamayon 4y agoI do exactly that for storing the result thumbnails for some of the dbs in my reverse image search engine (SauceNAO). Non compressed zip files allow quickly/easily seeking to and accessing component files without extraction. A few tens to hundreds of thousands per zip file works great. Millions would probably not be too different, but would use more resources/take more time when loading the zip file index.
- mthoms 4y agoInteresting. Have you ever considered SQLite file storage? I'm wondering how it would compare. https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
- Xamayon 4y agoHaven't looked into it, but it sounds like it would work similarly (with some nice benefits such as also being able to easily store other metadata/etc). Feasibility would depend on how quickly the indexes and such load, and the resource consumption associated with opening/closing dozens of them at a time 24/7. In my screwy case there are hundreds of thousands of zip files which are randomly accessed on the fly to grab one or two thumbnails at a time. The random access speed on unloaded files is critical, and for zip files it's extremely quick.
- greatgib 4y agoGood remainder again that "the cloud is just someone's else computer"! In my experience, GDrive is a piece of crap with a lot of weird behaviors and easy ways to lose your data if you sync your computer with it. The worse here, as said by multiple persons, is not to have a limit. A limit on their service is fair. It is that this limit is undocumented, and that their key selling point is to shout everywhere that if you pay you will have "unlimited" storage. And that it will scale more easily than using your own "not cloud" backups.
- Gigachad 4y agoDo they advertise unlimited storage? When I looked, the top plan was 30tb. To hit the file cap, you have to have 30tb of files averaging no more than 6mb.
- ElectricalUnion 4y agoIn theory, if name metadata is "free", you can make a infinite ammount of 0 byte files with very large paths and tags.
- Gigachad 4y agoWhats the chance someone created some FUSE mount for storing data in 0 byte files.
- lazide 4y ago100% I imagine. GDrive does have API rate limiting which either way is going to make it slow/useless for REALLY large data storage.
- Gigachad 4y agoI've noticed over the years they have been cutting off all the unlimited stuff. Docs used to be unmeasured and unlimited, now they actually count the doc size to your storage, which is still effectively unlimited under non abuse use cases.
- mcherm 4y agoHaving a limit of 5 million files is perfectly reasonable. Failing to document that such a limit exists and refusing to publicly confirm it (which apparently is STILL the case) is extraordinarily poor customer service/communication. Google KEEPS setting new records for poor customer communication, to the point where I (and much of the HN crowd) now expect it. Android developer banned from the app store? There is no meaningful way to appeal but you'll probably never be able to find out why. Your best hope is to post on HN and hope someone with power at Google notices. Leadership at Google ought to recognize this; they ought to make an effort to improve the channels by which "customers" can communicate with Google. But I see no signs that they are even aware of the issue; I see no effort to change anything. I would try to tell them but... there's no communication channel. Maybe I should post about it on HN.
- _yb2s 4y agoI think it’s basically the attitude “customer support is expensive and annoying, and since we’re Google and can do whatever we want, let’s skip it.” This kind of made sense when they had a massive tech advantage such that their products also needed less support. Now they seem to have lost that edge and are just another competitor offering products that come with poor support.
- dingusdew 4y ago[dead]
- jollofricepeas 4y agoI’d like to draw parallels to the old cable titans with local monopolies who also had the same customer service tactics as well. 1. Offer a cheap (free) service 2. Gain significant market penetration 3. Provide little to no customer service 4. Raise prices 5. Let the product/services degrade until significant public outcry or regulator involvement
- jrootabega 4y agoCustomers? Them thinking of us as peasants, or even cattle, might better explain these attitudes.
- harshaw 4y agoThe challenge with running cloud storage is that you have to think around the corners for usage and shape customer behavior with pricing. Seems like google didn't want to do this or was too lazy (sorry). Millions of files will always be a problem. The metadata costs more for these users, it's impossible to manage, hard to cleanup, etc. The problem with google is if they fuckup their service they make it the customers problem. Other places if they fuckup, its more viewed as a one way door. You can sunset old products with (in this case, unlimited files), but you never put in a new restriction.
- SMAAART 4y ago"Don't be evil" ---> You either die a hero, or live long enough to see yourself become the villain.
- benhurmarcel 4y agoI wonder how that works for companies using Google Workspace. My company has Workspace users close to 6 digits I believe, I'd think we collectively store way more than a few million files.
- danpalmer 4y agoIt's a per-user limit.
- benhurmarcel 4y agoAnd how does that work with Shared Drives then? They don’t belong to 1 user.
- danpalmer 4y agoI believe each user can create up to 5 million files in the shared drive. I work at a company with many more than 5 million files overall where this has not been an issue.
- slackfan 4y agoJust another unlimited(*) with the asterisk in giant bolf and the explanation text in 2point type service. Yay.
- deleted 4y ago[deleted]
- ourmandave 4y agoInitial thought is the ones who are surprised are the ones who sending a notification email to wouldn't have been noticed. Like my dad has 300+ unread emails with who knows how many gigs of attachments.
- lopkeny12ko 4y agoI wonder if you could create a block-level virtual filesystem backed by Google Drive so that you could store many small logical files in one physical remote "block" (file).
- smileybarry 4y agoMaybe, but most storage providers don’t expose a seek/range API for their consumer stuff, so you’re doomed to fetch & upload the whole block every time. For about a decade, Dropbox was the only one to offer delta sync, OneDrive added that a few years ago, and they’re the only two IIRC that do delta syncing. The rest basically redownload & reupload.
- Reptur 4y agoSeems like an Engineering issue more than a User issue. They could just take the node_modules folders and zip them up behind the scenes without changing the user interaction.
- exabrial 4y agoOnce again: Don’t use Google for anything crucial or critical. Not Google Cloud, Google Docs, Google Drive, even Gmail is becoming a liability. Real Engineering involves developing forward looking designs and maintaining backwards compatibility. It involves a release schedule. It involves communication channels and releases notes. It’s hard. It’s unsexy. Google treats their product lineup with the seriousness of a social media platform. They don’t care about your puny business; even if it means the world to you, it means nothing to them.
- rsync 4y agoI see it speculated, downthread, that this is a response to modern web-dev and node (?) creating millions of files, etc. I can’t comment on that but I do know that modern, encrypted, archive tools such as duplicity and borg and restic “chunk” your source files into thousands (potentially millions) of little files. We see tens of billions of files on our zpools and have “normal” customers breaking 1B … and the cause is typically duplicity or borg.
- jeffbee 4y agoAt a previous company the biggest cause of files in a particular context was that "oh my zsh" was installed for each of thousands of engineers by default. It often seems as though "modern framework" just means it is an abusive piece of junk.
- Dylan16807 4y agoBillions seems weird for normal customers. Duplicity appears to put its chunks into 25MB files, I know for sure that restic puts them into 4MB or more recently 16MB files, and Borg looks like it puts them into 500MB files.
- rsync 4y agoNot average, but normal. What I mean is, they just have a plain old rsync.net account and pay no premium nor incur any penalty for using all of those inodes. You are correct: the average customer does not use billions of inodes.
- Dylan16807 4y agoWell I interpreted "normal" to mean "does not have many petabytes of data", and if those tools are reaching a billion files with smaller amounts of data then I'm very confused about how it's happening.
- squokko 4y agoThis is why despite G Suite being in many ways a superior product, it's made almost no inroads in Corporate America vs. Microsoft Office. Enterprises need to be able to specify a business workflow and depend on it, and if there are nasty surprises it fucks with their money. Microsoft software is much worse than many competitors but it's documented, the behavior doesn't change suddenly, and it's backwards compatible.
- LightBug1 4y agoI know Google employees are reading this ... don't you recognise this sh!tshow and communicate it internally? I'll never understand how such a large organisation can let this kind of stuff happen.
- marcinzm 4y agoProbably the usual corporate reason that the nail that sticks out gets hammered down and that the messenger is often shot. Especially with all the layoffs it's better to not be the one in that position. And even if you do convince them it's a bad idea and don't get punished for it what do you gain? A half sentence on your performance review? Risk not worth the potential reward.