12 ms·

5M item limit for Google Drive: File unable to generate or upload due to 403

165 points by throwaway2056 4y ago

vivegi 4y ago
Rug pull. That's what this is. The Google Drive landing page (https://www.google.com/intl/en-US/drive/ https://www.google.com/intl/en-US/drive/) as of April 1, 2023 still doesn't mention the 5M per user maximum files count limit. Is it an April fool's joke? /s. Do better $GOOG.
a2tech 4y ago
Interesting that this was enabled with apparently no outreach. Maybe Google needed some of those employees that it recently laid off
- londons_explore 4y ago
  And it would have been pretty simple to run a query for users with over 4 million objects, and fire off an email to them all saying "We are about to implement a limit, and you will be impacted. Here are potential workarounds/mitigations". I suspect though that the simple process of dropping a few hundred customers who will be directly impacted an email requires approval from 27 middle managers, and it's easiest to just ignore them.
  - shadowgovt 4y ago
    More often, this kind of error out of Google comes from the left hand not knowing what the right hand is doing. Joe or Jane Noogler has just been brought into the team, and their starting project was to improve indexing. They succeeded in doing so by attaching the Drive data to an indexing service created after Drive existed. It's working great and has a 10-20X speedup. But oops... That new indexing system has a 5-million-element hard limit built in, and nobody caught it until they went to production. J. Noogler is too new to have realized this could have been an issue so they never started the escalation / customer messaging processs.
- hirako2000 4y ago
  Precisely what I felt skimming through the comments on the issue. A lot of high scale businesses have taken a huge risk. A company like Google had over 10y long tenure engineers who were swept by the layoffs. The argument that the year prior was a hiring spree only makes it worse: now you have hands to throw at the problem but unlikely the right pairs, which, given the climate of job loss fear, aren't likely to admit they are incompetent. Off topic, but this isn't an isolated case of mega size top tier companies seemingly dropping the ball more far often and/or with greater blast radius impacts. See github leaking secrets publicly on their own git repo. Call me paranoid, but to me one of these things is going on: 1/ tipping point of too many under qualified for the job engineers running things, due to C-suites looking at keyboard monkeys the way they see factory workers, naively applying "cost cutting" measures, plucking through spreadsheets who "are the biggest seemingly disposable weights" and fire those as part of waves of x% layoffs 2/ Overworked remaining know-how crews - added the weight of dealing with inspiring politicians turned enginners for the juicy 200k + rsu package who, demand better articulated instructions, knowledge transfers and more comprehensive formal documentation materials, so that they too can shine too - on top. Competent folks having less and less time but more more tears and blue bags below their eyes finally realising that ain't worth it as it keeps getting worse anyway. 3/ sabotage Or all those, since there could be some smowball effects there. Call me paranoid, that it isn't what's going on, we clearly aren't seeing the slow fall of major infrastructures the few who were there to build them now having mostly packed their bag and long gone, replaced by swaths of bootcamp trophy coders hoarding cloud certificates like North korean generals like to collect shiny pins. Maybe it isn't, the future is still in the making anyway, but I felt glimpses of that several years ago, then about each passing year, now every few months. like an seeing accelerating meteroite seemingly going away but which keeps looking bigger and bigger. More so related to the topic: just drop drive as a centralized storage, there are e2e solutions out there and open source scripts to entirely migrate out of these wall gardens. You won't miss google sheets: you can still use it!
londons_explore 4y ago
5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers of objects were probably causing those machines to OOM. 5M items, at about 1000 bytes per item (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about the amount of RAM most tasks will be given. I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota. This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.
- londons_explore 4y ago
  It could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because you want to downrank blank 'untitled documents' in search results, because they are easily accidentally made). You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce. However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).
- cabirum 4y ago
  More like an arbitrary number from the old times "cause 5M items ought to be enough for everybody."
  - charcircuit 4y ago
    >from the old times That doesn't make sense because this is a new restriction.
    - 8organicbits 4y ago
      I think the parent is referring to the 42 year old meme quote "640K ought to be enough for anybody". https://www.computerworld.com/article/2534312/the--640k--quote-won-t-go-away----but-did-gates-really-say-it-.html https://www.computerworld.com/article/2534312/the--640k--quo...
steponlego 4y ago
Sounds like they want to protect against resource exhaustion. Touch a file, it's empty with a size of zero bytes. But depending on the block size of the storage device, it's likely to be between 4k and up to even 1M.
- ec109685 4y ago
  They should incorporate a minimum size for every file when calculating how much quota you have. Because they have no retrieval fees for gsuite, I bet it is expensive dealing with drives with tons of small files. Agree they are rolling this out terribly. They should have granted everyone 2x the number of files they currently have in their drive quota (or 4M, whichever was smaller).
- mayli 4y ago
  Imagine having the filename and other non-quota-able attributes as an unlimited storage.
  - steponlego 4y ago
    I like this. Simply store your data as the filename in 64 byte chunkers. Free unlimited drive space.
encoderer 4y ago
Tangent: I was wrong to ever consider it sus that Google includes gsuite with their cloud revenue. Clearly it’s being used that way.
- amf12 4y ago
  So does Microsoft. Intelligent Cloud = Azure + Office 365
duxup 4y ago
What is Google Drive... for? I see folks describe Google Drive errors / limits like this and people describe "a business critical operational system" and I wonder. Is Google Drive even supposed to do this thing? Not pointing the finger back at them, if Google doesn't make it clear they probably should, but at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.
- Spooky23 4y ago
  It’s a general purpose file and document store. People run significant organizations on Google Drive. Like anything, people can push limits where it’s no longer productive to extend.
- chimeracoder 4y ago
  > at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service. That may be how you view the product, but Google Drive absolutely is used as an enterprise cloud service by many large companies. And as a result, there are lots of applications that integrate with its API, etc., to serve those enterprise use cases.
  - duxup 4y ago
    I don't doubt people do it. Humans are industrious and users will push things to their limits. I just wonder .... is that what Google built Drive for? / how much of that they were thinking of.
    - unmole 4y ago
      Google Drive is part Google Workspace which is a paid enterprise offering.
      - duxup 4y ago
blitzar 4y ago
They also have what looks like an arbitary limit on "Shared Drives" The number of stored files: 400,000
- danpalmer 4y ago
  This is true, but you can just create more shared drives. It looks like storage quota is shared across all drives on the account.
  - blitzar 4y ago
    I would need to check, but I dont think my (total) storage quota in GB is affected by shared drives. Nevertheless, there must be something related to their storage model that ranks count of files > storage size.
bobsmooth 4y ago
Just checked, my C drive has a bit more than 1 million files. What are you doing that you have 5 million separate files in your Google Drive?
- jackson1442 4y ago
  If you're using Google Earth Engine or a similar tool, it's generally highly recommended to use Google Drive as the storage backend for it - especially since Google sells Enterprise plans with "as much storage as you need." Geospatial artifacts can be very large, and also can be spread across thousands and thousands of files (20+ zoom levels on Earth, with enough 256x256 tiles to cover the planet at each zoom level, for example).
- hirako2000 4y ago
  Many use some rsync-like tools to use gdrive as a flat backup system. Perfectly fine use case. You would be over a million as a single user. Some account have thousands of users. See some comments on the tracker, some orgs have a multi sites set up across an entire nation. I wouldn't be surprised if 10k+ volunteers at some NGOs are on that sort of plan and use it as their personal storage solution on top of all the things they sync for document counts heavy work.
- justin_oaks 4y ago
  Nobody should ever need more than 640K of RAM, right? Just because your workflow doesn't involve lots of tiny files doesn't mean other people's don't. Besides, most of those commenting in the issue are talking about their organization having hit the limit, not individuals.
  - kulahan 4y ago
    Why is an organization using a primarily-personal file storage/transfer system for real work
    - shadowgovt 4y ago
      The same reason anyone uses anything: because it works until it doesn't.
pdw 4y ago
As one of the comments points out, Google sells 30TB accounts. With a 5M file limit, you need an average file size of 6MB to fill your drive.
- shadowgovt 4y ago
  This checks out; in general, the 30TB consumers are media storage, editing, and archival.
fbn79 4y ago
Maybe Google cloud using NTFS disks (Maximum number of files on disk: 4,294,967,295)? :))
- overthrow 4y ago
  That would be 5 billion, not 5 million
dkjaudyeqooe 4y ago
I'm surprised it knows how to count that low.
EricE 4y ago
So are there different limits for Google for Business/Workspace/whatever they are branding it this week? Or is this for all Google drive accounts?
deleted 4y ago
[deleted]
danpalmer 4y ago
From reading the thread it sounds like this is 5M per user. Google Drive appears to be targeted at "human" usage, i.e. people uploading or creating files. I would guess that this is also worked into the cost – that the assumptions of the amount of work a human can do are a part of the price formulation. The reporter of this bug seems to be using this as a storage backend for software though, which I don't believe is the intended use-case. Looking at S3 pricing, just the storage is ~5x the cost of Google Drive, and then you need to add transfer and API calls on top of that. I don't personally think that there are reasonable use-cases for human users with 5 million files. There may be some specialist software that produces data sets that a human might want to back up to Google Drive, but that software is unlikely to run happily on drive streamed files so even those would be unlikely to be stored directly on Drive. (Disclaimer, I work at Google, not on Drive, this is my personal reading and interpretation of the public info, I don't have any inside info here)
- TazeTSchnitzel 4y ago
  What if you check out a few large git repos in a directory synced with Google Drive?
  - danpalmer 4y ago
    I thought about this, but I accidentally put the Webkit repo (~1m files) into Dropbox once and it didn't go well. Git needs a consistent view of the filesystem across many files to work, and that's not really possible with that many files stored in a sync system that works per-file like Drive does. I put Git in the category of specialist software that shouldn't be operated in Drive. By all means tar and backup, that should be effective, but there's no need to store all the individual files.
    - TazeTSchnitzel 4y ago
      Storing a git working tree in a cloud drive does seem like a bad idea, but the actual .git directory itself might be okay? I think most of the files in there are immutable or infrequently changed.
mike503 4y ago
Part of this seems like some of the wonky design that seems to be under the hood of Drive. Just looking around at how rclone has to do things, the difference between shared and individual drives, the fact I couldn't even originally find a way in the console to see my shared drive usage (even though I was the owner of it and it was on my account) it just seems like there is a very very wonky odd design to how the system works. Not like an object store under the hood with a layer of "document" logic applied, but rather what started as initially some API-esque thoughts on document storage that grew into a monstrosity. I wouldn't be shocked if there's just a ton of tech debt there.
paxys 4y ago
While a 5 million file limit sounds reasonable at first glance, it isn't just the free tier that is affected. Google Drive also sells storage plans for up to 30 TB for an individual user, and there is no mention of such a limitation anywhere on any plan. If I bought and actually wanted to use that 30 TB (or heck even 5 TB), it isn't unreasonable to want to store a few million files. Regardless of the technical reasons behind it, not mentioning this restriction up front is before the sale is inexcusable.
- Gigachad 4y ago
  I did the math and to reach 5M files in 30 TB, they all have to be averaging 6MB. If you actually have this many files, you should be using S3 or mongodb.
  - chii 4y ago
    if you were using google drive to store images (e.g., you automated a camera system to take high res pictures), you can easily reach this limit. Of course, it definitely is much nicer to have S3 over google drive for such a solution, but if you didn't want to write software (which you'd need for S3), and simply used the google drive's sync'ing features...
    - Gigachad 4y ago
      In this case you can just delete the oldest files, or tar/zip them in to one file. I doubt anyone needs instant granular access to 5 million images. The chance of hitting this limit is so insanely slim and the workarounds are so trivial that I don't really see any issues here. I bet there would have been some abuse where someone tried to store data in file names and metadata which is not counted to your quota.
throwaway81523 4y ago
It seems like there is a much lower limit for "download all". There is a 100 or so item directory that I want to download, and "download all" tries to wrap them in a zip file and fails. I think it wants to pre-scan all the files for viruses before zipping, but is unwilling to do that many. So I have to download the files one at a time.
- deleted 4y ago
  [deleted]
- bobbylarrybobby 4y ago
  I've definitely downloaded folders with 100 items before. I think it might have to do with how big the individual files are because g drive won't create a zip that's larger than 2GB.
- dublinben 4y ago
  In the future, you can use rclone to easily download entire folders from Google Drive. https://rclone.org/ https://rclone.org/
  - throwaway81523 4y ago
    Hmm, thanks, that looks promising, but it appears to be an all singing all dancing complicated client that does two way transfers to gdrive and expects you to have a google account. I don't have a google account (don't want to enroll one), and am just trying to download a public folder that someone gave me a link to. I'll keep looking to see if I can find a way to use it.
synctheship 4y ago
Well this is scary, I wonder if all users/plans are impacted? I have north of 8.5million files in GDrive and I luckily have not hit an error yet.
- danpalmer 4y ago
  What are you using it for!?! Honestly, I'm interested to hear what your use-case is as I struggled to come up with one for this many files.
  - synctheship 4y ago
    We've been using GDrive in one form or another since 2014 when they released Google Drive for Work. Primary use case nowadays for my team is an old legacy tool(been in place since 2014) that connects to our ERP and other internal tools to generate reports and export data with combined metadata from multiple tools/sources. Then if needed, makes a copy to different share drives and teams, assigns access rights for internal and external users. We have custom web front ends that allow them to interact with these exports along with fat clients if they need to manipulate a larger dataset of files. My last check had the service account that I'm responsible for somewhere around 8.7 Million files with an average size around 9.2MB. We have multiple of these "accounts", with similar use-cases across other teams. This is excluding the normal use of Gdrive with normal "rank-and-file" office works managing standard office docs. We've felt the squeeze from google over the past few years, so we've already started migrating off of google services about 2.5 years into a 4 year project.
    - danpalmer 4y ago
      Yeah this doesn't seem to be in line with the expected use or target market of Drive. It sounds like your tool is probably creating all the files acting as one service account, and therefore hitting the limit if it is indeed per user. I think the rule of thumb I would use is that if you're creating the files in an automated way, and then accessing them outside of Drive interfaces, in all cases, then it's probably a better fit for a web app with cloud storage backend. Google Cloud Storage or S3 would work well for this.
ccheney 4y ago
Someone must've copied their node_modules folder into Google Drive by mistake...
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
up2isomorphism 4y ago
The key is that every company is very happy to lose this kind of customers.

165 points by throwaway2056 4y ago

Rug pull. That's what this is. The Google Drive landing page (https://www.google.com/intl/en-US/drive/ https://www.google.com/intl/en-US/drive/) as of April 1, 2023 still doesn't mention the 5M per user maximum files count limit. Is it an April fool's joke? /s. Do better $GOOG.

a2tech 4y ago

Interesting that this was enabled with apparently no outreach. Maybe Google needed some of those employees that it recently laid off

londons_explore 4y ago
And it would have been pretty simple to run a query for users with over 4 million objects, and fire off an email to them all saying "We are about to implement a limit, and you will be impacted. Here are potential workarounds/mitigations". I suspect though that the simple process of dropping a few hundred customers who will be directly impacted an email requires approval from 27 middle managers, and it's easiest to just ignore them.
- shadowgovt 4y ago
  More often, this kind of error out of Google comes from the left hand not knowing what the right hand is doing. Joe or Jane Noogler has just been brought into the team, and their starting project was to improve indexing. They succeeded in doing so by attaching the Drive data to an indexing service created after Drive existed. It's working great and has a 10-20X speedup. But oops... That new indexing system has a 5-million-element hard limit built in, and nobody caught it until they went to production. J. Noogler is too new to have realized this could have been an issue so they never started the escalation / customer messaging processs.
hirako2000 4y ago
Precisely what I felt skimming through the comments on the issue. A lot of high scale businesses have taken a huge risk. A company like Google had over 10y long tenure engineers who were swept by the layoffs. The argument that the year prior was a hiring spree only makes it worse: now you have hands to throw at the problem but unlikely the right pairs, which, given the climate of job loss fear, aren't likely to admit they are incompetent. Off topic, but this isn't an isolated case of mega size top tier companies seemingly dropping the ball more far often and/or with greater blast radius impacts. See github leaking secrets publicly on their own git repo. Call me paranoid, but to me one of these things is going on: 1/ tipping point of too many under qualified for the job engineers running things, due to C-suites looking at keyboard monkeys the way they see factory workers, naively applying "cost cutting" measures, plucking through spreadsheets who "are the biggest seemingly disposable weights" and fire those as part of waves of x% layoffs 2/ Overworked remaining know-how crews - added the weight of dealing with inspiring politicians turned enginners for the juicy 200k + rsu package who, demand better articulated instructions, knowledge transfers and more comprehensive formal documentation materials, so that they too can shine too - on top. Competent folks having less and less time but more more tears and blue bags below their eyes finally realising that ain't worth it as it keeps getting worse anyway. 3/ sabotage Or all those, since there could be some smowball effects there. Call me paranoid, that it isn't what's going on, we clearly aren't seeing the slow fall of major infrastructures the few who were there to build them now having mostly packed their bag and long gone, replaced by swaths of bootcamp trophy coders hoarding cloud certificates like North korean generals like to collect shiny pins. Maybe it isn't, the future is still in the making anyway, but I felt glimpses of that several years ago, then about each passing year, now every few months. like an seeing accelerating meteroite seemingly going away but which keeps looking bigger and bigger. More so related to the topic: just drop drive as a centralized storage, there are e2e solutions out there and open source scripts to entirely migrate out of these wall gardens. You won't miss google sheets: you can still use it!

londons_explore 4y ago

5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers of objects were probably causing those machines to OOM. 5M items, at about 1000 bytes per item (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about the amount of RAM most tasks will be given. I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota. This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.

londons_explore 4y ago
It could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because you want to downrank blank 'untitled documents' in search results, because they are easily accidentally made). You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce. However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).
cabirum 4y ago
More like an arbitrary number from the old times "cause 5M items ought to be enough for everybody."
- charcircuit 4y ago
  >from the old times That doesn't make sense because this is a new restriction.
  - 8organicbits 4y ago
    I think the parent is referring to the 42 year old meme quote "640K ought to be enough for anybody". https://www.computerworld.com/article/2534312/the--640k--quote-won-t-go-away----but-did-gates-really-say-it-.html https://www.computerworld.com/article/2534312/the--640k--quo...

steponlego 4y ago

Sounds like they want to protect against resource exhaustion. Touch a file, it's empty with a size of zero bytes. But depending on the block size of the storage device, it's likely to be between 4k and up to even 1M.

ec109685 4y ago
They should incorporate a minimum size for every file when calculating how much quota you have. Because they have no retrieval fees for gsuite, I bet it is expensive dealing with drives with tons of small files. Agree they are rolling this out terribly. They should have granted everyone 2x the number of files they currently have in their drive quota (or 4M, whichever was smaller).
mayli 4y ago
Imagine having the filename and other non-quota-able attributes as an unlimited storage.
- steponlego 4y ago
  I like this. Simply store your data as the filename in 64 byte chunkers. Free unlimited drive space.

encoderer 4y ago

Tangent: I was wrong to ever consider it sus that Google includes gsuite with their cloud revenue. Clearly it’s being used that way.

amf12 4y ago
So does Microsoft. Intelligent Cloud = Azure + Office 365

duxup 4y ago

What is Google Drive... for? I see folks describe Google Drive errors / limits like this and people describe "a business critical operational system" and I wonder. Is Google Drive even supposed to do this thing? Not pointing the finger back at them, if Google doesn't make it clear they probably should, but at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service.

Spooky23 4y ago
It’s a general purpose file and document store. People run significant organizations on Google Drive. Like anything, people can push limits where it’s no longer productive to extend.
chimeracoder 4y ago
> at least as an individual I always see Google Drive as a single user cloud light system that allows for some sharing and organizational functions... but still not an "industrial" type cloud service. That may be how you view the product, but Google Drive absolutely is used as an enterprise cloud service by many large companies. And as a result, there are lots of applications that integrate with its API, etc., to serve those enterprise use cases.
- duxup 4y ago
  I don't doubt people do it. Humans are industrious and users will push things to their limits. I just wonder .... is that what Google built Drive for? / how much of that they were thinking of.
  - unmole 4y ago
    Google Drive is part Google Workspace which is a paid enterprise offering.
    - duxup 4y ago

blitzar 4y ago

They also have what looks like an arbitary limit on "Shared Drives" The number of stored files: 400,000

danpalmer 4y ago
This is true, but you can just create more shared drives. It looks like storage quota is shared across all drives on the account.
- blitzar 4y ago
  I would need to check, but I dont think my (total) storage quota in GB is affected by shared drives. Nevertheless, there must be something related to their storage model that ranks count of files > storage size.

bobsmooth 4y ago

Just checked, my C drive has a bit more than 1 million files. What are you doing that you have 5 million separate files in your Google Drive?

jackson1442 4y ago
If you're using Google Earth Engine or a similar tool, it's generally highly recommended to use Google Drive as the storage backend for it - especially since Google sells Enterprise plans with "as much storage as you need." Geospatial artifacts can be very large, and also can be spread across thousands and thousands of files (20+ zoom levels on Earth, with enough 256x256 tiles to cover the planet at each zoom level, for example).
hirako2000 4y ago
Many use some rsync-like tools to use gdrive as a flat backup system. Perfectly fine use case. You would be over a million as a single user. Some account have thousands of users. See some comments on the tracker, some orgs have a multi sites set up across an entire nation. I wouldn't be surprised if 10k+ volunteers at some NGOs are on that sort of plan and use it as their personal storage solution on top of all the things they sync for document counts heavy work.
justin_oaks 4y ago
Nobody should ever need more than 640K of RAM, right? Just because your workflow doesn't involve lots of tiny files doesn't mean other people's don't. Besides, most of those commenting in the issue are talking about their organization having hit the limit, not individuals.
- kulahan 4y ago
  Why is an organization using a primarily-personal file storage/transfer system for real work
  - shadowgovt 4y ago
    The same reason anyone uses anything: because it works until it doesn't.

pdw 4y ago

As one of the comments points out, Google sells 30TB accounts. With a 5M file limit, you need an average file size of 6MB to fill your drive.

shadowgovt 4y ago
This checks out; in general, the 30TB consumers are media storage, editing, and archival.

fbn79 4y ago

Maybe Google cloud using NTFS disks (Maximum number of files on disk: 4,294,967,295)? :))

overthrow 4y ago
That would be 5 billion, not 5 million

dkjaudyeqooe 4y ago

I'm surprised it knows how to count that low.

EricE 4y ago

So are there different limits for Google for Business/Workspace/whatever they are branding it this week? Or is this for all Google drive accounts?

deleted 4y ago

[deleted]

danpalmer 4y ago

From reading the thread it sounds like this is 5M per user. Google Drive appears to be targeted at "human" usage, i.e. people uploading or creating files. I would guess that this is also worked into the cost – that the assumptions of the amount of work a human can do are a part of the price formulation. The reporter of this bug seems to be using this as a storage backend for software though, which I don't believe is the intended use-case. Looking at S3 pricing, just the storage is ~5x the cost of Google Drive, and then you need to add transfer and API calls on top of that. I don't personally think that there are reasonable use-cases for human users with 5 million files. There may be some specialist software that produces data sets that a human might want to back up to Google Drive, but that software is unlikely to run happily on drive streamed files so even those would be unlikely to be stored directly on Drive. (Disclaimer, I work at Google, not on Drive, this is my personal reading and interpretation of the public info, I don't have any inside info here)

TazeTSchnitzel 4y ago
What if you check out a few large git repos in a directory synced with Google Drive?
- danpalmer 4y ago
  I thought about this, but I accidentally put the Webkit repo (~1m files) into Dropbox once and it didn't go well. Git needs a consistent view of the filesystem across many files to work, and that's not really possible with that many files stored in a sync system that works per-file like Drive does. I put Git in the category of specialist software that shouldn't be operated in Drive. By all means tar and backup, that should be effective, but there's no need to store all the individual files.
  - TazeTSchnitzel 4y ago
    Storing a git working tree in a cloud drive does seem like a bad idea, but the actual .git directory itself might be okay? I think most of the files in there are immutable or infrequently changed.

mike503 4y ago

Part of this seems like some of the wonky design that seems to be under the hood of Drive. Just looking around at how rclone has to do things, the difference between shared and individual drives, the fact I couldn't even originally find a way in the console to see my shared drive usage (even though I was the owner of it and it was on my account) it just seems like there is a very very wonky odd design to how the system works. Not like an object store under the hood with a layer of "document" logic applied, but rather what started as initially some API-esque thoughts on document storage that grew into a monstrosity. I wouldn't be shocked if there's just a ton of tech debt there.

paxys 4y ago

While a 5 million file limit sounds reasonable at first glance, it isn't just the free tier that is affected. Google Drive also sells storage plans for up to 30 TB for an individual user, and there is no mention of such a limitation anywhere on any plan. If I bought and actually wanted to use that 30 TB (or heck even 5 TB), it isn't unreasonable to want to store a few million files. Regardless of the technical reasons behind it, not mentioning this restriction up front is before the sale is inexcusable.

Gigachad 4y ago
I did the math and to reach 5M files in 30 TB, they all have to be averaging 6MB. If you actually have this many files, you should be using S3 or mongodb.
- chii 4y ago
  if you were using google drive to store images (e.g., you automated a camera system to take high res pictures), you can easily reach this limit. Of course, it definitely is much nicer to have S3 over google drive for such a solution, but if you didn't want to write software (which you'd need for S3), and simply used the google drive's sync'ing features...
  - Gigachad 4y ago
    In this case you can just delete the oldest files, or tar/zip them in to one file. I doubt anyone needs instant granular access to 5 million images. The chance of hitting this limit is so insanely slim and the workarounds are so trivial that I don't really see any issues here. I bet there would have been some abuse where someone tried to store data in file names and metadata which is not counted to your quota.

throwaway81523 4y ago

It seems like there is a much lower limit for "download all". There is a 100 or so item directory that I want to download, and "download all" tries to wrap them in a zip file and fails. I think it wants to pre-scan all the files for viruses before zipping, but is unwilling to do that many. So I have to download the files one at a time.

deleted 4y ago
[deleted]
bobbylarrybobby 4y ago
I've definitely downloaded folders with 100 items before. I think it might have to do with how big the individual files are because g drive won't create a zip that's larger than 2GB.
dublinben 4y ago
In the future, you can use rclone to easily download entire folders from Google Drive. https://rclone.org/ https://rclone.org/
- throwaway81523 4y ago
  Hmm, thanks, that looks promising, but it appears to be an all singing all dancing complicated client that does two way transfers to gdrive and expects you to have a google account. I don't have a google account (don't want to enroll one), and am just trying to download a public folder that someone gave me a link to. I'll keep looking to see if I can find a way to use it.

synctheship 4y ago

Well this is scary, I wonder if all users/plans are impacted? I have north of 8.5million files in GDrive and I luckily have not hit an error yet.

danpalmer 4y ago
What are you using it for!?! Honestly, I'm interested to hear what your use-case is as I struggled to come up with one for this many files.
- synctheship 4y ago
  We've been using GDrive in one form or another since 2014 when they released Google Drive for Work. Primary use case nowadays for my team is an old legacy tool(been in place since 2014) that connects to our ERP and other internal tools to generate reports and export data with combined metadata from multiple tools/sources. Then if needed, makes a copy to different share drives and teams, assigns access rights for internal and external users. We have custom web front ends that allow them to interact with these exports along with fat clients if they need to manipulate a larger dataset of files. My last check had the service account that I'm responsible for somewhere around 8.7 Million files with an average size around 9.2MB. We have multiple of these "accounts", with similar use-cases across other teams. This is excluding the normal use of Gdrive with normal "rank-and-file" office works managing standard office docs. We've felt the squeeze from google over the past few years, so we've already started migrating off of google services about 2.5 years into a 4 year project.
  - danpalmer 4y ago
    Yeah this doesn't seem to be in line with the expected use or target market of Drive. It sounds like your tool is probably creating all the files acting as one service account, and therefore hitting the limit if it is indeed per user. I think the rule of thumb I would use is that if you're creating the files in an automated way, and then accessing them outside of Drive interfaces, in all cases, then it's probably a better fit for a web app with cloud storage backend. Google Cloud Storage or S3 would work well for this.

ccheney 4y ago

Someone must've copied their node_modules folder into Google Drive by mistake...

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

up2isomorphism 4y ago

The key is that every company is very happy to lose this kind of customers.