5 ms·

I don’t think it’s reasonable. I think it’s poor engineering. Imagine if a hard drive had a limit on the number of files, not just the size. Would that be reas

I don’t think it’s reasonable. I think it’s poor engineering. Imagine if a hard drive had a limit on the number of files, not just the size. Would that be reasonable. Or imagine if someone designed a storage protocol and submitted it to IETF with such a limitation saying “it doesn’t affect the vast majority of users so we set this arbitrary limit to make our lives easier.” It would not make it through review.

Kye 4y ago
File systems do have a limit on number of files, though as far as I know they're all quite a bit higher than anything in the millions.
- Wowfunhappy 4y ago
  Of course, with a physical disk there are also things you can do to get around those limitations, like switch to a different filesystem or use multiple filesystems on one disk. Using the cloud leaves you exposed to the whims of the cloud provider. They select and switch technologies based on their needs instead of yours.
  - Kye 4y ago
    If you manage to hit five billion files (which seems to be the standard limit), you might be in the market for object storage rather than new frontiers in file systems.
    - yjftsjthsd-h 4y ago
      > five billion files (which seems to be the standard limit), Article says 5 million? Still a big number, but smaller than 5 billion.
      - Kye 4y ago
        We're talking about file systems.
        yjftsjthsd-h 4y ago
        Then where are we getting 5 billion? The most common filesystem limits listed seem to be 2^32, which is a touch over 4 billion, and 2^64, which is Big. (i.e. I can see now what I think you meant, but "5 billion" is a much closer match to "5 million" than 2^32)
        Kye 4y ago
        My brain doesn't do numbers so good, so 4,294,967,295 became 4,924,967,295 and I rounded up instead of down.
        yjftsjthsd-h 4y ago
        Ah, that makes sense. Completely understandable:)
  - amf12 4y ago
    > Of course, with a physical disk there are also things you can do to get around those limitations, like switch to a different filesystem or use multiple filesystems on one disk. So, use another Google account? If someone's workflow depends on one Google account, sounds like they need to understand the concept of sharding.
- prepend 4y ago
  Of course they have limits, but the limits should be based on the design. And cloud should have a phenomenally high level.
viraptor 4y ago
https://en.wikipedia.org/wiki/Comparison_of_file_systems https://en.wikipedia.org/wiki/Comparison_of_file_systems almost every filesystem has a limit. Some document it, some not. Some limit it per directory rather than globally. There are also path limits which are effectively a file number limits as well. But the limit itself is very normal and would make it through reviews without issues.
- Kwpolska 4y ago
  NTFS and ext4 have a 2^32 limit according to this table. The Reddit poster mentioned in the article [0] has 2 TB worth of space. I’m not sure if that’s decimal or binary TB, but if we assume binary, you get an average of 512 bytes per file, which is pretty reasonable (and on a real drive, you might be limited by sector size). But even if you had a 20 TB hard drive, and you wanted a lot of files smaller than 5120 bytes, you can just create multiple partitions. Maybe a little inconvenient, but you can still use your drive to the fullest. Not so much with Google’s 5 million limit (for a 2 TB Google Drive, the minimum average file size is around 430 KB [2]) [0] https://www.reddit.com/r/google/comments/123fjx8/google_has_applied_a_5_million_items_limit_for/ https://www.reddit.com/r/google/comments/123fjx8/google_has_... [1] https://www.wolframalpha.com/input?i=%282%2F2%5E32%29+TiB+in+bytes https://www.wolframalpha.com/input?i=%282%2F2%5E32%29+TiB+in... [2] https://www.wolframalpha.com/input?i=%282%2F5000000%29+TiB+in+bytes https://www.wolframalpha.com/input?i=%282%2F5000000%29+TiB+i...
- yjftsjthsd-h 4y ago
  Sure, everything has some limit. But the lowest limit I see on that page is 2^32, which is just a touch higher than 5 million. (/s) And far more importantly, none of those filesystems just decided one day to reduce the limit, which is kind of a big deal. (Although funny, in a horrible way; imagine booting up your machine one day and it tells you that your home filesystem needs to drop a few million files before it can mount. Can you imagine?)
  - viraptor 4y ago
    > none of those filesystems just decided one day to reduce the limit Yes and no. This works differently from one filesystem to another, but search for "running out of inodes" to find many ways this can happen in reality, way below the technical limit of the filesystem itself. You can end up in situations where you can't create a new file and need to remove more than one to solve the situation. (Or even where removing many files doesn't seem to make a difference)
Thaxll 4y ago
It's poor engineering to not understand that everything has limits.
- prepend 4y ago
  Everything has limits. The limits should be based on hardware limitations, not arbitrary limits. The limits should increase over time.
mindslight 4y ago
Said like someone who has never run out of inodes! There's a reason `df -i` exists.
IggleSniggle 4y ago
Without a limit, you could design a storage system using empty files on google drive with limitless storage potential.
- jjnoakes 4y ago
  Why couldn't Google just count the file name length and perhaps a small constant metadata size against your quota?
  - acdha 4y ago
    That is a limit. It’s a different one but it tacitly accepts that some check is needed.
    - aflag 4y ago
      They already had a total size limit. The complaint is that they introduced a new one without really telling anyone.
    - Kwpolska 4y ago
      Not a limit as small as Google set, and not a limit added overnight without any previous announcement or grace period. If you want to limit storing data in names of blank files, you could say “blank files, files smaller than 1024 bytes, and folders are counted as 1024 bytes for quota purposes”. A minimum billable size seems fair to me, and there is prior art (e.g. AWS S3 has minimum sizes for some storage classes).
- dwaite 4y ago
  on real filesystems, directories are effectively files and take up space. I see no reason for them not to have zero byte files take from the storage limits due to metadata.
dboreham 4y ago
> Imagine if a hard drive had a limit on the number of files NPM developer? <ducks>
- victortroz 4y ago
  pnpm helped me a lot with this.

Kye 4y ago

File systems do have a limit on number of files, though as far as I know they're all quite a bit higher than anything in the millions.

Wowfunhappy 4y ago
Of course, with a physical disk there are also things you can do to get around those limitations, like switch to a different filesystem or use multiple filesystems on one disk. Using the cloud leaves you exposed to the whims of the cloud provider. They select and switch technologies based on their needs instead of yours.
- Kye 4y ago
  If you manage to hit five billion files (which seems to be the standard limit), you might be in the market for object storage rather than new frontiers in file systems.
  - yjftsjthsd-h 4y ago
    > five billion files (which seems to be the standard limit), Article says 5 million? Still a big number, but smaller than 5 billion.
    - Kye 4y ago
      We're talking about file systems.
      - yjftsjthsd-h 4y ago
        Then where are we getting 5 billion? The most common filesystem limits listed seem to be 2^32, which is a touch over 4 billion, and 2^64, which is Big. (i.e. I can see now what I think you meant, but "5 billion" is a much closer match to "5 million" than 2^32)
        Kye 4y ago
        My brain doesn't do numbers so good, so 4,294,967,295 became 4,924,967,295 and I rounded up instead of down.
        yjftsjthsd-h 4y ago
        Ah, that makes sense. Completely understandable:)
- amf12 4y ago
  > Of course, with a physical disk there are also things you can do to get around those limitations, like switch to a different filesystem or use multiple filesystems on one disk. So, use another Google account? If someone's workflow depends on one Google account, sounds like they need to understand the concept of sharding.
prepend 4y ago
Of course they have limits, but the limits should be based on the design. And cloud should have a phenomenally high level.

viraptor 4y ago

https://en.wikipedia.org/wiki/Comparison_of_file_systems https://en.wikipedia.org/wiki/Comparison_of_file_systems almost every filesystem has a limit. Some document it, some not. Some limit it per directory rather than globally. There are also path limits which are effectively a file number limits as well. But the limit itself is very normal and would make it through reviews without issues.

Kwpolska 4y ago
NTFS and ext4 have a 2^32 limit according to this table. The Reddit poster mentioned in the article [0] has 2 TB worth of space. I’m not sure if that’s decimal or binary TB, but if we assume binary, you get an average of 512 bytes per file, which is pretty reasonable (and on a real drive, you might be limited by sector size). But even if you had a 20 TB hard drive, and you wanted a lot of files smaller than 5120 bytes, you can just create multiple partitions. Maybe a little inconvenient, but you can still use your drive to the fullest. Not so much with Google’s 5 million limit (for a 2 TB Google Drive, the minimum average file size is around 430 KB [2]) [0] https://www.reddit.com/r/google/comments/123fjx8/google_has_applied_a_5_million_items_limit_for/ https://www.reddit.com/r/google/comments/123fjx8/google_has_... [1] https://www.wolframalpha.com/input?i=%282%2F2%5E32%29+TiB+in+bytes https://www.wolframalpha.com/input?i=%282%2F2%5E32%29+TiB+in... [2] https://www.wolframalpha.com/input?i=%282%2F5000000%29+TiB+in+bytes https://www.wolframalpha.com/input?i=%282%2F5000000%29+TiB+i...
yjftsjthsd-h 4y ago
Sure, everything has some limit. But the lowest limit I see on that page is 2^32, which is just a touch higher than 5 million. (/s) And far more importantly, none of those filesystems just decided one day to reduce the limit, which is kind of a big deal. (Although funny, in a horrible way; imagine booting up your machine one day and it tells you that your home filesystem needs to drop a few million files before it can mount. Can you imagine?)
- viraptor 4y ago
  > none of those filesystems just decided one day to reduce the limit Yes and no. This works differently from one filesystem to another, but search for "running out of inodes" to find many ways this can happen in reality, way below the technical limit of the filesystem itself. You can end up in situations where you can't create a new file and need to remove more than one to solve the situation. (Or even where removing many files doesn't seem to make a difference)

Thaxll 4y ago

It's poor engineering to not understand that everything has limits.

prepend 4y ago
Everything has limits. The limits should be based on hardware limitations, not arbitrary limits. The limits should increase over time.

mindslight 4y ago

Said like someone who has never run out of inodes! There's a reason `df -i` exists.

IggleSniggle 4y ago

Without a limit, you could design a storage system using empty files on google drive with limitless storage potential.

jjnoakes 4y ago
Why couldn't Google just count the file name length and perhaps a small constant metadata size against your quota?
- acdha 4y ago
  That is a limit. It’s a different one but it tacitly accepts that some check is needed.
  - aflag 4y ago
    They already had a total size limit. The complaint is that they introduced a new one without really telling anyone.
  - Kwpolska 4y ago
    Not a limit as small as Google set, and not a limit added overnight without any previous announcement or grace period. If you want to limit storing data in names of blank files, you could say “blank files, files smaller than 1024 bytes, and folders are counted as 1024 bytes for quota purposes”. A minimum billable size seems fair to me, and there is prior art (e.g. AWS S3 has minimum sizes for some storage classes).
dwaite 4y ago
on real filesystems, directories are effectively files and take up space. I see no reason for them not to have zero byte files take from the storage limits due to metadata.

dboreham 4y ago

> Imagine if a hard drive had a limit on the number of files NPM developer? <ducks>

victortroz 4y ago
pnpm helped me a lot with this.