3 ms·

On a related topic, I've had a ton of problems finding a cloud storage system that will reliably handle files around 100-200gb. Does anyone have a recommendatio

On a related topic, I've had a ton of problems finding a cloud storage system that will reliably handle files around 100-200gb. Does anyone have a recommendation for a service that can handle that file size with ease?

brian_herman__ 7y ago
Have you tried amazon s3? What is your use case? Can you elaborate? edit:Grammar
toomuchtodo 7y ago
Any object storage system (S3, Backblaze B2, Azure Blob, GCP) should be able to handle those file sizes with proper chunking into smaller parts (limit details below per object store). S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.... (Max object size: 5TB, Max single multipart size: 5GB) Backblaze B2: https://www.backblaze.com/b2/docs/large_files.html https://www.backblaze.com/b2/docs/large_files.html (Max file size: 10TB, Max single object size: 5GB) Azure: https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs https://docs.microsoft.com/en-us/rest/api/storageservices/un... (Max file size: 4.75 TB, Max single block size: 100MB) Google: https://cloud.google.com/storage/quotas https://cloud.google.com/storage/quotas (Max file size: 5TB, doesn't appear there is a lower limit for objects to be composed into a single object, docs could be better in this regard) @khc: Terminology updated to be more clear for S3
- reaperducer 7y ago
  I'd drop Microsoft from that list because the request for for something "reliable." It's reliable enough, if you can get it to Microsoft's cloud. But for the last six months I've struggled putting very large files into Azure, using five different connections from five different providers in three locations. Small files are no problem. But large ones take two, three, or four tries.
  - toomuchtodo 7y ago
    I don't disagree Azure is hot garbage, just listing for completeness. S3 and Backblaze B2 are my go-to object stores.
- khc 7y ago
  S3 max single object size is 5TB. The 5GB limit is for single part upload API. You can use the multi-part API to upload single object in multiple parts.
- BrandonY 7y ago
  Hi, GCS engineer here. The lower limit on composing objects is one source object, in which case you are not so much composing as you are copying with style. Zero source objects is an error. I will file a note about the docs, thanks.
  - toomuchtodo 7y ago
    Thank you so much! Is there a limit on size of objects besides the 5TB max described in the docs, similar to other object stores where multiparts have lower limits than the total composed object?
    - BrandonY 7y ago
      There is also a lower bound of 0 bytes :) You can compose a 4TB object with a 1 byte object, or you can compose 32 150GB objects, just so long as the destination object doesn't go over 5 terabytes.
      - toomuchtodo 7y ago
        Brilliant! Thank you again!
Taek 7y ago
Sia, the system described above, comfortably handles files of 100-200 gb, and with the recent Skynet link also enables filesharing for files up to 400 GB.
rsync 7y ago
"On a related topic, I've had a ton of problems finding a cloud storage system that will reliably handle files around 100-200gb. Does anyone have a recommendation for a service that can handle that file size with ease?" rsync.net gives you an empty, ZFS filesystem that you can do anything you like with.[1] I believe the file-size limit is 16 exbibytes (264 bytes). rsync.net can also talk to every cloud storage provider[2] because rclone[3] is built into the platform: ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir [1] http://rsync.net http://rsync.net [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html [3] https://rclone.org/ https://rclone.org/
- mkl 7y ago
  Looks like HN ate the * characters in 2**64 16 exbibytes is 2^64 bytes.

brian_herman__ 7y ago

Have you tried amazon s3? What is your use case? Can you elaborate? edit:Grammar

toomuchtodo 7y ago

Any object storage system (S3, Backblaze B2, Azure Blob, GCP) should be able to handle those file sizes with proper chunking into smaller parts (limit details below per object store). S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.... (Max object size: 5TB, Max single multipart size: 5GB) Backblaze B2: https://www.backblaze.com/b2/docs/large_files.html https://www.backblaze.com/b2/docs/large_files.html (Max file size: 10TB, Max single object size: 5GB) Azure: https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs https://docs.microsoft.com/en-us/rest/api/storageservices/un... (Max file size: 4.75 TB, Max single block size: 100MB) Google: https://cloud.google.com/storage/quotas https://cloud.google.com/storage/quotas (Max file size: 5TB, doesn't appear there is a lower limit for objects to be composed into a single object, docs could be better in this regard) @khc: Terminology updated to be more clear for S3

reaperducer 7y ago
I'd drop Microsoft from that list because the request for for something "reliable." It's reliable enough, if you can get it to Microsoft's cloud. But for the last six months I've struggled putting very large files into Azure, using five different connections from five different providers in three locations. Small files are no problem. But large ones take two, three, or four tries.
- toomuchtodo 7y ago
  I don't disagree Azure is hot garbage, just listing for completeness. S3 and Backblaze B2 are my go-to object stores.
khc 7y ago
S3 max single object size is 5TB. The 5GB limit is for single part upload API. You can use the multi-part API to upload single object in multiple parts.
BrandonY 7y ago
Hi, GCS engineer here. The lower limit on composing objects is one source object, in which case you are not so much composing as you are copying with style. Zero source objects is an error. I will file a note about the docs, thanks.
- toomuchtodo 7y ago
  Thank you so much! Is there a limit on size of objects besides the 5TB max described in the docs, similar to other object stores where multiparts have lower limits than the total composed object?
  - BrandonY 7y ago
    There is also a lower bound of 0 bytes :) You can compose a 4TB object with a 1 byte object, or you can compose 32 150GB objects, just so long as the destination object doesn't go over 5 terabytes.
    - toomuchtodo 7y ago
      Brilliant! Thank you again!

Taek 7y ago

Sia, the system described above, comfortably handles files of 100-200 gb, and with the recent Skynet link also enables filesharing for files up to 400 GB.

rsync 7y ago

"On a related topic, I've had a ton of problems finding a cloud storage system that will reliably handle files around 100-200gb. Does anyone have a recommendation for a service that can handle that file size with ease?" rsync.net gives you an empty, ZFS filesystem that you can do anything you like with.[1] I believe the file-size limit is 16 exbibytes (264 bytes). rsync.net can also talk to every cloud storage provider[2] because rclone[3] is built into the platform: ssh user@rsync.net rclone s3:/some/bucket rsync/home/dir [1] http://rsync.net http://rsync.net [2] https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html [3] https://rclone.org/ https://rclone.org/

mkl 7y ago
Looks like HN ate the * characters in 2**64 16 exbibytes is 2^64 bytes.