4 ms·

Amazon S3 - Object Size Limit Now 5 TB

60 points by jeffbarr 16y ago

bobf 16y ago
Although I wouldn't argue there is an immediate need for 5TB objects, raising the previous 5GB limit is undoubtedly useful.
- mashmac2 16y ago
  I definitely see a connection between this and their recent addition of Cluster GPU instances... big rendering files, scientific data sets, etc... http://aws.amazon.com/about-aws/whats-new/2010/11/15/announcing-cluster-gpu-instances-for-amazon-ec2/ http://aws.amazon.com/about-aws/whats-new/2010/11/15/announc...
fredoliveira 16y ago
Thanks for bringing the discussion over to HN, Jeff. While I don't have a use for the new object size limits (the data we store at S3 is on average of only a few Kb per object - even though we store millions of them), I can't wait to see what people (mainly the scientific community) do with this. Bumping the object limit to 5TB means huge datasets can now be stored, processed straight from and added back to S3 - removing size limits is the final piece of the puzzle to "trivialize" big data processing. PS: It's been fun, and exciting, and unexpected to follow Amazon AWS' growth in the past few years. I remember back when AWS was first announced thinking of how odd it was for a company like Amazon to open its infrastructure like you guys were doing. A little while later, we were fortunate enough to be among the very first companies to use EC2 (Goplan was the first 3rd party product to run on EC2 back in the day - it got us on Wired Magazine). These days, AWS has become an integral part of most startups I see and products I work with. That is a huge deal.
gruseom 16y ago
How do you access a file that large? Presumably you don't have to stream it from S3 to EC2 in its entirety in order to do computations with it. Does that mean you can have random access into it in order to work with a chunk at a time?
- amock 16y ago
  Yes. "For example, you could stream terabytes of data off of a genomic sequencer as it is being created, store the final data set as a single object and then analyze any subset of the data in EC2 using a ranged GET"
- btn 16y ago
  S3's REST API supports the HTTP 'Range' header to pluck bits of an object out (with a similar facility in its SOAP API).
cperciva 16y ago
Hmm... 5 TB is enough to store 12 trillion digits of Pi.
- jeffbarr 16y ago
  So what are you waiting for?
  - cperciva 16y ago
    I'm busy playing with other things on EC2 right now. :-)

60 points by jeffbarr 16y ago

bobf 16y ago

Although I wouldn't argue there is an immediate need for 5TB objects, raising the previous 5GB limit is undoubtedly useful.

mashmac2 16y ago
I definitely see a connection between this and their recent addition of Cluster GPU instances... big rendering files, scientific data sets, etc... http://aws.amazon.com/about-aws/whats-new/2010/11/15/announcing-cluster-gpu-instances-for-amazon-ec2/ http://aws.amazon.com/about-aws/whats-new/2010/11/15/announc...

fredoliveira 16y ago

Thanks for bringing the discussion over to HN, Jeff. While I don't have a use for the new object size limits (the data we store at S3 is on average of only a few Kb per object - even though we store millions of them), I can't wait to see what people (mainly the scientific community) do with this. Bumping the object limit to 5TB means huge datasets can now be stored, processed straight from and added back to S3 - removing size limits is the final piece of the puzzle to "trivialize" big data processing. PS: It's been fun, and exciting, and unexpected to follow Amazon AWS' growth in the past few years. I remember back when AWS was first announced thinking of how odd it was for a company like Amazon to open its infrastructure like you guys were doing. A little while later, we were fortunate enough to be among the very first companies to use EC2 (Goplan was the first 3rd party product to run on EC2 back in the day - it got us on Wired Magazine). These days, AWS has become an integral part of most startups I see and products I work with. That is a huge deal.

gruseom 16y ago

How do you access a file that large? Presumably you don't have to stream it from S3 to EC2 in its entirety in order to do computations with it. Does that mean you can have random access into it in order to work with a chunk at a time?

amock 16y ago
Yes. "For example, you could stream terabytes of data off of a genomic sequencer as it is being created, store the final data set as a single object and then analyze any subset of the data in EC2 using a ranged GET"
btn 16y ago
S3's REST API supports the HTTP 'Range' header to pluck bits of an object out (with a similar facility in its SOAP API).

cperciva 16y ago

Hmm... 5 TB is enough to store 12 trillion digits of Pi.

jeffbarr 16y ago
So what are you waiting for?
- cperciva 16y ago
  I'm busy playing with other things on EC2 right now. :-)