7 ms·
Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
- MartinMond 14y agoSo how many objects should have been corrupted/lost according to their SLA and how many actually did get corrupted?
- HeyImAlex 14y ago>how many objects should have been corrupted/lost according to their SLA Anywhere from 20 to 200M per year depending on how many people use RRS...
- jeffbarr 14y agoNote that RRS is "Reduced Redundancy Storage" - an S3 option that offers 99.99% durability, as opposed to the 99.999999999% durability offered by Standard S3.
- gphil 14y agoIt's staggering that out of 2 trillion only 20 might be corrupted, assuming they lived up to their SLA. 2000000000000 - (.99999999999 * 2000000000000) = 20 Makes me feel pretty OK about having backups there.
- angersock 14y agoDo objects tend to get lost in clusters, or what?
- Smrchy 14y agoIt would be really interesting to know the average size of an object and visualize the amount of harddisks it takes to store all this data.
- ihsw 14y agoTo put this into perspective it's 2.7B new objects per day (assuming 1 Trillion objects averaged over 365 days). Assuming each object is 100KB (generous estimate, after compression) that would be 270GB per day -- or assuming ten levels of redundancy and striped across three RAID storage devices (per level of redundancy) then 8.1TB per day. I'm not familiar with their hard disk procurement policies but it wouldn't be difficult to assume they've been purchasing 1TB drives, so 10 new disk drives per day just for keeping ahead of growth. Furthermore let's assume their disk drive failure churn rate is 10% per day so another 1 new disk drive for parts replacement (so 11 disk drives per day). These are really loose numbers not based on any actual data (or any personal experience at all) but just napkin math, so take it all with a grain of salt.
- stilldavid 14y agoI'm not convinced that 100KB is a great estimate on file size, but either way you're off by a few zeroes. It's not 270GB per day, it's 270TB. Even if each object were just one byte, that would be 2.7GB. 100KB is one hundred thousand bytes. So it's quite a bit more than eleven drives per day!
- ihsw 14y agoYou are correct, that would be 270TB. After applying the same shoddy math with each object being 100KB -- 270TB with 10 levels of redundancy across 3 RAID drives resulting in 8,100TB per day. This would be 8,100 drives (at 1TB per drive), or 8,910 drives after 10% being dead-on-arrival. The math is sketchy, so let's cut it down by 10x (10KB per object): 891 drives per day. Keep in mind this is just for S3 and it doesn't account for existing drives failing, growth, or what other services require (eg: EC2, RDS, Cloudwatch, Cloudfront, etc).
- deleted 14y ago[deleted]
- SpikeGronim 14y agoAverage size is irrelevant. A handful of 1G objects dwarf hundreds of 1 byte objects when computing the average. The overal distribution is interesting. There are actually three: GET sizes, PUT sizes, and stored sizes. They are not identical distributions, especially since the as the PUT size distribution has changed it becomes out of synch with the stored size distribution. Wish I could tell you more, there are some fascinating data points in there but, you know, NDA. Source: form S3 employee.
- cinquemb 14y agowaiting for the comparison to the total outstanding us derivative exposure :P
- jeffbarr 14y agoDollars and objects don't compare very well.
- TallboyOne 14y agoWhat is an 'object' in this sense?
- chuckmans3 14y agoConsidering we don't hear about problems that often, this is quite an impressive feat of engineering. You really don't think about it until we get a hiccup and half the Internet goes down.
- raulonkar 14y agowhat do u mean by objects?
- jeffbarr 14y agoAn object is a single blob of data. S3 objects vary in size from 1 byte up to 5 Terabytes. They can be uploaded with a single PUT, or with multiple PUTs in series or in parallel (which we call multipart upload). They can be downloaded as a unit, in full (GET) or in part (range GET).
- tomschlick 14y agoAnyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...
- ceejayoz 14y agoNetflix is unlikely to store customer data on S3. They use it for movie files IIRC, but customer data would go in a proper database of some sort. Dropbox does use S3, but I think you're probably doing the fairly standard human mistake of not realizing just how big a trillion is. Dropbox hit 100M users in November of 2012, so for Dropbox to use up two trillion objects each user would need to have 20,000 of them. Dropbox does deduplication, has lots of inactive/minimal users, etc., so they're probably a percent or two of S3 objects.
- saurik 14y agoNetflix stores most of their user data in Cassandra (at the AWS conference last year I attended a talk by them, and specifically asked after some of the things they were storing). (At one point, the stuff I was storing/logging for Cydia actually represented over a percent, maybe it was even over two percent, of all objects in S3; now I'm between 0.1% and 0.5%.)
- charlesju 14y agoI think you are desensitized to what a trillion means. http://demonocracy.info/infographics/usa/us_debt/us_debt.html http://demonocracy.info/infographics/usa/us_debt/us_debt.htm...
- NewAccnt2 14y agoOr ~1 year of money at the current rate of money being printed by the fed ($85 Billion / month)
- fooyc 14y ago1.1 Million Requests/Second seems especially low for Amazon. The average server can serve more than 1,000 static objects / second easily.
- sylvinus 14y ago"If you added one S3 object every 60 hours starting at the Big Bang, you'd have accumulated almost two trillion of them by now." That actually sounds underwhelming! IMHO our brains have an easier time thinking "hey, 1 every 60 hours that's not much" compared to figuring out the universe is really incredibly old ;-)
- jeffbarr 14y agoI'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
- sylvinus 14y agoAwesome ;-) How about comparing to the lifetime of one person? With the US average life expectancy (78 years) that means you'd have to upload 800 objects/second for your whole life to get to 2 trillon :) Edit: One S3 object for each fish in the ocean (3 to 4 trillion) could also be a nice future milestone (if also slightly underwhelming :) Edit2: I also love the eye blink as a unit of time. Each time you blink (average is around 15 times a minute), XXX more objects will have been uploaded to/requested on S3
- Fenster 14y agoIf each object was written on a single standard 8 1/2 x 11 inch piece of paper and stacked up, the stack would be about 125,000 miles high (about half the distance to the moon).
- oskarth 14y agoImagine each object being a person, and imagine a room large enough to fit 10 000 people, the size of a small village, standing side by side, barely being able able to touch each other's hands, spanning roughly 15-20 km. Now imagine putting 9 999 people behind every person. And finally, imagine stacking 9 999 on each of these million people's head. That's twice the height a normal airplane flies at. Then double that number. That's how many objects have been created so far. --- A bit long, but I think the roughly human scale numbers, 999s and even more effect makes it almost imaginable. EDIT: Another one - you could fill the whole of manhattan with cents and still have money to spare. (Manhattan is 87.46 km^2 and a coin is less than about 40mm^2)
- spullara 14y agoI am surprised that the number of requests per second is this low — especially if this includes PUTs. There must be a pretty huge multiple between CloudFront and S3 that keeps this in check.
- ConceitedCode 14y agoAnyone know what technology S3 uses to meet this kind of demand?
- deleted 14y ago[deleted]
- thelegit 14y agoHard drives.. Jk... but seriously, good question! it's gotta just be a massive feet of engineering
- upthedale 14y agoWow, Windows Azure seems to be beating it by a long way. Genuine surprise... I assumed it would be much closer. 9 months ago, they announced they stored twice this amount - 4 trillion objects. A year before that, 1 Trillion. Given that previous rate of growth, we can expect they have a lot more than this now. They also announced peaks of 880,000 requests a second. Whilst Amazon wins here, I'd say its fair to assume this number has increased in those 9 months. http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/20/windows-azure-storage-4-trillion-objects-and-counting.aspx http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/...
- theatraine 14y agoBear in mind that Azure storage includes "Blobs, Disks/Drives, Tables, and Queues" however S3 is only blobs (other services like Amazon's DynamoDB would be analogous to table storage). Hence, it's not an apples-to-apples comparison.
- upthedale 14y agoFair point. Do we have any other numbers that might make it a more apples-to-apples comparison?
- kenneth_reitz 14y agoI love S3, and have been experimenting with using it as a true key value store. https://github.com/kennethreitz/elephant https://github.com/kennethreitz/elephant It's going quite well so far :)