7 ms·
Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
- leesalminen 6y ago> That’s almost 13,000 objects for each person in the world, or 50 objects for every one of the roughly two trillion galaxies (according to this 2021 estimate) in the Universe. That’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.
- TedDoesntTalk 6y ago> It’s like the new Library of Alexandria. Presumably the Library of Alexandria didn’t have access control lists
- jcims 6y agoAnd look what happened...
- herodoturtle 6y agoThere's a joke in here about OVH...
- leesalminen 6y agoFor sure ;) but I also imagine that the number of objects with a public-read ACL (potentially mis-configured) would still dwarf Alexandria.
- koolba 6y agoIf it did then Caesar wouldn’t have had permission to burn it.
- tmotwu 6y ago> It’s like the new Library of Alexandria. Would be very unfortunate if the S3 story ends like the Library of Alexandria.
- jliptzin 6y agoWhat’s way more mind boggling to me is that there are at least 260 galaxies per person in the world. I am not shocked at all that S3 is home to any number of cat pictures many of which are probably duplicates.
- anderspitman 6y agoFound the IPFS user.
- iamgopal 6y agoI don’t know, but to me 260 per person seems less than what I would guess.
- intsunny 6y agoCan we update the title? 100T reads as "100 terabytes" instead of "100 trillion" (as per the source).
- sneak 6y ago100 terabytes is 100TB. The T prefix without a unit means 100 trillion, just as 100TB means 100 trillion bytes. I think it's clear as-is.
- woudsma 6y agoI read it as 100 trillion. 100TB would be the 'right' suffix for terabytes IMHO.
- alpaca128 6y ago> 100T reads as "100 terabytes" No, it doesn't. Not more than it reads terameters or teragrams. SI prefixes were not invented specifically for computer memory.
- nrmitchi 6y ago100 terabytes would be '100TB', not '100T'. Further, '100 Terabyte Objects' wouldn't really make sense as a sentence anyways.
- globular-toast 6y agoIt should be read as "100 tera-objects", meaning 100 trillion objects. Writing "K" seems to be so widespread these days (Youtube likes/dislikes are written as 1.5K, for example), I'm surprised there is any confusion here.
- TedDoesntTalk 6y ago> S3 is designed to provide “11 9’s” (99.999999999%) of durability Has anyone lost data on S3 or know anyone who has?
- gnrlst 6y agoThere's different storage tiers, with different redundancy. So if anyone has an anecdote, would be curious to know which storage tier was used (e.g. single availability zone)
- ZiiS 6y agoIf it works as designed I guess 1000 have; but how would you know. Did your perfect code running on an perfect CPU and RAM connected though a perfect network upload the file with 11 9's certainty?
- donavanm 6y agoThis is a real problem, but it is partially accounted for: > To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value. It’s not as obvious with multipart uploads where each chunk is (iirc) individually checksumd and stored, but only a single md5 is returned in the “etag”
- pauldino 6y agoNot to me, but it’s happened. https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanently/answer/Scott-Bonds https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...
- rhacker 6y agoI imagine that in most cases it's a giant archive - data that no one really cares about (I mean they do in principal). Then there's probably like 0.1% that's probably operationally important data - which is actually huge. This is just a guess based on my own usage of it over the last 15 years at different companies.
- bluedino 6y agoDo they de-dupe at all?
- jgalt212 6y agooops, I think you found their secret sauce for cost control and economies of scale.
- weird-eye-issue 6y agoThere is no way
- jgalt212 6y agoprove me wrong.
- babelfish 6y agoYou have the burden of proof for your claim.
- tpmx 6y agoIt's obviously technically possible.
- weird-eye-issue 6y ago"Technically possible" that they do it, sure. But I would disagree it is possible that it is S3's "secret sauce".
- tpmx 6y agoBecause?
- weird-eye-issue 6y ago
- corban1 6y agoI am really getting afraid of this centralization of data.
- stingraycharles 6y agoAt the same time, the S3 api is so ubiquitous that there implementations for it in multiple clouds, it frequently comes with hardware storage vendors as well, and there are projects like minio that provide a solution to host your own s3 compatible data architecture. All things considered, I’m much more concerned about the centralization of eg browser technologies than this.
- klodolph 6y agoEh… the thing about cloud storage is that it can be efficient in ways that smaller players cannot. A system like Amazon S3 can pack its hard drives to the brim with data and pay a fairly small overhead for replicas & unused drives. If you try to decentralize the data, or keep your data yourself, you end up with something that uses more resources, has worse durability, or is worse along some other axis. I am not so worried about centralization yet. There is still healthy competition between different cloud storage vendors. My impression is that cheap storage is used to make other cloud services more attractive—which means that Amazon’s incentives here are somewhat aligned with mine (as a customer). Personally I am much more worried about the centralization of the shopping experience.
- ghgdynb1 6y agoI think it’s fine as long as people know what they’re getting themselves into. I’m sure I would’ve found whatever was happening on Parler abhorrent, but in the same way Merkel and Macron were concerned by the tech deplatforming that followed the capitol riots, so am I. I don’t think filecoin will ever demolish a fantastic business like S3 based on stealing away customers. The interesting value proposition from the protocol is probably more about censorship resistance and adjacent issues.
- astuyvenberg 6y agoS3 may be my favorite AWS product. It's the epitome of "It just works" design.
- ignoramous 6y agoOne of the tenets for the S3 team is/was "indistinguishable from perfect." I guess they're living up to it. https://www.allthingsdistributed.com/2010/05/amazon_s3_reduced_redundancy_storage.html https://www.allthingsdistributed.com/2010/05/amazon_s3_reduc...
- PaulHoule 6y agoI am worried. We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing. It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.
- herodoturtle 6y agoAt this rate, won't AWS run out of storage space before they run out of UUIDs?
- PaulHoule 6y agoor the waste heat boils the oceans... or something
- pradn 6y agoIs this a real problem? It should be possible for Amazon to update their code to generate IDs a different way, and for them to update their client libraries well ahead of a time when this becomes a problem. Yes, code that relies on the ID being a particular format will break. But users have to deal with deprecated client library features all the time. It's reasonable to give a few year grace period before old APIs are shut down. UUIDs are usually not simply random numbers. So it's possible to tell when a particular standard of UUID is used. Amazon's code can branch based on this detection, or even something simpler like "UUID-type", a new field that would be initialized by default to the old UUID type. I simply don't see this as a problem for anyone.
- gonzo41 6y agoAll AWS need to do is add something like a namespace. Shazzam! problem solved for your lifetime.
- gabagool 6y agoIs there any specific reason one could not just put two UUID-4's alongside one another? Also as another commenter mentioned, if they are tied to a namespace prefix, that delays collisions.
- simonebrunozzi 6y agoIf you're curious about some "S3 internals", this is a talk I gave in 2009 [0], based on a ton of interesting stuff that S3's then-GM Alyssa Henry [1] shared with me. [0]: https://vimeo.com/7330740 https://vimeo.com/7330740 [1]: http://itc.conversationsnetwork.org/shows/detail5273.html http://itc.conversationsnetwork.org/shows/detail5273.html
- jodrellblank 6y agoThis recent interview with Werner Vogels[1] touches on S3 design priorities as well: https://cacm.acm.org/magazines/2021/3/250706-a-second-conversation-with-werner-vogels/fulltext https://cacm.acm.org/magazines/2021/3/250706-a-second-conver... https://news.ycombinator.com/item?id=26365873 https://news.ycombinator.com/item?id=26365873 [1] which you commented on, but others might not have seen.
- tw04 6y ago>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled I have a tough time taking him seriously reading this - I'm not sure if he's being intentionally ignorant or intentionally misleading. I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical. Their goal is 5-9's of UPTIME - the expectation is 100% data durability. You would get fired if you architected a solution keeping two copies of data in one datacenter. Object storage absolutely has a place and I'm happy amazon was able to push a quasi-standard for the industry. Object storage prior to S3 was a hodge-podge of proprietary plays (EMC Centera, Bytecast, etc) and sort-of standards that nobody really used (CDMI). I just wish they did a better job of fairly representing it vs. turning every opportunity into a sales pitch spreading FUD about the alternatives.
- ctvo 6y ago
- codegeek 6y agoOne of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3. aws s3 sync ... And you are in business.
- astuyvenberg 6y agoWhen people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer. But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
- selfhoster11 6y agoMinio and other S3-compatibles make moving off of AWS that much simpler, though obviously it might be difficult if you're using Amazon-only S3 features extensively.
- astuyvenberg 6y agoEspecially if you layer on additional functionality, like triggering a lambda function to resize an image when it hits the S3 bucket.
- luhn 6y agoYeah, even if you don't use any vendor-specific features, there's data gravity—It's hard to move your data from one location to another. AWS and others increase this cost even further by making egress so expensive.
- astuyvenberg 6y agoPrecisely - vendor lock-in generally happens at the data level. For example, AWS finally turned off their last Oracle DB server only a year or two ago.
- throwawaysea 6y ago> tens of millions of requests per second How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware? Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?
- ttul 6y agoI’m wildly guessing here, but I assume that there are many tiers of load balancing to turn your initial request into the message that reaches the thing that gives back your file data. They also likely do have specialized hardware in various parts of the AWS infrastructure. A service that can handle tens of millions of requests per second can probably outscale most DDoS attacks without further engineering.
- zokier 6y ago> At some point isn't contacting this service really just contacting a handful of IP addresses? "Handful". S3 has 283 subnets allocated to it, biggest one being a /15, so there are some few hunderd thousand IP addresses reserved for S3. While impossible to say how many of them are actively in use, I imagine large fraction of them. The AWS IP ranges are documented in https://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json
- dehrmann 6y agoI wonder how many objects have been lost and how many have been corrupted.
- TheGuyWhoCodes 6y agoS3 is a great service but it's a double edge sword. It's easy to start with and "cheap" until you scale and then you come to realize you pay a ton for writes and you have a latency of 100ms+ to retrieve data. Even S3 select which on face value seems great is absurdly slow. Then they offer you Fsx or elastic cache as a proxy.
- rootsudo 6y agoWow, has it really been 15 years?
- dna_polymerase 6y ago> All of those decisions remain valid and code written on launch day will still work just fine today. And this is why GCP will never come close to AWS.
- joana035 6y agoI wonder how much of ceph inspired s3...
- notacoward 6y agoVery little, I'm sure. S3 and Ceph were both introduced to the world in 2006, which means they had been in development at the same time - or perhaps S3 even a bit before, since it was in a far more mature state by then. Also, while S3 and Ceph's RADOS both supposedly handle "objects" they're very different kinds of objects with very different APIs. And of course the metadata layers are completely different to serve different needs. It's practically certain that they both drew on the same common ancestors, but it's more likely that lessons learned from S3 informed subsequent development on Ceph than the other way around.