4 ms·

Does anyone have rough numbers on s3 scale? Most storage classes have redundant copies as well so 3x things maybe from that. YouTube / S3 - a few services seem

• majewsky 4y ago
  If you're building one of these object storages for yourself (say, using Ceph or OpenStack Swift), a plausible configuration is 12 drives with 8 TB each, or 96 TB per node [1]. Let's round that to 100 TB for simplicity. You obviously want to spread your replicas across multiple datacenters, so let's just say you'll put 10 of these servers (i.e. around half a rack full) each into three different datacenters. That's a very modest deployment in terms of floor space, and yet already has a capacity in the low petabyte range (3 PB raw capacity, 1 PB net capacity). On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet. [1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.
  • QuinnyPig 4y ago
    Erasure coding drops the drive requirements massively; they talked about it at reinvent 2021.
• oxymoron 4y ago
  There is a talk by the S3 VP on Youtube which mentions some rough numbers, I think it’s from re:invent 2019. Also, they mention 100 trillion objects here https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it-is-still-day-1-after-5475-days-100-trillion-objects/ https://aws.amazon.com/blogs/aws/amazon-s3s-15th-birthday-it...