Actual answer: There is almost NO company that really needs that much data. This has mostly just become a pissing match. In general, companies (especially st

Actual answer: There is almost NO company that really needs that much data. This has mostly just become a pissing match. In general, companies (especially startups) are way better off making sure they have a small amount of high-quality, accurate, data than a huge pile-o-dung that they think they're going to use magical AI/ML pixie dust to do something with. That said, if you really think you must, spend effort on good deduping/transcoding (relatively easy with images/video), and consider some far lower-cost storage options than S3, which is pretty pricey no matter what you do. If S3 is a good fit, I hear good things about Wasabi, but haven't used it myself. If you have the technical ability (non-trivial, you need someone who really understands, disk and system I/O, RAID Controllers, PCI lane optimization, SAN protocols and network performance (not just IP), etc.) and the wherewithal to invest, then putting this on good hardware with something like say, ZFS at your site or a good co-lo will be WAY cheaper and probably offer higher performance than any other option, especially combined with serious deduping. (Look carefully at everything that comes in once and you never have to do it again.) Also, keep in mind that even-numbered RAID levels can make more sense for video streaming, if that's a big part of the mix. The MAIN thing: Keep in mind that understanding your data flows is way more important than just "designing for scale". And really try to not need so much data in the first place. (Aside: I'm was cofounder and chief technologist of one of the first onsite storage service providers - we built a screamer of a storage system that was 3-4x as fast, and scaled 10x larger than IBM's fastest Shark array, at less than 10% of the cost. The bad news - we were planning to launch the week of 9/11 and, as self-funded, ran out of money before the economy came back. The system kicked ass, though.)