5 ms·
10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. The
by acchow 3y ago
10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops.
There's tons of OSS to manage all layers of this stuff now.
- colechristensen 3y agoYou can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.
- icedchai 3y agoWhat about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.
- fh973 3y agoUse a distributed file system or object store with what you need, like Quobyte or Ceph.
- colechristensen 3y agoPlenty of tools for that on premises. Are you actually asking?
- icedchai 3y agoNo, I'm just mentioning these things to show that the decision is more complex than a "4U server" vs "S3." S3/AWS gives you a ton of stuff you'd have to implement yourself. Plus you need to hire more people to manage it.
- colechristensen 3y agoMinio does most of S3 for example, duh you would have to do some stuff yourself but you wouldn’t be implementing everything yourself from scratch, far from it. https://min.io/ https://min.io/
- kazen44 3y agohalf of these are already managed by something like ZFS.. (the only one i can think of which is not inherent in the Filesystem is High availability). But one should build that kind of redundancy on a higher level anyways. (multiple storage pools, replicating writes etc). or, if you go fibre channel, just replicate the writes across multiple storage systems and be done with it. (this requires a fibre channel network though, which is neither easy nor cheap).
- simonjgreen 3y agoSlight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of mirrors" (RAID10) gives you the safest and the highest performance array (at scales where these decisions are being made which most people would see anyway). The common argument against is the cost (usableCapacity = totalCapacity/2) but my counter argument is that RAID5 is a timebomb with a secret countdown. RAID5 is great if you don't need the data on the array. Source: >20 years running complex large arrays
- buildbot 3y agoOr “software” raid 10 with ZFS mirror dev’s all slapped into one pool?
- simonjgreen 3y agoPrecisely that
- wongarsu 3y agoWhat's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?
- zokier 3y agoWith Ceph, and I imagine other similar solutions, you can skip RAID altogether and manage redundancy at higher level over plain disks. RAID makes sense if you specifically want to have conventional filesystem on top of it, but for object storage its not necessary.
- simonjgreen 3y agoYou're applying the same principles, just at the object level rather than the filesystem level. You still have to choose a redundancy pattern. But I do agree with you, it does make the problem significantly easier/different. ZFS and Ceph are marvelous creations.
- aljarry 3y agoBut that's on a single RAID in a single location. You'd probably need at least one failover, and backups for that as well, if you'd like to target higher availability / durability. S3 keeps multiple copies in multiple geographic locations.
- jlokier 3y agoWhen you are distributing over multiple geographic locations, and much of the data can tolerate a longer round trip time for access, you can extend the RAID pattern across multiple locations, instead of within a location. Provided you have enough network bandwidth for recovery, you don't need a full RAID at each individual site. Having more locations is actually helpful for this. Because of the independence of different locations, the risk of data loss is in some ways lower than a local RAID system where all drives might be fried at the same time by a single event.