5 ms·

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ ther

by pmlnr 6y ago

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, and ZFS across multiple machines is nasty as far as I'm aware, but I could be wrong. HDFS would work, but the latency can be completely random there. [^1]: https://www.hpe.com/uk/en/storage/apollo-4000.html https://www.hpe.com/uk/en/storage/apollo-4000.html So unless you are desperate to save money in the long run, stick to the cloud, and let someone else sweat about the filesystem level issues :) EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc. EDIT2: at 10PB HDFS would be happy; buy 3 racks of those apollos, and you're done. We started struggling at 1000+ nodes first; now, with 2400 nodes, nearly 250PB raw capacity, and literally a billion filesystem objects, we are slow as f*, so plan carefully.

monstrado 6y ago
MinIO is an option as well and would allow you to transition from testing in S3 to your own MinIO cluster seamlessly.
- dividedbyzero 6y ago
  Does it scale that far?
  - pas 6y ago
    Likely not that "gracefully". But Ceph absolutely does, and has an S3 gateway.
    - tempest_ 6y ago
      Using Ceph like S3 could be a bit tricky if all of that 10pb is very small files. Redhat did an interesting series of blog posts about Ceph and getting it to 1 billion objects https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond https://www.redhat.com/en/blog/scaling-ceph-billion-objects-...
- goliatone 6y ago
  I wonder if anyone can comment if they have experience running minio at scale. It would be a pleasant surprise if a “simple” minio cluster could handle such workload
secabeen 6y ago
You can also get units like this direct from Western Digital/HGST. We have a system with 3 of their 4U60 units, and they weren't all that expensive. Ordering direct from HGST, we only paid a small premium on top of the cost of the SAS drives.
- Terretta 6y ago
  This is the answer that worked for us storing petabytes a decade ago. We collaborated with OEMs and also shared/compared notes with Backblaze on rackable mass storage for commodity drives. Backblaze published a series of iterations of designs of multi-drive chassis, and one of the OEMs would make them for other buyers as well. If you’re doing this route, read through those for considerations and lessons learned. Performance was > 10x better than enterprise solutions. A policy to “leave dead disks dead” aka “let them rot” as said elsewhere in this thread kept maintenance cheap. The secret sauce part making this viable for commercial online storage hosting (we hosted video) was we used disks as JBOD with an in-house meta index with P2P health awareness to place objects redundantly across disks, chassis, racks, colocation providers, and regions.
more_corn 6y ago
Don't buy HPE gear. Qualify the gear with sample units from a few competing vendors and you'll see why.
- specktr 6y ago
  I’ve done qualification on hpe and various competing vendors and honestly haven’t seen dramatic differences in terms of performance and failure rates. From my experience the biggest difference was with vendor support services rather than the actual hardware. I’d be curious to hear more about your qualification experience with this particular vendor if you’d be willing.
walrus01 6y ago
> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc. I am a huge advocate of hosting stuff yourself on bare metal you own, but this is a ridiculous statement. Any drive in that class should come with a 3 or 5 year warranty. And the manual labor and hassle time to replace one (you have hundreds of thousands of dollars of storage and no ready to go cold spares on a shelf?!?!) is infinitesimal.
- closeparen 6y ago
  If the colo is far and there’s plenty of headroom, it might not justify much urgency.
  - cortesoft 6y ago
    That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.
    - Terretta 6y ago
      Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.
      - walrus01 6y ago
        'cattle not pets' is not a valid argument when you're the owner and operator of the bare metal hardware. Do you also recommend that ISPs not replace failed fans in core and edge routers and optical transport systems? Let things with dual power supplies run for six months on one failed power supply? Also you've clearly never interacted with cattle, sheep, goats, llamas or alpacas, which absolutely do get things like veterinary care and vaccinations. Large animal vet is a whole specialty and they spend lots of time working on animals other than horses. No trips to the vet???
lars_francke 6y ago
I'd be interested to learn more about your HDFS usage and your experience at that scale. Would you be willing to have a chat? If so, my email is in my profile.
metabrew 6y ago
When we set up user content storage of images and mp3s for Last.fm back in 2006ish we used MogileFS (from the bradfitz LJ perl days) running on our own hardware. 3/4/5/6u machines stuffed full of disks. I still think it's an elegant concept – easy to grok, easy to debug, easy to reason about. No special distributed filesystem to worry about. Don't take this as an endorsement of the MogileFS perl codebase in 2021, but worth considering this style of storage system depending on your precise needs.

by pmlnr 6y ago

monstrado 6y ago

MinIO is an option as well and would allow you to transition from testing in S3 to your own MinIO cluster seamlessly.

dividedbyzero 6y ago
Does it scale that far?
- pas 6y ago
  Likely not that "gracefully". But Ceph absolutely does, and has an S3 gateway.
  - tempest_ 6y ago
    Using Ceph like S3 could be a bit tricky if all of that 10pb is very small files. Redhat did an interesting series of blog posts about Ceph and getting it to 1 billion objects https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond https://www.redhat.com/en/blog/scaling-ceph-billion-objects-...
goliatone 6y ago
I wonder if anyone can comment if they have experience running minio at scale. It would be a pleasant surprise if a “simple” minio cluster could handle such workload

secabeen 6y ago

You can also get units like this direct from Western Digital/HGST. We have a system with 3 of their 4U60 units, and they weren't all that expensive. Ordering direct from HGST, we only paid a small premium on top of the cost of the SAS drives.

Terretta 6y ago
This is the answer that worked for us storing petabytes a decade ago. We collaborated with OEMs and also shared/compared notes with Backblaze on rackable mass storage for commodity drives. Backblaze published a series of iterations of designs of multi-drive chassis, and one of the OEMs would make them for other buyers as well. If you’re doing this route, read through those for considerations and lessons learned. Performance was > 10x better than enterprise solutions. A policy to “leave dead disks dead” aka “let them rot” as said elsewhere in this thread kept maintenance cheap. The secret sauce part making this viable for commercial online storage hosting (we hosted video) was we used disks as JBOD with an in-house meta index with P2P health awareness to place objects redundantly across disks, chassis, racks, colocation providers, and regions.

more_corn 6y ago

Don't buy HPE gear. Qualify the gear with sample units from a few competing vendors and you'll see why.

specktr 6y ago
I’ve done qualification on hpe and various competing vendors and honestly haven’t seen dramatic differences in terms of performance and failure rates. From my experience the biggest difference was with vendor support services rather than the actual hardware. I’d be curious to hear more about your qualification experience with this particular vendor if you’d be willing.

walrus01 6y ago

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc. I am a huge advocate of hosting stuff yourself on bare metal you own, but this is a ridiculous statement. Any drive in that class should come with a 3 or 5 year warranty. And the manual labor and hassle time to replace one (you have hundreds of thousands of dollars of storage and no ready to go cold spares on a shelf?!?!) is infinitesimal.

closeparen 6y ago
If the colo is far and there’s plenty of headroom, it might not justify much urgency.
- cortesoft 6y ago
  That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.
  - Terretta 6y ago
    Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.
    - walrus01 6y ago
      'cattle not pets' is not a valid argument when you're the owner and operator of the bare metal hardware. Do you also recommend that ISPs not replace failed fans in core and edge routers and optical transport systems? Let things with dual power supplies run for six months on one failed power supply? Also you've clearly never interacted with cattle, sheep, goats, llamas or alpacas, which absolutely do get things like veterinary care and vaccinations. Large animal vet is a whole specialty and they spend lots of time working on animals other than horses. No trips to the vet???

lars_francke 6y ago

I'd be interested to learn more about your HDFS usage and your experience at that scale. Would you be willing to have a chat? If so, my email is in my profile.

metabrew 6y ago

When we set up user content storage of images and mp3s for Last.fm back in 2006ish we used MogileFS (from the bradfitz LJ perl days) running on our own hardware. 3/4/5/6u machines stuffed full of disks. I still think it's an elegant concept – easy to grok, easy to debug, easy to reason about. No special distributed filesystem to worry about. Don't take this as an endorsement of the MogileFS perl codebase in 2021, but worth considering this style of storage system depending on your precise needs.