5 ms·

I ran a ~ 0.5 PB Ceph cluster for a few years, on quite old spinning disk hardware (bought second-hand.) It was great: it just worked, coped very well with hard

I ran a ~ 0.5 PB Ceph cluster for a few years, on quite old spinning disk hardware (bought second-hand.) It was great: it just worked, coped very well with hardware failures, told the operator what was happening. An extremely solid, well-engineered system. My thanks to the Ceph team :)

yankcrime 7y ago
Yup, adding to this: For about four years we ran a similar size Ceph cluster that supported a burgeoning public cloud platform with users making extensive use of block and object. We did this on a range of hardware from creaky second-hand SuperMicro boxes to newer all-flash Quanta-based hardware. Through questionable hardware selection as well as standard operational challenges such as upgrades and scaling, Ceph never let us down. The only time we had a major problem turned out to be our fault. The creaky machines we were using at the start 'lost' half of their memory during a round of power failure testing. We didn't have monitoring in place to spot this, and unfortunately it manifest after we lost a node which triggered a significant rebalancing operation across the cluster. With several machines missing 50% of their RAM, this quickly descended into a horrendous disk thrashing exercise. Again, all credit due to Ceph - we were able to coax the cluster back into life with no data loss.