Sure. It's an extraordinarily complex system that's difficult to engineer correctly. It provides extraordinary durability, but the radius of failure isn't obv

Sure. It's an extraordinarily complex system that's difficult to engineer correctly. It provides extraordinary durability, but the radius of failure isn't obvious. Pro tip, it's the entire cluster. As such, an issue with an OSD in one pool could potentially cause the entire cluster to have issues. Recovery is difficult and there's no support unless you have a subscription from Redhat and also run RHEL plus their stable distro of Ceph (RedHat Storage or whatever). IIRC, they quoted me $90k for a petabyte of raw disk. I haven't messed with it much in the last couple of years. Bluestore looked really promising. I've thought about taking a look at rook, but haven't yet. If I were in a position to deploy a bunch of storage on bare metal again, I'd likely go with ceph. I do know that $GLORIOUS_FORMER_EMPLOYER ended up making the migration to ScaleIO and report being happy with it and having good performance.