4 ms·

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bo

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

chrisbolt 12y ago
> Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object? Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc). http://ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types http://ceph.com/docs/master/rados/operations/crush-map/#crus...
epistasis 12y ago
Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week. And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.
- ghshephard 12y ago
  What's the sweet spot for Ceph cluster redundancy - Original + 3 Copies? I realize the answer is, "It depends on how much redundancy you want" - but I'm guessing that there is a general best practice that Ceph administrators have adopted?
  - epistasis 12y ago
    Ceph is a Consistent and Partition tolerant system in the CAP formulation, at the expense of Availability. Fortunately for my use cases I've not needed high availability, which would require two separate clusters and replication between them. I've mostly seen people use triplicate pools. For semi-warm storage I've been testing erasure coded pools, with a triplicate cache tier on top, and had good experiences on my test cluster.

chrisbolt 12y ago

> Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object? Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc). http://ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types http://ceph.com/docs/master/rados/operations/crush-map/#crus...

epistasis 12y ago

Yes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week. And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.

ghshephard 12y ago
What's the sweet spot for Ceph cluster redundancy - Original + 3 Copies? I realize the answer is, "It depends on how much redundancy you want" - but I'm guessing that there is a general best practice that Ceph administrators have adopted?
- epistasis 12y ago
  Ceph is a Consistent and Partition tolerant system in the CAP formulation, at the expense of Availability. Fortunately for my use cases I've not needed high availability, which would require two separate clusters and replication between them. I've mostly seen people use triplicate pools. For semi-warm storage I've been testing erasure coded pools, with a triplicate cache tier on top, and had good experiences on my test cluster.