5 ms·
Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement G
by epistasis 12y ago
Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)
- ghshephard 12y agoA Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes. Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?
- chrisbolt 12y ago> Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object? Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc). http://ceph.com/docs/master/rados/operations/crush-map/#crush-map-bucket-types http://ceph.com/docs/master/rados/operations/crush-map/#crus...
- epistasis 12y agoYes, there's no more ticking time bomb so there's no urgency to replace the drive, nor any urgency to max out the drive for 15 hours straight during the rebuild. It can be eased into the cluster over the course of days or a week. And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.
- ghshephard 12y agoWhat's the sweet spot for Ceph cluster redundancy - Original + 3 Copies? I realize the answer is, "It depends on how much redundancy you want" - but I'm guessing that there is a general best practice that Ceph administrators have adopted?
- epistasis 12y agoCeph is a Consistent and Partition tolerant system in the CAP formulation, at the expense of Availability. Fortunately for my use cases I've not needed high availability, which would require two separate clusters and replication between them. I've mostly seen people use triplicate pools. For semi-warm storage I've been testing erasure coded pools, with a triplicate cache tier on top, and had good experiences on my test cluster.
- insaneirish 12y agoSo, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584 https://twitter.com/tbmcmullen/status/566856092185923584
- walrus 12y agoDoes this cause problems in practice? I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.
- cmurf 12y agoCeph isn't ready for production. GlusterFS is, and it also behaves more like a file system, while Ceph is more object storage. They serve different use cases but GlusterFS is more general purpose I think. No MDS required, scales fairly linearly. They're using it at Facebook with XFS and Btrfs underneath depending on workload. At home you can build little ARM gluster clusters, so it kinda even scales down! http://events.linuxfoundation.org/sites/events/files/slides/Vault%20CephFS%20fsck.pdf http://events.linuxfoundation.org/sites/events/files/slides/...
- nigwil_ 12y ago"Ceph isn't ready for production." - this is not the case. Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API. Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored. [1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.