3 ms·

That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you desi

That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you design correctly, you can have a very resilient storage system. Source: I come from enterprise storage and have been running Ceph in production (successfully) for ~5 years.

halbritt 9y ago
Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
- antongribok 9y ago
  > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
  - halbritt 9y ago
    Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.

3 ms·

That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you desi

by antongribok 9y ago

halbritt 9y ago
Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
- antongribok 9y ago
  > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
  - halbritt 9y ago
    Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.