3 ms·

Coming from an enterprise background for a couple of years: network block storage screws you one way or another either way. It's a single point of failure that

Coming from an enterprise background for a couple of years: network block storage screws you one way or another either way. It's a single point of failure that no matter how reliable it is, will always go down at some point or have failures that affect your application layer.

llama052 9y ago
Sounds like you've been in environments that have done network block storage incorrectly. If you do multi-path, redundant/replicated network storage with proper configurations in place, you shouldn't have a single point of failure. Ceph oddities aside. Everything has issues given a long enough time span though. Each layer has an accounted level of risk, and it definitely is a balancing act.
antongribok 9y ago
That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you design correctly, you can have a very resilient storage system. Source: I come from enterprise storage and have been running Ceph in production (successfully) for ~5 years.
- halbritt 9y ago
  Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
  - antongribok 9y ago
    > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
    - halbritt 9y ago
      Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.

llama052 9y ago

Sounds like you've been in environments that have done network block storage incorrectly. If you do multi-path, redundant/replicated network storage with proper configurations in place, you shouldn't have a single point of failure. Ceph oddities aside. Everything has issues given a long enough time span though. Each layer has an accounted level of risk, and it definitely is a balancing act.

antongribok 9y ago

That blanket statement is absolutely not true. With Ceph you can distribute data across hosts, switches, racks, rows, fabrics, rooms, datacenters. If you design correctly, you can have a very resilient storage system. Source: I come from enterprise storage and have been running Ceph in production (successfully) for ~5 years.

halbritt 9y ago
Ceph has some weird failure modes, though. Your radius of failure absolutely includes the entire cluster, regardless of how resilient its been made. Sure, you're insulated from hardware failures, but not from bugs in the underlying system. Data loss is rare, but having a cluster go unavailable does happen.
- antongribok 9y ago
  > Ceph has some weird failure modes No argument. > radius of failure absolutely includes the entire cluster Completely agree, same goes for any other storage system, network-based or otherwise. I'm simply disagreeing with a blanket statement that all network block storage is unreliable, and that is simply not true. Don't do silly things with Ceph, don't just have a single network fabric, don't buy cheap switches that drop packets on the floor at medium load, don't test things in production.
  - halbritt 9y ago
    Oh, do test things in production. Can’t understate the need for good network, especially with the traffic amplification of replication. I was running 2x10GE and had plans to scale up the cluster network to 2x or 3x10GE. My biggest issue was the MTTR of a failed cluster. If one is RBD mirroring to a different cluster, recovery time may be hours or days. That said, I had one cluster run continuously for a couple years with basically zero administration. It had great performance and good reliability other than that one week-long service impacting incident as a result of a bug. I’d definitely run it again.