4 ms·

Awesome. I actually think that there is a lot of potential for using block storage over FDB to bring extreme fault tolerance to legacy applications without rear

Awesome. I actually think that there is a lot of potential for using block storage over FDB to bring extreme fault tolerance to legacy applications without rearchitecting them. Because block devices are unshared (and you can use FDB to ensure that robustly) conventional OS caching works well, and you can get relatively normal performance. Of course FDB storage engine is currently not very well optimized for this use case, but since FDB is scalable you can "just use more" I wrote more about this on the FDB forums: https://forums.foundationdb.org/t/block-storage-full-fault-tolerance-for-legacy-applications/182 https://forums.foundationdb.org/t/block-storage-full-fault-t...

geofft 8y ago
Would this provide additional benefits over Ceph, which does no-single-point-of-failure networked block storage and is already a production-ready system? It wouldn't get you the locking you want (multiple attempts to boot the VM end in mutual exclusion at the block storage layer), I believe, but I think that's probably better done with an etcd or something if you want that.
- voidmain 8y ago
  Unless you have serious simulation or model checking tools, your ad hoc combination of etcd and ceph will almost certainly be buggy. I'm not 100% positive it is even possible in the asynchronous model. And when you decide you want, say, regional or rack aware fault tolerance, you have three more distributed systems (ceph, etcd, and your combination) to make meet that goal. When you restore a backup of the contents of ceph, what do you do with the lock state in etcd? Do you know what happens when (say) etcd runs out of memory? If you wanted to ship an on premise version of your solution, can you teach your customers to admin all these systems? FDB's mission is to get under all of your state storage so that you only need one system to do all this operational stuff correctly. A block device is most useful for people who are trying to eliminate the last bits of state not in FDB from their infrastructure. Another huge payoff is the new "satellite" replication mode coming (I'm told) with FDB 6.0, which will give you "the best of both worlds" of synchronous and asynchronous replication across regions. A block device will let your legacy systems usually fail over across regions automatically without losing a single write.
  - emmericp 8y ago
    RBD (Ceph block devices) supports exclusive locking. The RBD kernel client has support for that since kernel 4.9. Older kernels can use rbd-nbd to use the user space implementation of RBD. Async replication is also available with rbd-mirror.

geofft 8y ago

Would this provide additional benefits over Ceph, which does no-single-point-of-failure networked block storage and is already a production-ready system? It wouldn't get you the locking you want (multiple attempts to boot the VM end in mutual exclusion at the block storage layer), I believe, but I think that's probably better done with an etcd or something if you want that.

voidmain 8y ago
Unless you have serious simulation or model checking tools, your ad hoc combination of etcd and ceph will almost certainly be buggy. I'm not 100% positive it is even possible in the asynchronous model. And when you decide you want, say, regional or rack aware fault tolerance, you have three more distributed systems (ceph, etcd, and your combination) to make meet that goal. When you restore a backup of the contents of ceph, what do you do with the lock state in etcd? Do you know what happens when (say) etcd runs out of memory? If you wanted to ship an on premise version of your solution, can you teach your customers to admin all these systems? FDB's mission is to get under all of your state storage so that you only need one system to do all this operational stuff correctly. A block device is most useful for people who are trying to eliminate the last bits of state not in FDB from their infrastructure. Another huge payoff is the new "satellite" replication mode coming (I'm told) with FDB 6.0, which will give you "the best of both worlds" of synchronous and asynchronous replication across regions. A block device will let your legacy systems usually fail over across regions automatically without losing a single write.
- emmericp 8y ago
  RBD (Ceph block devices) supports exclusive locking. The RBD kernel client has support for that since kernel 4.9. Older kernels can use rbd-nbd to use the user space implementation of RBD. Async replication is also available with rbd-mirror.