That's true. I've tried ceph, swift and riakcs. Our requirements were two servers, the storage is several terabytes, and losing data because of an unknown bug

That's true. I've tried ceph, swift and riakcs. Our requirements were two servers, the storage is several terabytes, and losing data because of an unknown bug in the filesystem wasn't an option. Such filesystems are quorum-based and certainly need more than two servers with terabytes of data. More than what we could afford. Also if you lose quorum, the system hangs to ensure atomicity. But we don't need such properties. Also after some testing, those filesystem resulted about 5-10% slower than traditional filesystems, with few nodes of course. They are designed to scale with many nodes.