I use Ceph in my homelab. Small cluster of five all-in-one noses with ten OSDs each. I went this way since it gives me a lot more expandability and fault tolera

I use Ceph in my homelab. Small cluster of five all-in-one noses with ten OSDs each. I went this way since it gives me a lot more expandability and fault tolerance than ZFS can. I use min_size 2, size 3. My use case is both CephFS and RBD for a small two node OpenStack cluster. I have found CephFS to be rather performant for my use cases, enough that I had to get a 10Gbps switch. I am not machine or the bandwidth on that switch, but my individual clients use more than 1Gbps. OpenStack and Ceph tie together wonderfully. I have my VMs backed by NVMe drives and my VMs are snappy. Recovery is quick too. I am using crappy first gen xeon-d boards and even with those I hit 8Gbps recovery on those drives. Ceph shines when you have a lot of parallel access. It is recommended to have at least ten nodes for a production cluster so recoveries so not take too long. If you have a lot of clients Ceph is king. I used ZFS in the past as a simple Fileserver before using Ceph. It worked well, I could saturate a 1Gbps link, however I found the vdev resize limitation too restricting at times when I wanted to expand by a little bit. It is pretty easy to manage, though I find Ceph very easy to manage as well. For my backup server which is a target for BorgBackup I went with btrfs for the better flexibility it offers with resizing arrays.