3 ms·

Could a ZFS server farm be a good alternative to a smallish Ceph installation ? If I wanted to build a storage layer for a small private DC - is Ceph the way to

Could a ZFS server farm be a good alternative to a smallish Ceph installation ? If I wanted to build a storage layer for a small private DC - is Ceph the way to go? ZFS? Or maybe there's something else? I'd like to have storage details hidden away from storage layer so that applications can write to a mount or something similar - what is the best solution for such problem these days, if one needs resiliency and redundancy baked in ?

HeadsUpHigh 6y ago
Zfs sounds like a good solution although I'm not experienced with Ceph. Btrfs is ab option too but it has sone pitfalls.
sekh60 6y ago
I use Ceph in my homelab. Small cluster of five all-in-one noses with ten OSDs each. I went this way since it gives me a lot more expandability and fault tolerance than ZFS can. I use min_size 2, size 3. My use case is both CephFS and RBD for a small two node OpenStack cluster. I have found CephFS to be rather performant for my use cases, enough that I had to get a 10Gbps switch. I am not machine or the bandwidth on that switch, but my individual clients use more than 1Gbps. OpenStack and Ceph tie together wonderfully. I have my VMs backed by NVMe drives and my VMs are snappy. Recovery is quick too. I am using crappy first gen xeon-d boards and even with those I hit 8Gbps recovery on those drives. Ceph shines when you have a lot of parallel access. It is recommended to have at least ten nodes for a production cluster so recoveries so not take too long. If you have a lot of clients Ceph is king. I used ZFS in the past as a simple Fileserver before using Ceph. It worked well, I could saturate a 1Gbps link, however I found the vdev resize limitation too restricting at times when I wanted to expand by a little bit. It is pretty easy to manage, though I find Ceph very easy to manage as well. For my backup server which is a target for BorgBackup I went with btrfs for the better flexibility it offers with resizing arrays.
- azalemeth 6y ago
  How many computers do you have at home, in your homelab? Do they also heat your house? Where do you store them?
  - sekh60 6y ago
    I have 5 Ceph nodes, 2 OpenStack nodes, 1 backup target, 3 switches, 1 pfsense router, 1 freeipa server and my desktop. The homelab is kept in the furnace room. It is almost completely on atom or xeon-d boards, so the power usage hard drives aside isn't too bad. The room does get a bit warm though, even when the furnace is off. They are kept in a 36U rack.
throw0101a 6y ago
> Could a ZFS server farm be a good alternative to a smallish Ceph installation ? ZFS and Ceph work at different 'layers'. ZFS provides redundancy with-in a server, so if drive(s) die then the service on that service can continue to run without interruption. Ceph provides redundancy between servers, so if drive(s), servers, or even entire racks/ToR switches die then things keep going. Ceph is generally for much larger scales (e.g., OpenStack, HPC) than ZFS, which is usually done on NFS or SMB servers. Until relatively recently, Ceph was also only accessible at a block layer, so you'd have to put a file system on top of it (i.e, Ceph gave you a /dev/sdX), but somewhat recently CephFS has become/declared 'stable'. Regardless, you still need three machines for quorum for Ceph. If you just want a file share, then it's probably unnecessarily complex.
3np 6y ago
I have a glusterfs cluster where the bricks are consisting of zfs datasets. You can so it similarly with zeph and zvols.

HeadsUpHigh 6y ago

Zfs sounds like a good solution although I'm not experienced with Ceph. Btrfs is ab option too but it has sone pitfalls.

sekh60 6y ago

I use Ceph in my homelab. Small cluster of five all-in-one noses with ten OSDs each. I went this way since it gives me a lot more expandability and fault tolerance than ZFS can. I use min_size 2, size 3. My use case is both CephFS and RBD for a small two node OpenStack cluster. I have found CephFS to be rather performant for my use cases, enough that I had to get a 10Gbps switch. I am not machine or the bandwidth on that switch, but my individual clients use more than 1Gbps. OpenStack and Ceph tie together wonderfully. I have my VMs backed by NVMe drives and my VMs are snappy. Recovery is quick too. I am using crappy first gen xeon-d boards and even with those I hit 8Gbps recovery on those drives. Ceph shines when you have a lot of parallel access. It is recommended to have at least ten nodes for a production cluster so recoveries so not take too long. If you have a lot of clients Ceph is king. I used ZFS in the past as a simple Fileserver before using Ceph. It worked well, I could saturate a 1Gbps link, however I found the vdev resize limitation too restricting at times when I wanted to expand by a little bit. It is pretty easy to manage, though I find Ceph very easy to manage as well. For my backup server which is a target for BorgBackup I went with btrfs for the better flexibility it offers with resizing arrays.

azalemeth 6y ago
How many computers do you have at home, in your homelab? Do they also heat your house? Where do you store them?
- sekh60 6y ago
  I have 5 Ceph nodes, 2 OpenStack nodes, 1 backup target, 3 switches, 1 pfsense router, 1 freeipa server and my desktop. The homelab is kept in the furnace room. It is almost completely on atom or xeon-d boards, so the power usage hard drives aside isn't too bad. The room does get a bit warm though, even when the furnace is off. They are kept in a 36U rack.

throw0101a 6y ago

> Could a ZFS server farm be a good alternative to a smallish Ceph installation ? ZFS and Ceph work at different 'layers'. ZFS provides redundancy with-in a server, so if drive(s) die then the service on that service can continue to run without interruption. Ceph provides redundancy between servers, so if drive(s), servers, or even entire racks/ToR switches die then things keep going. Ceph is generally for much larger scales (e.g., OpenStack, HPC) than ZFS, which is usually done on NFS or SMB servers. Until relatively recently, Ceph was also only accessible at a block layer, so you'd have to put a file system on top of it (i.e, Ceph gave you a /dev/sdX), but somewhat recently CephFS has become/declared 'stable'. Regardless, you still need three machines for quorum for Ceph. If you just want a file share, then it's probably unnecessarily complex.

3np 6y ago

I have a glusterfs cluster where the bricks are consisting of zfs datasets. You can so it similarly with zeph and zvols.