3 ms·

FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST https://wiki.freebsd.org/HAST There are also other solutions which you can

by olavgg 10y ago

FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.

cmurf 10y ago
Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it can be a day due to capacity growing faster than performance.
- zokier 10y ago
  Is the performance of Ceph/Gluster comparative to ZFS performance? I would have thought them to be inherently massively slower
  - olavgg 10y ago
    It is slower, but still probably good enough for many use cases. https://software.intel.com/en-us/blogs/2013/10/25/measure-ceph-rbd-performance-in-a-quantitative-way-part-i https://software.intel.com/en-us/blogs/2013/10/25/measure-ce... http://www.intel.com/content/dam/www/public/us/en/documents/reference-architectures/120315-analyzing-ceph-cluster-optimize-storage-costs.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...
- zimbatm 10y ago
  Isn't the rebuilding argument valid on the machine level as well though? Granted if your blocks are redundant it's possible to copy from multiple sources but rebuilding is still limited to the network speed.
  - zzzcpan 10y ago
    On a properly designed distributed system disk-replacement costs are spread across all machines and in time, making performance degradation a non-issue.
X86BSD 10y ago
People will do whatever they want to do. And my comment is not a knock on HAST. But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain. Wether it's iSCSI, HAST, NFS, HAST or any combination thereof. The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's. Assign half the disks in a mirror to HBA-1 the other half to HBA-2. Use one nic for public networking, use the other for a direct P2P Gig-E link between server A and server B. Use zfs replication over the private network link using the granularity you feel most comfortable with. 30 minutes? 15 minutes? 5? It's up to you. This is the simplest, and IMO most reliable way to go about this. When you start adding in iSCSI, NFS, HAST etc you are simply making things to complex, introducing multiple points of failure, and the pain when something does break? Ill pass. I'd rather have to manually intervene and fail over to server B if server A fails and vice versa than deal with all the pain of additional things that could break or introduce split brained issues etc. But it's a free world, use whatever solution fits your comfort zone.
- zzzcpan 10y ago
  The biggest issue with your approach is consistency. You just cannot guarantee it unless you are using proper distributed algorithms on every level.
- Annatar 10y ago
  But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain. Rarely have truer words been written. Highly available, highly reliable storage means keeping it as simple as possible, but no simpler. I wonder how long will it take before computer professionals at large realize this?

by olavgg 10y ago

cmurf 10y ago

Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it can be a day due to capacity growing faster than performance.

zokier 10y ago
Is the performance of Ceph/Gluster comparative to ZFS performance? I would have thought them to be inherently massively slower
- olavgg 10y ago
  It is slower, but still probably good enough for many use cases. https://software.intel.com/en-us/blogs/2013/10/25/measure-ceph-rbd-performance-in-a-quantitative-way-part-i https://software.intel.com/en-us/blogs/2013/10/25/measure-ce... http://www.intel.com/content/dam/www/public/us/en/documents/reference-architectures/120315-analyzing-ceph-cluster-optimize-storage-costs.pdf http://www.intel.com/content/dam/www/public/us/en/documents/...
zimbatm 10y ago
Isn't the rebuilding argument valid on the machine level as well though? Granted if your blocks are redundant it's possible to copy from multiple sources but rebuilding is still limited to the network speed.
- zzzcpan 10y ago
  On a properly designed distributed system disk-replacement costs are spread across all machines and in time, making performance degradation a non-issue.

X86BSD 10y ago

People will do whatever they want to do. And my comment is not a knock on HAST. But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain. Wether it's iSCSI, HAST, NFS, HAST or any combination thereof. The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's. Assign half the disks in a mirror to HBA-1 the other half to HBA-2. Use one nic for public networking, use the other for a direct P2P Gig-E link between server A and server B. Use zfs replication over the private network link using the granularity you feel most comfortable with. 30 minutes? 15 minutes? 5? It's up to you. This is the simplest, and IMO most reliable way to go about this. When you start adding in iSCSI, NFS, HAST etc you are simply making things to complex, introducing multiple points of failure, and the pain when something does break? Ill pass. I'd rather have to manually intervene and fail over to server B if server A fails and vice versa than deal with all the pain of additional things that could break or introduce split brained issues etc. But it's a free world, use whatever solution fits your comfort zone.

zzzcpan 10y ago
The biggest issue with your approach is consistency. You just cannot guarantee it unless you are using proper distributed algorithms on every level.
Annatar 10y ago
But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain. Rarely have truer words been written. Highly available, highly reliable storage means keeping it as simple as possible, but no simpler. I wonder how long will it take before computer professionals at large realize this?