3 ms·

Any recommendations for alternatives, and what their relative weak points are in your experience?

takeda 10y ago
So far I heard a lot about GlusterFS and Ceph being great but so far I only seen MooseFS in production. MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master. They have commercial offering that includes HA, but I never used so hard to comment about it. You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.
- zhynn 10y ago
  Oh yeah, MooseFS. I forgot about that one.
zhynn 10y ago
I only have experience running Lustre and Gluster. Gluster was a mess, to be honest, though that was several years ago. Lustre (and we run Intel Enterprise Lustre) has been pretty solid. Most HPC outfits run lustre over ZFS, actually, so you get the benefits of both. Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case. The others that I know of (but haven't tried) are: - OrangeFS (http://www.orangefs.org/ http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/ http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/ http://www.openafs.org/, http://www.stacken.kth.se/project/arla/ http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/ http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/ https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/ http://qumulo.com/ There are probably many more, but these are the ones I know about.
- pinewurst 10y ago
  Qumulo can't hit the numbers. It's just NFS, running on ho-hum hardware with an Ethernet cluster interconnect. It's like slow Isilon if cheaper and with less functionality.
- __jal 10y ago
  People may use Gluster for HPC; I don't know. But it really isn't suited. I run two Gluster clusters for more boring enterprisey reliability. It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc. All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.

So far I heard a lot about GlusterFS and Ceph being great but so far I only seen MooseFS in production. MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master. They have commercial offering that includes HA, but I never used so hard to comment about it. You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.

zhynn 10y ago
Oh yeah, MooseFS. I forgot about that one.

zhynn 10y ago

I only have experience running Lustre and Gluster. Gluster was a mess, to be honest, though that was several years ago. Lustre (and we run Intel Enterprise Lustre) has been pretty solid. Most HPC outfits run lustre over ZFS, actually, so you get the benefits of both. Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case. The others that I know of (but haven't tried) are: - OrangeFS (http://www.orangefs.org/ http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/ http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/ http://www.openafs.org/, http://www.stacken.kth.se/project/arla/ http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/ http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/ https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/ http://qumulo.com/ There are probably many more, but these are the ones I know about.

pinewurst 10y ago
Qumulo can't hit the numbers. It's just NFS, running on ho-hum hardware with an Ethernet cluster interconnect. It's like slow Isilon if cheaper and with less functionality.
__jal 10y ago
People may use Gluster for HPC; I don't know. But it really isn't suited. I run two Gluster clusters for more boring enterprisey reliability. It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc. All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.