5 ms·

It's been amusing to see all the apologia and simple ass-covering following this. Lustre is a monstrosity - badly designed, poorly implemented, very hard to co

It's been amusing to see all the apologia and simple ass-covering following this. Lustre is a monstrosity - badly designed, poorly implemented, very hard to configure and keep running or even get adequate performance under other than a single limited use case. Good riddance to bad rubbish.

yourapostasy 10y ago
Any recommendations for alternatives, and what their relative weak points are in your experience?
- takeda 10y ago
  So far I heard a lot about GlusterFS and Ceph being great but so far I only seen MooseFS in production. MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master. They have commercial offering that includes HA, but I never used so hard to comment about it. You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.
  - zhynn 10y ago
    Oh yeah, MooseFS. I forgot about that one.
- zhynn 10y ago
  I only have experience running Lustre and Gluster. Gluster was a mess, to be honest, though that was several years ago. Lustre (and we run Intel Enterprise Lustre) has been pretty solid. Most HPC outfits run lustre over ZFS, actually, so you get the benefits of both. Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case. The others that I know of (but haven't tried) are: - OrangeFS (http://www.orangefs.org/ http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/ http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/ http://www.openafs.org/, http://www.stacken.kth.se/project/arla/ http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/ http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/ https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/ http://qumulo.com/ There are probably many more, but these are the ones I know about.
  - pinewurst 10y ago
    Qumulo can't hit the numbers. It's just NFS, running on ho-hum hardware with an Ethernet cluster interconnect. It's like slow Isilon if cheaper and with less functionality.
  - __jal 10y ago
    People may use Gluster for HPC; I don't know. But it really isn't suited. I run two Gluster clusters for more boring enterprisey reliability. It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc. All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.
didip 10y ago
In your opinion, what would be better than Lustre? Ceph, Gluster, HDFS?
- koolba 10y ago
  Those choices are apples and oranges. Particularly HDFS as you can't really use it as a native fs.
- khm 10y ago
  GPFS.
photon-torpedo 10y ago
Any opinion on BeeGFS?
- usernam 10y ago
  We used BeeGFS since it was called FhGFS on a cluster of 32 nodes for a total capacity ranging from 256tb-1Pb. BeeGFS is very easy to configure and maintain, outperforming Lustre for every task we threw it at. In fact, we never transitioned lustre into production. Lustre was often slower than plain NFS for the cases where all storage could be mounted by a single node(!). Unfortunately, it's prior licensing scheme prevented larger initial adoption. A good chunk of large scale setups I've seen have adopted Ceph, but I cannot comment on performance.
  - pinewurst 10y ago
    BeeGFS is great. I originally evaluated it when it was FhGFS and am still involved with a large production BeeGFS cluster deployment. Any knucklehead can install it and it works transparently with excellent performance. It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it.
    - hpcjoe 10y ago
      > It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it. Its getting there quickly. BeeGFS v6 has metadata mirroring via buddy groups, and data mirroring the same way. A few things are still not there, but it is rapidly getting closer. Being open source helps as well. Definitely recommend it, and have deployed it and supported it many places in my previous life: http://scalableinformatics.com http://scalableinformatics.com .
arcanus 10y ago
> or even get adequate performance under other than a single limited use case. Hmm? From the article: " In the core HPC market, Lustre has 75 of the Top 100 systems on the Top 500 rankings, and is used on nine of the top ten systems." That is a strong argument for it being performant. That 'single limited use case' you mention is... high performance computing.
- convolvatron 10y ago
  right, so they took a vertical that no one else wanted and served it kind of badly. I don't know about the last few years, but Lustre used to be famous for eating up a huge amount of time in the field, even given that its only supposed to be relevant for really large files. it definitely served a need that nothing else did, but from an armchair position we could collectively do better
- pinewurst 10y ago
  That's an artifact of the limited choices available. Plus the core HPC market is both notoriously inbred and tolerant of low system reliability and high delicacy that the enterprise wouldn't let in the door. And no, "single limited use case" I mean, is a relatively small number of very large files that need to be sequentially streamed out/in as fast as possible. That's a small subset of HPC. GPFS is the gold standard but it's expensive. Lustre is more a byproduct of the HPC vendors trying to get more margin by using open source. They've drunk from the poisoned chalice in that the amount of time, money and effort required to get Lustre to acceptability (limited as that may be) has been far more than GPFS license/support cost. Even from the brain-eating zombie that is today's IBM.
  - lo_stronzo 10y ago
    > GPFS is the gold standard but it's expensive. Confirming! And going on a small tangent (ha!). Our previous configuration was Lustre and XFS/NFS; the former was the scratch file system for HPC applications and the latter for home directories and what not. Lustre was definitely a beast (in a good sense), but we'd occasionally get bit by a work load high in metadata operations - which would bring Lustre to its knees due to latency. XFS/NFS was great for its purpose (no HPC workloads), but we'd also get bit by a user or users reading seemingly tiny, innocuous files (inadvertently in parallel) which would cause load averages to spike; surprisingly the latency wasn't as bad as the Lustre workload mentioned above. Not too drink the GPFS kool aid here, but it's definitely solved both problems above. It has its issues, but definitely handles the common I/O patterns seen on our cluster.

yourapostasy 10y ago

Any recommendations for alternatives, and what their relative weak points are in your experience?

takeda 10y ago
So far I heard a lot about GlusterFS and Ceph being great but so far I only seen MooseFS in production. MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master. They have commercial offering that includes HA, but I never used so hard to comment about it. You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.
- zhynn 10y ago
  Oh yeah, MooseFS. I forgot about that one.
zhynn 10y ago
I only have experience running Lustre and Gluster. Gluster was a mess, to be honest, though that was several years ago. Lustre (and we run Intel Enterprise Lustre) has been pretty solid. Most HPC outfits run lustre over ZFS, actually, so you get the benefits of both. Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case. The others that I know of (but haven't tried) are: - OrangeFS (http://www.orangefs.org/ http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/ http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/ http://www.openafs.org/, http://www.stacken.kth.se/project/arla/ http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/ http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/ https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/ http://qumulo.com/ There are probably many more, but these are the ones I know about.
- pinewurst 10y ago
  Qumulo can't hit the numbers. It's just NFS, running on ho-hum hardware with an Ethernet cluster interconnect. It's like slow Isilon if cheaper and with less functionality.
- __jal 10y ago
  People may use Gluster for HPC; I don't know. But it really isn't suited. I run two Gluster clusters for more boring enterprisey reliability. It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc. All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.

didip 10y ago

In your opinion, what would be better than Lustre? Ceph, Gluster, HDFS?

koolba 10y ago
Those choices are apples and oranges. Particularly HDFS as you can't really use it as a native fs.
khm 10y ago
GPFS.

photon-torpedo 10y ago

Any opinion on BeeGFS?

usernam 10y ago
We used BeeGFS since it was called FhGFS on a cluster of 32 nodes for a total capacity ranging from 256tb-1Pb. BeeGFS is very easy to configure and maintain, outperforming Lustre for every task we threw it at. In fact, we never transitioned lustre into production. Lustre was often slower than plain NFS for the cases where all storage could be mounted by a single node(!). Unfortunately, it's prior licensing scheme prevented larger initial adoption. A good chunk of large scale setups I've seen have adopted Ceph, but I cannot comment on performance.
- pinewurst 10y ago
  BeeGFS is great. I originally evaluated it when it was FhGFS and am still involved with a large production BeeGFS cluster deployment. Any knucklehead can install it and it works transparently with excellent performance. It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it.
  - hpcjoe 10y ago
    > It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it. Its getting there quickly. BeeGFS v6 has metadata mirroring via buddy groups, and data mirroring the same way. A few things are still not there, but it is rapidly getting closer. Being open source helps as well. Definitely recommend it, and have deployed it and supported it many places in my previous life: http://scalableinformatics.com http://scalableinformatics.com .

arcanus 10y ago

> or even get adequate performance under other than a single limited use case. Hmm? From the article: " In the core HPC market, Lustre has 75 of the Top 100 systems on the Top 500 rankings, and is used on nine of the top ten systems." That is a strong argument for it being performant. That 'single limited use case' you mention is... high performance computing.

convolvatron 10y ago
right, so they took a vertical that no one else wanted and served it kind of badly. I don't know about the last few years, but Lustre used to be famous for eating up a huge amount of time in the field, even given that its only supposed to be relevant for really large files. it definitely served a need that nothing else did, but from an armchair position we could collectively do better
pinewurst 10y ago
That's an artifact of the limited choices available. Plus the core HPC market is both notoriously inbred and tolerant of low system reliability and high delicacy that the enterprise wouldn't let in the door. And no, "single limited use case" I mean, is a relatively small number of very large files that need to be sequentially streamed out/in as fast as possible. That's a small subset of HPC. GPFS is the gold standard but it's expensive. Lustre is more a byproduct of the HPC vendors trying to get more margin by using open source. They've drunk from the poisoned chalice in that the amount of time, money and effort required to get Lustre to acceptability (limited as that may be) has been far more than GPFS license/support cost. Even from the brain-eating zombie that is today's IBM.
- lo_stronzo 10y ago
  > GPFS is the gold standard but it's expensive. Confirming! And going on a small tangent (ha!). Our previous configuration was Lustre and XFS/NFS; the former was the scratch file system for HPC applications and the latter for home directories and what not. Lustre was definitely a beast (in a good sense), but we'd occasionally get bit by a work load high in metadata operations - which would bring Lustre to its knees due to latency. XFS/NFS was great for its purpose (no HPC workloads), but we'd also get bit by a user or users reading seemingly tiny, innocuous files (inadvertently in parallel) which would cause load averages to spike; surprisingly the latency wasn't as bad as the Lustre workload mentioned above. Not too drink the GPFS kool aid here, but it's definitely solved both problems above. It has its issues, but definitely handles the common I/O patterns seen on our cluster.