4 ms·

ironic to see comments here criticizing Lustre, from people who don't seem to understand it at all. Lustre is for large HPC clusters. It's not for your deskto

ironic to see comments here criticizing Lustre, from people who don't seem to understand it at all. Lustre is for large HPC clusters. It's not for your desktop; it's not for your video editing suite. It is the only way to provide a filesystem that scales to support, for instance, a compute job of 100k ranks, all writing checkpoints and snapshots periodically. No, Hadoop doesn't do it. Ceph is for performance-and-scaling-insensitive cloud installs. Lustre is for when you expect to saturage 100 100Gb IB links to storage. it works remarkably well for its use case (though even on HPC clusters, MDS performance can be a problem).

dekhn 9y ago
actually, Hadoop-like systems do precisely this (a compute job of 100k ranks, all writing checkpoints and snapshots periodically). 1.25Tbyte/sec isn't a particularly large amount of data. (btw: makes more sense to saturate 1000 10Gb ethernet, economically speaking, at this point)
- epistasis 9y ago
  1000 10Gb ethernet doesn't usually make sense in the situations that are using Lustre and Infiniband; they actually need the reduced latency for inter-process communication. It may be fine for ethernet in front of the disks, but they still need to get the bits to the processors that are connected by infiniband. Rewriting large existing codebases to use a different paradigm for storage and distribution may make economic sense, but it's a risky proposition. And when you get one shot at building a Top 500 cluster, taking that risk and delivering a cluster that can't run the code you need it to may mean that all that economic spend was wasted.
  - dekhn 9y ago
    The 10Gb's are for node to storage. The IB is for low latency. Having both in the same machine is a win. Wasting IB for storage is dumb, because storage times are orders of magnitude higher than the IB latency. Ultimately, rewriting your application to not require low-latency is not a risky proposition. it's in fact the least risky.
    - gnufx 9y ago
      That makes no sense to an HPC person (who has actually been evaluating low-latency Ethernet recently). How do you even separate the MPI-IO traffic over a different fabric from other MPI in, say, Open MPI, and how could maybe doubling network cost and complexity be a win if you could? And then we have to re-write the applications somehow? There's literature on improved performance running HDFS over Infiniband and Hadoop over Lustre and PVFS at least. And from up the thread, for a discussion of file systems for high performance checkpointing of MPI jobs (which is what's relevant to HPC) see, for instance http://nowlab.cse.ohio-state.edu/static/media/publications/abstract/rajachan-hpdc13.pdf http://nowlab.cse.ohio-state.edu/static/media/publications/a...
      - dekhn 9y ago
        An MPI implementation is free to spread traffic over available transports; there's nothing in the design of MPI that prevents such a configuration. The improved performance of HDFS over Infiniband comes at a price (the aggregation switches are much more expensive and challenging to configure; there are far fewer network engineers who understand them). As for the paper you linked to- I see numerous problems. First is the fact that HPC codes spend 60% of their time checkpointing to deal with MTBFs. This is not a problem in data warehouse applications- failures of components do not affect the job's progress (except in the form of slower progress). Second, the paper has the checkpoint data buffered in local RAM then spilled out to local or remote storage. That's fine, except that you're still at danger if the node crashes before the data is committed to true backing storage (or has a way to recover the spill process after repair/reboot).
        gnufx 9y ago
        The question was how you'd configure a concrete MPI implementation to separate MPI-IO traffic from the rest if it was a good idea. We have the Infiniband (or similar) fabric, and we understand it -- in my case better than the Ethernet one we're proposed to have to understand as well, and we're typically not "network engineers" doing this stuff. I disagree that Infiniband is more challenging anyway, having had and known about bad problems with cluster Ethernet. Then this 10G Ethernet get a whole 10% extra bandwidth on top of current HPC fabrics while presumably adding jitter. HPC codes don't generally spend 60% checkpointing, and "data warehouse applications" typically aren't of interest to us, but we can do mapreduce-type things fast. See the bits about redundancy and the advantage of RDMA in the checkpointing filesystem.
- skynetv2 9y ago
  Hadoop is embarrassingly parallel IO, Lustre can do that, but it does much more than that specific workload.
hpcjoe 9y ago
Actually Lustre isn't the only game in town for this. BeeGFS (http://beegfs.com http://beegfs.com) does a very good job at this as well, has better small to large scaling, understandable by mere-mortal error messages, doesn't require a specific (ancient) kernel or distro for the server ... So does GPFS ... er ... Spectrum Scale by IBM. There are a few others that fit in this, but Lustre is not alone here.
- frozenport 9y ago
  To my knowledge GPFS is IBM only.
  - Infernal 9y ago
    It's an IBM commercial product, if that's what you mean by "IBM only". You can install it on any hardware though, we run it on a combination of SuperMicro, Dell, and DDN currently.
- angry_octet 9y ago
  Also Panasas (http://www.panasas.com/ http://www.panasas.com/).
- gnufx 9y ago
  I'm not sure why the filesytem server kernel is relevant or how RHEL 7.3's is ancient. I'm not aware of MPI-IO support for BeeGFS. Are there any installations of it comparable with, say, the filesystems of the big HPC sites like Tri-labs?

dekhn 9y ago

actually, Hadoop-like systems do precisely this (a compute job of 100k ranks, all writing checkpoints and snapshots periodically). 1.25Tbyte/sec isn't a particularly large amount of data. (btw: makes more sense to saturate 1000 10Gb ethernet, economically speaking, at this point)

epistasis 9y ago
1000 10Gb ethernet doesn't usually make sense in the situations that are using Lustre and Infiniband; they actually need the reduced latency for inter-process communication. It may be fine for ethernet in front of the disks, but they still need to get the bits to the processors that are connected by infiniband. Rewriting large existing codebases to use a different paradigm for storage and distribution may make economic sense, but it's a risky proposition. And when you get one shot at building a Top 500 cluster, taking that risk and delivering a cluster that can't run the code you need it to may mean that all that economic spend was wasted.
- dekhn 9y ago
  The 10Gb's are for node to storage. The IB is for low latency. Having both in the same machine is a win. Wasting IB for storage is dumb, because storage times are orders of magnitude higher than the IB latency. Ultimately, rewriting your application to not require low-latency is not a risky proposition. it's in fact the least risky.
  - gnufx 9y ago
    That makes no sense to an HPC person (who has actually been evaluating low-latency Ethernet recently). How do you even separate the MPI-IO traffic over a different fabric from other MPI in, say, Open MPI, and how could maybe doubling network cost and complexity be a win if you could? And then we have to re-write the applications somehow? There's literature on improved performance running HDFS over Infiniband and Hadoop over Lustre and PVFS at least. And from up the thread, for a discussion of file systems for high performance checkpointing of MPI jobs (which is what's relevant to HPC) see, for instance http://nowlab.cse.ohio-state.edu/static/media/publications/abstract/rajachan-hpdc13.pdf http://nowlab.cse.ohio-state.edu/static/media/publications/a...
    - dekhn 9y ago
      An MPI implementation is free to spread traffic over available transports; there's nothing in the design of MPI that prevents such a configuration. The improved performance of HDFS over Infiniband comes at a price (the aggregation switches are much more expensive and challenging to configure; there are far fewer network engineers who understand them). As for the paper you linked to- I see numerous problems. First is the fact that HPC codes spend 60% of their time checkpointing to deal with MTBFs. This is not a problem in data warehouse applications- failures of components do not affect the job's progress (except in the form of slower progress). Second, the paper has the checkpoint data buffered in local RAM then spilled out to local or remote storage. That's fine, except that you're still at danger if the node crashes before the data is committed to true backing storage (or has a way to recover the spill process after repair/reboot).
      - gnufx 9y ago
        The question was how you'd configure a concrete MPI implementation to separate MPI-IO traffic from the rest if it was a good idea. We have the Infiniband (or similar) fabric, and we understand it -- in my case better than the Ethernet one we're proposed to have to understand as well, and we're typically not "network engineers" doing this stuff. I disagree that Infiniband is more challenging anyway, having had and known about bad problems with cluster Ethernet. Then this 10G Ethernet get a whole 10% extra bandwidth on top of current HPC fabrics while presumably adding jitter. HPC codes don't generally spend 60% checkpointing, and "data warehouse applications" typically aren't of interest to us, but we can do mapreduce-type things fast. See the bits about redundancy and the advantage of RDMA in the checkpointing filesystem.
skynetv2 9y ago
Hadoop is embarrassingly parallel IO, Lustre can do that, but it does much more than that specific workload.

hpcjoe 9y ago

Actually Lustre isn't the only game in town for this. BeeGFS (http://beegfs.com http://beegfs.com) does a very good job at this as well, has better small to large scaling, understandable by mere-mortal error messages, doesn't require a specific (ancient) kernel or distro for the server ... So does GPFS ... er ... Spectrum Scale by IBM. There are a few others that fit in this, but Lustre is not alone here.

frozenport 9y ago
To my knowledge GPFS is IBM only.
- Infernal 9y ago
  It's an IBM commercial product, if that's what you mean by "IBM only". You can install it on any hardware though, we run it on a combination of SuperMicro, Dell, and DDN currently.
angry_octet 9y ago
Also Panasas (http://www.panasas.com/ http://www.panasas.com/).
gnufx 9y ago
I'm not sure why the filesytem server kernel is relevant or how RHEL 7.3's is ancient. I'm not aware of MPI-IO support for BeeGFS. Are there any installations of it comparable with, say, the filesystems of the big HPC sites like Tri-labs?