3 ms·

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me). Supercomputer system MT

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me). Supercomputer system MTBF numbers aren't high at all. Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades. These systems are built with the likes of dubious metafilesystems (I'm talking to you, Lustre) sitting on top of shaky file systems on top of junk-bin HW RAID systems almost guaranteed to corrupt your data over time. I think your overall statement is valuable with the global search/replace of "enterprise" for "HPC" - airline reservations or credit card processing. That's all I'm saying - HPC people (at least currently) are massively overrated as developers and as admins. Maybe it's all just a plan to extract energy from Seymour Cray's pulsar-like rotating remains?

nickpsecurity 10y ago
" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was knowledgeable of this area. It could've all gone downhill since. "That's all I'm saying - HPC people (at least currently) are massively overrated as developers and as admins. " I'll agree with that. It's one of reasons field built so much tooling to make up for it. ;)
- drauh 10y ago
  Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)
  - nickpsecurity 10y ago
    Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while. https://www.mellanox.com/related-docs/solutions/ppt_ceph_mellanox_ceph_day.pdf https://www.mellanox.com/related-docs/solutions/ppt_ceph_mel... http://www.snia.org/sites/default/files/JohnKim_CephWithHighPerformanceNetworks_V2.pdf http://www.snia.org/sites/default/files/JohnKim_CephWithHigh... Also, look at Sector/Sphere which was made by UDT designer for distributed, parallel workloads for supercomputing. It has significant advantages over Hadoop. It's used with high-performance links to share data between supercomputing centers. http://sector.sourceforge.net/index.html http://sector.sourceforge.net/index.html http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%20v1.pdf http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%2...
    - ymse 10y ago
      Native RDMA support for Ceph is still a ways off. The current implementation requires disabling Cephx authentication, which is a no-go in any environment where you can't completely trust every client (e.g. "cloud", where most current deployments/users live). It also hasn't seen much development since the initial proof-of-concept (still highly experimental). That said, IPoIB should work just fine, and the main bottleneck currently is (Ethernet) latency. I'm running a couple of 1,6PB clusters (432 * 4TB) and can only get 20-60MBps on a single client with a 4kB block size, but got bored of benchmarking after saturating 5 concurrent 10Gb clients with a 4MB block size. I do expect the RDMA situation to improve substantially over the next year or so, even if authentication will still be unsupported. The latter generally isn't a problem in HPC where stuff like GPFS lives (where you also have to trust every client). And they clearly want that market now that CephFS is finally deemed production ready.
  - pinewurst 10y ago
    At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSIX, trusting Ceph at this point is an issue unless, as you say, MTBF isn't a concern. You might want to try BeeGFS, a similar logical model but much simpler to implement, performance up to a very high level, and a record of reliable HPC deployments (as oxymoronic as that sounds). If you can do with object then certainly exorcise Lustre from your environment in favor of Ceph (or try Scality if non-OS isn't an issue). Lustre's only useful as a jobs program anyway - keeping people occupied who'd otherwise be bodging up real software.
- notacoward 10y ago
  What does "potential for reliability" even mean? Even Lustre, which you malign and I have maligned even more, has potential for reliability if they just fix a few hundred egregious design flaws.
  - nickpsecurity 10y ago
    I haven't had a chance to run it. I also don't have clear data on it's userbase. I just know it's been used in supercomputing centers for a while doing large jobs fast over fast WAN's. So, potentially reliable if I see more users telling me it is in various situations.

nickpsecurity 10y ago

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was knowledgeable of this area. It could've all gone downhill since. "That's all I'm saying - HPC people (at least currently) are massively overrated as developers and as admins. " I'll agree with that. It's one of reasons field built so much tooling to make up for it. ;)

drauh 10y ago
Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)
- nickpsecurity 10y ago
  Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while. https://www.mellanox.com/related-docs/solutions/ppt_ceph_mellanox_ceph_day.pdf https://www.mellanox.com/related-docs/solutions/ppt_ceph_mel... http://www.snia.org/sites/default/files/JohnKim_CephWithHighPerformanceNetworks_V2.pdf http://www.snia.org/sites/default/files/JohnKim_CephWithHigh... Also, look at Sector/Sphere which was made by UDT designer for distributed, parallel workloads for supercomputing. It has significant advantages over Hadoop. It's used with high-performance links to share data between supercomputing centers. http://sector.sourceforge.net/index.html http://sector.sourceforge.net/index.html http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%20v1.pdf http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%2...
  - ymse 10y ago
    Native RDMA support for Ceph is still a ways off. The current implementation requires disabling Cephx authentication, which is a no-go in any environment where you can't completely trust every client (e.g. "cloud", where most current deployments/users live). It also hasn't seen much development since the initial proof-of-concept (still highly experimental). That said, IPoIB should work just fine, and the main bottleneck currently is (Ethernet) latency. I'm running a couple of 1,6PB clusters (432 * 4TB) and can only get 20-60MBps on a single client with a 4kB block size, but got bored of benchmarking after saturating 5 concurrent 10Gb clients with a 4MB block size. I do expect the RDMA situation to improve substantially over the next year or so, even if authentication will still be unsupported. The latter generally isn't a problem in HPC where stuff like GPFS lives (where you also have to trust every client). And they clearly want that market now that CephFS is finally deemed production ready.
- pinewurst 10y ago
  At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSIX, trusting Ceph at this point is an issue unless, as you say, MTBF isn't a concern. You might want to try BeeGFS, a similar logical model but much simpler to implement, performance up to a very high level, and a record of reliable HPC deployments (as oxymoronic as that sounds). If you can do with object then certainly exorcise Lustre from your environment in favor of Ceph (or try Scality if non-OS isn't an issue). Lustre's only useful as a jobs program anyway - keeping people occupied who'd otherwise be bodging up real software.
notacoward 10y ago
What does "potential for reliability" even mean? Even Lustre, which you malign and I have maligned even more, has potential for reliability if they just fix a few hundred egregious design flaws.
- nickpsecurity 10y ago
  I haven't had a chance to run it. I also don't have clear data on it's userbase. I just know it's been used in supercomputing centers for a while doing large jobs fast over fast WAN's. So, potentially reliable if I see more users telling me it is in various situations.