Well most HPC installation I know are just running RHEL6/7 (except for NERSC which is using Cray linux based on SUSE still I think?) which I wouldn't peg as par

Well most HPC installation I know are just running RHEL6/7 (except for NERSC which is using Cray linux based on SUSE still I think?) which I wouldn't peg as particularly exotic. The HA thing is know, typically Lustre is just a piece of the file system, along with local disks and GPFS (and now towards Ceph), and HPSS/tape (which can be a tier of GPFS). I'm not so sure the hope for "one filesystem to rule them all" will ever really work out, but Ceph is the best positioned.