4 ms·

In HPC/HTC, Lustre is very common, especially in DOE labs. I've never tried to install it, but I don't think my colleagues who have are some kind of special gen

• jpgvm 11y ago
  Most HPC labs run distributions that go to quite a lot of pain to ensure Lustre works well. The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues. That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install. Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park. Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.
  • batbomb 11y ago
    Well most HPC installation I know are just running RHEL6/7 (except for NERSC which is using Cray linux based on SUSE still I think?) which I wouldn't peg as particularly exotic. The HA thing is know, typically Lustre is just a piece of the file system, along with local disks and GPFS (and now towards Ceph), and HPSS/tape (which can be a tier of GPFS). I'm not so sure the hope for "one filesystem to rule them all" will ever really work out, but Ceph is the best positioned.