4 ms·

I do not quite get this. How does this enable someone to run ray or metaflow on a typical batch scheduled HPC system (slurm or alike)? Inter node communication

by riedel 3y ago

I do not quite get this. How does this enable someone to run ray or metaflow on a typical batch scheduled HPC system (slurm or alike)? Inter node communication is done via the lustre file system, right?

linksnapzz 3y ago
I think it said that data access is via Lustre, and communication is by Nvidia MLNX NCCL, which seems to be some kind of nvidia gpu-specific MPI type library; it would seem to be doing RDMA from GPU to GPU via fabric interconnects, so far as I can tell...
vtuulos 3y ago
Metaflow integrates with AWS Batch which many folks use for serious HPC. Internode scheduling happens through the multinode scheduling supported by AWS Batch. networking via EFA etc. We'll blog more about this soon but you can certainly give it a try today! https://github.com/outerbounds/metaflow-ray https://github.com/outerbounds/metaflow-ray