12 ms·

I'd have thought MPI-IO is the relevant standard for HPC i/o (perhaps under something like HDF5). I'd expect HPC applications with serious i/o requirements to b

by gnufx 9y ago

I'd have thought MPI-IO is the relevant standard for HPC i/o (perhaps under something like HDF5). I'd expect HPC applications with serious i/o requirements to be using that anyway.

noahdesu 9y ago
Applications using MPI-IO are still typically run on top of a parallel file system. The MPI-IO API is, in simple terms, intended to create I/O workloads that should perform well on top of the POSIX I/O interface (e.g. large, sequential I/O). But my understanding is that MPI-IO can have some major drawbacks, like a huge amount of compute-side network utilization used to reorganize and stage I/O.
- gnufx 9y ago
  The PFS under MPI-IO clearly doesn't have to be a POSIX one; PVFS is stateless. I can't comment on that specific intention, which isn't in the standard's rationale. Where is it documented? You'd have to argue with Rob Latham about whether "MPI is a good setting for parallel i/o".
glennklockwood 9y ago
It is and people do use it, but it ultimately still exists as software that sits on top of POSIX in the vast majority of cases. It can clean up really bad POSIX I/O by buffering and reordering, but doing so induces latency, memory overheads, and consumes other resources that don't always pay dividends at all scales.
- gnufx 9y ago
  ROMIO unfortunately requires global locking on Lustre, but I don't see how that's relevant to the use of standard or other high-level (HDF, NetCDF, ...) non-POSIX interfaces. You can swap in PVFS, or something else with a different architecture. (So what if object stores are on _local_ POSIX-y filesystems?) I'd have thought the salient feature is collective i/o, not cleaning up silly POSIX i/o of some sort. Nothing's perfect, but what should replace the non-blocking collective MPI-IO under5, for instance? I wonder what are the major NERSC applications that don't do sane i/o and should. The published workload characterization doesn't help.

by gnufx 9y ago

I'd have thought MPI-IO is the relevant standard for HPC i/o (perhaps under something like HDF5). I'd expect HPC applications with serious i/o requirements to be using that anyway.

noahdesu 9y ago

Applications using MPI-IO are still typically run on top of a parallel file system. The MPI-IO API is, in simple terms, intended to create I/O workloads that should perform well on top of the POSIX I/O interface (e.g. large, sequential I/O). But my understanding is that MPI-IO can have some major drawbacks, like a huge amount of compute-side network utilization used to reorganize and stage I/O.

gnufx 9y ago
The PFS under MPI-IO clearly doesn't have to be a POSIX one; PVFS is stateless. I can't comment on that specific intention, which isn't in the standard's rationale. Where is it documented? You'd have to argue with Rob Latham about whether "MPI is a good setting for parallel i/o".

glennklockwood 9y ago

It is and people do use it, but it ultimately still exists as software that sits on top of POSIX in the vast majority of cases. It can clean up really bad POSIX I/O by buffering and reordering, but doing so induces latency, memory overheads, and consumes other resources that don't always pay dividends at all scales.

gnufx 9y ago
ROMIO unfortunately requires global locking on Lustre, but I don't see how that's relevant to the use of standard or other high-level (HDF, NetCDF, ...) non-POSIX interfaces. You can swap in PVFS, or something else with a different architecture. (So what if object stores are on _local_ POSIX-y filesystems?) I'd have thought the salient feature is collective i/o, not cleaning up silly POSIX i/o of some sort. Nothing's perfect, but what should replace the non-blocking collective MPI-IO under5, for instance? I wonder what are the major NERSC applications that don't do sane i/o and should. The published workload characterization doesn't help.