On the other hand, I've run multi-hundred-node HPC stateless HPC clusters long term off an (rpm-based) NFS root with little trouble once I could get the cluster

On the other hand, I've run multi-hundred-node HPC stateless HPC clusters long term off an (rpm-based) NFS root with little trouble once I could get the cluster vendors out of the picture. Places like the Tri-Labs appear to -- or have done? -- with larger systems, even if they need something like https://computation.llnl.gov/projects/spindle https://computation.llnl.gov/projects/spindle at scale. What flattened NFS wasn't executables but i/o of data files, BLAST being the worst offender; staging those in array jobs was definitely worthwhile. I strongly prefer the trades-off of stateless systems in that context, though I might prefer to do it off a parallel filesystem, and clearly your kilometreage varies. [From experience ~10 years ago, I'd say avoid GPFS more than NFS, but doubtless it's improved, and I hold no brief for NFS.]