4 ms·

Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We

Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We use SLURM as our job scheduler. It supports "job arrays", which allows a user to submit a large number of jobs (up to 1,000) with a single "batch script", which is a shell script that gets run on a compute node, in a cgroup with the requested amount of memory and CPU. A job array simply runs this on many (for us, up to 1,000) cgroups, spread out across nodes; the batch script is able to check environment variables to see its "array index", which the user uses to split up work. Once people found this, they took advantage of our site-wide MATLAB license, launching hundreds or thousands of MATLAB instances at around the same time. MATLAB likes to create a 'prefdir' inside `~/.matlab/RXXXXy` ("R2017b", for example). Thousands of MATLABs, all making directories, inside the same common directory, on network-mounted storage. Our Isilon cluster was floored. We found the environment variable which controlled where the prefdir is placed. We changed it to a path in $SCRATCH (the user's scratch space), which is on Lustre. Our Luster environment held up for a while, but was similarly floored. In all of these cases, the issue appears to be many clients trying to take a short lock on the `~/.matlab/RXXXXy` directory, to make their subdirectory. "It worked fine for one, or for a hundred, so it must be OK?"

gnufx 8y ago
I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.

gnufx 8y ago

I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.